Latest AI JobsBrowse Jobs
A

Site Reliability Engineer, AI Observability

Appnovation TechnologiesToronto, MontrealTechnology
Apply on company site
Salary
Not listed
Job type
Full-time
Posted
6 Oct 2026
Status
Open · checked 7d ago

About this role

Based in Toronto or Montreal, this full-time Site Reliability Engineer role involves maintaining a shared observability platform for large language model applications serving a global life sciences client. The position suits an operations professional with strong infrastructure experience who wants to build monitoring, alerting, and service levels from scratch.

Role Overview

Appnovation Technologies seeks a Site Reliability Engineer to manage a self-hosted platform built on AWS Kubernetes. The role focuses on operating a technology stack that includes Langfuse, ClickHouse, PostgreSQL, Redis, and Argo CD. In addition to daily maintenance, the successful candidate will establish comprehensive monitoring and write operational documentation for internal teams.

Key Responsibilities

Troubleshoot and fix issues across ClickHouse, PostgreSQL, Redis, ingestion workers, and the underlying Kubernetes infrastructure.
Manage ClickHouse operations using the operator model, covering replication, Keeper quorum, shard topology, and S3 storage tiering.
Test and document backup and restore procedures to verify recoverability against potential failure modes.
Plan, rehearse, and execute safe software upgrades in lower environments with reliable rollback strategies.
Establish monitoring, alerting, and service level definitions from the ground up.
Update infrastructure as code through Kubernetes manifests, Helm values, and Argo CD applications.
Author and maintain clear runbooks and standard operating procedures for incident management.
Handle internal onboarding, user support, and ticket triage for teams relying on the platform.
Automate repetitive operational workflows.
Coordinate with platform engineers on software migrations and deployment rollbacks.

Requirements

Practical background managing Kubernetes on AWS, utilizing Helm and Argo CD.
Production experience operating ClickHouse, including replication, topology management, and backup routines.
Demonstrated history of designing monitoring, alerting, and service levels from the beginning.
Background executing safe software upgrades and migrations with tested rollback procedures.
Operational knowledge of PostgreSQL and Redis to resolve queue and metadata issues.
Proficiency in technical writing, including runbooks and post-incident documentation.

Nice to Have

Background in observability engineering involving OpenTelemetry, Grafana dashboards, and alerting design.
Integration work with OIDC or enterprise single sign-on solutions.
Experience using GitHub Actions for deployment pipelines.
Familiarity with large language model observability platforms such as Langfuse, LangSmith, or Arize Phoenix.
Prior background working in life sciences, pharmaceuticals, or other regulated sectors.
Previous consulting experience.

Skills and Tools

Kubernetes
AWS
Helm
Argo CD
ClickHouse
PostgreSQL
Redis
OpenTelemetry
Grafana
GitHub Actions
Langfuse

About the Company

Appnovation Technologies operates as a global digital partner delivering strategy, design, engineering, and managed services. The company functions as an equal opportunity employer that values workplace diversity and provides recruitment accommodations upon request.

How to Apply

Interested candidates should submit their application through the official Appnovation Technologies careers page.

Skills and tech stack

KubernetesAWSHelmArgo CDClickHousePostgreSQLRedisOpenTelemetryGrafanaGitHub ActionsLangfuseLLM

Questions candidates ask

Where is this job located?

The position is based in Toronto or Montreal.

What technologies are used in the observability platform?

The platform is built on Langfuse, self-hosted on Kubernetes using AWS, with ClickHouse for analytics, PostgreSQL for metadata, Redis for the ingestion queue, and Argo CD for delivery.

Will the person in this role be building new features or only maintaining existing systems?

The role involves both. While maintenance is required, the platform currently lacks monitoring, alerting, and service levels, which the engineer will build from scratch.

Is prior experience in the life sciences industry required?

It is not strictly required, but experience in life sciences, pharmaceuticals, or another regulated industry is preferred.

How should I apply for this position?

Candidates should apply directly through the official Appnovation Technologies careers page.

Ready to apply?

The application is on the official Appnovation Technologies careers page.

Apply on company site