Latest AI JobsBrowse Jobs
Culture Amp logo

Staff Applied AI Scientist

Culture AmpSydneyData Science
Apply on company site
Salary
Not listed
Job type
Full-time
Posted
9 Jun 2026
Status
Open · checked 7d ago

About this role

Culture Amp is seeking a Staff Applied AI Scientist to join the team in Sydney. This position suits an experienced AI professional with a strong engineering background who wants to elevate production AI systems and enable engineering teams.

Role Overview

Culture Amp is hiring a Staff Applied AI Scientist to establish observability and evaluation frameworks for the Coach AI system. This role focuses on continuous production evaluation, diagnosing performance shifts, and driving measurable quality improvements at scale. The successful candidate will own the end-to-end feedback loop and empower the broader engineering organization to build and run their own evaluations.

Key Responsibilities

Own the end-to-end feedback loop by establishing a rigorous cycle of prompt engineering, scale evaluation, and continuous improvement.
Build LLM-powered analysis tools that diagnose performance shifts, offer deep insights, and automate recommendations for system-level enhancements.
Contribute to context engineering by optimizing retrieval, session memory, context assembly, and compression for multi-turn agentic flows.
Design and execute evaluations utilizing sampling, LLM-as-a-judge, and labelling systems over de-identified production traces for longitudinal monitoring.
Drive eval-driven agentic orchestration by contributing to agent architecture and letting findings trigger structural changes.
Manage model and provider selection by balancing quality, latency, and cost trade-offs.
Design and monitor safety guardrails, PII handling, content-safety, and jailbreak resistance for sensitive coaching data.
Enable product and engineering teams to run independent evaluations through reusable frameworks, tooling, and documentation.
Partner closely with the AI Coach team, product, data science, and people science functions.
Stay updated on current research regarding LLMOps, evaluation, and observability.

Requirements

Experience in building and transitioning production agentic systems, including context engineering, RAG, memory, cost, and model selection.
Proven expertise in analyzing AI product performance in production and executing changes to improve the product.
Hands-on experience with LLM evaluation in production using LLM-as-judge, eval datasets, and human-in-the-loop labelling.
Familiarity with observability tooling for LLM and agentic systems such as Langfuse.
Experience tracking longitudinal measurement, metrics, baselines, and regression detection.
AI-native daily practice utilizing agentic coding tools like Claude Code or Cursor with strong technical writing skills.

Nice to Have

Built or scaled an evaluation and observability practice across multiple teams.
Experience evolving existing enterprise codebases with AI.
Postgraduate degree in machine learning, computer science, applied mathematics, or a related field.
Public writing, speaking engagements, or open-source contributions in evaluation, observability, or LLMOps.

Skills and Tools

LLM
RAG
Langfuse
Claude Code
Cursor
LLMOps
Prompt Engineering
Agentic Orchestration

Salary and Benefits

Equity via the Employee Share Option Program.
Learning programs and coaching for professional growth.
Quarterly refresh days, a monthly wellbeing and lifestyle allowance, and an extended end-of-year break.
Inclusive parental leave starting from day one.
MacBook and a home workspace setup budget.
Five annual social impact days.

About the Company

Culture Amp provides an employee experience platform used by thousands of companies globally, helping organizations transform employee engagement, drive performance management, and develop high-performing teams. The company operates hubs in multiple regions, including Australia, and expects team members to work from their local office an average of two days a week.

How to Apply

Interested candidates should apply directly on the official Culture Amp careers page.

Skills and tech stack

LLMRAG

Questions candidates ask

What is the primary focus of the Staff Applied AI Scientist?

The primary focus is establishing observability and evaluation frameworks for the Coach AI system, managing the end-to-end feedback loop, and enabling the wider engineering organization to run evaluations.

Where is this job located?

This role is based out of the Culture Amp hub in Sydney, with a hybrid work expectation averaging two days a week in the local office.

What tools and technologies are relevant to this role?

Relevant tools and technologies include LLMs, RAG systems, Langfuse for observability, and agentic coding tools like Claude Code and Cursor.

Are medical insurance benefits available for this role in Sydney?

Medical insurance coverage is currently available for US and UK employees only.

What are the working hour or attendance expectations?

Campers are asked to work from their local Culture Amp office an average of two days a week.

Ready to apply?

The application is on the official Culture Amp careers page.

Apply on company site