Latest AI JobsBrowse Jobs
Culture Amp logo

Staff Applied AI Scientist

Culture AmpMelbourneData Science
Apply on company site
Salary
Not listed
Job type
Full-time
Posted
9 Jun 2026
Status
Open · checked 7d ago

About this role

Culture Amp is seeking a Staff Applied AI Scientist to join the Data Science team in Melbourne. This role focuses on scaling production evaluation and observability frameworks for the Coach AI system, making it an ideal match for an experienced AI engineer passionate about LLM performance.

Role Overview

The Staff Applied AI Scientist will spearhead continuous production evaluation and observability for the Culture Amp Coach AI system. This position involves taking ownership of the end-to-end feedback loop, building automated analysis tools, and enabling the broader engineering organization to maintain high-quality AI products. The successful candidate will work hybridly from the Melbourne office, collaborating closely with product, data science, and people science teams to map measured quality to real customer value.

Key Responsibilities

Own the end-to-end feedback loop by establishing rigorous cycles of prompt engineering, evaluation at scale, and continuous improvement.
Build LLM-powered analysis tools that diagnose performance shifts, provide deep insights, and automate recommendations for prompt or system-level enhancements.
Contribute to context engineering by designing and optimizing retrieval, memory across sessions, context assembly, and compression.
Design and run evaluations using sampling, LLM-as-a-judge, and labeling systems over de-identified production traces for longitudinal monitoring.
Contribute to agentic orchestration, letting evaluation findings drive structural changes like adding self-check steps or altering tool selection.
Make model and routing decisions balancing quality, latency, and cost trade-offs.
Design input and output guardrails, PII handling, content safety, and jailbreak resistance for sensitive coaching data.
Enable product and engineering teams to run their own evaluations through reusable frameworks, tooling, and documentation.

Requirements

Experience building and turning production agentic systems, including context engineering, RAG, memory, cost, and model selection.
Proven background in analyzing AI or data product performance in production and executing improvements.
Hands-on LLM evaluation experience in production, including LLM-as-judge, eval datasets, human-in-the-loop labeling, and scoring against thresholds.
Familiarity with observability tooling for LLM and agentic systems such as traces, prompt management, and production monitoring platforms like Langfuse.
Experience with longitudinal measurement, including metrics, baselines, regression detection, and quality tracking.
AI-native daily practice utilizing agentic coding tools like Claude Code or Cursor on multi-step tasks.
Strong technical writing and communication skills with a track record of building system capabilities and teaching others.
Must be legally authorized to work in Australia for the duration of employment.

Nice to Have

Experience building or scaling an evaluation and observability practice across multiple teams.
Background in evolving existing enterprise codebases with AI integration.
A postgraduate degree in Machine Learning, Computer Science, Applied Mathematics, or a related field.
Public writing, speaking engagements, or open-source contributions in evaluation, observability, or LLMOps.

Skills and Tools

Large Language Models (LLMs)
Retrieval-Augmented Generation (RAG)
LLMOps and Observability Tools (Langfuse)
Prompt Engineering and Context Management
Agentic Systems Orchestration
LLM-as-a-Judge Evaluation Frameworks

Salary and Benefits

Equity through the Employee Share Option Program.
Learning programs, coaching, and a monthly lifestyle and wellbeing allowance.
Quarterly refresh days and an extended end-of-year break.
Inclusive parental leave from day one.
MacBook and a home workspace setup budget.
Five annual social impact days.

About the Company

Culture Amp is the world’s leading employee experience platform, empowering over 6,000 companies and 25 million employees to transform engagement and develop high-performing teams. Recognized by Forbes and Fast Company as an innovative industry leader, Culture Amp is backed by prominent venture capital funds and operates globally with hubs in the US, UK, Germany, and Australia.

How to Apply

Interested candidates should apply directly on the official Culture Amp careers page.

Skills and tech stack

LLMRAG

Questions candidates ask

Where is this role located?

This role is based out of the Culture Amp Melbourne hub, with team members working from the local office an average of two days a week.

What are the working authorization requirements?

Candidates must be legally authorized to work in Australia for the duration of employment.

What technologies and tools will I use?

You will work extensively with Large Language Models, RAG systems, LLM observability and tracing tools like Langfuse, and agentic coding assistants.

What is the primary focus of the Staff Applied AI Scientist?

The primary focus is establishing observability and evaluation frameworks for the Coach AI system, managing the end-to-end feedback loop, and enabling the engineering organization to run evaluations.

What perks and benefits are provided?

Benefits include equity through an Employee Share Option Program, learning and coaching programs, a monthly wellbeing allowance, quarterly refresh days, inclusive parental leave, and home workspace setup support.

Ready to apply?

The application is on the official Culture Amp careers page.

Apply on company site