Principal AI Engineer - Context - Agents and Context
About this role
Writing production code, designing telemetry and evaluation frameworks, and owning the end-to-end improvement loop for the Context Engine form the daily scope of this hybrid position. The team builds the knowledge layer that AI agents use to interact with enterprise data inside Elasticsearch, extracting information into a structured AI Index and serving it via public APIs, MCP tools, and framework integrations. Engineers in this group collaborate closely with data science, product, and UX colleagues to shape how agents, automations, and skills behave in production environments.
Ownership spans offline evaluations, customer telemetry, safe rollouts, and versioning for prompts and skills. The codebase relies heavily on TypeScript, and development happens in the open alongside the broader Elastic Stack. Candidates who thrive here typically bring a pragmatic, low-ego working style, deep experience with eval-driven development, and the ability to balance rigorous measurement with shipping production code in a distributed, async-first organization.
What you will do
- Manage the production improvement loop for the Context Engine by analyzing offline evaluations, customer conversations, and telemetry to identify and fix failure modes.
- Establish safety protocols for iterating on agents and skills, including prompt versioning, regression coverage, and staged evaluations.
- Design telemetry systems that capture agent traces, tool calls, and knowledge retrieval data to feed dashboards and evaluation loops.
- Collaborate with the data science team to define evaluation strategies, golden datasets, and quality gates for latency and cost.
- Elevate team standards by reviewing pull requests and architectural designs, mentoring engineers on evaluation-driven development, and authoring technical proposals.
What you need
- Bring over 10 years of software engineering experience with a strong focus on shipping and operating AI-driven products on live production traffic.
- Demonstrate a proven track record of eval-driven product improvement by diagnosing LLM behavior from traces and implementing measurable fixes.
- Possess direct experience building stateful agents with memory and safely iterating on prompts, skills, and tools in production.
- Show familiarity with Model Context Protocol (MCP) concepts, including the creation and exposure of public MCP servers and tools.
- Demonstrate experience designing telemetry for AI systems to guide engineering and product decisions.
- Maintain strong backend engineering skills in either Python or TypeScript for building APIs, stateful workflows, and data pipelines.
- Exhibit a pragmatic, low-ego working style suited for collaboration within a distributed, async-first team.
Nice to have
- Experience with agent frameworks such as LangGraph, CrewAI, or the Claude Agent SDK
- Agentic retrieval experience including knowledge representation
- Practical Elasticsearch experience
Benefits
- Competitive pay based on the work you do
- Health coverage for you and your family in many locations
- Flexible locations and schedules
- Generous number of vacation days each year
- Donation matching up to $2000 or local currency equivalent
- Up to 40 hours each year for volunteer projects
- Minimum of 16 weeks of parental leave
Skills and tech stack
Questions candidates ask
What is the remote policy for this role?
This is a hybrid role with the ability to craft your calendar with flexible locations and schedules.
What is the compensation structure?
Compensation consists of a base salary ranging from €80,400 to €127,200 EUR with no variable compensation component.
What technologies does the team use?
The codebase is written in TypeScript and Python, utilizing Elasticsearch, MCP tools, and various agent frameworks.
What kind of team environment should I expect?
You will work as part of a distributed, async-first team alongside data scientists, backend engineers, product managers, and UX designers.