Staff Software Engineer, Agent Quality
About this role
Databricks needs to solve the challenge of accurately evaluating and continuously improving its complex AI agents as the organization expands its first-party offerings. The AI Research organization is building models and multi-agent systems that unlock enterprise data, requiring dedicated infrastructure to measure performance reliably. As a founding member of the new agent quality team, this engineer will design and scale the developer workflows, tooling, and benchmarking systems that let researchers and engineers evaluate agents rigorously across their lifecycle.
Working as part of this new initiative involves building a feedback loop where evaluation results directly inform agent training and iterative development. Success in the role requires comfort navigating ambiguous boundaries between research and product teams, along with a strong background in distributed systems or data pipelines. Candidates who thrive here possess a high bar for technical quality and the ability to shape long-term technical direction across multiple teams in New York City.
What you will do
- Stand up foundational evaluation infrastructure for Genie Agents to enable rigorous benchmarking, regression detection, and quality measurement.
- Build the feedback loop that connects evaluation results back into agent improvement across production signals, training, and iterative development.
- Shape the long-term technical direction for agent quality infrastructure as the company expands its first-party agents and agent development platform.
- Design pragmatic and rigorous systems that generate trustworthy and reproducible signals for complex applications.
- Partner closely with both researchers and engineers to turn ideas into robust internal platforms across ambiguous research and product boundaries.
What you need
- Bring at least 6 years of industry experience building software systems.
- Demonstrate strong Python programming skills with a background in building production or research infrastructure.
- Possess experience building or operating distributed systems, data pipelines, or large-scale infrastructure focused on reliability and operational maturity.
- Show an ability to design pragmatic, rigorous systems that yield trustworthy and reproducible signals for complex applications.
- Maintain comfort working across ambiguous research and product boundaries while partnering with multiple teams.
Nice to have
- Experience with devtools, CI/CD platforms, testing frameworks, observability tooling, or benchmarking infrastructure
- Familiarity with how LLM or agent quality is measured through evals, experimentation platforms, or production monitoring
Skills and tech stack
Questions candidates ask
What is the expected salary range for this role?
The local pay range for this position in New York City is $200,000 to $265,000 USD. Actual compensation packages are based on factors such as job-related skills, depth of experience, relevant certifications, and specific work location.
What team will this engineer join?
This person will be a founding member of a new team focused on evaluating and continuously improving Databricks' AI Agents within the AI Research organization. The team works on scaling infrastructure and developer workflows to drive an evaluation flywheel.
What kind of technical background is required?
Candidates need 6 or more years of industry experience building software systems alongside strong Python programming skills. Experience building or operating distributed systems, data pipelines, or large-scale infrastructure with a focus on reliability and correctness is also required.