Senior / Staff AI Engineer
About this role
Engineers in this group spend their days designing and building infrastructure that powers large-scale agentic workflows, synthetic data generation, and evaluation pipelines. Team members construct orchestration systems, distributed compute setups, and LLM infrastructure that make non-deterministic AI workloads observable, reproducible, and scalable.
Snorkel AI operates out of the Stanford AI Lab tradition, focusing on the data and environments behind frontier and agentic artificial intelligence. This position exists to support a foundational shift toward agent-first workflows where models interact with tools, environments, and other agents over long-running trajectories. Candidates who thrive here combine strong distributed systems fundamentals with practical experience operating AI workloads in production.
Individuals in this role collaborate closely with research, product, and engineering groups to turn experimental setups into reliable, reusable platform capabilities. Successful engineers possess a strong background in backend or infrastructure engineering and enjoy tackling complex challenges related to probabilistic system behavior, telemetry, and developer experience.
What you will do
- Design and build infrastructure for running large-scale agentic workloads, multi-step agents, and simulated environments.
- Build scalable synthetic data generation and automated labeling systems to support high-quality training and evaluation datasets.
- Design evaluation infrastructure for measuring AI system behavior, including reproducible experiments and regression detection.
- Develop orchestration and distributed compute systems for running millions of AI experiments reliably across heterogeneous environments.
- Build and operate LLM infrastructure covering routing, rate limiting, retries, caching, and cost attribution across model providers.
- Instrument agent and model workloads to capture traces, tool calls, environment state, and performance metrics.
- Improve developer experience by building APIs, SDKs, and workflow abstractions that transition workloads from local development to production.
What you need
- Five or more years of experience building production software systems in AI infrastructure, ML platforms, distributed systems, data platforms, or backend infrastructure.
- Practical experience operating non-deterministic AI or ML workloads in production or at significant scale.
- Experience building infrastructure for experimentation, evaluation, model development, synthetic data, or agentic workflows.
- Strong proficiency in Python alongside experience building production-quality APIs, services, and developer tooling.
- Strong background in distributed systems and cloud platforms including compute orchestration, storage, and networking.
- Familiarity with workflow or distributed execution frameworks such as Prefect, Airflow, Dagster, Ray, or Kubernetes.
- Solid understanding of production system fundamentals including observability, telemetry, reliability, performance, and cost management.
Nice to have
- Experience building or operating LLM or agent infrastructure, including model gateways, agent runtimes, or multi-agent systems.
- Experience building evaluation or experimentation platforms for LLMs, agents, or other probabilistic systems.
- Experience with synthetic data generation, automated labeling, data refinement, or dataset quality systems.
- Experience building reinforcement learning environments, agent simulations, benchmarks, or other environment-based evaluation systems.
- Experience running large-scale distributed AI workloads across containers, Kubernetes, serverless compute, or sandboxes.
Skills and tech stack
Questions candidates ask
What locations are available for this role?
This position offers hybrid work arrangements in New York City, NY and San Francisco, CA.
What kind of systems will I be building?
You will build infrastructure for running large-scale agentic workloads, synthetic data generation, automated labeling systems, evaluation infrastructure, and distributed compute platforms.
What is the primary focus of the AI Platform team?
The team builds the infrastructure and systems that power AI development at scale, focusing on synthetic data generation, evaluation, agentic workflows, simulation environments, LLM infrastructure, and distributed compute.