About this role
Scaling reliable AI inference infrastructure to handle growing platform demand requires engineers who can eliminate single points of failure across distributed systems. Harvey needs a Staff Software Engineer to lead the design and implementation of the Model Infrastructure platform, building the underlying systems that power every AI request across the organization.
Daily responsibilities include improving the Unified Model Controller and Model Selector platform to automatically route traffic based on quality, reliability, latency, cost, and compliance. The engineering group operates with intensity and values decisiveness alongside simplicity while transforming legal and professional services. Ideal candidates bring extensive background in distributed systems and want to build foundational platforms that support frontier agentic AI workloads.
What you will do
- Lead the design and implementation of Harvey's Model Infrastructure platform to ensure high availability and low latency for AI inference.
- Improve the Unified Model Controller and Model Selector platform to intelligently route traffic and detect model degradations.
- Develop systems for capacity management, failover, provisioning, and traffic engineering across multiple external AI providers.
- Integrate new model providers while maintaining provider APIs and SDKs for rapid adoption of emerging frontier models.
- Enhance observability through end-to-end telemetry, alerting, health dashboards, cost reporting, and token usage analytics.
- Collaborate with Product Engineering and AI Research to support model launches, experimentation, evaluation, and training infrastructure.
- Drive infrastructure efficiency through utilization optimization, capacity planning, and cost visibility.
What you need
- Bring 7 or more years of software engineering experience focused on building large-scale distributed systems.
- Demonstrate experience designing and operating highly available production services at scale.
- Possess strong programming skills in languages such as Go, Java, Python, Rust, or C++.
- Have a deep understanding of cloud infrastructure, distributed systems, networking, and observability.
- Show documented experience leading technical projects across multiple engineering teams.
Nice to have
- Experience with AI infrastructure, LLM serving, or machine learning platforms
- Experience with model routing, inference gateways, or policy-based serving systems
- Experience working with OpenAI, Anthropic, Azure OpenAI, Fireworks, Baseten, or open-source LLMs
- Experience with Kubernetes, cloud infrastructure, and service mesh technologies
- Experience with data infrastructure technologies such as Kafka, Spark, Flink, Airflow, or Iceberg
Benefits
- Offers Equity
- Offers Bonus
Skills and tech stack
Questions candidates ask
What is the compensation range for this role?
The salary range for this position is $231,000 to $340,000 USD. Additional compensation includes equity and bonus offerings.
Where is the position located?
This role is based in San Francisco. The posting does not mention remote work options.
What technologies and tools will I work with?
You will work with programming languages like Go, Java, Python, Rust, or C++. The stack also incorporates various AI providers and infrastructure tools such as OpenAI, Anthropic, Azure OpenAI, Fireworks, Baseten, and Kubernetes.
Who will I collaborate with on the team?
You will partner closely with AI Research, Product Engineering, Infrastructure, and external model providers. The role also involves leading cross-functional technical initiatives and mentoring engineers across the organization.