Performance Engineer, Inference Engine
About this role
Anthropic builds reliable, interpretable, and steerable AI systems that serve millions of users and power internal research. The inference engine acts as the foundational software layer between accelerator kernels and the routing layer, managing token paths, memory allocation, batching, and model state across all accelerator and cloud platforms.
As a performance engineer on this team, the selected individual will optimize throughput, cost, reliability, and latency at scale. Daily responsibilities involve analyzing hardware bottlenecks, maximizing device utilization, and coordinating forward passes while collaborating closely with safety and research teams to maintain model quality and robust production safeguards.
What you will do
- Build and optimize the in-house inference engine to improve throughput, cost, reliability, and latency across various accelerator and cloud platforms.
- Maintain a strong working mental model of LLM inference by understanding how prefill and decode phases utilize compute, memory, and interconnect resources.
- Observe and profile system performance first, formulate hypotheses, test changes in the code, and measure the resulting impact.
- Keep device utilization high by eliminating overheads that cause accelerators to wait.
- Manage model state efficiently by caching and reusing data whenever it is more cost-effective than recomputing.
- Collaborate closely with safeguards and safety teams to ensure the inference engine supports production safety systems without compromising robustness.
What you need
- Possess a working mental model of LLM inference including accelerator compute, memory, interconnect, and host responsibilities.
- Demonstrate the ability to ramp up fast in deep, unfamiliar systems and ship consequential changes quickly.
- Write strong systems programming code in Rust, C++, or similar languages with careful attention to code quality and testing.
- Apply analytical approaches to performance by profiling first, testing hypotheses, and measuring code changes.
- Display low ego through asking naive questions, receiving feedback well, and picking up tasks outside the standard job description.
- Enjoy pair programming and care about the societal impacts of software engineering work.
Nice to have
- Experience inside an LLM serving engine with knowledge of GPU and accelerator programming abstractions
- Familiarity with OS internals and language modeling with transformers
- Experience building an allocator, cache, scheduler, or high-bandwidth transport
- Fluency in Rust
- Experience making systems reproducible through determinism, replay, or property-based tests
Benefits
- Competitive compensation and benefits
- Optional equity donation matching
- Generous vacation
- Parental leave
- Flexible working hours
- Lovely office space for collaboration
Skills and tech stack
Questions candidates ask
What is the remote work policy for this role?
The role follows a location-based hybrid policy where staff are expected to be in one of the offices at least 25% of the time, though some roles may require more in-office presence.
What is the compensation range for this position?
The annual salary range for this role is between $350,000 and $850,000 USD.
Does the company offer visa sponsorship?
Yes, visa sponsorship is available, and the company retains an immigration lawyer to help make every reasonable effort to secure a visa for an offered candidate.
What kind of engineering culture and collaboration style does the team have?
The team operates as a single cohesive unit focusing on large-scale research efforts, enjoys pair programming, and frequently hosts research discussions to pursue high-impact work.