Research Engineer / Performance Engineer, RL Distributed Systems
About this role
Anthropic is building reliable, interpretable, and steerable AI systems, focusing heavily on reinforcement learning to help Claude reason, write code, and act autonomously over long horizons. Within the RL Engineering group, the Distributed Systems team builds and operates the massive, concurrent architecture that powers training, sampling, and environment execution across large fleets of accelerators and hosts.
As a Research Engineer on this team, you will tackle the current performance limits of the system, working across scheduling, data movement, storage, networking, and observability. Because frontier RL runs behave as unusually demanding distributed systems where hardware fails and research needs shift rapidly, this role suits versatile generalists who can reason from first principles. Successful candidates thrive on moving between infrastructure layers, solving complex failures, and ensuring that compute capacity translates directly into learning progress.
What you will do
- Design, build, and operate large-scale distributed systems that drive reinforcement learning across training, sampling, and environment execution.
- Identify and eliminate system bottlenecks across scheduling, data movement, storage, networking, and coordination layers.
- Implement robust fault tolerance mechanisms including failure detection, isolation, and recovery to sustain long-running jobs without manual intervention.
- Develop resource management and autoscaling frameworks to dynamically match shifting compute demands during active runs.
- Create comprehensive observability tools to diagnose throughput drops, stalls, and unexpected execution behavior.
- Produce automation routines and safe operational interfaces for engineers and automated tools to manage runs securely.
- Collaborate with researchers and performance engineers to guarantee training correctness and prevent unintended nondeterminism.
- Eliminate underlying failure classes through incident reviews, rigorous testing, redesign, and clear technical documentation.
What you need
- Demonstrate strong software engineering skills utilizing Python alongside at least one systems programming language such as Rust, C++, or Go.
- Bring proven experience designing, building, and operating large-scale distributed systems within production environments.
- Possess a deep understanding of distributed systems fundamentals covering consistency, coordination, consensus, failure modes, and recovery.
- Apply quantitative reasoning to analyze throughput, latency, and resource costs across compute, memory, storage, and network boundaries.
- Show expertise in debugging complex failures spanning numerous hosts and services, including unreproducible local issues.
- Communicate complex technical concepts clearly through written design documents and detailed incident writeups.
Nice to have
- Experience running ML training or inference infrastructure at scale
- Experience across several layers of the stack, such as scheduling, storage, networking, and orchestration
- Experience building schedulers, autoscalers, or resource management systems
- Experience with container orchestration such as Kubernetes, and with sandboxed or virtualized code execution at scale
- Experience with high-performance networking, RDMA, or collective communication libraries
Skills and tech stack
Questions candidates ask
What is the remote work policy for this role?
Anthropic maintains a location-based hybrid policy. Staff are expected to be in one of the designated offices at least 25% of the time, though specific roles may require additional office time.
What is the compensation range for this position?
The annual salary range for this position is $500,000 to $850,000 USD.
What programming languages and technical skills are required?
Candidates must have strong software engineering skills in Python and at least one systems language such as Rust, C++, or Go, alongside experience operating large-scale distributed systems in production.
Does Anthropic provide visa sponsorship?
Yes, visa sponsorship is available. If an offer is made, the company makes every reasonable effort to secure a visa with the help of a retained immigration lawyer.