Tech Lead Manager, GPU Cluster Infrastructure
About this role
The day-to-day work for the Tech Lead Manager, GPU Cluster Infrastructure involves setting the technical direction and roadmap for the compute platform. This individual will be responsible for deciding which systems are run, how jobs are scheduled and stored across different providers, and defining key performance metrics such as utilization and failure rates. A significant part of the role involves deep technical work, including owning the architecture for scheduling and storage, and debugging complex failures that span multiple layers, such as node health or multi-node job hangs. Furthermore, this role includes the responsibility of hiring and mentoring a small team of senior engineers.
This position exists to ensure the infrastructure can scale effectively to support FAR.AI's growing research needs, particularly as experiments move to frontier open-weight models and AI agents working directly on the cluster. The role bridges the gap between cutting-edge research and robust, scalable infrastructure management. It is ideal for a technical leader who enjoys both hands-on system design and team development, thrives in an environment where infrastructure is integral to the research process, and is motivated to build systems that accelerate scientific progress.
What you will do
- Set the platform's technical direction and own its roadmap for the GPU cluster infrastructure.
- Determine the systems to be run, including scheduling and storage across various providers.
- Define metrics for performance, such as utilization and queue wait times, and failure rates.
- Own the architecture for scheduling and storage design.
- Debug complex failures that cross layers, such as node health, GPU, and fabric faults.
- Hire and grow a small team of senior engineers.
- Establish the security direction for the shared cluster.
What you need
- Possesses the technical expertise required to manage and operate large-scale GPU cluster infrastructure.
- Demonstrated ability to own and drive the technical roadmap for a complex platform.
- Experience in designing and implementing scheduling and storage solutions for distributed systems is necessary.
- Proven capability in debugging intricate, multi-layered system failures is required.
- Experience in leading and growing a team of senior engineers will be a key asset.
- A strong understanding of infrastructure as code practices is beneficial.
- Experience in setting security posture for shared computational environments is important.
Nice to have
- Experience with AI agents working directly on clusters
Benefits
- $225K – $325K salary
Skills and tech stack
Questions candidates ask
Is this position remote?
This role is based in a Remote (International) location.
What is the salary range for this role?
The salary offered for this position ranges from $225K to $325K.
What is the primary technology stack involved?
The infrastructure involves running research on a mix of bare-metal and managed Kubernetes GPU clusters from multiple providers.
What is the team structure like?
This role is within the Foundations team, which manages the compute platform, tools, and workflows.