About this role
The xAI team builds advanced AI systems designed to accurately understand the universe and aid humanity in its pursuit of knowledge. Operating in a small, highly motivated environment with a flat organizational structure, the group relies on hands-on contributors who value engineering excellence and curiosity. This Director of Site Operations role exists to own node and rack uptime for the company's AI supercompute cluster, ensuring exceptional reliability across multiple 24-7 sites.
The incoming leader will direct a large organization of site managers, shift supervisors, technicians, and site reliability engineers while driving fault mitigation and system recovery at scale. Ideal candidates possess a strong background in managing complex compute environments and demonstrate the ability to balance rigorous operational oversight with cross-functional collaboration. Anyone stepping into this position must maintain a strong work ethic, prioritize effectively, and communicate concisely with teammates.
What you will do
- Manage node, rack, and cluster health across five or more 24-7 sites to maintain exceptional uptime and meet customer service level agreements.
- Direct an organization of over 250 personnel, including site managers, shift supervisors, and technicians operating across four shifts.
- Oversee the systematic recovery of failed hardware through physical and command-line interventions to minimize repair times.
- Collaborate with facilities operations, network engineering, and tenant representatives to limit downtime and coordinate cluster upgrades.
- Direct external vendors through hardware rework and field operations to ensure capacity work matches cluster requirements.
- Lead site reliability engineering teams to monitor cluster health, conduct root cause analyses, and establish reliability procedures.
- Drive continual improvement initiatives by utilizing operational data to optimize team resources and enhance performance metrics.
- Lead incident response during cluster-impacting events by providing clear direction and maintaining communication with partners.
What you need
- Hold a bachelor's degree alongside at least seven years of experience in large-scale operations, including five years leading people leaders of technical teams.
- Alternatively, possess ten or more years of large-scale operations experience with at least five years leading people leaders of technical teams.
- Demonstrate a proven ability to lead large, multi-site, 24-7 operations organizations in fast-paced environments.
- Bring deep expertise in server hardware, cluster reliability, and data center technologies throughout their lifecycle.
- Show experience supporting compute-heavy environments such as artificial intelligence, machine learning, or high-performance computing.
- Maintain the physical capability to handle data center tasks, including lifting up to 50 pounds unassisted and standing for extended periods.
- Willingness to travel frequently to data center locations and work extended hours or weekends as needed.
Nice to have
- Familiarity with tooling and automation platforms such as Jira, Python, and Bash used for cluster monitoring.
Skills and tech stack
Questions candidates ask
What is the primary focus of the Director of Site Operations?
The role owns node and rack uptime for the AI supercompute cluster and acts as the extreme owner of cluster health and customer service level agreements across multiple sites operating 24-7.
How large is the team managed by this position?
The director leads a 250-plus person organization consisting of site managers, shift supervisors, technicians, and the site reliability engineering team.
What physical and travel demands are associated with the job?
The position requires frequent travel to data center locations, the physical capability to lift up to 50 lbs unassisted, and a willingness to work extended hours or weekends.