Sr. AI Engineer, Supercomputing, Special Programs - Top Secret Clearance
About this role
The day-to-day work for the Sr. AI Engineer, Compute Infrastructure, Special Programs involves designing and building essential tooling and working alongside cleared engineers at classified sites to ensure operational effectiveness. This role is critical because the quality of the infrastructure built directly dictates how well these engineers can operate in secure environments where direct support is unavailable. The core responsibility is ensuring all tooling is deterministic, complete, well-tested, and foolproof.
This position exists to provide robust, reliable deployment and infrastructure solutions for SpaceX's special programs. The engineer will focus on creating deployment generators, managing pipeline automation, and implementing profile-driven rendering across various environments, including public cloud, on-premise, and air-gapped targets. Individuals who excel in building highly reliable, automated systems under strict operational constraints would thrive in this role.
What you will do
- Design and build the deployment generator.
- Create and maintain platform documentation.
- Build the bundle pipeline.
- Implement profile-driven rendering for public cloud, enterprise on-prem, and classified air-gap targets based on profile selection.
- Build and maintain the cross-profile CI matrix.
- Build the testing and validation framework.
- Integrate with the existing Supercompute team's operators, Helm charts, and CRDs by consuming their work as inputs to the generator without modifying production infrastructure.
- Own the monitoring stack migration.
What you need
- A Bachelor's degree in a relevant technical field is required.
- Candidates must possess more than five years of experience working with Go Lang and/or Python.
- A minimum of five years of experience working with Kubernetes or similar tooling for automating, developing, and scaling containerized applications is necessary.
- The ability to obtain and maintain a Top Secret or Top Secret SCI clearance is a mandatory requirement.
- Experience building CI/CD pipelines and deployment automation tools such as Buildkite or GitHub Actions is preferred.
- Familiarity with container image management is required.
- Experience with Infrastructure-as-Code tools like Pulumi or Terraform, along with an understanding of state management tradeoffs, is beneficial.
- Understanding of Linux systems is a fundamental requirement.
Nice to have
- Experience building air-gapped or disconnected deployment tooling where all dependencies must be pre-staged.
- Familiarity with GPU infrastructure, including NVIDIA drivers, CUDA, NCCL, InfiniBand/RoCE networking.
- Experience with PXE boot, cloud-init, squashfs, or other bare metal provisioning technologies.
- Own the bare metal provisioning pipeline.
Benefits
- Comprehensive medical, vision, and dental coverage
- Access to a 401(k) retirement plan
- Short and long-term disability insurance
- Life insurance
- Paid parental leave
- 3 weeks of paid vacation
- 10 or more paid holidays per year
Skills and tech stack
Questions candidates ask
What is the remote policy for this position?
The posting indicates that 20% travel may be required to government sites. Employees must be willing to work extended hours and weekends as needed.
What is the salary range for this role?
The salary range for the Sr. AI Engineer is listed as $220,000.00 to $350,000.00 per year.
What is the required security clearance?
This position requires successfully obtaining and maintaining a Top Secret Security Clearance as a condition of employment.
What is the team environment like?
The role involves working with cleared engineers at classified sites, where the quality of the built tooling directly determines the engineers' operational effectiveness.