Sr. AI Engineer, Supercomputing, Special Programs - Top Secret Clearance
About this role
This role is suited for an experienced AI Engineer who enjoys building robust, deterministic infrastructure for complex systems. The individual will be instrumental in developing the tooling and deployment mechanisms that allow cleared engineers to operate effectively in secure, classified environments. The core focus is ensuring that the infrastructure built is complete, well-tested, and foolproof, as its quality directly impacts the operational effectiveness of the engineering teams. This position exists to bridge the gap between AI/ML development and the secure, reliable deployment of those systems across various targets, including public cloud and air-gapped sites.
Responsibilities involve designing critical components like the deployment generator and building comprehensive testing and validation frameworks. The engineer will integrate with existing platform components by consuming their outputs to drive the generation process without altering production infrastructure. A key duty includes owning the migration of the monitoring stack and managing the cross-profile CI matrix. This role requires a methodical approach to infrastructure automation and a commitment to creating reliable systems under strict operational constraints.
Candidates who thrive on solving complex infrastructure challenges, possess deep experience in automation, and are comfortable working with security protocols are well-suited for this opportunity. Success in this role depends on the ability to engineer solutions that are reliable, secure, and capable of functioning in highly constrained environments.
What you will do
- Design and build the deployment generator.
- Create and maintain platform documentation.
- Build the bundle pipeline.
- Implement profile-driven rendering for public cloud, enterprise on-prem, and classified air-gap targets based on profile selection.
- Build and maintain the cross-profile CI matrix.
- Build the testing and validation framework.
- Integrate with the existing Supercompute team's operators, Helm charts, and CRDs by consuming their work as inputs to the generator without modifying production infrastructure.
- Own the monitoring stack migration.
What you need
- A Bachelor's degree in a field such as computer science, mathematics, computer engineering, physics, data science, or engineering discipline is required.
- Candidates must possess more than five years of experience working with Go Lang and/or Python.
- Five or more years of experience working with Kubernetes or similar tooling for automating, developing, and scaling containerized applications is necessary.
- The role requires the ability to obtain and maintain a Top Secret or Top Secret SCI clearance.
- Experience building CI/CD pipelines and deployment automation, such as Buildkite or GitHub Actions, is preferred.
- Familiarity with container image management is a desired skill.
- Experience with Infrastructure-as-Code tools like Pulumi or Terraform, along with an understanding of state management tradeoffs, is beneficial.
- A solid understanding of Linux systems is mandatory.
Nice to have
- Experience building air-gapped or disconnected deployment tooling where all dependencies must be pre-staged
- Familiarity with GPU infrastructure, including NVIDIA drivers, CUDA, NCCL, InfiniBand/RoCE networking
- Experience with PXE boot, cloud-init, squashfs, or other bare metal provisioning technologies
- Own the bare metal provisioning pipeline
Benefits
- Comprehensive medical, vision, and dental coverage
- Access to a 401(k) retirement plan
- Short and long-term disability insurance
- Life insurance
- Paid parental leave
- 3 weeks of paid vacation
- 10 or more paid holidays per year
Skills and tech stack
Questions candidates ask
What is the required security clearance for this role?
This position requires successfully obtaining and maintaining a Top Secret Security Clearance as a condition of employment. While the clearance may not be immediately necessary upon hire, the company encourages initiating the application process promptly.
What is the salary range for this position?
The salary range for the Sr. AI Engineer is listed between $220,000.00 and $350,000.00 per year. The actual level and base salary will be determined on a case-by-case basis based on knowledge, skills, education, and experience.
What is the remote work policy?
The posting indicates that 20% travel may be required to government sites. This position is based in Hawthorne, CA.
What is the primary technology stack used?
The core programming languages mentioned are Go Lang and/or Python, and the infrastructure tooling includes Kubernetes, Buildkite, GitHub Actions, Pulumi, and Terraform.