This position offers a remote international setup and suits an experienced infrastructure specialist looking to support advanced AI safety research. The role involves managing and scaling GPU cluster infrastructure to accelerate experimentation at FAR.AI.
Role Overview
This position is for a Senior Software Engineer focusing on GPU Cluster Infrastructure at FAR.AI, operating remotely internationally. The role sits on the Foundations team to support non-profit AI safety research by running high-performance compute platforms, automating research workflows, and scaling experimental capabilities.
Key Responsibilities
Operate the Kubernetes GPU fleet daily, handling node lifecycles, upgrades, driver and image rollouts, safe rollbacks, and capacity planning.
Own batch scheduling and multi-tenancy, managing queues, quotas, priorities, preemption, gang scheduling, and fair share across research teams.
Design and operate storage beneath the fleet, spanning high-performance shared filesystems for datasets and checkpoints, object storage tiers, quotas, and backups.
Ensure fault tolerance for multi-node training runs by managing node health, automated draining, debugging NCCL and fabric issues, tracking down stragglers and flaky GPUs, and building checkpoint patterns.
Harden platform security through identity and access management, network policies, secrets management, workload isolation, and sandboxing for autonomous AI agents.
Integrate new capacity online by acceptance-testing providers on fabric, NCCL, and storage throughput, enforcing SLAs, and utilizing infrastructure as code.
Collaborate directly with research teams on infrastructure issues, turning recurring problems into platform fixes while participating in the on-call rotation, runbooks, and postmortems.
Requirements
Three or more years of experience in systems or infrastructure engineering on production Linux, running GPU, HPC, or large-scale batch platforms, with ownership of at least one system from design through operation.
Experience operating production Kubernetes for GPU workloads with a batch layer on top such as Slurm, Kueue, or Volcano, including quotas, priority, preemption, and node health.
Practical background owning infrastructure as code and observability for a production fleet, provisioning with Terraform or Ansible, deploying with Helm and ArgoCD, and monitoring with Prometheus or equivalents.
Strong programming capability in infrastructure languages like Python, Go, Rust, or C++, maintaining automation and services as shared code.
Strong written communication skills for technical documentation, design docs, incident summaries, and vendor escalations.
Nice to Have
Real depth in distributed training infrastructure including multi-node PyTorch, NCCL debugging, NVIDIA node stacks, InfiniBand or RoCE fabrics, and topology-aware placement.
Experience with distributed storage systems like VAST, Weka, Lustre, Ceph, or object storage at scale.
Cluster security expertise covering admission control, RBAC, node and container hardening, sandboxed runtimes like gVisor, Kata, or Firecracker, and agent isolation.
Familiarity with scheduler internals, Kubernetes scheduler plugins, custom controllers, and multi-provider platforms.
Skills and Tools
Linux
Kubernetes
Slurm
Kueue
Volcano
Terraform
Ansible
Helm
ArgoCD
Prometheus
Python
Go
Rust
C++
PyTorch
NCCL
NVIDIA GPU Operator
DCGM
InfiniBand
RoCE
VAST
Weka
Lustre
Ceph
gVisor
Kata
Firecracker
Salary and Benefits
Salary range of $150K to $275K.
Health insurance with 94% of the premium paid by the organization, starting within one month of the start date.
Retirement plan including a 401(k) with up to a 2% match.
25 days of Paid Time Off (PTO) per year, accrued weekly.
Up to 10 days of paid sick leave per year.
About the Company
FAR.AI is a non-profit AI research institute dedicated to ensuring advanced AI is safe and beneficial. Founded in July 2022, the organization facilitates AI safety research, conducts pre-deployment testing for frontier developers, provides independent evaluations for governments, and supports the broader technical community through grants and co-working spaces.
How to Apply
To apply for this role, please visit the official company careers page.
