Staff Software Engineer - AI Compute, Together Cloud
About this role
Scaling globally distributed AI infrastructure presents significant bottlenecks in high-performance GPU compute, low-latency data center networking, and fault-tolerant scheduling across thousands of specialized hardware units. Together AI addresses these systemic challenges by developing an end-to-end AI Native Cloud platform that powers rapid LLM inference, reinforcement learning, and fine-tuning workloads.
As a Staff Software Engineer for AI Compute, the incoming technical leader will build and architect the foundational IaaS layer that virtualizes state-of-the-art machine learning hardware, configures InfiniBand and RoCEv2 fabrics, and automates data center bootstrapping from blank pages to production readiness. Working directly on the core infrastructure powering both internal SaaS offerings and external self-serve Kubernetes and Slurm clusters, this person will define global scheduling policies, streamline GPU and network virtualization stacks, and establish engineering standards that enable the broader organization to ship reliable systems at scale.
What you will do
- Manage the GPU and network virtualization stack, including hypervisor, kernel, and software-defined networking components that maintain strong isolation across heterogeneous hardware.
- Design and roadmap the in-datacenter IaaS layer, Kubernetes operators, and provisioning libraries for virtual machines, parallel filesystems, and InfiniBand partitions.
- Architect the distributed GPU scheduling and global management plane responsible for scaling cluster limits and automating capacity onboarding across dozens of facilities.
- Build automated monitoring and remediation strategies for fault detection, node isolation, and recovery to sustain distributed training and inference workloads through hardware failures.
- Drive technical direction by leading design reviews, resolving cross-cutting architectural dependencies, and defining engineering standards across multiple teams.
- Mentor senior and junior engineers while elevating organizational expertise in virtualization, data center networking, and infrastructure automation.
What you need
- Seven or more years of professional software development experience backed by expert proficiency in at least one backend language, preferably Golang.
- Demonstrated track record of independently owning the architecture of large distributed systems from initial design to scaled production environments.
- Extensive experience building and operating globally distributed microservice architectures across major cloud providers like AWS, Azure, or GCP.
- Expert systems knowledge spanning concurrency, memory management, performant input-output operations, and global-scale compute, networking, and storage.
- Proven technical leadership through mentoring senior peers, conducting design reviews, and driving cross-team alignment without direct reporting lines.
- Hands-on operational experience with infrastructure automation tools such as Terraform and Ansible, observability stacks like Prometheus and Grafana, and CI/CD pipelines.
Nice to have
- Deep Kubernetes internals experience including custom operators, plugins, or custom schedulers
- Extensive experience with virtual machines and hypervisors such as QEMU/KVM, cloud-hypervisor, and VFIO
- Familiarity with data center networking technologies like VLAN, VXLAN, VPC, and OVS/OVN
- Prior work on high-performance compute, storage, or GPU and InfiniBand virtualization
- Experience building large-scale Infrastructure-as-a-Service or Platform-as-a-Service systems
Benefits
- Competitive compensation ranging from $260,000 to $300,000 base salary
- Startup equity
- Health insurance
- Flexibility in terms of remote work
Skills and tech stack
Questions candidates ask
What is the remote work policy for this role?
The company offers flexibility in terms of remote work for this full-time position based out of San Francisco.
What is the salary range for this position?
The US base salary range for this full-time role is $260,000 to $300,000, supplemented by equity and benefits.
What core technologies will I work with daily?
You will work with Golang, Kubernetes, Slurm, InfiniBand, RoCEv2 fabrics, and various virtualization and infrastructure automation tools.
How is individual compensation determined?
Salary ranges are determined by location, level, and role, with individual compensation finalized based on experience, skills, and job-related knowledge.