Latest AI JobsBrowse Jobs
B

Software Engineer - Inference Platform

BasetenSan FranciscoEPD
Apply on company site
Salary
$180K – $360K
Job type
Full-time
Work mode
Remote
Experience
Mid · 3+ yrs
Posted
5 Oct 2026
Status
Open · checked 8d ago

About this role

Software Engineer- Inference Platform is a full-time engineering position located in San Francisco with Baseten, suited for distributed systems engineers and product-minded generalists who enjoy owning production systems.

Role Overview

The Software Engineer- Inference Platform position at Baseten involves building the distributed runtime that powers large-scale LLM inference. Team members work across the entire stack, spanning from customer-facing deployment developer experiences to the orchestration systems running on Kubernetes and GPU workloads. This role suits professionals who want to make complex infrastructure simple and reliable while focusing on speed, reliability, and cost-efficiency.

Key Responsibilities

Build infrastructure and orchestration systems that deploy and run large-scale distributed LLM inference, handling routing, autoscaling, scheduling, and runtime management.
Design, build, and operate Model APIs with a focus on advanced inference capabilities like structured outputs, tool or function calling, and multimodal serving.
Implement platform fundamentals including API versioning, validation, usage metering, quotas, and authentication.
Instrument deep observability through metrics, traces, and logs while building repeatable benchmarks for speed, reliability, and quality.
Debug and harden complex production systems spanning Kubernetes, distributed runtimes, networking, and GPU workloads.
Partner with Inference Performance engineers to roll out new optimizations to customers.
Own projects end to end from architecture through deployment, monitoring, and iteration on user feedback.

Requirements

Bachelor's, Master's, or Ph.D. in Computer Science, Engineering, or a related field, or equivalent practical experience.
Three or more years of experience building and operating distributed systems, backend infrastructure, or large-scale APIs focused on reliability, latency, and scale.
Demonstrated track record of owning low-latency, reliable backend services including rate limiting, authentication, quotas, metering, and migrations.
Infrastructure instincts covering profiling, tracing, capacity planning, and SLO management.
Comfort debugging performance and reliability issues across multiple stack layers.
Strong focus on developer experience and how systems are utilized by users.
Eagerness to learn new languages, frameworks, and systems with an interest in inference engineering.
Excellent written communication and collaboration skills for creating clear design docs and cross-functional work.

Nice to Have

Experience with or contributions to LLM inference engines and frameworks such as vLLM, SGLang, TensorRT-LLM, TGI, or Dynamo.
Deep Kubernetes experience including operators and custom resources, plus familiarity with service meshes or API gateways.
Experience with distributed scheduling, autoscaling, or service orchestration.
Experience operating GPU workloads in production.
Background in developer-facing infrastructure or APIs, or contributions to open-source infrastructure or machine learning systems.
Familiarity with observability tooling, CI/CD systems, or release automation.

Skills and Tools

Kubernetes
Distributed Runtimes
LLM Inference Engines (vLLM, SGLang, TensorRT-LLM, TGI, Dynamo)
APIs
GPU Workloads

Salary and Benefits

Salary range between $180K and $360K with offered equity.
100% coverage of medical, dental, and vision insurance for employees and dependents within the U.S.
Flexible PTO policy including a company-wide Winter Break from Christmas Eve to New Year's Day.
Paid parental leave.
Fertility and family-building stipend through Carrot.
Company-facilitated 401(k) for U.S. employees.
Exposure to various machine learning startups for learning and networking.

About the Company

Baseten powers mission-critical inference for AI companies such as Cursor, Notion, OpenEvidence, Abridge, Clay, Gamma, and Writer. The organization unites applied AI research, flexible infrastructure, and developer tooling to help companies operating at the frontier of AI bring models into production. Backed by investors like Altimeter Capital, Conviction Partners, and Spark Capital, Baseten provides equal employment opportunities and maintains a diverse and inclusive workplace.

How to Apply

Apply on the official company careers page.

Skills and tech stack

KubernetesvLLMSGLangTensorRT-LLMTGIDynamoLLMSparkMachine LearningDistributed Systems

Questions candidates ask

What is the salary and equity for the Software Engineer- Inference Platform role?

The salary ranges from $180K to $360K, and the position includes offered equity.

Where is the position located?

The role is located in San Francisco.

Is prior machine learning or LLM experience mandatory?

Prior machine learning or LLM experience is not required, though experience with model serving or inference systems is considered a plus.

What level of experience is required for this position?

Candidates must have 3+ years building and operating distributed systems, backend infrastructure, or large-scale APIs.

What benefits are provided for U.S. employees?

U.S. employees receive 100% coverage of medical, dental, and vision insurance for themselves and dependents, and a company-facilitated 401(k).

Ready to apply?

The application is on the official Baseten careers page.

Apply on company site