Latest AI JobsBrowse Jobs
B

Software Engineer - Inference Performance

BasetenSan FranciscoEPD
Apply on company site
Salary
$180K – $360K
Job type
Full-time
Work mode
Remote
Experience
Senior
Posted
5 Oct 2026
Status
Open · checked 7d ago

About this role

This role suits engineers who want to make demanding artificial intelligence workloads run faster and more efficiently in a fast-paced startup environment. The Software Engineer- Inference Performance position is based in San Francisco at Baseten.

Role Overview

The Software Engineer- Inference Performance position focuses on optimizing large-scale artificial intelligence workloads and making inference engines run faster and more efficiently. Professionals in this role work across the complete technology stack, from inference engines and runtimes to scheduling, serving, and routing.

Key Responsibilities

Implement and productionize advanced inference techniques such as quantization, speculative decoding, KV-cache reuse, chunked prefill, LoRA, guided generation for structured outputs, and custom scheduling algorithms.
Profile and optimize inference operations from kernel launch overhead and memory layout up to request scheduling and cache-aware routing.
Translate technical performance improvements into cost savings by increasing tokens per GPU-hour and hardware utilization.
Bring up and tune new model architectures on new hardware platforms quickly following their release.
Construct benchmarking frameworks to measure real-world performance across varying model architectures, batch sizes, sequence lengths, and hardware configurations.
Contribute upstream to open-source inference engines like vLLM, SGLang, and TensorRT-LLM while partnering with internal teams.

Requirements

Bachelor's, Master's, or Ph.D. degree in Computer Science, Engineering, Mathematics, or a related field.
Experience with general-purpose programming languages including Python or C++.
Familiarity with large language model optimization techniques such as quantization, speculative decoding, and continuous batching.
Working knowledge of machine learning libraries like PyTorch, TensorRT, or TensorRT-LLM.
Demonstrated interest and experience working with large language models.
Deep understanding of graphics processing unit architecture.

Nice to Have

Proficiency in performance enhancement for software systems, specifically related to large language models.
Prior contributions to vLLM, SGLang, TensorRT-LLM, or other inference engines.
Background in large-scale distributed serving involving autoscaling, load balancing, and multi-region or multi-cloud capacity.
Experience writing or optimizing graphics processing unit kernels using CUDA, Triton, CUTLASS, or similar tools.
Practical experience with quantization methods like FP8, FP4, AWQ, or GPTQ, or speculative decoding in production environments.
Strong adherence to software engineering principles and a track record of deploying artificial intelligence and machine learning inference solutions.

Skills and Tools

Python
C++
PyTorch
TensorRT
TensorRT-LLM
vLLM
SGLang
CUDA
Triton
CUTLASS

Salary and Benefits

Salary range of $180K to $360K with equity offers
Full coverage of medical, dental, and vision insurance for employees and dependents within the United States
Flexible paid time off policy alongside a company-wide winter break closure
Paid parental leave
Fertility and family-building stipend provided through Carrot
Company-facilitated 401(k) program for United States employees
Exposure to various machine learning startups

About the Company

Baseten powers mission-critical inference for artificial intelligence companies by uniting applied AI research, flexible infrastructure, and developer tooling. The organization recently raised a Series F funding round led by Altimeter Capital, Conviction Partners, and Spark Capital.

How to Apply

To apply for this position, please visit the official Baseten careers page to submit your application.

Skills and tech stack

PythonC++PyTorchTensorRTTensorRT-LLMvLLMSGLangCUDATritonCUTLASSLLMLLMsSparkMachine Learning

Questions candidates ask

What is the salary range for the Software Engineer- Inference Performance role?

The salary ranges from $180K to $360K with included equity offers.

Where is this job located?

This position is located in San Francisco.

What programming languages are required?

Applicants need experience with one or more general-purpose programming languages, specifically Python or C++.

Are there specific machine learning libraries mentioned in the requirements?

Yes, candidates should have strong familiarity with machine learning libraries, especially PyTorch, TensorRT, or TensorRT-LLM.

What educational background is required?

A Bachelor's, Master's, or Ph.D. degree in Computer Science, Engineering, Mathematics, or a related field is required.

Ready to apply?

The application is on the official Baseten careers page.

Apply on company site