Latest AI JobsBrowse Jobs
RunPod logo

Senior ML Systems Engineer, Inference

RunPodRemote - USAEngineering
Apply on company site
Salary
$150K - $220K
Job type
Full-time
Work mode
Remote
Posted
25 Sept 2026
Status
Open · checked 7d ago

About this role

RunPod is seeking a Senior ML Systems Engineer, Inference to join its remote team in the USA. This full-time hands-on engineering role suits an experienced professional with a strong background in Python and ML systems who wants to lead the effort in making the platform the fastest and most cost-efficient place to run large language models.

Role Overview

In this position, the engineer owns LLM serving performance from end to end. The role focuses on measuring, understanding, and enhancing performance across diverse models, hardware generations, and workloads. The individual finds real bottlenecks, fixes them, and turns those solutions into production-ready runtimes that benefit a million developers.

Key Responsibilities

Establish methods to measure inference performance, including time to first token, throughput, inter-token latency, and cost per token, while building rigorous and repeatable tooling.
Profile and diagnose performance issues throughout the serving stack, covering scheduling, memory management, kernels, and interconnect.
Enhance serving efficiency for state-of-the-art models on single-node and multi-node GPU deployments.
Translate findings into production-ready configurations, runtimes, and defaults that automatically benefit customers.
Collaborate with product and infrastructure teams to define how inference is offered on the platform.
Monitor the fast-moving inference ecosystem and open-source community to determine what to adopt, build, or contribute back.
Trace bottlenecks directly in the serving engine or runtime and execute fixes beyond basic configuration tuning.

Requirements

Minimum of 5 years of professional system engineering experience.
Deep, hands-on production or benchmark-scale experience with serving engines like vLLM or SGLang.
Strong software engineering capabilities in Python, with comfort working inside performance-critical codebases.
Solid understanding of the drivers of LLM inference performance, such as batching, parallelism, memory, and the trade-offs between throughput and latency.
Practical experience with modern inference optimization methods, including speculative decoding, quantization, or distributed serving.
Rigorous approach to benchmarking and performance analysis, alongside familiarity with GPU profiling tools.
Ability to clearly articulate results in writing to guide decision-making.

Nice to Have

Experience tuning or writing GPU kernels in Triton or CUDA.
Prior contributions to machine learning systems or inference projects.
Familiarity with high-speed networking and multi-node GPU systems.
Background working at an organization where latency and inference cost served as core business metrics.

Skills and Tools

Python
LLM serving engines (vLLM, SGLang)
GPU profiling tools
Optimization techniques (quantization, speculative decoding, distributed serving)
CUDA or Triton (preferred)

Salary and Benefits

Competitive base pay ranging from $150,000 to $220,000, narrowed during the interview based on experience, qualifications, and location.
Meaningful equity through stock options in a fast-growing company.
Generous medical, dental, and vision insurance plans.
Flexible paid time off policy.
$1,200 home office and equipment stipend for workspace setup.
Remote work environment utilizing Slack for collaboration.

About the Company

Runpod operates as the AI Developer Cloud, providing a platform where over one million developers experiment, train, fine-tune, deploy, and scale artificial intelligence. The company closed a $100M Series A in June 2026 and operates as a small, remote-first team focused on ownership, speed, and scale. The organization is an equal opportunity employer committed to a diverse, inclusive workplace, welcoming eligible candidates to work in the United States without visa sponsorship availability.

How to Apply

Interested candidates should visit the official Runpod careers page at runpod.io to submit an application for this remote position.

Skills and tech stack

PythonLLMCUDA

Questions candidates ask

What is the salary range for the Senior ML Systems Engineer, Inference role?

The competitive base pay ranges from $150,000 to $220,000, depending on factors such as candidate experience, qualifications, and location.

Is this position remote?

Yes, this is a remote position based in the USA, operating within a remote-first team environment.

What experience is required for this job?

Candidates must have 5 or more years of professional system engineering experience, along with deep hands-on experience using vLLM, SGLang, or a comparable serving engine in production or at benchmark scale.

Does Runpod offer visa sponsorship for this role?

Runpod welcomes qualified candidates eligible to work in the United States, but is currently unable to sponsor employment visas.

What benefits are included besides salary?

Benefits include stock options, generous medical, dental, and vision plans, flexible paid time off, and a $1,200 home office and equipment stipend.

Ready to apply?

The application is on the official RunPod careers page.

Apply on company site