Latest AI JobsBrowse Jobs
Anyscale logo

Distributed LLM Inference Engineer

AnyscaleSan FranciscoEngineering
Apply on company site
Salary
$170K - $245K USD
Job type
Full-time
Work mode
Remote
Posted
27 May 2026
Status
Open · checked 7d ago

About this role

Role Overview

The Distributed LLM Inference Engineer position at Production in San Francisco offers a remote work option for engineering professionals focused on cutting-edge AI infrastructure. This role suits individuals passionate about pushing the performance boundaries of large-scale machine learning inference and building market-leading systems.

Key Responsibilities

Build and iterate rapidly alongside product teams to deliver comprehensive batch and online inference solutions at scale for customers and open-source Ray users.
Connect across the technology stack by integrating Ray Data and large language model engines to deliver low-cost, high-scale machine learning inference optimizations.
Collaborate with open-source communities, integrate tools like vLLM, and contribute improvements directly back to open-source projects.
Monitor cutting-edge research and open-source advancements, putting industry best practices into practical implementation and expansion.

Requirements

Experience operating machine learning inference at high scale, focusing on high throughput and low latency.
Practical familiarity with deep learning concepts and frameworks such as PyTorch.
Strong foundational knowledge regarding distributed systems and the inherent challenges of machine learning inference.

Nice to Have

Practical knowledge of machine learning systems.
Prior hands-on experience utilizing Ray.
Collaborative background working with open-source LLM engines, including vLLM and TensorRT-LLM.
Past code contributions to deep learning frameworks like PyTorch and TensorFlow.
Contributions directed toward deep learning compilers such as Triton, TVM, and MLIR.
Previous professional exposure working directly with GPUs and CUDA.

Skills and Tools

PyTorch
TensorFlow
Large Language Models (LLMs)
Distributed Systems
Ray and Ray Data
vLLM and TensorRT-LLM
GPUs and CUDA

Salary and Benefits

The salary range for this position is $170,000 to $245,000 USD per year, determined through a market-based, data-driven approach. Eligible team members also receive stock options and comprehensive benefits including:

Healthcare plans with 99% of premiums covered by the company for both employees and their dependents.
401(k) retirement plan.
Education and wellbeing stipend.
Paid parental leave and fertility benefits.
Paid time off and commute reimbursement.
Fully covered in-office meals.

About the Company

Production is commercializing Ray, an open-source framework designed for scalable machine learning and distributed computing. Major global companies utilize Ray in their technical stacks to accelerate artificial intelligence applications. The organization is backed by prominent investors including Andreessen Horowitz, NEA, and Addition, having raised over $250 million to date.

How to Apply

Interested candidates should visit the official company careers page to submit an application for this role.

Skills and tech stack

PyTorchTensorFlowLLM

Questions candidates ask

What is the salary range for the Distributed LLM Inference Engineer?

The target salary ranges from $170,000 to $245,000 USD per year, based on market data.

Is this position remote?

Yes, this role is listed with remote work options based out of San Francisco.

What technical skills are required?

Candidates need familiarity with large-scale machine learning inference, deep learning frameworks like PyTorch, and a solid understanding of distributed systems.

What benefits are included with the job?

Benefits include stock options, 99% employer-covered healthcare for employees and dependents, a 401(k) plan, education and wellbeing stipends, paid parental leave, fertility benefits, paid time off, commute reimbursement, and covered in-office meals.

What open-source technologies are used in this role?

Engineers work extensively with Ray, Ray Data, and open-source LLM engines such as vLLM.

Ready to apply?

The application is on the official Anyscale careers page.

Apply on company site