About this role
Role Overview
The Performance Engineer, Inference position at Sarvam AI is a full-time, on-site role based in Bengaluru within the infrastructure department. This senior-level opportunity is well-suited for a seasoned machine learning systems professional with over five years of experience who wants to drive production-scale serving performance.
Key Responsibilities
In this capacity, individuals take end-to-end ownership of the production serving path for large distributed models. Responsibilities include integrating artifacts from model and kernel teams into an active multi-node and multi-tenant stack. Professionals read, modify, and extend existing serving frameworks when stock behaviors do not match workload requirements. The work involves building, training, and tuning speculative decoding models and ensuring accurate performance metrics for latency and throughput.
Requirements
Candidates must bring a minimum of five years of experience in machine learning systems, with at least two years specifically focused on production-scale inference serving. Applicants need source-level fluency in at least one major serving framework such as SGLang, vLLM, NVIDIA Dynamo, or TensorRT-LLM, alongside reading-level familiarity with the remaining three. Essential qualifications include hands-on experience serving models exceeding 100 billion parameters across multi-node environments, deep knowledge of distributed serving stacks, and expertise in KV cache internals. Candidates should possess a working command of parallelization primitives, multi-tenant serving models, profiling tools, and possess read-and-modify capabilities in C++ and CUDA. On-call ownership experience for inference SLOs is also required.
Nice to Have
Preferred backgrounds include upstream open-source contributions to major serving frameworks and direct operational experience with Dynamo or llm-d at scale. Candidates who have operated a forked runtime in production, published or shipped specialized speculator work, or managed Mixture-of-Experts serving at scale, long-context scenarios, or multilingual workloads will find their skills align well with the team.
Skills and Tools
About the Company
Sarvam AI operates a multi-node, multi-tenant fleet of advanced hardware to serve diverse model families, ranging from small and large LLMs to Mixture-of-Experts, Indic automatic speech recognition, text-to-speech, streaming, and multimodal models.
How to Apply
Interested candidates should visit the official Sarvam AI careers page to submit their application for the Performance Engineer, Inference role.
Skills and tech stack
Questions candidates ask
Where is this job located?
This full-time role is based on-site in Bengaluru.
What is the experience level required for this role?
Applicants must have 5+ years of experience in ML systems, with at least 2 years dedicated to inference serving at production scale.
What core technologies and frameworks are required?
Candidates need source-level fluency in at least one of SGLang, vLLM, NVIDIA Dynamo, or TensorRT-LLM, as well as proficiency in C++ and CUDA.
What does the Performance Engineering team own at Sarvam AI?
The team owns the metrics the company plans against, including serving speed, operating costs, and overall GPU utilization across the fleet.
How should I apply for this position?
You should apply directly through the official Sarvam AI careers website.