Senior Machine Learning Engineer, LLM Inference Optimization
About this role
Engineers who thrive on tackling hard infrastructure problems and optimizing large-scale AI systems will find a natural fit within the Nebius Token Factory team. This group builds fast, reliable, and cost-efficient inference services for frontier models, owning the entire path from model artifacts to production deployment.
As a Senior Machine Learning Engineer on the Applied AI team in London, you will focus on improving latency, throughput, memory efficiency, GPU utilization, and cost per token without compromising model quality. Your day-to-day work involves running engine comparisons, debugging performance regressions, and partnering closely with kernel and platform engineers to diagnose bottlenecks across the model code, runtime, and cluster layers.
Nebius operates globally as a cloud infrastructure provider for the AI economy, giving engineers direct ownership over compute, storage, networking, and applied AI. This role suits individuals who enjoy hands-on optimization projects, writing reproducible benchmarks, and delivering measurable performance gains for real-world workloads.
What you will do
- Own end-to-end optimization work for specific model families, customer endpoints, or serving backends.
- Optimize LLM and VLM endpoints to maximize latency, throughput, memory efficiency, and GPU utilization while controlling cost per token.
- Deploy, configure, benchmark, and extend inference engines such as vLLM, SGLang, and TensorRT-LLM.
- Build and productionize model-compression workflows including quantization, distillation, and accuracy recovery.
- Implement advanced acceleration techniques like speculative decoding, KV-cache optimization, prefix caching, and continuous batching.
- Partner with GPU kernel and platform engineers to diagnose bottlenecks across model code, kernels, runtime, and infrastructure layers.
- Write clear design documents, performance reports, rollout plans, and customer-facing technical explanations.
What you need
- Strong engineering skills in Python and PyTorch.
- Hands-on experience deploying or optimizing high-throughput transformer inference systems for LLMs or VLMs.
- Practical knowledge of modern inference stacks such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, or equivalent systems.
- Solid understanding of transformer inference bottlenecks including KV cache, attention, memory bandwidth, and batching.
- Ability to reason quantitatively about trade-offs involving latency, throughput, quality, and cost.
- Strong communication skills to collaborate effectively with research, kernel, infrastructure, product, and customer teams.
Nice to have
- Experience with quantization-aware training, post-training quantization, FP8, INT8, INT4, AWQ, GPTQ, or SmoothQuant.
- Experience with distillation, speculative decoding, EAGLE, Medusa, or other inference acceleration methods.
- Experience with agentic workloads including tool calling, structured outputs, streaming APIs, and high concurrency.
- Familiarity with CUDA or Triton.
- Open-source contributions to projects like vLLM, SGLang, TensorRT-LLM, FlashInfer, or PyTorch.
Benefits
- Competitive compensation
- Career growth and learning opportunities
- Flexibility and ownership
- Collaborative and innovative culture
- Opportunity to work on impactful AI projects
- International environment and talented teams
Skills and tech stack
Questions candidates ask
What is the focus of the optimization work?
You will focus on improving latency, throughput, memory efficiency, GPU utilization, and cost per token while maintaining model quality and reliability. Your work spans model internals, inference engines, serving architecture, and benchmarking.
Which inference tools and frameworks will I use?
You will deploy, configure, benchmark, and extend inference engines such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, NVIDIA Dynamo, or similar systems.
Who will I collaborate with on the team?
You will work closely with GPU kernel engineers and platform engineers to evaluate serving configurations, resolve performance and quality regressions, and diagnose bottlenecks across multiple infrastructure layers.
Are there opportunities for career growth?
The company offers career growth and learning opportunities within a fast-moving environment characterized by trust, real ownership, and a focus on meaningful impact.