About this role
Managing serverless serving pools, configuring the fleet layer with Kubernetes Gateway API, and building the bare-metal platform underneath consume the daily routine of the Inference Infrastructure Architect. Operating as part of the founding China team at Telnyx, this remote professional works to maximize inference throughput per GPU-dollar while keeping latency and reliability SLOs on a proprietary B300 fleet. The individual shapes both internal AI-agent traffic and external customer inference platforms.
Success in this role requires a background in owning production LLM serving under strict traffic and latency constraints. The right person brings deep operational command of engines like vLLM or SGLang, alongside systems-level performance engineering skills and the ability to evaluate, benchmark, and integrate open-source components into production.
What you will do
- Operate and expand the B300 hardware fleet efficiently while minimizing cost per token and meeting strict latency and reliability service-level objectives.
- Build serverless serving pools using vLLM and SGLang engines featuring continuous batching, prefix caching, and low-precision serving.
- Architect the fleet layer with llm-d and NVIDIA Dynamo on the Kubernetes Gateway API for KV-cache-aware routing and prefill/decode disaggregation.
- Manage the underlying Kubernetes bare-metal platform utilizing GPU Operator, topology-aware scheduling, and Kata Containers for isolated tenants.
- Drive weight logistics and elasticity through peer-to-peer model distribution, warm pools, and inference-metric-driven autoscaling.
- Maintain the dedicated tenant tier, GPU-hour metering, private networking, and the serving infrastructure for model-tuning loops.
- Instrument observability pipelines with DCGM, Prometheus, and OpenTelemetry while conducting capacity planning grounded in roofline math.
What you need
- Demonstrated ownership of production LLM serving under meaningful traffic and latency constraints with a clear explanation of bottlenecks and architectural changes.
- Hands-on experience operating Kubernetes on GPU fleets end to end, covering GPU Operator, device plugins, and topology-aware placement.
- Deep operational command of vLLM or SGLang covering parallelism, quantization, batch settings, and KV-cache configurations.
- Performance engineering expertise at the system level involving engine and DCGM metrics analysis, roofline reasoning, and precise deployment sizing.
- Proficiency in Python and Go for automation, coupled with real Linux, networking, and storage depth.
- Professional proficiency in English and Chinese with the ability to write clear runbooks and design documents.
Nice to have
- Fine-tuning and RL infrastructure including LoRA pipelines and evaluation harnesses
- Real-time voice latency work focused on sub-second time-to-first-token budgets on live calls
- Multi-region deployments and data-residency requirements
Benefits
- Access to the newest hardware running on a globally expanding B300 fleet built end to end by the company
- Opportunity to build a greenfield inference platform as a founding member of the China team
- Open-source first environment supporting upstream contributions and conference travel
- Remote work arrangement based in mainland China with no relocation required
Skills and tech stack
Questions candidates ask
What is the remote work policy for this position?
This role is based in mainland China with no relocation required, allowing candidates to work remotely with a global, async-friendly team.
What hardware and technology stack will I be working with?
You will work with the company's own B300 fleet and an open-source stack that includes Kubernetes on bare metal, vLLM, SGLang, and OpenAI-compatible endpoints.
What are the primary responsibilities regarding inference throughput?
You must maximize useful inference throughput per GPU-dollar while meeting latency and reliability SLOs, and continuously reduce the cost per token.
What should I include when I apply for this position?
Applicants should include a brief description of an inference system they personally improved, detailing the bottleneck, the intervention, and the measured result.