Staff Backend Engineer - K8 (Envoy)
About this role
The infrastructure team at Coupang builds the next-generation AI Inference Gateway platform to serve mission-critical machine learning and generative AI workloads at scale. As a Staff Backend Engineer based in Bengaluru, you will design and develop high-performance request routing, model endpoint abstraction, traffic shaping, load balancing, and failover mechanisms for inference services. You will also collaborate closely with platform and product leaders to shape scalable infrastructure across cloud and on-prem environments.
This role exists to drive the technical vision for reliable AI infrastructure, handle complex production challenges, and establish common platform standards across the organization. An ideal candidate brings extensive experience in distributed systems, a strong grasp of Kubernetes and Envoy-related technologies, and a hands-on background in languages like Go, Java, or Python.
What you will do
- Architect and build the next-generation AI Inference Gateway platform to support large-scale machine learning and generative workloads.
- Design request routing, model endpoint abstraction, load balancing, caching, and policy enforcement mechanisms for inference services.
- Develop critical infrastructure components using Go, Java, or Python while prioritizing performance and operational excellence.
- Implement multi-tenant platform features including authentication, authorization, quota management, rate limiting, and governance controls.
- Partner with machine learning platform, model serving, data, and product engineering teams to ensure seamless workload operations.
- Lead architecture reviews, elevate engineering standards, and mentor senior engineers across multiple squads.
- Investigate complex production issues, conduct root-cause analysis, and deploy long-term architectural fixes.
What you need
- Bring at least 8 years of professional software development experience.
- Possess 5 or more years of experience designing and operating large-scale distributed systems in production environments.
- Demonstrate strong hands-on programming expertise in Go, Java, or Python.
- Show a proven track record of building highly available, mission-critical platform or infrastructure services.
- Have direct experience designing API platforms, service gateways, service mesh, or large-scale networking infrastructure.
- Maintain a deep understanding of microservices architecture and cloud-native technologies.
- Bring production experience working with Kubernetes and containerized workloads, alongside public cloud operations on AWS, Azure, or GCP.
Nice to have
- Experience building AI/ML inference platforms, LLM gateways, model serving infrastructure, or GPU-accelerated workloads
- Deep expertise in Kubernetes ecosystem technologies such as Gateway API, Ingress Controllers, Service Mesh like Istio, Linkerd, or Envoy
- Familiarity with inference serving frameworks including vLLM, Triton Inference Server, TensorRT-LLM, Ray Serve, KServe, or SGLang
- Understanding of high-performance networking, gRPC, HTTP/2, streaming protocols, and service proxy architectures
- Experience with distributed data systems such as Kafka, Cassandra, Redis, or MongoDB
Skills and tech stack
Questions candidates ask
What is the remote work policy for this role?
Coupang operates on a hybrid work model where employees are required to work at least 3 days in the office per week, with the flexibility to work from home 2 days a week depending on role requirements.
What technologies will I be working with?
You will develop critical infrastructure components in Go, Java, or Python, while working with technologies like AWS, Kafka, Kubernetes, Kubeflow, Argo CD, and gRPC.
What will I actually be working on?
You will architect and build Coupang's next-generation AI Inference Gateway platform, design high-performance request routing and model endpoint abstraction, and develop multi-tenant platform capabilities.