Role Overview
Handshake is seeking a Senior Software Engineer, LLM Platform to join the Data and ML Platform team in San Francisco, CA. This full-time position suits an experienced developer who wants to build shared data, machine learning, and large language model infrastructure powering production workflows and Handshake AI.
Key Responsibilities
Build and operate LiteLLM-based AI gateways and shared LLM clients.
Own provider and model onboarding, routing, failover, rate limits, and capacity planning.
Build self-service virtual-key, model-access, budget, and credential-management workflows.
Establish service-level objectives, observability, cost attribution, and alerts for production LLM traffic.
Safely qualify and roll out new models, providers, software development kits, and gateway configurations.
Support hosted and self-hosted inference through a consistent platform interface.
Partner with product, AI, and forward-deployed engineering teams to turn recurring delivery problems into paved-platform capabilities.
Contribute to the broader Data and ML Platform, including workflow orchestration, model serving, shared cloud infrastructure, and developer tooling.
Participate in team on-call and support, improving runbooks, automation, and reliability across owned platform services.
Requirements
Strong production software engineering experience in Python, TypeScript, Go, or a similar language.
Experience building or operating high-throughput API gateways, proxies, or multi-tenant platform services.
Hands-on experience with Kubernetes, Terraform, CI/CD, and production service ownership.
Practical experience with large language model provider APIs, including streaming, long-running requests, retries, timeouts, cancellation, and rate limits.
Experience with authentication, quotas, credential management, tenant isolation, and auditability.
Experience building observability for distributed systems and leading production incident response.
Experience with usage metering, cost attribution, capacity planning, or financial operations.
Experience with data or machine learning platform systems such as BigQuery, Airflow, streaming pipelines, model serving, or machine learning observability.
Strong judgment in ambiguous environments and a bias toward simple, reliable systems that scale.
Nice to Have
Experience with LiteLLM, Portkey, or a comparable multi-provider AI gateway.
Familiarity with vLLM, Modal, Ray, Triton, PyTorch, or graphical processing unit-backed serving.
Experience with Temporal or another durable-execution system for long-running large language model requests.
Background in evaluation, batch inference, fine-tuning, reinforcement learning, or other post-training infrastructure.
Knowledge of agent runtimes, sandbox infrastructure, model context protocol, tool use, or coding-agent infrastructure.
Skills and Tools
Programming Languages: Python, TypeScript, Go
Infrastructure and Cloud: Kubernetes, Terraform, Spacelift, GCP, Cloudflare, Tailscale
Data and Orchestration: BigQuery, Airflow, Dataflow/Beam, Datastream, PostgreSQL, Redis
AI and Observability: LiteLLM, OpenAI, Anthropic, Gemini, OpenRouter, vLLM, Modal, Anyscale/Ray, Datadog, Arize, Temporal
Salary and Benefits
Annual Salary: $176,000 to $220,000 USD
Equity in a fast-growing company
Financial Wellness: 401(k) match, competitive compensation, financial coaching
Family Support: Paid parental leave, fertility benefits, parental coaching
Wellbeing: Medical, dental, and vision, mental health support, $500 wellness stipend
Growth: $2,000 learning stipend, ongoing development
Remote and Office Perks: Internet, commuting, and free lunch and gym in the San Francisco office
Time Off: Flexible paid time off, 15 holidays, and 2 flex days
Connection: Team outings and referral bonuses
About the Company
Handshake was founded on the belief that everyone deserves a path to a great career, regardless of where they went to school or who they know. The company powers 25 million job seekers, over 1 million employers, and 1,600 educational institutions. In 2025, Handshake launched Handshake AI, working directly with frontier artificial intelligence lab researchers to create evaluations, publish benchmarks, and advance data boundaries.
How to Apply
To apply for this position, please visit the official company careers page at joinhandshake.com/careers.
