Senior Software Engineer, Machine Learning Platform
About this role
Designing scalable ML and AI infrastructure on AWS, building shared platform capabilities for LLM and agentic workloads, and developing evaluation frameworks for non-deterministic AI systems occupy the daily work. Chime Financial, Inc employs the Machine Learning Platform team to build and operate the infrastructure, tooling, and developer experience that powers machine learning across the company.
Engineers on this team design scalable systems spanning traditional machine learning and emerging AI workloads, such as model training, feature computation, real-time inference, foundation-model access, evaluation, and agentic orchestration. The group focuses on creating secure, reliable, and reusable platform capabilities that help teams choose the right approach, from conventional predictive models to LLM-powered and multi-step agentic systems, while maintaining strong standards for evaluation, observability, governance, privacy, and cost efficiency.
What you will do
- Design, build, and operate scalable ML and AI infrastructure on AWS.
- Design and operate shared platform capabilities for LLM and agentic workloads including model access, prompt and configuration lifecycle, retrieval, tool integration, state management, and workflow orchestration.
- Build evaluation frameworks for non-deterministic AI systems such as offline benchmarks, regression testing, online quality signals, human feedback, and failure analysis.
- Establish observability, reliability, and governance for models and agents covering traces, model and prompt versions, tool calls, latency, token usage, quality, safety, privacy, and cost.
- Build distributed training, batch inference, and large-scale processing systems using frameworks like Ray or Spark.
- Build and maintain infrastructure as code using Terraform while supporting and evolving feature stores and feature pipelines.
- Partner closely with Data Science and Engineering teams to improve developer experience and participate in on-call rotations to support production systems.
What you need
- Possess a working knowledge of the machine learning development lifecycle from data preprocessing and model training to evaluation, deployment, and monitoring.
- Demonstrate experience designing distributed systems and large-scale data or compute platforms on AWS using frameworks such as Spark or Ray.
- Bring 5+ years of experience in ML or AI infrastructure, platform engineering, distributed systems, or production ML systems.
- Show working knowledge of LLM application patterns including retrieval-augmented generation, structured outputs, tool calling, agent orchestration, and evaluation of non-deterministic systems.
- Have experience designing production systems that integrate ML or foundation models through reliable APIs, workflows, and data contracts.
- Maintain hands-on experience with CI/CD pipelines, DevOps practices, infrastructure as code, containerization, and orchestration technologies like Docker and Kubernetes.
- Exhibit strong programming skills in Python, Go, Scala, Java, or similar languages alongside a solid understanding of software engineering fundamentals.
Nice to have
- Experience shipping LLM-powered or agentic systems to production.
- Experience with model gateways, prompt lifecycle management, retrieval or vector search, tool execution, and agent orchestration frameworks.
- Experience building evaluation, tracing, and observability capabilities for non-deterministic AI systems.
- Familiarity with managed or self-hosted foundation model infrastructure, such as Amazon Bedrock, SageMaker, or equivalent platforms.
- Experience operating GPU-based workloads and optimizing training or inference performance and cost, with CUDA experience as a plus.
Benefits
- Backup child, elder, and pet care and subsidized commuter benefits for eligible employees
- Comprehensive health, financial, and wellbeing benefits designed to support you at every stage of life
- Generous vacation policy and company-wide paid days off
- 1% of your time off to support local community organizations of your choice
- Annual wellness stipend to use towards eligible wellness related expenses
- Up to 22 weeks of paid parental leave for birthing parents and 12 weeks of paid parental leave for non-birthing parents with access to family planning reimbursement
Skills and tech stack
Questions candidates ask
What is the remote work or in-office policy?
The in-office work policy is designed to keep you connected with four days a week in the office and Fridays from home for those near one of the offices, plus team and company-wide events depending on location. Whether you are coming in regularly or are part of a fully remote program, you stay engaged with your work and teammates.
What is the salary range for this role?
The base salary offered for this role and level of experience is between $187,000 and $259,000 USD. Full-time employees may also be eligible for bonuses, competitive equity, and benefits.
What tools and technologies will I use on the Machine Learning Platform team?
Engineers work with infrastructure and tools on AWS using frameworks such as Ray, Spark, Terraform, Kinesis, Kafka, Flink, Docker, and Kubernetes. Programming languages utilized include Python, Go, Scala, and Java.
What kind of projects does the team work on?
The team designs and builds scalable systems spanning traditional machine learning and emerging AI workloads, including model training, feature computation, real-time inference, foundation-model access, evaluation, and agentic orchestration. They also build evaluation frameworks for non-deterministic AI systems and establish observability, reliability, and governance for models and agents.
The application is on the official Chime Financial, Inc careers page.
Apply on company site