About this role
Enterprise clients often struggle to move beyond experimental proofs of concept and deploy scalable AI systems into production. Robots and Pencils solves this bottleneck by embedding forward-deployed engineers directly with client teams to build real infrastructure.
This staff machine learning engineer will operate and train models on Amazon SageMaker utilizing custom AWS Trainium silicon. The work requires going deep into the hardware and compiler stack, connecting low-level execution details with PyTorch training code to build production-grade pipelines.
Ideal candidates are pragmatic engineers who refuse to treat accelerators as black boxes and excel at turning abstract training requests into cost-aware, optimized systems.
What you will do
- Train and operate models on Amazon SageMaker using AWS Trainium as the underlying compute infrastructure.
- Write and optimize PyTorch training code with a deep awareness of NeuronCore architecture, compiler behavior, memory limits, and throughput tradeoffs.
- Diagnose training failures caused by hardware or compiler issues rather than standard model or data bugs.
- Translate high-level training requests into complete, cost-aware end-to-end pipelines on SageMaker.
- Tune distributed training runs to maximize throughput and minimize costs on SageMaker training infrastructure.
- Collaborate directly with client and internal engineering teams to deliver production training workloads.
What you need
- Demonstrate strong, hands-on PyTorch experience, including background in distributed or multi-device training.
- Bring production experience utilizing Amazon SageMaker for model training and or inference.
- Possess comfort working close to the hardware layer to debug device-specific compilation issues.
- Maintain solid Python fundamentals and operate effectively within a client-facing, production engineering environment.
Nice to have
- AWS Trainium or Inferentia (Neuron SDK) experience
Skills and tech stack
Questions candidates ask
What is the remote work policy?
This position is US Remote.
What does the team actually build?
The team builds production AI systems for enterprise clients as an AWS Partner, working forward-deployed and embedded directly with client engineering teams.
What is the primary tech stack for this role?
The stack centers around Amazon SageMaker, AWS Trainium utilizing custom silicon, PyTorch, and Python.
Who does this engineer work with day-to-day?
This person works directly with client and internal engineering teams to scope and deliver real production training workloads.
The application is on the official Robots and Pencils careers page.
Apply on company site