Machine Learning Engineer (Evals and Voice Models)
About this role
You will spend your days designing evaluation foundations, training voice models, and building automated quality monitoring systems across customer communication products. The team builds AI voice agents, messaging assistants, and real-time guidance tools that handle phone calls, SMS, and WhatsApp conversations for thousands of companies globally. This person will actually work on voice models, agent capability benchmarks, LLM-as-judge systems, failure analysis, and regression testing pipelines to measure accuracy, resolution quality, and safety.
The role exists because Aircall needs to establish shared metrics, test sets, and tooling so product teams can ship updates confidently without reinventing evaluation methods. Someone who thrives in an execution-focused, data-driven environment with a background in applied machine learning, voice models, and rigorous evaluation frameworks will do well here.
What you will do
- Design and document comprehensive evaluation frameworks for AI agents across voice, chat, and messaging.
- Train and fine-tune voice models including TTS, ASR, and speech-to-speech using production and synthetic data.
- Assess AI-generated solutions across training pipelines, experimentation setups, debugging processes, and optimization strategies.
- Design annotation guidelines and workflows for human-labeled evaluation data while calibrating LLM-as-judge systems against human raters.
- Build and maintain live quality monitoring for deployed AI agents to track accuracy, resolution rate, and safety signals.
- Own the metric contract for every published AI metric including definitions, populations, grains, rollups, and validity windows.
- Build release gates and offline regression suites that every AI surface must pass before shipping changes.
What you need
- Hold a Bachelor of Science in Computer Science, Machine Learning, Statistics, or a related field.
- Bring three or more years of experience in machine learning engineering or applied ML within eight or more years of overall experience.
- Demonstrate strong experience in evaluating supervised, unsupervised, LLM, and deep learning models.
- Possess hands-on experience in failure analysis and evaluating large language models.
- Show direct experience building automated evaluation systems.
- Exhibit strong communication skills to articulate complex technical concepts to diverse audiences.
- Demonstrate hands-on experience training or fine-tuning voice and speech models.
Nice to have
- Master of Science or PhD in Computer Science, Machine Learning, Statistics, or a related field
- Experience evaluating LLMs or agentic systems like LLM-as-a-judge and RAG evaluation
- Experience with synthetic data generation and prompt engineering
- Experience training or fine-tuning voice models at scale with familiarity in model distillation or low-latency inference optimization
Benefits
- Competitive salary package and benefits
- Work-life balance
- Fast-learning environment with strong team spirit
- Cosmopolite and multicultural mindset with 45+ nationalities
Skills and tech stack
Questions candidates ask
What is the salary range for this role?
The base salary range is $181,000 to $250,000 USD.
Where is the position located?
The role is based out of the San Francisco office.
What kind of products will I be working on?
You will work on evaluation foundations for AI voice agents, messaging agents, and assistive tools that handle voice, WhatsApp, and SMS communications.
Does the hiring process involve artificial intelligence tools?
Aircall may use AI tools to support parts of the hiring process such as reviewing applications, analyzing resumes, or assessing responses, though final hiring decisions are made by humans.
The application is on the official Aircall.io, Inc. careers page.
Apply on company site