Latest AI JobsBrowse Jobs
Deepgram logo

Senior Software Engineer - Model Evaluation & AI Systems

DeepgramUSA - RemoteEngineering
Apply on company site
Salary
$180K - $240K USD
Job type
Full-time
Work mode
Remote
Posted
28 Jul 2026
Status
Open · checked 7d ago

About this role

Deepgram is seeking a remote Senior Software Engineer to lead model evaluation and AI systems in the United States. This position suits an experienced developer eager to build automated testing infrastructure and ensure top-tier speech and voice AI model performance.

Role Overview

The Senior Software Engineer for Model Evaluation & AI Systems will join the team responsible for validating the quality of speech, audio, and multilingual models before customer release. This position creates the evaluation and quality assurance surfaces ensuring models meet batch and streaming performance targets. The role involves building pipelines, harnesses, and test frameworks to catch quality issues, while partnering with Research to establish reproducible, automated checks.

Key Responsibilities

Define and build evaluation methodologies for speech-to-text, text-to-speech, LLM, RAG, agent, and multimodal systems.
Design, build, and maintain automated evaluation pipelines across batch and streaming environments focusing on correctness and reproducibility.
Build scalable evaluation infrastructure running against production models and GPU clusters.
Translate Research benchmarks and expected model metrics into automated, enforceable pass and fail gates.
Build and operate canaries and continuous-monitoring systems to detect production quality regressions.
Partner with DevOps and Infrastructure to set up ephemeral test environments and result-aggregation infrastructure.
Work alongside Research, model training, inference, and product teams to provide trusted evaluation signals for release decisions.
Integrate evaluation and quality gates into CI/CD pipelines for continuous verification.
Participate in code reviews, technical design discussions, and engineering practice improvements.

Requirements

Degree in Computer Science, AI, Applied Math, or a related field, or equivalent experience.
Professional software or quality assurance engineering experience with a track record of shipping test infrastructure or evaluation systems.
Solid backend or scripting experience in a language such as Python, Rust, Go, or similar.
Experience designing and building automated test pipelines, evaluation frameworks, or data-processing systems.
Strong analytical skills with comfort reasoning about metrics, thresholds, and statistical variation.
Ability to handle ambiguous technical challenges and communicate effectively across teams.

Nice to Have

Hands-on experience evaluating modern AI systems including LLMs, RAG pipelines, agents, or multimodal models.
Experience with React Native or other cross-platform mobile frameworks for tooling development.
Experience building ML infrastructure used by other teams or external users.
Appreciation for evaluation quality, consistency, and reproducibility.
Experience with voice, audio, speech recognition, or real-time systems and metrics like WER, MOS, or latency and TTFB.
Prior involvement in open-source projects.
Experience acting as a technical bridge across multiple teams and platforms.
Familiarity with cloud infrastructure, containerized environments, and monitoring tools like Grafana.

Skills and Tools

Python
Rust
Go
LLMs and RAG pipelines
Automated test pipelines and evaluation frameworks
CI/CD integration
Cloud infrastructure and monitoring tools

Salary and Benefits

Annual Salary: $180,000 - $240,000 USD

About the Company

Deepgram provides real-time APIs for speech-to-text, text-to-speech, and production-grade voice agents. Over 200,000 developers and 1,300 organizations build voice offerings powered by Deepgram. The company operates with an AI-first mindset where active experimentation and adaptation to fast-paced technological changes are fundamental to daily operations.

How to Apply

Visit the official Deepgram careers page to submit an application for this remote position.

Skills and tech stack

PythonReactRustLLMRAG

Questions candidates ask

What is the salary range for this remote role?

The annual salary for the Senior Software Engineer position is between $180,000 and $240,000 USD.

Is this job fully remote?

Yes, this position is designated as USA - Remote.

What programming languages are expected for backend and scripting?

Candidates should have solid backend or scripting experience in languages such as Python, Rust, Go, or similar.

What does the Model Evaluation team do at Deepgram?

The team validates the quality of speech, audio, and multilingual models before customer release, building automated evaluation pipelines, harnesses, and monitoring systems.

What operating mindset does Deepgram expect from team members?

Deepgram expects an AI-first mindset where team members actively use, experiment with, and build advanced AI tools into their everyday work.

Ready to apply?

The application is on the official Deepgram careers page.

Apply on company site