About this role
Building and shipping production-grade automated validation frameworks and evaluation pipelines across the AI lifecycle forms the core of daily responsibilities. The engineering team designs AI-powered platforms that sit at the center of critical healthcare decisions, creating a trusted data foundation for health plans using Databricks and MLflow. Operating in a platform-oriented capacity, this practitioner builds custom validation frameworks and automated testing harnesses to evaluate orchestration logic, manage non-deterministic behavior, and ensure safe degradation under uncertainty.
This position exists because scaling agentic, LLM-driven systems requires embedded guardrails and evaluation signals to guarantee safe and predictable real-world behavior. Ideal candidates view quality as a core engineering discipline rather than a downstream checkpoint, bringing automation-first mindsets to distributed systems and backend software engineering.
What you will do
- Build and ship production-grade, automated validation frameworks, test harnesses, and evaluation pipelines across the entire AI lifecycle.
- Design and evolve an AI testing platform integrated with Databricks and MLflow to enable repeatable testing, traceability, and auditability.
- Create large-scale, scenario-based test suites with hundreds to thousands of cases to validate agentic workflows end-to-end, covering edge cases and failure modes.
- Validate orchestration behavior and stress-test non-deterministic system behavior prior to production deployment.
- Embed quality by design by defining system contracts, guardrails, and safe-degradation patterns at key boundaries.
- Define measurable quality signals for LLM systems, such as grounding, hallucinations, relevance, latency, and cost, and integrate them into CI/CD pipelines as automated quality gates.
- Partner closely with AI, platform, security, and delivery teams to translate mission needs into clear quality criteria and trade-offs.
What you need
- Bring seven or more years of software engineering experience focusing primarily on backend or platform systems.
- Demonstrate proven experience designing and implementing AI testing automation in production environments rather than just executing tests.
- Show strong proficiency in Python and TypeScript within modern AI engineering stacks.
- Possess hands-on experience with AI-powered systems, including LLM-based or agentic workflows and non-deterministic behavior.
- Understand CI/CD integration deeply, including embedding automated tests and quality gates into deployment pipelines.
- Maintain a solid understanding of AWS cloud-native architectures.
- Demonstrate a track record of engineering for quality, reliability, governance, and safety as core system design principles.
Nice to have
- Experience with Databricks-native environments and Medallion architecture.
- Experience using MLflow for model evaluation, lineage tracking, and auditability.
- Exposure to observability tools like Datadog, Prometheus, or Grafana for distributed AI workflows.
- Familiarity with LLM evaluation techniques, guardrails, and policy enforcement frameworks.
- Formal training or certification in AI/ML systems such as ISTQB AI Testing, AWS ML Specialty, or Google ML Engineer.
Skills and tech stack
Questions candidates ask
What kind of systems will I be testing in this role?
You will work on AI-powered platforms that sit at the center of critical healthcare decisions, specifically focusing on agentic, LLM-driven systems. Your work will involve validating orchestration logic, tool use, memory, and decision behavior to ensure these systems operate safely and predictably in real-world environments.
Is this a traditional manual QA position?
This is not a traditional QA role. It is an automation-first, platform-oriented position focused on establishing repeatable, scalable quality practices, building custom validation frameworks, and embedding guardrails directly into system design and deployment pipelines.
What tools and architectures will I use daily?
You will design and own an AI testing platform operating natively within a Databricks Medallion architecture. The tech stack includes Python, TypeScript, MLflow for evaluation and lineage, and AWS cloud-native environments.
The application is on the official Abacus Insights careers page.
Apply on company site