About this role
Engineers who thrive on solving complex validation challenges for non-deterministic systems will find a natural home in this role. Anaplan is expanding its engineering capacity in London to focus on quality assurance for emerging GenAI and agentic workflows. The team builds scenario planning and analysis platforms utilized by major global enterprises, requiring robust mechanisms to maintain accuracy and trust in AI features.
As the Senior SDET for AI, the chosen candidate will architect evaluation frameworks, design automated prompt testing systems, and execute adversarial red-teaming to mitigate hallucinations and biases. This newly created position exists to establish rigorous quality standards across LLM-powered applications. Success in the role requires deep familiarity with GenAI behaviour, strong automation skills in Python, and the analytical mindset needed to measure model safety, consistency, and relevance.
What you will do
- Design comprehensive test plans, evaluation frameworks, and quality metrics tailored for LLM-powered applications and agentic workflows.
- Write robust, end-to-end automated test suites in Python covering evaluations, API checks, UI checks, and performance tests.
- Build automated prompt testing systems and regression suites to detect unintended variations in model behavior.
- Establish multi-dimensional evaluation criteria to track GenAI quality across accuracy, relevance, safety, consistency, and latency.
- Conduct adversarial testing and red-teaming to uncover model hallucinations, biases, and security vulnerabilities.
- Create internal tools and testing frameworks that empower software engineering teams to validate their own GenAI implementations.
- Implement real-time monitoring and alerting systems to flag quality degradation for AI features operating in production.
- Own the overarching quality and testing strategy while maintaining high-fidelity test datasets and defining QA best practices.
What you need
- Demonstrated understanding of generative artificial intelligence technologies, including large language models, prompt engineering, and application patterns.
- Strong hands-on coding experience utilizing Python.
- Expertise in test automation frameworks and scripting across Python, JavaScript, Selenium, and Pytest.
- Comprehensive background in software testing methodologies spanning functional, integration, regression, performance, and security testing.
- Proven capability to design test cases and evaluation criteria specifically for non-deterministic systems.
- Analytical and problem-solving abilities paired with close attention to detail.
- Familiarity with API testing tools like Postman and REST Assured alongside backend testing principles.
- Practical experience integrating automated testing within CI/CD pipelines.
Skills and tech stack
Questions candidates ask
What will I be working on?
You will pioneer a new role focused exclusively on quality assurance for GenAI and agentic systems. This includes developing testing strategies, evaluation frameworks, and quality metrics specifically designed for LLM-powered applications.
What technical skills are required for this position?
The role requires a deep understanding of GenAI technologies including LLMs and prompt engineering, alongside strong hands-on Python coding experience. Candidates must also bring expert experience with test automation frameworks and a strong background in software testing methodologies.
How does the hiring process work regarding offers?
Anaplan does not extend offers to candidates without an extensive interview process with a member of the recruitment team and a hiring manager via video or in person. All offers are first extended verbally by an internal recruitment team member whenever possible and then followed up via written communication.