About this role
Ensuring the reliability, accuracy, and safety of large language models and agentic workflows presents a core challenge that this hire will solve. Anaplan seeks a Senior SDET for AI in London to pioneer quality assurance for their GenAI-infused scenario planning and analysis platform. Building robust evaluation frameworks specifically designed for non-deterministic systems requires a combination of traditional QA expertise and a deep understanding of generative artificial intelligence.
Working alongside software engineering teams, the person in this role will design comprehensive test plans, execute adversarial testing, and build automated prompt testing systems. Candidates who excel here possess strong Python skills, a background in software testing methodologies, and a passion for establishing QA best practices within fast-paced technology environments.
What you will do
- Design and architect comprehensive test plans, evaluation frameworks, and quality metrics designed specifically for LLM-powered applications and agentic workflows.
- Develop robust, end-to-end automated test suites in Python across evaluations, API checks, UI checks, and performance tests.
- Create automated prompt testing systems and regression suites to proactively detect unintended changes in model behaviour.
- Establish multi-dimensional evaluation criteria to measure and track GenAI quality across accuracy, relevance, safety, consistency, and latency.
- Perform adversarial testing and red-teaming to identify and mitigate model hallucinations, biases, or security vulnerabilities.
- Develop internal tools and testing frameworks that enable software engineering teams to seamlessly validate their own GenAI implementations.
- Deploy real-time monitoring and alerting systems to quickly detect and flag quality degradation of AI features in production.
- Own the overall quality and testing strategy for your workstream, maintaining high-fidelity test datasets and defining QA best practices.
What you need
- Demonstrate a deep understanding of GenAI technologies including LLMs, prompt engineering, and AI application patterns.
- Bring strong hands-on Python coding experience to the team.
- Possess expert experience with test automation frameworks and scripting using Python, JavaScript, Selenium, and Pytest.
- Showcase a strong background in software testing methodologies covering functional, integration, regression, performance, and security testing.
- Prove the ability to design test cases and evaluation criteria for non-deterministic systems.
- Exhibit strong analytical and problem-solving skills with meticulous attention to detail.
- Demonstrate experience with API testing tools like Postman or REST Assured alongside backend testing.
- Display strong experience integrating automated testing into CI/CD pipelines.
Skills and tech stack
Questions candidates ask
What will I actually be working on?
You will develop testing strategies, evaluation frameworks, and quality metrics specifically designed for LLM-powered applications and agentic workflows. Your tasks include performing adversarial red-teaming, building automated prompt testing systems, and deploying real-time production monitoring.
What technical background is required?
The role requires deep understanding of GenAI technologies like LLMs and prompt engineering, alongside strong hands-on Python coding experience. You will also need expert experience with test automation frameworks such as Selenium and Pytest.
Where is the position located?
The position is based in London, United Kingdom.
What is the hiring process like?
Anaplan does not extend offers to candidates without an extensive interview process with a member of their recruitment team and a hiring manager via video or in person. All offers are first extended verbally by a member of the internal recruitment team whenever possible.