About this role
Anaplan requires a dedicated QA Automation Engineer to solve the quality and reliability challenges inherent in generative AI and agentic systems. This newly established position focuses on developing specialized evaluation frameworks, testing strategies, and metrics to ensure LLM-powered applications perform accurately and securely.
Working within the AI team in London, this practitioner will build end-to-end automated test coverage across APIs, user interfaces, and performance workflows using Python. Responsibilities include running regression suites for prompt testing, curating golden datasets for edge cases, and conducting adversarial testing to catch model hallucinations and biases.
An ideal candidate brings solid test engineering expertise alongside a practical understanding of generative AI technologies and non-deterministic system evaluation. Individuals who thrive in collaborative environments and enjoy shaping quality standards for emerging technology will find this position rewarding.
What you will do
- Develop and execute end-to-end automated test coverage across evaluations, APIs, UIs, and performance tests using Python.
- Implement testing strategies for GenAI features, including conversational AI, agentic systems, and LLM-powered workflows.
- Run and maintain automated test suites for prompt testing and regression checks to detect unintended model behaviour.
- Support evaluation frameworks that measure GenAI performance across accuracy, relevance, safety, consistency, and latency.
- Contribute to curating high-quality test datasets and golden examples representing diverse user scenarios and edge cases.
- Assist in adversarial testing to identify model hallucinations or biases while supporting production alerting integration.
- Utilize internal tools and frameworks to help developers validate GenAI code and assist with user acceptance testing.
What you need
- Solid QA or test engineering experience with exposure to or a keen interest in AI and machine learning systems.
- Hands-on Python coding experience combined with familiarity in test automation frameworks and scripting.
- Good understanding of generative AI technologies, including large language models, prompt engineering, and application patterns.
- Practical knowledge of software testing methodologies such as functional, integration, regression, performance, and security testing.
- Ability to execute test cases and evaluation criteria specifically for non-deterministic systems.
- Experience with API testing tools and backend testing alongside familiarity with CI/CD pipeline integration.
Skills and tech stack
Questions candidates ask
What kind of systems will I be testing in this role?
You will focus exclusively on quality assurance for GenAI and agentic systems, developing testing strategies and evaluation frameworks specifically designed for LLM-powered applications. This involves working with conversational AI, agentic systems, and LLM-powered workflows.
What programming languages and tools are required?
The role requires hands-on Python coding experience along with solid experience using test automation frameworks and scripting such as Pytest, API, and UI testing. You will also use API testing tools like Postman and REST Assured for backend testing.
What specific quality metrics will I be measuring?
You will support and maintain evaluation frameworks to measure GenAI performance across accuracy, relevance, safety, consistency, and latency. Additionally, you will assist in adversarial testing to identify potential model hallucinations or biases.