About this role
Anaplan builds an AI-infused scenario planning and analysis platform designed to optimize business decision-making for global enterprises. The engineering group focuses on delivering reliable AI features that help major brands outpace their markets.
As the QA Automation Engineer for AI, the chosen candidate will pioneer quality assurance practices exclusively for generative AI and agentic systems. Responsibilities include building testing strategies, evaluation frameworks, and metric systems for large language models, alongside writing end-to-end automated tests in Python. The person filling this role needs a solid foundation in software testing combined with a strong grasp of non-deterministic AI behavior and prompt engineering.
What you will do
- Develop and execute end-to-end automated test coverage across evaluations, APIs, UIs, and performance tests using Python.
- Implement testing strategies for GenAI features, including conversational AI, agentic systems, and LLM-powered workflows within the designated workstream.
- Run and maintain automated test suites for prompt testing, including regression tests designed to detect unintended changes in model behaviour.
- Support and maintain evaluation frameworks to measure GenAI performance across accuracy, relevance, safety, consistency, and latency.
- Contribute to curating high-quality test datasets and golden examples that represent diverse user scenarios and edge cases.
- Assist in adversarial testing to identify potential model hallucinations or biases while supporting the integration of production alerting systems.
- Utilise internal tools and testing frameworks to help developers easily validate their GenAI code, and assist with user acceptance testing.
- Take accountability for automated test coverage within the assigned workstream by documenting procedures and known issues clearly.
What you need
- Bring solid quality assurance or test engineering experience, paired with exposure to or a keen interest in artificial intelligence and machine learning systems.
- Demonstrate hands-on coding experience using Python for test scripting and automation.
- Possess a good understanding of generative AI technologies, including large language models, prompt engineering, and application patterns.
- Showcase solid experience working with test automation frameworks and scripting tools such as Python, Pytest, API, and UI testing.
- Apply practical knowledge of software testing methodologies covering functional, integration, regression, performance, and security testing.
- Maintain the ability to execute test cases and evaluation criteria for non-deterministic software systems.
- Bring familiarity with API testing tools like Postman or REST Assured alongside general backend testing capabilities.
- Hold familiarity with continuous integration and continuous deployment pipelines and the integration of automated testing.
Skills and tech stack
Questions candidates ask
What kind of systems will I be testing in this role?
You will focus exclusively on quality assurance for generative AI and agentic systems. This involves testing conversational AI, agentic systems, and LLM-powered workflows.
What programming language and frameworks are required?
The role requires hands-on Python coding experience and solid experience with test automation frameworks like Pytest. You will also use API testing tools such as Postman and REST Assured.
How does the team evaluate GenAI performance?
The team supports and maintains evaluation frameworks to measure GenAI performance across accuracy, relevance, safety, consistency, and latency. You will also assist in adversarial testing to identify model hallucinations or biases.