AI Test Architect

Role & responsibilities

We are seeking an accomplished Test Architect with deep expertise in AI quality engineering and agentic AI systems. In this role you will own the end-to-end quality strategy for AI-powered products from LLM evaluation frameworks and multi-agent orchestration testing to scalable test infrastructure. You will operate at the intersection of software architecture, AI/ML, and enterprise quality engineering, shaping both the tools teams use and the standards they follow.

Architecture & Strategy

• Design and own the enterprise-wide Test Architecture for AI products including LLM-based applications, RAG pipelines, and autonomous agentic systems.
• Define quality gates, evaluation metrics, and acceptance criteria aligned with AI model behaviour, safety, and regulatory requirements.
• Architect scalable, cloud-native test infrastructure on Azure / AWS / GCP supporting continuous evaluation of non-deterministic AI outputs.
• Lead the selection and integration of AI testing toolchains into CI/CD pipelines (e.g. Playwright, Cypress, LangSmith, Evals frameworks).

Agentic AI & LLM Testing

• Build evaluation harnesses for multi-agent systems — covering agent reasoning chains, tool-use accuracy, memory recall, and goal completion rates.
• Design test strategies for agentic workflows built on frameworks such as AutoGen, LangGraph, CrewAI, or custom orchestration layers.
• Develop adversarial testing suites: prompt injection attacks, jailbreak probes, hallucination detection, and bias evaluation.
• Establish ground-truth benchmarking datasets and model regression pipelines to track LLM quality across fine-tuning cycles.

Product & Platform Engineering

• Drive design and delivery of internal testing products: AI Eval Platforms, automated quality dashboards, and AI observability tooling.
• Contribute to the product roadmap for quality tooling — writing PRDs, shaping backlogs, and aligning engineering with business outcomes.
• Integrate performance testing strategies (K6, JMeter) for AI API endpoints, latency SLAs, and throughput benchmarks.

Leadership & Enablement

• Lead, mentor, and grow a team of 8–15 QE engineers across automation, AI eval, and performance disciplines.
• Establish CoE standards: coding conventions, test design patterns (POM, BDD), and documentation norms across QE squads.
• Partner with AI Product Managers, Data Scientists, and MLOps engineers to co-own quality across the full model lifecycle.
• Present architecture decisions and quality health metrics to senior leadership and external stakeholders.