Gen AI Test Lead

Role Overview

We are seeking an experienced AI Test Lead to lead end-to-end testing and quality assurance of Generative AI applications. This role bridges AI quality engineering, test automation, and customer engagement. You will collaborate with QA Managers, AI/ML Engineers, Prompt Engineers, Full Stack Developers, and DevOps Engineers to validate production-ready, scalable AI applications against industry AI testing standards, while serving as the primary testing point of contact for customers.

Key Responsibilities

• AI Testing Strategy & Delivery Management
• Lead full lifecycle of AI testing activities: test strategy, planning, design, execution, defect management, and sign-off for GenAI applications
• Define and implement test strategies for LLM-powered applications including RAG systems, conversational AI, and AI agents
• Plan and manage functional, non-functional, and AI-specific testing: model evaluation, prompt regression, bias/fairness, robustness, hallucination detection, and explainability validation
• Establish test estimation, sprint-level test planning, milestones, and delivery timelines aligned with agile development cycles
• Track and report test coverage, quality metrics, and release readiness to QA leadership and stakeholders

Customer Engagement & Test Management

• Act as the primary testing point of contact for customers; lead requirement discussions, status reviews, and defect triage calls
• Translate customer business requirements and acceptance criteria into comprehensive AI test plans and test scenarios
• Manage day-to-day testing activities of the QA team: task allocation, progress monitoring, risk identification, and escalation management
• Present test results, quality dashboards, and risk assessments to customer stakeholders in business-focused language
• Drive UAT support, release sign-off, and post-release quality monitoring in coordination with customer teams

AI Testing Standards & Quality Governance

• Implement AI testing practices aligned with industry standards: ISO/IEC 42001 (AI management systems)
• Apply responsible AI and risk frameworks such as NIST AI RMF and EU AI Act requirements to test planning and evidence collection
• Define quality gates, entry/exit criteria, and audit-ready test documentation for AI systems
• Establish evaluation benchmarks and metrics for LLM outputs: accuracy, relevance, groundedness, toxicity, and consistency

Test Automation & Tooling

• Design and build test automation frameworks primarily using Python (Pytest, Selenium, Playwright, Requests)
• Automate AI/LLM evaluation pipelines using frameworks such as DeepEval, Ragas, LangSmith, or promptfoo
• Integrate automated test suites into CI/CD pipelines (GitHub Actions, Jenkins) with reporting and alerting
• Drive API, integration, and performance testing for AI services and microservices
• Mentor QA engineers on Python scripting, automation best practices, and AI testing techniques

Required Skills & Qualifications

Experience

• 8+ years of overall experience in software testing and quality assurance
• 12 years of hands-on experience in AI/ML and GenAI testing activities
• Proven experience interacting with customers and managing end-to-end testing activities and QA teams
• Track record of testing production-grade AI applications through complete release cycles

Technical Knowledge

• Strong understanding of LLMs (GPT, Claude, Llama), prompt engineering, RAG architectures, and AI agents from a testing perspective
• Hands-on expertise in AI-specific testing: model evaluation, bias/fairness testing, adversarial/robustness testing, hallucination detection, and prompt regression testing
• Strong automation skills with Python scripting; experience with Pytest, Selenium/Playwright, and API testing tools (Postman, Requests)
• Familiarity with LLM evaluation tools and frameworks: DeepEval, Ragas, LangSmith, Hugging Face evaluate, or equivalent
• Working knowledge of cloud platforms (AWS, Azure, GCP), CI/CD pipelines, and test management tools (Jira, Zephyr, TestRail)

AI Testing Standards

• Knowledge of AI testing and quality standards: ISO/IEC 42001, and ISTQB CT-AI syllabus
• Understanding of responsible AI frameworks and regulations: NIST AI RMF, EU AI Act, and data privacy considerations in AI testing

Education

• Bachelor’s or Master’s degree in Computer Science, Engineering, AI/ML, Data Science, or related field
• ISTQB Cer…