AI QA Engineer Consultant

Job Summary

AI QA Engineer / AI QA Consultant

Role Overview

We are seeking an AI QA Engineer / AI QA Consultant responsible for ensuring the quality, reliability, performance, robustness, and compliance of AI/ML, Generative AI, LLM, and Agentic AI solutions. This role operates within pharmaceutical/life sciences and manufacturing contexts, aligning AI testing, governance, and quality practices with regulatory expectations and production-readiness requirements. The ideal candidate blends deep AI model validation expertise with practical quality engineering discipline to enable safe, auditable AI deployments.

Key Responsibilities

• Develop and execute AI/ML model QA and evaluation strategies across the model lifecycle, from development through production.
• Evaluate ML models using metrics such as accuracy, precision, recall, F1 score, ROC/AUC, confusion matrix, and domain-specific thresholds.
• Test models across diverse datasets, edge cases, failure scenarios, and varying operating conditions to ensure robust performance.
• Perform GenAI/LLM evaluation covering accuracy, relevance, consistency, hallucination, groundedness, and response quality.
• Design and execute testing for AI agents and multi-agent workflows, including orchestration, tool calling, and output validation.
• Build test datasets, evaluation frameworks, test harnesses, and automated QA pipelines tailored to AI systems.
• Utilize Python, APIs, scripting, and automation to conduct AI testing and evaluation at scale.
• Work with LangChain, LangGraph, Langfuse, and similar AI observability and evaluation tools to monitor and validate agent behavior.
• Identify bias, model drift, hallucination, instability, and AI application/model failures, followed by root-cause analysis and remediation planning.
• Define test cases, user stories, acceptance criteria, quality thresholds, and production-readiness criteria in collaboration with stakeholders.
• Collaborate with Data Scientists, ML Engineers, Software Engineers, Product Owners, and business stakeholders to align on quality objectives.
• Document test results, model limitations, risks, defects, and evaluation outcomes in audit-ready formats suitable for regulatory review.
• Support Responsible AI, AI Governance, Data Integrity, and Quality-by-Design practices within project governance.

Must-Have Skills

• Machine Learning Model Evaluation
• GenAI / LLM Evaluation
• Agentic AI Multi-Agent Systems
• Python, APIs Automation
• LangChain, LangGraph, Langfuse
• Prompt Engineering LLM Testing
• AI Risk, Bias Robustness Testing
• Model / AI Application Regression Testing
• AI Test Strategy Test Case Design

Good to Have

• Pharmaceutical / Life Sciences experience
• GxP / CSV / Data Integrity knowledge
• Azure / AWS / GCP
• MLOps / LLMOps / ModelOps
• RAG and Vector Databases
• Responsible AI AI Governance
• CI/CD and automated testing
• AI Red-Teaming / Adversarial Testing Experience
• 5+ years of experience in AI/ML, GenAI, Model QA/Validation, Software QA, Data Science, or Quality Engineering.
• Experience in regulated, pharmaceutical, life sciences, manufacturing, or other quality-critical environments is preferred (not mandatory).

Niche Positioning

This role targets professionals specialized in AI Quality Engineering, AI Model Validation, GenAI Testing, LLM Evaluation, and Agentic AI Testing. It differentiates from traditional software QA by emphasizing AI/ML-driven quality assurance, governance, and production readiness. The candidate synthesizes AI/ML expertise with quality engineering disciplines to deliver auditable, trustworthy AI systems in regulated contexts.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.