QA AI Tester - Senior Associate
- Source: Naukri
- Pricewater House Coopers Service Delivery Center Bangalore
- On-site
- Bengaluru
- Full-Time
- Senior
Job Description
- Job Title: Senior QA AI Tester
- Job Summary
-
We are seeking an experienced and forward-thinking Senior AI Tester / QA Engineer to lead quality assurance efforts for AI-driven products and machine learning systems with over 4+ years’ experience. The ideal candidate will possess a deep understanding of software testing combined with hands-on experience in AI/ML model validation, data quality assurance, and automation testing frameworks. You will collaborate closely with data scientists, ML engineers, software developers, and DevOps teams to ensure the reliability, fairness, and performance of AI solutions through robust testing strategies and intelligent automation.
- Role & responsibilities
- Quality Assurance on AI/ML Systems
- Design and execute E2E test strategies for LLM-powered features, prompts, retrieval pipelines, and agent workflows. Test E2E AI workflows, from data ingestion to prediction APIs, ensuring functional and non-functional quality.
- Validate grounding, hallucination rates, embedding quality, and vector search accuracy.
- Evaluate LLM outputs for correctness, relevance, safety, and consistency.
- Conduct comprehensive validation of datasets ensuring data completeness, correctness, and representativeness.
- Define and monitor AI-specific quality metrics such as precision, recall, F1-score, model latency, and drift statistics.
- Lead root cause analysis for anomalies or performance degradation across the data and model lifecycle.
- Automation and Framework Development
- Design, develop, and maintain automation frameworks for AI systems, APIs, and model serving layers.
- Integrate automated validation pipelines into CI/CD workflows to support continuous testing of model updates.
- Collaborate with engineering and MLOps teams to establish test environments, mock data pipelines, and reproducible test scenarios.
- Utilize AI-assisted test generation tools and frameworks to improve coverage and accelerate testing cycles.
- AI Innovation and Continuous Improvement
- Explore and adopt AI-driven testing tools (e.g., GenAI-based test generation, intelligent defect prediction).
- Champion shift-left testing in AI development embedding quality checks early in the data preparation and model training stages.
- Collaborate with R&D to design simulated environments and synthetic data for stress and edge-case testing.
- Continuously evaluate and implement emerging tools, frameworks, and best practices in AI quality engineering.
- Communication and Collaboration
- Mentor QA engineers and promote best practices for AI/ML testing, data quality validation, and automation design.
- Work closely with cross-functional teams including AI researchers, DevOps, and software engineers — to drive a unified quality strategy.
- Communicate AI testing results, model performance metrics, and risks to stakeholders with clarity and actionable insights.
-
Work closely with AI engineers, data scientists, and platform engineers during test setup, execution and ongoing improvements
-
•Preferred candidate profile
- Qualifications
- Bachelor’s or Master’s degree in Computer Science, Data Science, or related field.
- 6+ years of experience in software testing or quality engineering, with 2+ years focused on AI/ML systems.
- Proficiency in programming languages such as Python, JavaScript, or Java, with experience using PyTest, Robot Framework, or similar tools.
- Strong understanding of AI/ML concepts — model lifecycle, data pipelines, performance metrics, and drift management.
- Experience with ML platforms such as TensorFlow, PyTorch, MLflow, or SageMaker.
- Familiarity with MLOps tools (Docker, Kubernetes, Airflow, Jenkins, Azure ML, etc.) and data validation frameworks (Great Expectations, Pandera, etc.).
- Knowledge of automation testing tools (Selenium, Playwright, Cypress) and API testing (Postman, REST Assured).
- Excellent analytical and problem-solving skills, with strong communication and leadership abilities.
- Experience testing Generative AI, NLP models, or LLMs.
- Familiarity with AI fairness, explainability, and governance frameworks.
- Exposure to AI-based testing tools or GenAI-powered automation platforms.
- Certification in AI/ML or Quality Engineering (ISTQB Advanced, TensorFlow Developer, etc.).
Skills
- drift
- Python
- javascript
- Java
- Pytest
- robot-frame-work
- Tensorflow
- PyTorch
- ml-flow
- mlflow
- Docker
- Kubernetes
- Airflow
- apache-airflow
- Jenkins
- microsoft-azure
- selenium
- Playwright
- Cypress
- postman
You missed this one - but you don't have to keep missing out. Try Club members get matched with open opportunities before they're gone, so you're always first in line, never last.
