AI Testing Specialist

We’re hiring an AI Test Engineer / AI Testing Specialist for a technology-driven organization.

Position: AI Testing Specialist
Experience: 5 -15yrs

Skills: Python Testing with experience in testing AI/ML systems or LLM-based applications

Location: Hyderabad

Work Mode: 5 Days WFO
Employment Type: Full-Time, Permanent with MNC

Job Summary

We are looking for an AI Test Engineer with hands-on experience in testing and evaluating LLM-based applications, RAG systems, Agentic AI architectures. The role involves building automated evaluation and regression frameworks, testing AI agents for accuracy, hallucinations and security vulnerabilities, and integrating AI quality gates into CI/CD pipelines.

Key Responsibilities & Requirements-

• Develop automated testing and evaluation pipelines using Python and pytest.
• Strong understanding of test automation, evaluation pipelines, and scripting in Python.
• Exposure to AI/ML system testing, LLM application testing, adversarial testing, and automated benchmarking is required.
• Develop adversarial and red-team testing suites to identify hallucinations, prompt injection vulnerabilities, edge-case failures, and unexpected agent behaviour.
• Build and maintain automated regression test suites to validate AI agent consistency across model updates, prompt changes, and skill-file modifications.
• Test AI agents within LangGraph-based architectures, including Refinement, Decision, and Coding Agents.
• Design and implement LLM evaluation frameworks using RAGAS, custom scoring functions, and automated accuracy benchmarking to validate AI/agent outputs.
• Integrate AI-specific testing and quality checks into CI/CD pipelines using GitHub Actions or comparable platforms.
• Demonstrate practical knowledge of LLM evaluation methodologies, RAG evaluation, and AI quality assessment.
• Have familiarity with RAG, LangChain, LangGraph, and multi-agent orchestration concepts.