AI Testing Consultant

We are looking for an AI Testing Consultant with handson experience in AI Agent–based solutions. The resource will be responsible for validating AI agents, testing workflows, and using Microsoft AI Foundry evaluators to measure quality, accuracy, and performance across different AI scenarios.

Key Responsibilities

• Perform testing and validation of AI Agents, including singleagent and multiagent workflows.
• Execute functional, nonfunctional, and scenariobased testing for AI-driven use cases.
• Use Microsoft AI Foundry evaluators to assess AI performance metrics such as accuracy, relevance, grounding, completeness, and safety.
• Analyze evaluator results and help identify gaps, risks, and improvement areas in AI responses.
• Validate AI outputs against business requirements and expected behaviors.
• Work closely with onshore teams, architects, and developers to support test strategy and defect resolution.
• Document test scenarios, test results, and evaluation findings clearly.

Required Skills & Experience

• 5+ years of experience in testing or validation roles, preferably in AI, ML, or intelligent automation.
• Handson experience testing AI agents, chatbots, or GenAI-based applications.
• Good understanding of AI evaluation metrics (e.g., precision, recall, relevance, groundedness, hallucination).
• Exposure to Microsoft AI Foundry evaluators or similar AI evaluation frameworks.
• Understanding of prompt-based systems and agent behaviors.
• Strong analytical skills to interpret AI evaluation results.
• Good communication and documentation skills.