AI/LLM QA Engineer - Agentic and Multi-Agent System Testing

AI/LLM QA Engineer - Agentic and Multi-Agent System Testing top of page AI/LLM QA Engineer - Agentic and Multi-Agent System Testing Hyderabad, Telangana, India Hybrid Full-time

About the Role

We are seeking a highly skilled AI/LLM QA Engineer with expertise in Agentic and Multi-Agent system testing to join our team. The ideal candidate will have a strong background in software QA/testing combined with hands-on experience in AI/ML or LLM-based applications. You will be responsible for validating the performance, accuracy, and robustness of next-generation AI systems, including orchestration-driven workflows and autonomous agents.

Requirements

AI/LLM QA Engineer - Agentic and Multi-Agent System Testing

Experience: 4+ years

Location: Hyderabad

Work Mode: Hybrid

Department: Engineering

Employment Type: Full-time

Notice Period: 15 days

Key responsibilities

• Design and execute test strategies for Agentic and Multi-Agent systems, ensuring reliability and scalability.
• Perform LLM evaluation using metrics such as Exact match / Soft match, BLEU, ROUGE, BERTScore, and semantic similarity using embeddings.
• Develop and maintain prompt testing frameworks and validate output consistency across use cases.
• Implement and evaluate guardrails for responsible AI behavior (hallucinations, toxicity, bias).
• Build automated test pipelines for LLM-driven workflows and orchestration frameworks.
• Collaborate with AI/ML engineers and product teams to define test cases, benchmarks, and acceptance criteria.
• Identify defects, performance issues, and edge cases in AI systems.
• Integrate testing processes into CI/CD pipelines.

Required skills and experience

• Hands-on experience in Agentic & Multi-Agent Testing.
• Expertise in LLM evaluation frameworks and methodologies.
• Strong understanding of prompt engineering validation, guardrails and safety mechanisms, and output evaluation using embeddings and NLP metrics.
• 3+ years in Software QA/Testing.
• Minimum 2+ years of experience in AI/ML or LLM-based systems.
• Proficiency in Python or TypeScript/JavaScript.

Good to have

• Experience with orchestration frameworks such as LangChain, LangGraph, LlamaIndex, DSPy, or OpenAI Assistants/Actions.
• Familiarity with CI/CD tools like GitLab and Jenkins.
• Knowledge of monitoring tools like Grafana and test management/tracking tools like Jira and X-Ray.
• Exposure to AI Quality Engineering practices.

About the Company

Our client is a forward-thinking organization specializing in innovative AI solutions and technologies.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.