AI/LLM QA Engineer - Agentic & Multi-Agent Systems
- Source: Naukri
- Qentelli
- On-site
- Hyderabad
- Full-Time
- Mid level
Job Description
About the Role
We are seeking a highly skilled AI/LLM QA Engineer with expertise in Agentic and Multi-Agent system testing to join our team. The ideal candidate will have a strong background in software QA/testing combined with hands-on experience in AI/ML or LLM-based applications. You will be responsible for validating the performance, accuracy, and robustness of next-generation AI systems, including orchestration-driven workflows and autonomous agents.
Key Responsibilities
- Design and execute test strategies for Agentic and Multi-Agent systems, ensuring reliability and scalability.
- Perform LLM evaluation using metrics such as:
o Exact match / Soft match
o BLEU / ROUGE
o BERTScore
o Semantic similarity using embeddings
- Develop and maintain prompt testing frameworks and validate output consistency across use cases.
- Implement and evaluate guardrails for responsible AI behavior (hallucinations, toxicity, bias).
- Build automated test pipelines for LLM-driven workflows and orchestration frameworks.
- Collaborate with AI/ML engineers and product teams to define test cases, benchmarks, and acceptance criteria.
- Identify defects, performance issues, and edge cases in AI systems.
- Integrate testing processes into CI/CD pipelines.
Mandatory Skills
Core AI/LLM Testing
- Hands-on experience in Agentic & Multi-Agent Testing
- Expertise in LLM evaluation frameworks and methodologies
- Strong understanding of:
o Prompt engineering validation
o Guardrails and safety mechanisms
o Output evaluation using embeddings and NLP metrics
Software QA Experience
- 3+ years in Software QA/Testing
- Minimum 2+ years of experience in AI/ML or LLM-based systems
Programming
- Proficiency in Python OR TypeScript/JavaScript
Orchestration Frameworks
Experience with at least one:
- LangChain / LangGraph
- LlamaIndex
- DSPy
- OpenAI Assistants / Action Or similar agentic orchestration frameworks
Tools & Platforms
- CI/CD: GitLab, Jenkins
- Monitoring: Grafana
- Test Management & Tracking: Jira, X-Ray
- Exposure to AI Quality Engineering practices
Skills
- bleu
- guardrails
- Python
- TypeScript
- microsoft-typescript
- javascript
- LangChain
- LangGraph
- LlamaIndex
- gitlab
- Jenkins
- Grafana
- JIRA
You missed this one - but you don't have to keep missing out. Try Club members get matched with open opportunities before they're gone, so you're always first in line, never last.
