AI/LLM QA Engineer (Agentic & Multi-Agent Systems)
- Source: Naukri
- Qentelli
- Hybrid
- Hyderabad
- Full-Time
- Mid level
Job Description
About the Role
We are seeking a highly skilled AI/LLM QA Engineer with expertise in Agentic and Multi-Agent system testing to join our team. The ideal candidate will have a strong background in software QA/testing combined with hands-on experience in AI/ML or LLM-based applications. You will be responsible for validating the performance, accuracy, and robustness of next-generation AI systems, including orchestration-driven workflows and autonomous agents.
Key Responsibilities:
• Design and execute test strategies for Agentic and Multi-Agent systems, ensuring reliability and scalability.
• Perform LLM evaluation using metrics such as:
o Exact match / Soft match
o BLEU / ROUGE
o BERTScore
o Semantic similarity using embeddings
• Develop and maintain prompt testing frameworks and validate output consistency across use cases.
• Implement and evaluate guardrails for responsible AI behavior (hallucinations, toxicity, bias).
• Build automated test pipelines for LLM-driven workflows and orchestration frameworks.
• Collaborate with AI/ML engineers and product teams to define test cases, benchmarks, and acceptance criteria.
• Identify defects, performance issues, and edge cases in AI systems.
• Integrate testing processes into CI/CD pipelines.
Mandatory Skills Core AI/LLM Testing :
• Hands-on experience in Agentic & Multi-Agent Testing
• Expertise in LLM evaluation frameworks and methodologies
• Strong understanding of:
o Prompt engineering validation
o Guardrails and safety mechanisms
o Output evaluation using embeddings and NLP metrics
Software QA Experience
• 3+ years in Software QA/Testing
• Minimum 2+ years of experience in AI/ML or LLM-based systems Programming
• Proficiency in Python OR TypeScript/JavaScript Orchestration Frameworks Experience with at least one:
• LangChain / LangGraph
• LlamaIndex
• DSPy
• OpenAI Assistants / Actions
• Or similar agentic orchestration frameworks Tools & Platforms
• CI/CD: GitLab, Jenkins
• Monitoring: Grafana
• Test Management & Tracking: Jira, X-Ray
• Exposure to AI Quality Engineering practices
Skills
- bleu
- guardrails
- Python
- TypeScript
- microsoft-typescript
- javascript
- LangChain
- LangGraph
- LlamaIndex
- gitlab
- Jenkins
- Grafana
- JIRA
You missed this one - but you don't have to keep missing out. Try Club members get matched with open opportunities before they're gone, so you're always first in line, never last.
