Walk-in || AI Test Engineer

We are seeking an experienced AI Test Engineer with 6+ years of overall testing experience and a minimum of 2 years of hands-on experience in AI/GenAI testing. The ideal candidate will be responsible for validating AI-powered applications, evaluating LLM responses, testing RAG-based solutions, ensuring Responsible AI compliance, and establishing AI quality assurance processes.

Role- AI Test Engineer

IT Exp- 6+

Location- Hyderabad

Key Responsibilities

AI Response Evaluation & Validation

• Evaluate AI/LLM responses using:

• Human Evaluation
• Automated Evaluation

• LLM-as-a-Judge techniques

• Validate AI outputs for:

• Accuracy
• Relevance
• Grounding
• Consistency

• Completeness

• Design and execute test scenarios for RAG-based applications.

• Identify and analyze:

• Hallucinations
• Bias
• Toxicity

• Inconsistent responses

• Perform functional, regression, and validation testing for AI features and AI-powered applications.

AI Evaluation Frameworks & Tooling

• Apply and utilize AI evaluation frameworks such as:

• RAGAS
• LangSmith (Good to Have)

• Similar AI evaluation and observability tools

• Define and monitor AI quality metrics including:

• Precision
• Recall
• Faithfulness
• Relevancy
• Toxicity

• Bias

• Support integration of AI evaluation processes into CI/CD pipelines.

Responsible AI & Governance

• Validate Responsible AI principles including:

• Fairness
• Safety
• Transparency

• Bias Detection

• Test system prompts, guardrails, and prompt templates.
• Execute prompt validation and jailbreak testing.
• Ensure compliance with organizational AI governance standards.

Test Data & Dataset Management

• Design and curate:

• Golden Datasets
• Adversarial Datasets

• Edge-case Datasets

• Prepare datasets for:

• Model Validation
• AI Evaluation

• Regression Testing

• Maintain dataset traceability and version control.

Collaboration & Quality Assurance

• Collaborate with AI Engineers, Product Owners, Business Teams, and SMEs.
• Analyze AI quality issues and provide actionable recommendations.
• Contribute to AI testing best practices, frameworks, and reusable assets.
• Support continuous improvement of AI testing processes.

Mandatory Skills

• Strong understanding of AI/LLM Testing concepts and AI Quality Assurance.
• Hands-on experience testing AI/GenAI-enabled applications.

• Experience in AI response evaluation techniques:

• Human Evaluation
• Automated Evaluation

• LLM-as-a-Judge

• Experience testing RAG-based applications.
• Ability to identify hallucinations, bias, inconsistencies, and unsafe responses.
• Experience in test data and dataset creation for AI validation.
• Knowledge of Responsible AI principles and governance models.
• Strong understanding of Functional Testing and Quality Engineering practices.
• Good analytical and problem-solving skills.
• Excellent communication and stakeholder management skills.

Good to Have

• Experience with RAGAS, LangSmith, or equivalent AI evaluation frameworks.
• Exposure to prompt engineering and system prompt design.
• Knowledge of Python-based evaluation pipelines and automation.

• Understanding of AI/ML lifecycle concepts including:

• Model Validation
• Data Drift

• Model Monitoring

• Experience with Azure OpenAI, Azure AI Foundry, AWS Bedrock, or Google Vertex AI.
• Knowledge of CI/CD integration for AI testing.