AI/LLM QA Engineer-Python

Job Summary

We are looking for an experienced AI/LLM QA Engineer with strong expertise in Python, Pytest Framework, and Software Testing for AI/ML applications. The ideal candidate will be responsible for validating Large Language Model (LLM) applications, designing test automation frameworks, evaluating model performance, and ensuring the quality and reliability of AI-driven solutions.

Key Responsibilities

Develop and maintain automated test frameworks using Python and Pytest.
Design and execute test cases for AI/LLM-based applications.
Perform functional, integration, regression, and end-to-end testing.
Validate LLM outputs using evaluation techniques such as accuracy, relevance, consistency, hallucination detection, and response quality assessment.
Create automated test scripts for APIs and AI workflows.
Work closely with AI/ML engineers to identify and resolve quality issues.
Develop test strategies for prompt engineering and LLM-based systems.
Analyze test results and generate detailed reports.
Ensure software quality throughout the SDLC.

Required Skills

Expert-level proficiency in Python Programming.
Strong experience with Pytest Framework.
Good understanding of Software Testing Frameworks and automation methodologies.
Experience in LLM Evaluation Techniques and AI model validation.
Knowledge of Software QA/Testing in AI/ML environments.
Familiarity with Python libraries such as Requests, Pandas, and testing utilities.
Understanding of API testing and REST services.
Strong debugging, analytical, and problem-solving skills.

Preferred Skills

Experience with Generative AI applications and prompt testing.
Exposure to cloud platforms (AWS/Azure/GCP).
Knowledge of CI/CD tools such as Jenkins, GitHub Actions, or GitLab CI.
Familiarity with AI observability and monitoring tools.