AI LLM QA Engineer-Python

Role Overview

We are looking for a highly motivated AI/LLM QA Engineer with strong expertise in Python-based test automation and LLM evaluation techniques. The ideal candidate will have hands-on experience in testing AI/ML or LLM-based systems, ensuring quality, reliability, and performance of intelligent applications. You will play a key role in building AI Quality Engineering frameworks, validating LLM outputs, and integrating testing pipelines within modern cloud and orchestration ecosystems.

Key Responsibilities

• Design, develop, and execute automated test frameworks for AI/LLM systems using Python.

• Perform LLM evaluation and validation using: o Semantic similarity o Embeddings-based scoring o Regression testing strategies for model outputs

• Build and maintain test suites using Pytest, TestNG, JUnit, Cucumber, or Behave.

• Conduct API testing and integration testing using libraries like boto3, requests, etc.

• Generate rich test reports using tools like pytest-html and enable parallel execution via pytest-xdist.

• Collaborate with data scientists and ML engineers to define evaluation benchmarks and test datasets.

• Implement end-to-end validation of AI workflows, including orchestration frameworks.

• Integrate tests into CI/CD pipelines (GitLab, Jenkins).

• Monitor system performance and quality metrics using tools like Grafana.

• Manage defect tracking, test cases, and reporting using Jira and X-Ray. Mandatory Skills Programming & Automation

• Strong hands-on experience with Python

• Experience with: o boto3, requests o pytest, pytest-html, pytest-xdist

• Knowledge of testing frameworks: o Pytest / TestNG / JUnit / Cucumber / Behave AI/LLM Testing Expertise

• Experience in Software QA/Testing for AI/ML or LLM-based systems • Strong knowledge of: o LLM evaluation techniques (semantic similarity, embeddings) o Regression testing for AI outputs o Prompt/output validation Orchestration Frameworks Experience with at least one: • LangChain / LangGraph • LlamaIndex • DSPy • OpenAI Assistants / Actions • Similar AI orchestration tools DevOps & QA Tools • CI/CD Tools: GitLab, Jenkins • Monitoring: Grafana • Test & Issue Tracking: Jira, X-Ray • Exposure to AI Quality Engineering practices Role &