AI/LLM QA Engineer - Python & AI Quality Engineering

Role Overview

We are looking for a highly motivated AI/LLM QA Engineer with strong expertise in Python-based test automation and LLM evaluation techniques. The ideal candidate will have hands-on experience in testing AI/ML or LLM-based systems, ensuring quality, reliability, and performance of intelligent applications.

You will play a key role in building AI Quality Engineering frameworks, validating LLM outputs, and integrating testing pipelines within modern cloud and orchestration ecosystems.

Key Responsibilities

• Design, develop, and execute automated test frameworks for AI/LLM systems using Python.
• Perform LLM evaluation and validation using:

o Semantic similarity

o Embeddings-based scoring

o Regression testing strategies for model outputs

• Build and maintain test suites using Pytest, TestNG, JUnit, Cucumber, or Behave.
• Conduct API testing and integration testing using libraries like boto3, requests, etc.
• Generate rich test reports using tools like pytest-html and enable parallel execution via pytest-xdist.
• Collaborate with data scientists and ML engineers to define evaluation benchmarks and test datasets.
• Implement end-to-end validation of AI workflows, including orchestration frameworks.
• Integrate tests into CI/CD pipelines (GitLab, Jenkins).
• Monitor system performance and quality metrics using tools like Grafana.
• Manage defect tracking, test cases, and reporting using Jira and X-Ray.

Mandatory Skills

Programming & Automation

• Strong hands-on experience with Python
• Experience with:

o boto3, requests

o pytest, pytest-html, pytest-xdist

• Knowledge of testing frameworks:

o Pytest / TestNG / JUnit / Cucumber / Behave

AI/LLM Testing Expertise

• Experience in Software QA/Testing for AI/ML or LLM-based systems
• Strong knowledge of:

o LLM evaluation techniques (semantic similarity, embeddings)

o Regression testing for AI outputs

o Prompt/output validation

Orchestration Frameworks

Experience with at least one:

• LangChain / LangGraph
• LlamaIndex
• DSPy
• OpenAI Assistants / Actions
• Similar AI orchestration tools

DevOps & QA Tools

• CI/CD Tools: GitLab, Jenkins
• Monitoring: Grafana
• Test & Issue Tracking: Jira, X-Ray
• Exposure to AI Quality Engineering practices

Secondary Skills (Nice to Have)

• Experience with AWS and data platforms:

o SageMaker, Redshift, Athena

• Databases:

o SQL Server, Teradata, Aurora DB

• Data tools:

o Alteryx, Data Lake technologies

• Understanding of data pipelines and large-scale data processing