AI Test Engineer (LLM and AI QA)

**About the Role**

We’re looking for an AI Tester to evaluate the quality, safety, and reliability of AI-powered products, including LLM applications, agents, and ML-driven features. You’ll design test strategies for non-deterministic systems and help ensure our AI behaves accurately, safely, and consistently in production.

**Responsibilities**

• Design and execute test plans for LLM, RAG, and agentic workflows

• Build evaluation datasets (golden sets, edge cases, adversarial prompts)

• Assess output quality: accuracy, relevance, hallucination, tone, and consistency

• Perform red-teaming: prompt injection, jailbreaks, data leakage, and bias testing

• Validate tool-calling, multi-step agent behavior, and guardrails

• Automate regression and evaluation pipelines (e.g., LLM-as-judge, scoring harnesses)

• Track metrics such as latency, cost per request, and failure rates

• Log, triage, and reproduce defects with clear prompts, inputs, and model versions

• Partner with ML engineers, product owners, and developers to improve prompts and models

• Document findings and contribute to release go/no-go decisions

**Required Qualifications**

• 4+ years in QA, test automation, or data/ML evaluation

• Solid understanding of how LLMs work (prompts, context windows, temperature, embeddings)

• Experience testing APIs and writing automated tests (Python preferred)

• Strong analytical skills and attention to detail

• Clear written communication for documenting non-deterministic behavior

**Preferred Qualifications**

• Experience with evaluation tools (e.g.,LangSmith, Ragas, DeepEval)

• Familiarity with RAG pipelines, vector databases, and agent frameworks

• Knowledge of AI safety, responsible AI, and compliance practices

• Cloud platform experience (Azure)

• Background in test automation frameworks