SDET 2 - AI Systems

Summary

Own quality for production AI systems at KnackLabs. You’ll work on a voice agent handling real banking conversations in Hindi-English and an enterprise AI chat platform with RAG, agent orchestration, and PII handling. This is an engineering role, not a traditional QA role. You write real code, build eval infrastructure, and ensure AI systems don’t break in production.

Location: Hyderabad, on-site - (Hybrid)

Responsibilities

Build eval harnesses and golden datasets for LLM response validation

Design LLM-as-judge pipelines that gate every prompt PR on automated eval scores

Validate RAG retrieval accuracy, embedding quality, and hallucination rates

Test multi-agent workflows, tool calling, and agentic AI pipelines end-to-end

Red-team AI systems with adversarial prompts, PII leak testing, and jailbreak scenarios

Build drift dashboards and observability so the team can track AI quality over time

Load and chaos test async LLM pipelines

Write production-grade code in Python or Java that survives senior engineer review

Own quality communication across sprint and release cycles

Qualifications

2 to 5 years QA/SDET experience with minimum 2.5 to 3 years in backend/API testing

Strong coding in Java, Python, or TypeScript

Hands-on with REST Assured, Pytest, or equivalent coded API frameworks

Understanding of RAG pipelines, LLM evaluation, and non-deterministic systems

Experience with tools like RAGAS, DeepEval, LangSmith, Langfuse, or Promptfoo is a strong plus

Understands microservices, async systems, and event-driven architecture

Builds automation frameworks from scratch, not just uses tools

CI/CD integration experience

Strong fundamentals can substitute for AI experience if you learn fast

Systems thinker who reasons about failure modes, retries, latency, and contracts

Performance testing exposure preferred (JMeter, k6)

Agile experience