AI / LLM Quality Engineer

Key Responsibilities

• Feedback Loop Management: Own the end-to-end pipeline that ingests

translation feedback (corrections, terminology flags, and style critiques) from

human reviewers and transforms it into actionable technical adjustments.

• Root Cause Analysis: Diagnose whether translation bugs are caused by base

LLM degradation, outdated prompt constraints, Retrieval-Augmented Generation

(RAG) issues, or contextual gaps.

• Systemic Prompt & Context Engineering: Refine system prompts, structural few-

shot examples, and dynamic context windows to ensure accurate regional

nuances, tone of voice, and localized industry terminology.

• Automated Evaluation Pipelines: Design and maintain automated “translation

quality gates” and regression suites using metrics like BLEU/TER, COMET, and

LLM-as-a-judge frameworks to evaluate updates before they go live.

• Cross-functional Collaboration: Partner with data annotation operations, product

managers, and software engineers to ensure LLM updates do not break core

application features or cause performance regressions.

Required Skills & Experience

• Programming & LLM Tooling: Strong proficiency in Python and familiarity with

LLM frameworks (e.g., LangChain, LlamaIndex) and API integration (OpenAI,

Anthropic, or open-source models).

• NLP & Localization Fundamentals: Deep understanding of natural language

processing concepts, machine translation evaluations, and

localization/globalization challenges (e.g., handling idioms, gender bias, and

cultural constraints).

• Data Ops & ML Evaluation: Proven experience building automated testing or

continuous evaluation (CI/CD) pipelines for AI applications.

• Experience: 2+ years of experience working as an LLM App Developer, AI Quality

Engineer, or Data Engineer within a localization/translation ecosystem.