Head of AI Evaluation

Head of AI Evaluation, LLM & AI Systems

Compensation: $400,000- $800,000 TC

Location: Hybrid New York City

A world-class, research-driven organization operating in a high-stakes decision-making environment.

We’re partnering with a highly selective organization to hire a senior leader who will define how AI systems are evaluated, trusted, and used to support consequential decisions.

This role sits at the intersection of AI, complex data, expert judgment, and research. The core challenge: when should an AI system actually be trusted?

You’ll take ambiguous, expert-driven questions and determine what data should exist, how it should be structured, how quality should be measured, and what evidence is needed before an AI capability can be relied upon.

What You’ll Do

• Own the evaluation strategy for LLMs and AI systems
• Build golden datasets, benchmarks, and evaluation frameworks
• Define quality metrics beyond surface-level accuracy
• Translate expert judgment into measurable criteria
• Connect offline evaluation with real-world system behavior
• Detect regressions, drift, hallucinations, and grounding failures
• Build lightweight pipelines, analysis tooling, and human-in-the-loop workflows
• Partner closely with researchers, engineers, and domain experts
• Shape standards for AI quality, trust, and decision readiness

What You’ll Bring

• Strong experience owning ML or AI evaluation
• Experience with benchmark datasets, experimentation, or evaluation frameworks
• Understanding of offline vs. online performance
• Experience with monitoring, drift detection, regression testing, or failure analysis
• Comfort evaluating qualitative or ambiguous model outputs
• Strong Python and data skills
• Experience with messy, unstructured, research, behavioral, or human-generated data
• Ability to turn ambiguous questions into structured, trusted data capabilities

Relevant backgrounds may include applied ML, ML evaluation, research engineering, knowledge graphs, scientific or biomedical data, decisioning, risk, ranking, or trust & safety.

Tech Stack

• Python
• LLMs / foundation models
• LLM and RAG evaluation
• Golden datasets and offline benchmarks
• Human-in-the-loop evaluation
• Model monitoring and regression testing
• Structured and unstructured data

Why Join?

You’ll help build the yardsticks that determine whether AI systems are genuinely improving decisions or quietly degrading them.

This is a rare opportunity to define evaluation standards from the ground up, work with exceptional technical and domain experts, and operate in an environment where AI quality and trust genuinely matter.

About People In AI

People In AI is a specialist recruitment partner focused exclusively on the global Artificial Intelligence and Machine Learning market, connecting exceptional talent with some of the most ambitious technical challenges in the industry.