Head of Evaluations (Legal AI Benchmarking)

Job Description Job Description: Head of Evaluations

Location: New York, NY / Remote

About The Company Newcode.ai is transforming how law firms and legal professionals use AI. The team works at the intersection of AI and legal innovation, building products for the future of legal work.

Note: As a fast-growing startup, not every process or playbook is fully established. Priorities can shift quickly. The ideal candidate is comfortable with ambiguity, takes ownership, and proactively creates structure.

Compensation & Benefits Position Overview

Discover more

jobs

Educational Resources

Engineering & Technology

We are seeking a highly analytical professional with a strong statistical background to lead evaluations. You will design, implement, and scale testing frameworks to ensure AI products meet high standards for legal reasoning, factual accuracy, regulatory compliance, and near-zero hallucination rates.

Key Responsibilities

• Design Legal Benchmarks for Contract Drafting, Information Extraction, Legal Research, and Contract Review
• Build, source, and maintain relevant datasets
• Audit AI Output: Review and score complex AI-generated legal text, contract analyses, and statutory interpretations for accuracy and precision; lay out a strategy
• Define Evaluation Metrics: Establish clear criteria for grading model performance, focusing on logical reasoning, citation accuracy, and safe abstention
• Collaborate with Engineering: Translate legal errors into actionable technical feedback for model fine-tuning

Requirements

• PhD or Masters in statistics, mathematics, machine learning, or equivalent
• Analytical Skills: Proven ability to break down complex statutory frameworks and case law into structured, logical data points
• Tech-Savviness: Proficiency in Python, pandas, numpy, Jupyter notebooks, and similar statistical models