Senior AI Evaluation Engineer

Engineering Expert

Role Overview

We are seeking experienced AI Evaluation Engineers specializing in engineering simulation and design to author and validate complex, simulation-based engineering problems for training and evaluating state-of-the-art AI agents. The role spans electrical, mechanical, control systems, aerospace, systems, and robotics engineering. You will create challenging, multi-constraint tasks that require agents to interpret requirements, navigate trade-offs, configure open-source simulation tools, diagnose failures, and iterate toward valid solutions. You will analyze agent execution logs, identify reasoning and tool-use gaps, and build objective automated graders.

Responsibilities

Author self-contained engineering design tasks with competing constraints, explicit optimization targets, validated reference solutions, and objective autograders.

Build, run, and validate problem environments using open-source simulation tools and custom Python test benches.

Analyze agent outputs and execution logs across repeated trials to identify systemic failure modes, such as misinterpreting simulator feedback, premature convergence, or physically impossible geometries.

Refine benchmark difficulty based on model performance data while avoiding ambiguity or missing information.

Collaborate with AI researchers, pod leads, and domain experts to integrate rigorous benchmarks into model-evaluation pipelines.

Requirements

A master’s degree or PhD in electrical, mechanical, or aerospace engineering, and 10+ years of hands-on engineering design experience. The description also lists a bachelor’s degree or equivalent practical experience in any field as an education pathway.

Proficiency in at least one relevant open-source simulation package, such as ngspice, PySpice, OpenFOAM, FEniCSx, CalculiX, python-control, CadQuery, build123d, OpenModelica, Cantera, or Gmsh, along with strong Python scripting skills.

Hands-on experience with modern LLMs or coding agents and evaluation concepts such as pass@k, failure-mode analysis, and nondeterministic behavior; ability to audit trajectory logs and isolate reasoning or tool-use failures.

Strong attention to physical plausibility, unit consistency, boundary conditions, convergence criteria, and technical documentation.

A stable, high-speed internet connection and a personal computer suitable for remote work.

Availability for 40 hours per week, four hours of overlap with PST, and weekend on-call availability. Part-time engagement is also noted as acceptable.