AI Evaluation-Benchmarking--W2 Remote contract

Software Engineer — AI Evaluation, Code Benchmarking & Automation

Role Overview

Help build and scale the tooling used to measure how well AI-powered software development tools and coding agents actually perform. You’ll develop evaluation harnesses, automate benchmark runs, and build code benchmarking frameworks that compare AI-generated code against established baselines and real-world engineering outcomes. This is a hands-on engineering role focused on ensuring evaluations are accurate, reproducible, meaningful, and scalable.

Key Responsibilities

• Build and integrate evaluation harnesses and automation for software development use cases, including turning real engineering artifacts such as merged pull requests into repeatable benchmark tasks.
• Design and execute code benchmarks to evaluate AI-generated code for correctness, functionality, code quality, maintainability, test coverage, performance, and efficiency.
• Compare AI-generated solutions against human-written/reference implementations and other AI-generated solutions using consistent benchmark criteria and automated test suites.
• Build versioned, repeatable processes to evaluate AI tools, coding agents, models, and harnesses, with reproducible run environments including pinned dependencies, containerized runs, and isolated Git worktrees.
• Develop automated testing and benchmarking workflows that validate generated code through unit tests, integration tests, static analysis, performance tests, and other execution-based evaluations.
• Validate and calibrate evaluation approaches against human judgment, ensuring that benchmark scores are not only repeatable but also accurately reflect real-world code quality and engineering outcomes.
• Support benchmarking across quality, productivity, performance, cost, and latency , including measuring the effort and resources required to achieve successful coding outcomes.
• Analyze results across repeated benchmark runs, identifying variance, failure patterns, regressions, performance differences, and cost per successful outcome.
• Investigate benchmark failures and evaluation inconsistencies, determine root causes, and improve the reliability and automation of evaluation workflows.
• Work closely with engineering and data teams to improve benchmarking infrastructure, evaluation methodologies, datasets, and reporting.
• Document benchmark methodology, results, limitations, and findings for both technical engineering teams and leadership audiences.

Required Skills & Experience

• Strong software engineering background with hands-on experience building automation, developer tooling, benchmarking, testing, or validation systems.
• Strong proficiency in Python , with experience in at least one additional language such as Java, JavaScript, TypeScript, or a similar programming language.
• Experience designing or implementing code benchmarks, automated code evaluation, software testing, or comparative analysis of code implementations .
• Strong understanding of software engineering fundamentals, including code quality, correctness, maintainability, testing, performance, and debugging.
• Solid working knowledge of Git , including branches, history, commits, worktrees, and repository workflows.
• Experience with Docker/containerization , APIs, development environments, CI/CD pipelines, and typical software engineering workflows.
• Experience building automated test and evaluation pipelines capable of executing and comparing multiple code implementations or solutions.
• Familiarity with AI/LLM/agent evaluation concepts, including why a judge can be consistent but incorrect, why a single run can be misleading, evaluation variance, and benchmark contamination.
• Experience analyzing benchmark results and determining whether a measurement is technically valid and representative of the underlying engineering outcome.
• Strong troubleshooting, analytical, and problem-solving skills with the ability to identify issues in both software implementations and evaluation methodologies.