Applied AI Research Engineer: Benchmarking & Performance Economics

Description

This is an applied research role, not a machine learning science role. You will not be advancing the state of the art in model architecture. You will be producing decision-grade numbers: benchmarks, cost models, and calculators that determine what we build, what we tell customers, and what we are willing to claim in public.

The work sits at the intersection of three disciplines that rarely overlap in one person — systems engineering, inference-stack depth, and honest experimental design. If you have ever read a vendor benchmark and immediately knew which confound made it meaningless, this is the seat for you.

What you’ll own

• The token economics of agentic execution. What it actually costs to run an autonomous agent to task completion, and how that compares to LangGraph, CrewAI, AutoGen, the OpenAI Agents SDK, Ray, and whatever exists six months from now. Tokens per call is the wrong unit and you know it, the unit is cost per completed task at a fixed success rate, and getting there means measuring steps, retries, tail latency, success rate, and run-to-run variance.
• Routing and model-selection economics. Where each routing strategy sits on the cost/quality frontier, what the router costs to run, and how a 5% misroute rate compounds into a 40% task failure rate across a thirty-step trajectory. Routing quality gets evaluated at the trajectory level here, not the request level.
• Small language model economics. The crossover analysis: at what volume, task narrowness, and quality tolerance does a fine-tuned 3B beat a frontier API call? You own the full stack: data curation, training, evaluation, serving, and the ongoing cost of drift and retraining, and the judgment to say “don’t train this” when the numbers say so.
• GPU and serving-stack modeling. A parameterized calculator that takes model size, quantization, batch size, context distribution, KV cache footprint, and concurrency, and returns throughput, memory headroom, latency percentiles, and cost per million tokens. Validated against measured ground truth, not derived from spec sheets. It accounts for prefix caching, because agentic workloads resend the same system prompt hundreds of times and a model that ignores that is wrong by a multiple.
• The cost of governance. What policy evaluation, guardrails, evaluator calls, and human-in-the-loop suspension actually cost in latency and tokens. Enterprise buyers assume the number is bad. Being able to state it precisely turns an objection into a differentiator.
• The benchmark harness as a product. Pinned versions, controlled cache state, variance reported rather than averaged away, full configuration captured with every result, reproducible by a third party. Most benchmark work in this industry does not survive scrutiny. Yours will be the asset that does.

What We’re Looking For

• You are constitutionally unwilling to publish a number you can’t defend. When the result is inconvenient, you report the result.
• Fair to the competition, to the point of discomfort. You will implement a rival framework properly, in its own idiom, and you will spend the extra two days doing it well. An unfair benchmark is worse than no benchmark, it’s a liability the moment someone reproduces it.
• You have been wrong in public and corrected it. We consider this a qualification, not a blemish. Someone who has never had a result challenged has never had a result that mattered.
• Scoping discipline. Most of this work arrives as an ill-posed question. Turning “is our routing better?” into a measurable experiment with a defined success criterion is the core skill, and it happens before any code is written.
• Suspicious of your own instruments. You assume the harness is lying until you’ve proven otherwise. You measure the measurement.
• Writer. The output of this role is arguments supported by evidence. A correct result that can’t be explained to an engineer, an architect, and a buyer has delivered a fraction of its value.

• Builder, not administrator. Akka is small enough that you write this playbook rather than inherit it. That should energize you.

• Real experimental design and statistics. Confidence intervals, confound control, sample sizing, and the judgment to know when a difference is noise. This is the most common gap in otherwise strong candidates and the one we’re least able to flex on.
• Inference stack depth, hands-on.…