Software Engineer in Test (AI Eval)

About InRhythm

InRhythm is a modern product consultancy and digital innovation firm helping Fortune 50 enterprises build and launch next-generation digital products and platforms.

Founded in 2002, InRhythm partners with global organizations to deliver high-impact solutions across product strategy, design, cloud-native applications, mobile, and web platforms.

At InRhythm, we believe exceptional products are created by exceptional people. We foster a culture built on innovation, collaboration, ownership, continuous learning, and meaningful impact.

Job Title: Lead Eval Engineering Consultant

Location: Bangalore, India

Work Mode: Hybrid

We’re hiring a Lead Eval Engineering Consultant (LLM/AI Evaluation) — if you have experience with Promptfoo, Braintrust, or LLM-as-judge systems.

The InRhythm Opportunity: Core (Foundry)

InRhythm is looking for a Lead Eval Engineering Consultant to join our Core Foundry team in Bangalore. At InRhythm, we do not just ship software, we guarantee outcomes. As an Eval Lead, you are embedded directly within a high-stakes AI-DLC delivery pod. You serve as the ultimate guardian of delivery integrity, moving beyond traditional QA frameworks to design, enforce, and execute verification contracts that make every delivered outcome provable before it leaves the engineering pipeline.

Your Impact

You will eliminate delivery risk at the source. By partnering with Spec Owners during the initial design phases, you ensure that loose requirements are never allowed to pass into development. Your work ensures that engineering pipelines emit bulletproof semantic evidence, preventing false-pass behaviors and structural gaps from escaping to production. You hold independent sign-off authority at the final gate, acting as the definitive voice on whether a feature has structurally proven its acceptance criteria.

What You Will Do:

● Own the Feature Eval Plan: Map every acceptance criterion to a dedicated evaluation path (automated test, static/gate output, human-judged, design review, or explicit risk mitigation) during the Define phase.

● Establish Sufficiency Thresholds: Define the quantitative and qualitative evidence bars required to mark a criterion as Confirmed, preventing the shifting of proof standards after code execution.

● Design and Calibrate Rubrics: Construct manual review frameworks, design-fidelity metrics, and LLM-as-judge graders, verifying their alignment and calibration against human judgment.

● Manage the Unverifiable-Criteria Log: Identify, document, and log features that cannot be programmatically or structurally proven, establishing named ownership and risk mitigations rather than allowing soft passes.

● Enforce False-Pass Controls: Design the algorithmic and semantic checks that catch illegitimate green results, serving as the human counterpart to grounding gates to eliminate trivial diffs and tautological test suites.

● Govern the Verification Contract: Oversee the integrity of the full testing pyramid, establishing what each layer must prove while collaborating with Technical Boundary Leads who oversee developer agent code authorship.

● Execute Phase Gate Sign-Offs: Audit task DAGs during planning, verify pipeline gate signals during execution, and deliver the final independent verdict (Confirmed, Missing, Divergent, or Unverifiable) at the Outcome Review stage.

What You Will Bring

Must Have Experience

● Production-Level Python: Deep engineering experience extending test harnesses, working with dashboard backends, manipulating pytest configurations, and writing clean GraphQL resolvers.

● Evaluation Methodology: Proven track record designing evaluation datasets, reference tests, and reproducible scoring harnesses for LLMs or agentic workflows using tools like Inspect, Promptfoo, or Braintrust.

● Statistical Honesty: Expert-level command of hypothesis testing, significance tracking, and false-pass control mechanisms with the ability to clearly demonstrate why passing test suites do not equate to verified outcomes.

● LLM-as-Judge Calibration: Practical experience building, testing, and optimizing model-based graders, including statistical proof of their alignment with human expert benchmarks.

● Technical Defensibility: Exceptional communication skills with a demonstrated ability to defend objective technical verdicts directly to client counter parties and engineering leads under timeline pressure.

**Nice to Have Experience…