AI Engineer — Retrieval & Evaluation

athenaprep ResourcesOur StoryGuideBlogProfilePricing

HelpdeskGo to Dashboard

ResourcesOur StoryGuideBlogProfilePricingHelpdeskGo to Dashboard

All open roles

AIActively hiringPosted 3 months ago

AI Engineer — Retrieval & Evaluation

Make Ask Athena genuinely trustworthy. This is retrieval over our own question bank and a student’s own notes — where a confidently wrong answer costs someone marks.

Apply for this role

Experience 2–5 yrs

Location Remote (India)

Arrangement Remote

Employment Full-time

Department AI

Responsibilities

• Improve retrieval quality and answer grounding across our content corpus
• Build evaluation harnesses so answer quality is measured, not argued about
• Cut hallucination rates and make refusals honest rather than evasive
• Keep inference fast and cost-sane at student scale
• Work with Content to turn subject expertise into better grounding data

Requirements

• 2+ years shipping ML or LLM features to real users
• Hands-on with embeddings, vector stores and retrieval pipelines
• Rigour about evaluation — you measure before and after, always
• Strong Python; comfortable in a TypeScript codebase

Preferred Skills Nice to have, not required — apply even if none of these are true of you.

• Experience with Upstash Vector, pgvector or similar
• Background in education or assessment
• Published work, open-source contributions or writing on RAG

What success looks like

How we’ll judge this role together — shared with you on day one, not at review time.

• Measurable reduction in ungrounded answers against a held-out eval set
• Median answer latency stays under two seconds
• Cost per assisted question trends down quarter over quarter

Growth path

• 1

First 90 days

Stand up the evaluation harness and establish the quality baseline.

• 2

Year one

Own the full AI surface — retrieval, grounding and cost.

• 3

Beyond

Define AI strategy as the product’s intelligence layer deepens.

Skills you’ll use

• Python
• RAG
• Vector search
• LLM evaluation
• Prompt engineering

Benefits

• Remote-first

Work from anywhere in India, with hours you actually control.

• Meaningful ownership

Early team members share in what they help build.

• Learning budget

Books, courses and conferences that make you better at the craft.

• Full AthenaPrep access

Every plan unlocked, for you and your immediate family.

• Equipment that helps

The machine and setup you need to do focused work.

• Time that’s genuinely off

Generous leave, and no expectation of being reachable on it.

Hiring timeline

• 1

You apply

We reply within 3 days

A short note on why this problem interests you beats a long CV. Links to things you’ve made help most.

• 2

Intro conversation

30 minutes

A relaxed call about your work, what you want next, and what we’re actually building. Questions go both ways.

• 3

A real problem

Take-home or live

A small, time-boxed piece of work close to what the role actually involves. No trick puzzles, no unpaid project work.

• 4

Conversation with the founder

45 minutes

Values, ownership, and how you’d want the first ninety days to go.

• 5

Offer

Within a week

Clear terms, clear scope, and a start date that works for you.

Questions

Yes. We care much more about evidence of craft than years of experience — a project you built and can explain in depth counts for more than a familiar company name.

Yes, remote-first across India, and async by default. We write decisions down so nobody needs to be online at a particular hour to stay in the loop.

Usually under three weeks from application to offer. We reply to every application within three days, even when the answer is no.

Yes, and they are real work with real ownership, not shadowing. Strong interns are regularly offered full-time roles.

Join the talent community below. We open roles as we grow, and the people already in the pool are the first ones we contact.

Think this is you?

A short note on why this problem interests you beats a long CV. If you’re close but not an exact match, apply anyway — we’d rather read your note than miss you.

Apply for AI Engineer — Retrieval & Evaluation Join talent community

Apply now