AI Operations Specialist

Job Description

Description

The Community you will join:
We’re the AI & ML Enablement team in Supply Tech. We help teams across the company use AI to do real work, and prove that it works. Right now, we’re helping business and field teams use AI to take on the review and quality work they run, and helping product teams understand whether an AI feature works before it reaches our users. Day to day, we move manual review processes to AI and put people where their judgment matters most. We design new workflows so AI does the work from the start. We also build the labeled datasets that show whether the AI is right, and we keep everything running as models and policies change. Our methods and projects change as the models do, and we expect the scope of this role to grow with them.
The difference you will make:
You’d work alongside our team on whichever problems AI could take on, whether that’s a business team with a process that’s slow, expensive or inconsistent, or a product team getting an AI feature ready to launch. The job looks much the same either way. You start with how the work really gets done, help build an AI version of it, and test it until you trust the results. Then you hand it off and use what worked on the next problem. The mix of projects will keep changing as AI takes on more of the work, so we care more about how you work than about a fixed list of projects. This is a hands-on role. You help build the workflow, ship it, and stay on to make sure it works.

Duties

Help the team decide which projects will drive the largest business impact, and make the case for the ones worth doing.
Sit with the team that owns a problem, map how the work gets done today, and work out how AI can do it and where people step in for the hard cases.
Build the AI workflow: write the prompts, set up the pipeline, and keep it at the agreed quality level, watching for drift once it’s live.
Build the test sets, measure accuracy, and calibrate LLM judges against human reviewers.
Support product teams in agreeing on what “good enough to launch” means for an AI feature, and bring the evidence for the go/no-go call.
Turn unclear policy into criteria a reviewer can apply consistently, working with product, Legal and Policy.
Work with subject matter experts to write labeling guidelines and produce human-verified labels on real data.
Run labeling at scale with vendor agents, and check their work against agreement metrics and your own spot checks.
Make what you build reusable and document what you learn, so the next team gets to production faster.
Keep our methods current as models, tools and vendor agents change, and retire what stops working.
Own the workstreams you’re assigned, from setup through quality checks. Model infrastructure and product roadmaps stay with the teams that own them.

Requirements

5-7 years in operations, program management, analytics, consulting or applied AI, where you were responsible for results.
You use AI every day and have built something with it that has actually moved the needle, such as a workflow, an automation or an analysis. You know where it breaks, and you keep pushing it further.
You’re comfortable with numbers: accuracy, precision and recall, sampling. SQL and notebooks help. If you haven’t used them yet, you’ll pick them up quickly with AI assistance.
You’re curious about how work actually gets done, including the awkward exceptions, and you’ve taken a messy process and made it measurable.
You’ve worked on quality in some form, such as review guidelines, QA or audits.
You write clearly and can work with business leads, PMs and engineers in the same week.
You’re fine working with limited direction, and you try things yourself instead of waiting for someone to build them. A technical background isn’t required.
You don’t need to have designed evaluations or calibrated LLM judges before. We’ll teach you, and you’ll work with a team that has done it many times.
Preferred experience
Hands-on SQL, Python or API experience.
Trust and safety, content moderation, annotation operations or other human-in-the-loop work at scale.
Fine-tuning models or calibrating LLM judges.
Background in a two-sided marketplace, travel, or another operations-heavy business.
Experience setting up intake and prioritization for a new team.

Compensation

Req ID