Faculty Subject Matter Expert- AI Evaluation (Remote, $50/hour)

We are hiring experts to test how well advanced AI systems reason over the published literature in their own field. If you have written research of your own and have assessed other people’s work — as a graduate researcher, in industry, or in a university post — this is work you will recognize.

$50/hour, remote, 10 hours a week.

About Pareto.AI

Pareto.AI is a human data platform connecting experienced professionals with leading AI researchers to help build safer, more reliable AI systems grounded in real-world expertise.

Project overview

This project focuses on improving the ability of advanced AI systems to reason over professional and academic documents. We are building an AI assistant capable of analysing documents in your field, answering realistic professional or academic questions, and producing structured, expert-level responses reflective of real-world work.

In practice, the work looks like setting an exam and marking it — except the candidate is a frontier AI model, the questions come out of your own field’s published literature, and the marking has to be rigorous enough to train on.

Who are we looking for?

Researchers, academics and graduate students tend to be especially well suited, since their day-to-day work already involves reading and synthesising dense documents, and assessing whether an argument holds up.

Practicing professionals, faculty or graduate researchers with a demonstrated academic or professional record in one of the following domains:

• Science and academic fields — PhD researchers, postdocs, faculty, lecturers, published researchers
• Healthcare — physicians, pharmacists, physician assistants, nurse practitioners, clinical researchers, allied health specialists, medical and clinical faculty
• Software Engineering — backend, full-stack and ML engineers, data scientists, DevOps and cloud engineers, CS graduate researchers
• Finance — financial analysts, applied mathematicians, accountants, actuaries, academic finance, economics and mathematics instructors

Scope of work

Selected experts will design and evaluate realistic tasks reflecting professional workflows in their domain. Examples include:

• Assembling document sets from your field — published research literature, case studies, technical papers, filings — and posing realistic professional or academic questions to an AI assistant
• Working across long-document and multimodal formats (tables, charts, images, diagrams)
• Evaluating AI-generated responses for accuracy and completeness
• Grading model outputs and providing structured feedback

Requirements

• A master’s degree or higher in your field, or an equivalent professional credential or license. PhD, postdoctoral or faculty background strongly preferred
• Published or written research output of your own — peer-reviewed papers, conference proceedings, a completed thesis, or substantial technical reports in your field
• A teaching, supervision or assessment record — university teaching or TA work, thesis supervision, peer review, exam marking, or curriculum and rubric design
• Professional or academic experience applying that expertise to real documents or cases
• Ability to identify issues and risks in complex, domain-specific scenarios
• Strong written communication skills

Compensation and Commitment

• $50/hour for approved submissions
• Flexible, remote participation (10 hours/week)
• Initial work expected within the first week after acceptance
• Additional ad-hoc expert tasks may also be offered at the same rate

Compliance Note

If you are based in the states of California, Washington, or New York MUST have an Established Business Entity (e.g., an LLC) for your application to be considered.

Pay: $50.00 per hour

Expected hours: 10.0 – 15.0 per week

Application Question(s):

• Have you completed a master’s degree or higher (or hold an equivalent professional credential or license)?
• Do you have experience teaching, supervising, peer-reviewing, or assessing academic or technical work?
• Do you have prior experience with AI training, data annotation, or evaluating AI model outputs?
• If you are based in the states of California, Washington or New York, do you have an established business entity (e.g. an LLC), or are you willing to set one up?

Work Location: Remote