Medical Reasoning Expert - Clinical AI Evaluation

About Pareto.AI

Pareto.AI is a human data platform connecting experienced professionals with leading AI researchers to help build safer, more reliable AI systems grounded in real-world expertise.

Project overview

This project focuses on improving advanced AI systems for complex clinical reasoning rooted in diagnostic judgment, differential formulation, and evidence-based management decisions.

We are building AI capable of reasoning through realistic clinical presentations, weighing competing diagnoses, and producing structured medical analysis reflective of professional practice.

Who are we looking for?

We are seeking physicians, resident physicians, and clinical educators who regularly exercise diagnostic and management judgment in professional practice.

Relevant backgrounds may include:

• Internal medicine, family medicine, or general practice
• Hospital medicine, emergency medicine, or critical care
• Any medical or surgical specialty or subspecialty
• Residency or fellowship training in any discipline
• Clinical education, examining, or academic medicine
• Clinical informatics, digital health, or medical AI evaluation

Strong candidates often have experience teaching, examining, grading, or evaluating clinical reasoning and written medical analysis.

Scope of work

Selected experts will design and evaluate challenging clinical tasks rooted in diagnostic reasoning and structured judgment. Examples include:

• Constructing realistic clinical scenarios from professional experience
• Diagnostic reasoning and differential formulation
• Investigation selection and interpretation
• Management and treatment decision analysis
• Multi-step clinical reasoning tasks
• Evaluating AI-generated clinical responses for accuracy and patient safety
• Grading model outputs and providing structured feedback

Minimum requirements

• Doctor of Medicine (MD), MBBS, MBChB, DO, or equivalent medical degree
• Active or currently completing medical qualification
• Demonstrated clinical reasoning at a professional level
• Excellent written communication and analytical precision

Compensation and Commitment

• $75/hour for approved submissions
• Flexible, remote participation (10 hours/week)
• Initial work expected within the first week after acceptance
• Additional ad-hoc expert tasks may also be offered at the same rate.

Compliance Note

If you are based in the states of California, Washington, or New York MUST have an Established Business Entity (e.g., an LLC) for your application to be considered.

Pay: $75.00 per hour

Work Location: Remote