Volunteer AI Engineer (Evaluations & Benchmarking) — AI Apps for Human Communication

Crossing Party Lines (CPL) provides opportunities, training, and support for civil, respectful discourse between Americans with dissimilar ideologies. Our primary offering is the CPL meetup regularly scheduled, facilitated meetings to which all are welcome. At our meetups, attendees discuss current hot topics while facilitators focus the meetings to teach and model skills and create a safe space for members to

• Be heard by and listened to by people with different political views.
• Express views without negative repercussions.
• Learn from one another through active listening.
• Develop curiosity about perspectives that may be new and unfamiliar to them.
• Find connection with members who espouse different views.
• Develop skills required for civil, respectful discourse.

In 2016, two groups named “Crossing Party Lines” started independently of one another on opposite ends of the country, only a few months apart. They joined forces to bring civil, respectful conversation to the rest of the country.,

Help build the rigorous evaluation pipelines and benchmarks that ensure conversational AI agents remain objective, depolarizing, emotionally safe, and constructive.

Crossing Party Lines is seeking a detail-oriented AI Engineer specializing in Evaluations (Evals) to join our Apps and Agents team. Evaluating conversational agents designed for politically charged, high-stakes dialogue requires moving beyond basic accuracy checks. You will define, implement, and automate evaluation suites that measure nuanced qualities such as neutrality, tone calibration, reframing effectiveness, cognitive empathy, and resistance to bias or sycophancy.

About The Apps And Agents Team Our team blends generative AI models and multi-agent workflows with Crossing Party Lines’ proprietary depolarization frameworks. As we prepare applications for Y Combinator and work toward broader public deployment, reliable evaluation is central to building tools that users and communities can trust.

What You’ll Do

• Design and maintain systematic evaluation datasets (golden sets) reflecting diverse, realistic conversational conflict and political viewpoints.
• Implement multi-dimensional evaluation rubrics—combining deterministic checks, reference-based metrics, and LLM-as-a-judge pipelines.
• Benchmark agent outputs against specific communication standards active listening, cognitive perspective-taking, de-escalation, and non-judgmental feedback.
• Run regression tests, track performance across prompt and model revisions, and integrate automated eval gates into CI/CD workflows.
• Conduct adversarial red-teaming to uncover subtle bias, conversational failure modes, hallucinations, or toxic drift.
• Collaborate with depolarization experts, product designers, and backend developers to calibrate evaluation criteria against real-world human judgment.

You May Be a Good Fit If You

• Have experience with Python and LLM application frameworks (e.g., LangChain/LangGraph, LlamaIndex, or raw API tool orchestration).
• Understand modern LLM evaluation concepts and tools (e.g., promptfoo, DeepEval, Ragas, Phoenix, LangSmith, or custom LLM-as-a-judge scorers).
• Have a keen analytical mindset and can translate qualitative human communication principles into clear quantitative metrics.
• Are comfortable analyzing edge cases across a wide spectrum of social and political perspectives without imposing personal viewpoints.
• Value transparency, scientific rigor, and reproducibility in AI benchmarking.

What You’ll Gain

• Direct experience solving one of the most critical challenges in production AI behavioral evaluation and alignment for multi-turn conversational agents.
• A key engineering role on early-stage AI products being prepped for high-profile accelerators and widespread adoption.
• Collaboration with communication experts, prompt engineers, and product builders.
• Professional references recognizing your technical rigor and contributions to production-grade AI reliability.

Time, location, and support

This is a remote volunteer position. You may complete most of the work on your own schedule.

We hope volunteers can contribute approximately 8 or more hours per week , although hours are flexible. Team members should also be available for one or two online team meetings per week .

You will report to Lisa Swallow, co-founder of Crossing Party Lines and lead of the Apps and Agents team. You…