AI Interaction Analyst | $30/hr Remote

Position: LLM AI Quality Analyst (Personalization) English

Type: Short-Term Contractor Assignment (4 months)

Compensation: $30/hour

Location: Remote

Commitment: Full-time availability, with 30 or 40 hours per week and at least 4 hours overlap with PST

Role Responsibilities

• Design and execute creative multi-turn conversational prompts, typically involving 1 5 turns, based on personal context and experiences.
• Evaluate AI responses based on the intent and requirements of the starting prompt.
• Assess whether personalization is accurate, relevant, and appropriately applied.
• Analyze responses for Grounding issues, including unsupported claims, incorrect personalization, flawed inferences, and hallucinations.
• Evaluate Integration quality to ensure personal information is incorporated naturally without unnecessary or robotic overnarration.
• Compare and stack-rank two AI model responses side-by-side based on helpfulness, usability, naturalness, and overall quality.

Requirements

• Minimum 1 year of relevant experience in data annotation, AI quality evaluation, content moderation, or a related analytical role is strongly preferred.
• Strong English proficiency, with the ability to read and write English at a high level.
• Experience or demonstrated ability in creative prompt engineering and multi-turn conversational testing.
• Strong understanding of personalization concepts, including identifying incorrect personalization, poor inferences, unsupported claims, and forced connections.

Application Process

• Apply via the Interest Check Form.
• Check email for next steps.