LLM – AI Quality Analyst (Personalization) – English

Vraify is hiring: LLM – AI Quality Analyst (Personalization) – English

Apply directly through the Vraify job portal:

https://vraify.com/jobs?post=llm-ai-quality-analyst-personalization-english

Open the link and select “Apply for this role” to submit your application.

About the role

Evaluate how accurately and naturally AI uses personal context to create helpful responses. This is a remote contractor assignment for four months, starting immediately. Pay: USD $30/hour.

Location

Remote, USA (approved states only; eligibility to be confirmed during screening).

Responsibilities

• Design and run conversations of one to five turns to test how Gemini uses personal context.
• Evaluate whether responses address the original prompt and apply personalization appropriately.
• Identify unsupported personal claims, incorrect inferences, and forced connections.
• Assess whether personal information is integrated naturally without unnecessary explanations.
• Compare and rank two responses side by side for accuracy, helpfulness, ease of use, and overall quality.
• Write clear rationales and detailed annotations referencing specific conversation turns.
• Verify model debug information to check whether chat summaries and data sources were used appropriately.
• Delete evaluation conversations as instructed to keep test interactions out of future chat history.
• Provide constructive feedback and collaborate with the team to maintain consistent evaluation quality.

Your profile

• At least one year of relevant experience.
• BS/BA degree or equivalent experience in Policy, Law, Ethics, Linguistics, Journalism, Computer Science, or a related analytical field.
• Strong English reading comprehension and clear, structured written communication.
• Willingness to use your primary personal Google account, not a testing account, and enable past Gemini conversations, Gmail, Google Search, and YouTube activity as personal data sources for evaluation.
• Strong analytical judgment and attention to detail when reviewing nuanced AI responses.
• Ability to design creative prompts based on your own experiences and write rationales supported by specific conversation turns.
• Strong communication and collaboration skills, with the ability to work independently remotely.
• Desktop or laptop with a reliable internet connection.
• Availability for a 30- or 40-hour weekly commitment, including four hours of overlap with PST. The applicable schedule will be confirmed during screening.
• Preferred: Experience in data annotation, AI quality evaluation, content moderation, or a related role.
• Preferred: Familiarity with prompt design, multi-turn conversations, and side-by-side response evaluation.

How to apply

Submit your application through Vraify:

https://vraify.com/jobs?post=llm-ai-quality-analyst-personalization-english

Pay: $30.00 per hour

Work Location: Remote