P

AI Data Annotator / AI Trainer

ProtocolClear

Management & Business Development

Today

New

Lagos Full Time IT & Telecoms Confidential

Share link

Share on WhatsApp Share on LinkedIn Share on Facebook Share on Twitter

Share via SMS

Job descriptions & requirements

About ProtocolClear

ProtocolClear is an independent, principal-led security, privacy, AI governance, and compliance consultancy serving regulated-industry clients across the United States and Nigeria. We are building an in-house data and model-evaluation team to label and assess the data that trains and tests AI systems. This is a remote role for a Lagos-based annotator who wants steady, serious work and the chance to grow with a fast-moving operation. We do careful, standards-driven work for clients who cannot afford sloppy data handling.

About the role

We label and evaluate data across domains: classification, entity and span tagging, image bounding-box and segmentation work, and reinforcement-learning-from-human-feedback (RLHF) tasks where you compare model responses, apply a preference rubric, and write the reasoning behind your rating. The data spans subjects; your job is the annotation craft, not a single field. This is an output-focused role measured on accepted batches, accuracy, agreement, and gold-task performance, not on hours logged. You will work inside a defined pipeline with guidelines, gold sets, and calibration sessions, and you will deliver consistently against a clear written quality standard. We are looking for someone who already operates at a professional standard and is productive from day one, not someone who needs to be taught what annotation is.

The standard your work is held to

Every batch you submit is checked against a written Definition of Done before it is accepted and paid. In short: we seed hidden gold/honeypot items to measure your accuracy and expect at least 95% on them; we measure how well your labels agree with consensus and expert review using kappa and expect at least 0.80 on objective tasks; we expect tight, consistent labels with no drift; on RLHF tasks we expect clear written rationales that engage the specific item and apply the rubric one axis at a time, with harmlessness prioritized where instructed; we expect strict confidentiality on client data under NDA inside approved tools only; and we do not accept annotations or rationales produced by ChatGPT, Claude, or any other AI tool unless a task explicitly tells you to use one. Work that fails any of these is rejected and redone before it is paid. If you already work to standards like these, you will feel at home; if this is new to you, this role is probably not the right fit.

What you will do

• Apply detailed annotation guidelines and taxonomies to text, image, and document data, including classification, entity and span tagging, and bounding-box or segmentation work.
• Compare and rank model responses, score them against rubrics, and write defensible rationales covering helpfulness, factuality, and safety/harm.
• Hit a daily throughput quota while holding the quality thresholds (kappa, gold-task accuracy, low rework).
• Pass gold/honeypot checks and take part in calibration sessions to keep agreement high.
• Escalate ambiguous or out-of-guideline items rather than guessing, and keep your own edge-case decision log so similar items are treated the same way.
• Handle all task data under NDA inside approved tools only, with strict care for any PII or sensitive content.
• Manage your own focus, self-checking, and file versioning so accepted work is never lost and deadlines are protected, and incorporate feedback within a defined revisions process.

What we are looking for (required)

• Hands-on data-annotation / AI-training experience and a real understanding of the craft, not just having “used a labeling tool once.”
• A working grasp of inter-annotator agreement (kappa), gold tasks/honeypots, consensus, and adjudication, and the ability to explain them in your own words.
• Experience with RLHF / model-response rating, including writing rationales that apply a rubric.
• Strong written English and sustained attention to detail over long labeling sessions.
• Comfort working to precise written standards and a pay structure where most pay is tied to accepted, quality-passed work.
• Willingness to be screen-proctored on designated tasks and to never use unauthorized …