RC-REGULATORY COMPLIANCE-GxP AI Model Consultant-Sr

Job highlights

• Consulting or client-facing experience preferred.
• Experience creating user stories,test scripts,validation plans,or AI solution prototypes.
• Practical experience with LangChain,LangGraph,and Langfuse.
• Experience evaluating GenAI / LLM-based applications and agents. . Strong documentation,analytical,and stakeholder communication skills.

Job Summary

We are seeking a highly experienced and detail-oriented AI Model Evaluation Consultant with strong expertise in working alongside AI/ML models, Generative AI (GenAI) systems, and multi-agent workflows within GxP-regulated and quality-critical environments. The ideal candidate will ensure the reliability, fairness, robustness, and regulatory compliance of AI solutions while engaging directly with cross-functional teams in designing, testing, and maintaining production-ready, inspection-ready AI systems.

This role is responsible for leading AI model consultation and development activities across the full model lifecycle, including model integration, training, testing, deployment, and post-deployment monitoring, in alignment with GxP principles, risk-based validation, and data integrity expectations. The consultant will play a key role in embedding Responsible AI, quality-by-design, and governance-by-design practices, enabling the safe, trustworthy, and explainable use of AI and agent-based solutions in regulated life sciences and pharmaceutical contexts.

Your key responsibilities

• Work on development of AI/ML and Agentic AI strategies by providing analysis, validation insights, and readiness assessments.
• Perform AI/ML model integration, evaluation and validation across development, training, testing, and deployment phases.
• Collaborate with data scientists and engineers to identify model improvement opportunities and risk mitigations.
• Design and execute testing strategies for AI agents and multi-agent workflows, covering orchestration logic, tool invocation, and output validation.
• Document evaluation results, limitations, risks, and acceptance criteria in a clear, audit-ready manner.
• Develop or review validation plans, user and functional requirements, risk assessments, requirements traceability matrices, test protocols, test evidence, deviation records, and validation summary reports.
• Establish acceptance criteria for data quality and model performance, including accuracy, sensitivity, specificity, precision, recall, F1 score, calibration, error rates, and other measures appropriate to the use case.
• Execute or independently review functional, integration, regression, negative, boundary, security-role, audit-trail, data-reconciliation, and user-acceptance testing for AI-enabled workflows.
• Challenge model reliability through reproducibility, robustness, bias, subgroup-performance, explainability, stress, and edge-case testing, with clear documentation of limitations and residual risk.
• Define production monitoring for model performance, data and concept drift, anomalous outputs, override patterns, human review, complaints, incidents, and other signals that may trigger investigation or revalidation.
• Evaluate proposed model updates, retraining, prompt changes, knowledge-base changes, infrastructure changes, and vendor releases through formal change control and impact assessment.
• Ability to perform root cause analysis for hallucinations, inaccurate outputs, agent failures, latency issues, and unexpected model behaviour.
• Assess model behavior across datasets, edge cases, and failure scenarios to identify bias, drift, and instability.
• Understand ML fundamentals - strong grasp of classification, regression, clustering, anomaly detection, and model performance metrics (Precision, Recall, F1, AUC).
• Evaluate LLM and GenAI systems - experience measuring accuracy, completeness, relevance, groundedness, faithfulness, consistency, and task completion.
• Analyze drift and model performance - ability to identify and investigate data drift, concept drift, model drift, and performance degradation in production environments.
• Assess RAG-based solutions - understand retrieval pipelines and be able to evaluate retrieval quality, citation accuracy, and response grounding.
• Work with AI observability platforms - experience with AI monitoring, tracing, evaluation logging, dashboards, and observability tools such as Langfuse, LangSmith, Arize, WhyLabs, MLflow, or similar platforms.
• Use LangChain, LangGraph, and Langfuse for age…