Senior AI/ML Engineer

Senior AI/ML Engineer — Clinical AI Quality & LLMOps

Role Purpose

We are looking for a Senior AI/ML Engineer — Clinical AI Quality & LLMOps to establish dedicated ownership of the quality, reliability, evaluation, optimization, and operational maturity of DoctusMind’s clinical AI capabilities.

• This role will sit at the intersection of AI/ML Engineering, Clinical AI Quality, LLMOps, AI Evaluation, Prompt Engineering, and Production AI Operations .
• The person will work closely with Product, Engineering, QA, Clinical, and other cross-functional teams to ensure that DoctusMind’s AI systems are clinically reliable, measurable, observable, cost-efficient, and production-ready.
• The role is not limited to prompt engineering. The expectation is that this person will build the systems, processes, evaluation frameworks, and engineering practices required to operate clinical AI reliably at scale .
• The initial focus will include AI stabilization, clinical AI quality, Chronic Care, and longitudinal intelligence , while establishing the foundation for future specialty, multimodal, agentic, retrieval, memory, and other clinical AI capabilities .

What Success Looks Like

The person will be accountable for five executive-level outcomes during the first 3–6 months.

1. AI Quality & Stabilization
2. Production AI Release Governance
3. AI Observability & Operational Performance
4. Clinical AI Capability Production Readiness

Core Responsibilities

1. Prompt Engineering & Management

• Design, optimize, and maintain production-grade prompts.
• Establish prompt versioning and lifecycle management.
• Build prompt regression suites.
• Analyze prompt-related failures.
• Optimize prompts for quality, consistency, latency, and cost.
• Maintain traceability between prompt changes and evaluation outcomes.

1. AI Evaluation & Clinical Quality

• Build AI evaluation frameworks for accuracy, relevance, consistency, safety, and reliability.
• Develop golden datasets and clinical test cases.
• Create automated evaluation workflows.
• Establish severity classification for AI failures.
• Perform systematic failure analysis and root-cause analysis.
• Monitor clinical AI quality continuously.
• Partner with Clinical and QA teams to validate high-risk workflows.

1. AI Regression & Release Management

• Establish AI regression testing as part of the release lifecycle.
• Define evaluation gates for prompts, models, retrieval, and configurations.
• Ensure AI changes cannot reach production without appropriate validation.
• Maintain AI version traceability.
• Establish rollback and post-release validation processes.

1. LLMOps & AI Observability

• Build and maintain AI-specific monitoring.
• Track quality, latency, failures, token consumption, and cost.
• Develop dashboards and operational reporting.
• Establish alerts for significant deviations.
• Support AI production incident investigation.
• Identify AI quality and performance drift.

1. AI Experimentation & A/B Testing

• Maintain environments for prompt and model experimentation.
• Conduct controlled A/B tests.
• Compare prompts, models, retrieval strategies, and AI workflows.
• Define experiment methodology and success criteria.
• Document findings and recommendations before production adoption.

1. Model Benchmarking & Selection

• Benchmark existing and emerging models against DoctusMind use cases.
• Evaluate models based on:
• Clinical quality
• Reliability
• Safety
• Latency
• Cost
• Scalability
• Recommend model selection and routing strategies.
• Evaluate fallback strategies for production resilience.

1. Cost & Token Optimization

• Analyze LLM/API consumption.
• Optimize prompt and context length.
• Evaluate caching opportunities.
• Optimize model selection and routing.
• Identify unnecessary or inefficient AI calls.
• Establish ongoing AI cost-performance monitoring.

1. Context Engineering, Retrieval & Memory

• Optimize how patient information, conversations, care plans, and clinical context are supplied to AI systems.
• Improve retrieval of relevant patient history and clinical context.
• Support RAG, embeddings, vector databases, and retrieval pipelines.
• Contribute to cross-session memory and longitudinal intelligence.
• Evaluate context quality and relevance as part of AI evaluation.

1. Clinical AI Capability Productionization

• Evaluate new AI capabilities before production.
• Support Chronic Care and longitudinal intell…