QA Engineering Manager AI Systems

QA Engineering Manager AI Systems

Healthcare AI • Playwright Automation • LLM Evaluation • Clinical Safety Validation

ABOUT THE ROLE

We’re building the next generation of AI Quality Engineering for healthcare.

This is a capability-building role, not a maintenance role. Rather than maintaining an existing QA organization, you’ll define, build, and lead a Quality Engineering function for AI-powered healthcare products from the ground up — establishing the standards, processes, automation, and AI validation practices that will support our platforms as they scale over the next several years.

This is a player–coach role. You are expected to provide technical leadership while remaining hands-on with automation, AI validation, and quality engineering activities when needed. We are open to candidates currently titled QA Manager, QA Architect, or Senior QA Engineer — what matters is the ability to build and lead this function.

THE MISSION

We are not looking for someone to manage a test backlog. We are looking for someone to build and own an AI Quality Engineering capability that will grow with the organization over the next 2–3 years.

Traditional software QA remains essential. But our long-term differentiator is AI quality — and this role will own it end to end.

CORE OWNERSHIP

• End-to-end software quality across web applications and integrated platform releases
• AI evaluation and validation — relevancy, faithfulness, contextual accuracy, hallucination detection, bias
• Clinical reasoning and safety validation for AI-generated treatment suggestions and clinical decision support
• Hallucination and guardrail testing for AI-driven clinical workflows
• AI regression testing as models, prompts, and workflows evolve
• Automation engineering using Playwright and JavaScript/TypeScript with full CI/CD integration
• Release readiness governance — QA release readiness assessments, risk registers, and sign-off processes
• Design and evolution of scalable AI evaluation frameworks and quality processes
• Quality KPIs, engineering dashboards, defect leakage trends, automation maturity, and executive reporting
• AI quality governance, evaluation standards, and engineering best practices

KEY RESPONSIBILITIES

Organizational leadership

• Build and scale the AI Quality Engineering function — define team structure, career paths, and engineering standards
• Hire, mentor, and develop QA engineers; lead capacity planning and resource allocation
• Establish engineering hiring standards, interview frameworks, and onboarding processes for the QA organization
• Drive consistent quality standards and governance across internal teams, client teams, and external delivery partners
• Represent the organization during client discussions, executive reviews, cross-vendor initiatives, technical escalations, and release governance meetings
• Provide independent quality recommendations and risk assessments to internal leadership and clients, even under demanding delivery timelines

Strategic responsibilities

• Define and execute a multi-year AI Quality Engineering roadmap aligned with product strategy, engineering maturity, and business growth
• Recommend tooling, frameworks, quality standards, and engineering practices to continuously improve AI Quality Engineering
• Partner with Engineering and Product leadership on quality strategy
• Evaluate and introduce emerging AI testing methodologies and tools
• Drive continuous innovation in AI validation practices
• Introduce modern quality engineering practices across the organization

AI and clinical quality engineering

• Design and implement AI evaluation frameworks using DeepEval, RAGAS, LangSmith, Promptfoo, or equivalent
• Validate GenAI/LLM-based clinical features including treatment guardrails, medication recommendations, and clinical task automation
• Develop AI regression test suites that detect quality degradation as models, prompts, and data evolve
• Define clinical safety testing protocols in collaboration with clinical SMEs and product stakeholders
• Build evaluation pipelines that scale across AI model updates and workflow changes

Automation and engineering

• Lead automation initiatives using Playwright and JavaScript/TypeScript for UI, API, and end-to-end testing
• Establish and maintain CI/CD-integrated automation pipelines supporting continuous quality validation
• Develop automation strategies supporting platform-specific and cross-platform testing
• Define and track automation c…