Senior AI QA Engineer

Test Engineer, AI Product Quality (US Healthcare Payer) Role Summary: We are seeking a highly motivated and skilled Test Engineer with 3-6 years of experience to join our team, focusing on ensuring the quality, reliability, and performance of our innovative native AI-driven products within the US healthcare payer platform. This role requires a strong foundation in software testing principles, a keen eye for detail, and a proactive mindset in approaching the unique challenges of AI/ML model validation and integration testing. You will collaborate closely with product managers, data scientists, and developers to deliver high-quality, impactful AI solutions to our members and internal stakeholders. • AI Product Quality Assurance: Specializes in the comprehensive testing and validation of AI/ML-powered applications and components, ensuring they meet functional, non-functional, and ethical requirements within the sensitive US healthcare payer context. • Healthcare Payer Domain Expertise: Leverages a strong understanding of healthcare payer operations to design relevant test scenarios and validate AI solutions’ impact on claims, member services, provider networks, and regulatory compliance. • Proactive Test Strategy & Design: Contributes to the development of robust test strategies, designs detailed test plans and cases, and proactively identifies potential failure points and data quality issues critical for AI/ML performance. • Automation & Efficiency: Drives the adoption and implementation of test automation frameworks and scripts, particularly for data validation, API testing, and AI model inference testing, to enhance efficiency and coverage. Builds reusable AI evaluation harnesses, synthetic test datasets, regression suites for prompts/agents, CI/CD integrated AI evaluation checks. • Collaborative & Communicative: Acts as a vital link between technical development teams (Data Scientists, Developers) and business stakeholders, clearly communicating test results, defects, and risks related to AI product quality. Partners with SMEs, Product, Data Science, Engineering, Architecture, AI Governance and Operations to review AI outputs, knowledge gaps, model drift and release readiness Essential Responsibilities: • Test Strategy & Planning: Work closely with Product Managers, Data Scientists, and Developers to understand AI product requirements, model functionalities, and system architecture. Develop comprehensive test strategies and detailed test plans for AI-driven features and end-to-end solutions. • Test Case Design & Execution: Design, develop, and execute a wide range of test cases, including functional, integration, regression, performance, and user acceptance tests, specifically for AI/ML models and their integration into existing payer systems. • AI/ML Model Validation Support: • Contribute to the validation of AI/ML model outputs, focusing on data quality, data drift, bias detection, and performance against defined metrics. • Design tests to evaluate model behavior under various data conditions and edge cases. • Work with Data Scientists to understand model limitations and interpretability, translating these into test scenarios. • AI Solution Quality & Governance Testing: Design and execute end-to-end testing of AI-driven applications, workflows, and data processes to ensure accuracy, reliability, usability, and compliance. Validate information quality, system behavior, approval workflows, auditability, and operational performance while ensuring seamless integration across business and technology components. • Test Automation Development: Design, develop, and maintain automated test scripts and frameworks for API testing, UI testing, data validation, and automated model inference testing using relevant tools and programming languages (e.g., Python, Java, Selenium, Postman). • Defect Management & Reporting: Identify, document, and track defects with clear, concise, and reproducible steps. Collaborate with development teams to ensure timely resolution and retesting. Provide regular, clear reports on test progress, defect status, and overall product quality to project stakeholders. • Data Validation & Quality: Focus on testing data pipelines and data transformations that feed AI models, ensuring data accuracy, completeness, and consistency. (Optional but highly valued: Design and execute ETL tests). • Performance & Scalability Testing: Participate in performance and scalability testing for AI applications, ensuring …