Shopify AI LLM QA Engineer (PLAYWRIGHT)

About The Opportunity A fast-scaling AI-powered e-commerce enablement platform, we build intelligent, LLM-driven QA systems that validate Shopify storefronts, automate regression testing, and ensure flawless generative experiences at scale. Our AI QA engineers work at the intersection ofShopify ecosystems, large language models, and test automation to ship reliable, customer-centric digital experiences across global merchant bases.

Role & Responsibilities

• Design and execute AI-powered test cases for Shopify themes, LLM-generated content, and AI-driven checkout flows.
• Build and maintain automated test suites using Playwright/Selenium integrated with LLM validation layers for dynamic content.
• Collaborate with AI/ML engineers to define test heuristics for LLM outputs—accuracy, hallucination detection, prompt fidelity, and tone alignment.
• Leverage synthetic test data and prompt engineering to simulate edge-case user journeys across Shopify stores.
• Integrate QA pipelines with CI/CD (GitHub Actions, Jenkins) to enable shift-left testing for AI-enhanced Shopify features.
• Monitor test coverage, flakiness, and LLM drift; recommend improvements to test architecture and validation logic.

Skills & Qualifications

• Must-Have
• Playwright
• Selenium
• Shopify Liquid
• LLM Validation
• CI/CD Pipelines
• Python
• Test Automation Frameworks
• GitHub Actions
• Preferred
• LangChain
• OpenAI API
• Shopify Admin API

Benefits & Culture Highlights

• 100% remote, flexible hours with async-first collaboration.
• Work directly on AI testing infrastructure for global Shopify merchants.
• Access to cutting-edge LLM tooling and testing frameworks—learn while you build.

Skills: engineers,github,selenium,python,automation,api,validation,playwright,shopify,javascript,testing