AI Automation Test Engineer

ABOUT THE ROLE

This is one of the most technically novel roles in the team. You will build a pipeline that takes a Jira ticket or product specification and produces a first-draft Playwright test using Claude, and an autonomous agent that monitors the live test suite, detects flakiness, diagnoses root causes, and proposes or applies fixes. The quality of your agent layer will directly determine how fast the rest of the team can move.

WHAT YOU’‘LL OWN

• Design and build the Claude-powered test generation pipeline: product spec or Jira ticket in, first-draft Playwright test out
• Build the test maintenance agent: continuously monitors the suite, detects flaky or failing tests, diagnoses causes, and proposes or auto-applies fixes with human-in-the-loop review
• Own the prompt engineering layer: maintain the prompt library, build an evaluation harness, and score generated test quality systematically
• Define the acceptance workflow: which AI-generated tests require engineer review before merge, and which meet the bar for automatic acceptance
• Instrument the agent layer: track cost per test generated, acceptance rate, time-to-fix for maintenance interventions, and overall flakiness reduction
• Partner closely with automation engineers to understand product domain nuance and improve generation quality for complex business logic
• Stay current with fast-moving LLM tooling and evaluate new approaches that could improve speed or quality

MUST HAVE

• Production experience with the Anthropic Claude API or OpenAI API specifically tool use, multi-step agent patterns, and structured output
• Ability to evaluate LLM output quality rigorously: you build evals, not just vibes checks
• Experience designing agentic systems that handle failure gracefully: retries, fallbacks, and human escalation paths
• Comfort with ambiguity and incomplete requirements you are building something that does not exist yet in most organizations

NICE TO HAVE

• Playwright or comparable E2E test framework experience you don’‘t need to be an expert but you must be able to read and reason about test code fluently
• Experience with test infrastructure: fixture factories, seed scripts, environment isolation
• Prior work building agents with persistent memory, multi-tool orchestration, or complex retry logic