Senior Software Engineer, Test

What you will work on

Our Vertical Search team’s mission is to deliver the best TV search experience: fast, comprehensive, and reliably relevant. For REA Vertical Search, that means helping users discover the right content across typed, voice, and assistant-driven experiences while supporting a growing set of subscription services and Roku Channel discovery surfaces. We partner with search engineers, ML engineers, product managers, and data partners to define quality strategy for relevance-sensitive experiences, build scalable automation and evaluation frameworks, and apply agentic automation to keep pace with expanding catalog breadth and use cases.

Role Overview

We are looking for a Lead, Search Relevance Quality to work closely with the Roku Vertical Search Team to define, measure, and improve Search and Assistant relevance quality. This person will lead quality strategy for relevance-sensitive experiences, build scalable automation and evaluation frameworks, and apply agentic automation to keep pace with expanding catalog breadth and use cases. The role requires balancing automated quality systems with pragmatic human-in-the-loop review where needed until confidence and coverage are high. Quality for Vertical Search will be measured with clear, trusted metrics rather than inferred only from downstream engagement signals. Search and Assistant launches will have reusable relevance-evaluation coverage, stronger release confidence, and faster debugging loops. The team can scale quality across a growing number of subscription services and Roku Channel experiences without linear growth in manual effort.

How AI is used at Roku

At Roku, AI agents do most of the keystroke-level coding. Engineers act as the technical leads for those agents, setting context, planning work, verifying outputs, and recovering when the agent goes off course. If you are excited about working this way, we want to talk to you.

At Roku, we’re embracing AI as a powerful tool to amplify human creativity, accelerate innovation, and deliver better results for our customers. Across teams, we’re exploring how AI can enhance our work and we’re only looking for people who are curious, adaptable, and excited to grow alongside these technologies.

Responsibilities

• Define the quality strategy for Vertical Search and Assistant relevance across Roku Premium subscription services, Roku Channel search journeys, and related discovery experiences
• Build scalable frameworks to measure search quality across query understanding, retrieval, ranking, result presentation, and assistant response quality
• Create automated evaluation pipelines for offline, pre-launch, and continuous post-launch quality monitoring using modern test infrastructure and agentic workflows
• Partner with search engineers, ML engineers, product managers, and data partners to define KPI hypotheses, guardrails, and learning plans for quality-related launches and experiments
• Develop quality signals and gold-standard datasets for vertical search, including subscription-service discovery, entity matching, ranking relevance, and assistant answer quality
• Use agentic automation to generate, maintain, and expand test coverage and evaluation assets while ensuring governance, traceability, and reliability of the resulting workflows
• Introduce the right mix of manual review and automation for ambiguous or high-impact experiences until the automated system is proven and trusted
• Drive root-cause analysis for relevance regressions and quality escapes, and convert findings into durable frameworks, dashboards, and release gates
• Lead end-to-end test and quality architecture for relevance-heavy search and assistant systems
• Design evaluation methodologies for search result quality, assistant response quality, and discovery experience consistency
• Build APIs, tools, datasets, and automation harnesses for offline and online quality validation
• Establish CI/CD quality gates for ranking, retrieval, and assistant-related regressions
• Partner with engineering to instrument systems for observability, diagnosis, and actionable quality telemetry
• Translate vague quality concerns into measurable test plans, labeled datasets, and scalable automation workflows
• Mentor engineers on quality engineering best practices and raise the technical bar for automation design, signal quality, and testability

Experience for success

• 8+ years in relevance quality engineering, or software engineering roles with…