Senior Test Architect – AI Platform Test Strategy, Technologies and Execution IRC306523

Description AI first platform development

Requirements About the Role

We are seeking a highly experienced Senior Test Architect/Consultant to lead the quality engineering strategy for our next-generation applications, with a heavy emphasis on intelligent agents, AI-driven workflows, and complex data ecosystems. In this role, you will bridge the gap between traditional quality assurance (automation, performance, security) and cutting-edge AI validation.

You will be responsible for designing comprehensive test strategies that address both functional and non-functional requirements (NFRs). Your mandate includes evaluating emerging test tools, building robust test harnesses, and ensuring the contextual accuracy, data security, and operational reliability of our AI agents and core platforms.

Key Responsibilities

Test Strategy & Planning: Design and author end-to-end test strategies and test plans covering all architectural concerns (functional, non-functional, security, and performance) for complex systems, specifically focusing on autonomous agents and AI-driven platforms.

AI Agent & Outcome Validation: Develop methodologies to test agentic behaviors, ensuring the accuracy of outcomes, contextual relevance of responses, and proper handling of edge cases/hallucinations.

Test Data Engineering: Architect strategies for test data creation, management, and synthetic data generation. Ensure rigorous data security, privacy compliance, and strict multi-tenant data separation within testing environments and LLM contexts.

Tool Evaluation & Harness Development: Continuously evaluate and onboard modern testing tools and frameworks. Build custom test harnesses and sandboxes specifically designed to interact with and evaluate intelligent agents and APIs.

Automation, Performance, & Security: Oversee the architecture of traditional test automation pipelines. Drive performance testing (load, stress, scalability) and security testing (vulnerability scanning, penetration testing integrations) to ensure enterprise-grade reliability.

Cross-Functional Leadership: Act as a consultant and technical leader, collaborating closely with developers, data scientists, product managers, and security teams to embed quality engineering principles early in the software development lifecycle (Shift-Left).

Required Qualifications Experience & Background:

10+ years of progressive experience in Quality Engineering, Test Architecture, or QA Consulting.

Proven track record in designing and implementing enterprise-level test strategies and master test plans from scratch.

Recent hands-on experience testing AI models, LLMs, or intelligent autonomous agents.

Technical Competencies:

AI & Agent Testing: Deep understanding of testing methodologies for contextual accuracy, prompt injection vulnerabilities, RAG (Retrieval-Augmented Generation) architectures, and non-deterministic outcomes.

Test Automation: Expert-level proficiency with modern test automation frameworks (e.g., Playwright, Cypress, Selenium) and API testing tools (Postman, REST Assured).

Performance & Security: Strong background in performance engineering (JMeter, Gatling, k6) and security testing concepts (OWASP Top 10, SAST/DAST, data masking).

Test Data Management: Expertise in formulating robust test data strategies, including synthetic data generation and enforcing strict data separation/privacy protocols.

Architecture & CI/CD: Ability to architect custom test harnesses and integrate automated quality gates into CI/CD pipelines (GitHub Actions, Jenkins, GitLab CI).

Preferred Skills Familiarity with LLM evaluation frameworks (e.g., LangSmith, TruLens, Ragas, promptfoo).

Experience in cloud-native environments (AWS, Azure, or GCP) and containerized applications (Docker, Kubernetes).

Strong programming skills in Python, Java, or Node.js.

Job responsibilities

**About The Role** We are seeking a highly experienced Senior Test Architect/Consultant to lead the quality engineering strategy for our next-generation applications, with a heavy emphasis on intelligent agents, AI-driven workflows, and complex data ecosystems. In this role, you will bridge the gap between traditional quality assurance (automation, performance, security) and cutting-edge AI validation.

You will be responsible for designing comprehensive test strategies that…