Software Development Engineer in Test - AI

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Software Development Engineer in Test – AI based in India. We are looking for a forward-thinking Software Development Engineer in Test specializing in AI to drive quality, reliability, and innovation across AI-powered applications and intelligent systems. In this role, you will go beyond traditional software testing by designing advanced evaluation frameworks for LLMs, AI agents, copilots, and RAG-based applications. You will combine strong automation engineering skills with hands-on AI expertise to validate complex workflows, improve system performance, and ensure safe and reliable AI outputs. Working closely with Engineering, AI/ML, Platform, Product, and DevOps teams, you will help integrate intelligent testing solutions throughout the software development lifecycle. You will also explore emerging AI frameworks, agentic architectures, and automation technologies to accelerate engineering productivity. This is an exciting opportunity to shape next-generation quality engineering practices in a collaborative, innovation-driven, and globally distributed environment.

Accountabilities

• AI/LLM Quality Engineering: Design, implement, and optimize testing strategies for AI-powered applications, chatbots, copilots, Retrieval-Augmented Generation (RAG) pipelines, and autonomous AI agents.
• AI Evaluation Frameworks: Develop and maintain AI evaluation (Evals) frameworks to assess hallucinations, groundedness, toxicity, bias, response quality, contextual relevance, prompt regressions, output consistency, and safety guardrails.
• Automated AI Validation: Build automated validation pipelines for LLM-generated outputs using structured assertions, semantic scoring, embeddings, and evaluation metrics. Create synthetic and benchmark datasets to validate different AI scenarios.
• Agentic Systems Testing: Validate multi-agent orchestration workflows, agent planning and execution, memory retention, context management, tool calling, function calling, and structured outputs. Test MCP (Model Context Protocol) integrations and interoperability.
• Prompt and Context Engineering: Contribute to prompt engineering, context engineering, and prompt optimization strategies to improve AI response quality, reliability, and consistency.
• Performance and Cost Optimization: Evaluate AI application latency, token consumption, throughput, scalability, and model usage costs to ensure efficient system performance.
• Automation Engineering: Develop and maintain scalable UI and API automation frameworks using Python, with tools such as Robot Framework, Playwright, Selenium, and Cypress. Build reusable testing libraries, utilities, and AI-assisted testing workflows.
• API and Application Testing: Automate testing across REST APIs, GraphQL APIs, event-driven systems, streaming services, webhooks, microservices, and micro-frontend architectures.
• RAG and AI Platform Validation: Validate RAG architectures, including document chunking strategies, embedding quality, retrieval accuracy, reranking, and vector search relevance. Work with vector databases such as Pinecone, Weaviate, ChromaDB, Qdrant, and FAISS.
• AI Agent Frameworks: Build, test, and validate AI agents using frameworks such as LangChain, LangGraph, CrewAI, AutoGen, Semantic Kernel, and OpenAI SDKs.
• DevOps and CI/CD Integration: Integrate automated testing and quality gates into CI/CD pipelines using GitHub Actions, Jenkins, GitLab CI/CD, or Azure DevOps.
• Cloud and Infrastructure Testing: Work with Docker, Kubernetes, and cloud environments across AWS, Azure, OCI, and GCP to validate application reliability, scalability, and deployment quality.
• Observability and Reliability: Monitor AI application behavior using logging, tracing, and observability platforms. Validate distributed systems reliability, identify performance bottlenecks, and contribute to shift-left quality engineering practices.
• Cross-Functional Collaboration: Partner with engineering, AI/ML, product, platform, and DevOps teams to embed quality throughout the development lifecycle and continuously improve testing standards.

Requirements

• Bachelor’s or Master’s degree in Computer Science, Software Engineering, Artificial Intelligence, Machine Learning, or a related field. Equivalent practical experience and demonstrable technical projects are also highly valued.
• Strong progr…