Senior AI Tools Testing Engineer

Job Title: Senior / Lead AI Tools & Testing Engineer (Agentic AI & LLM Evaluation)

Location: Dallas, TX (Hybrid)

Duration: Long‑Term

Experience:10 + Years

We seek a senior/lead AI engineer with hands‑on experience building, orchestrating, monitoring, and evaluating production‑grade Generative AI and Agentic AI applications. The ideal candidate is expert in Amazon Bedrock, AgentCore, LangGraph, LangChain, RAG, LLM observability, and evaluation frameworks. This role focuses on creating reliable, scalable, and production‑ready AI applications, emphasizing agent orchestration, AI testing, LLM evaluation, and continuous quality monitoring.

The candidate will collaborate with engineering, QA, and platform teams to establish best practices for building, testing, monitoring, and improving AI‑powered applications.

Key Responsibilities

1. Agentic AI & GenAI Engineering

• Design, develop, and deploy production‑grade agentic and generative AI applications.
• Build and orchestrate AI agents using LangGraph, LangChain, Amazon Bedrock, and AgentCore.
• Implement multi‑agent workflows, tool calling, state management, memory, and orchestration.
• Create scalable LLM‑powered applications and reusable AI components.

2. RAG & LLM Application Development

• Design and implement Retrieval‑Augmented Generation (RAG) solutions.
• Develop pipelines for document ingestion, chunking, embedding, retrieval, reranking, and context management.
• Optimize retrieval accuracy, response quality, latency, and cost.
• Apply grounding, hallucination reduction, and context‑relevance techniques.

3. LLM Evaluation & AI Testing

• Build evaluation frameworks for LLM‑based applications and agentic workflows.
• Automate evaluation with DeepEval and other LLM evaluation tools.
• Use LLM‑as‑a‑Judge methods to assess response quality, relevance, accuracy, faithfulness, and reasoning.
• Create evaluation datasets, test scenarios, benchmarks, and regression‑testing pipelines.
• Evaluate agent behavior, tool execution, workflow accuracy, and response consistency.
• Identify failure patterns and continuously improve AI application quality.

4. AI Observability & Monitoring

• Implement tracing, monitoring, and observability with OpenTelemetry, LangSmith, and Langfuse.
• Track LLM requests, agent execution flows, tool calls, latency, token consumption, errors, and performance.
• Establish debugging mechanisms for complex multi‑step agentic workflows.
• Develop monitoring practices to detect performance degradation and operational issues.

5. AWS AI Platform & Production Engineering

• Deploy AI applications using Amazon Bedrock and Bedrock AgentCore.
• Integrate foundation models, agent frameworks, APIs, and enterprise data sources.
• Apply production‑readiness practices covering scalability, reliability, security, and performance.
• Support CI/CD integration and automated evaluation of AI applications.
• Optimize model selection, inference costs, latency, and resource utilization.

6. Technical Leadership & Collaboration

• Provide technical guidance to engineering and QA teams.
• Establish reusable frameworks, implementation patterns, and best practices.
• Collaborate with architecture, DevOps, QA, and engineering teams.
• Participate in design discussions, code reviews, proof‑of‑concept development, and implementation.
• Mentor engineers and help teams adopt effective AI development and testing practices.

Required Technical Skills

Agentic AI & Orchestration

• LangGraph
• LangChain
• Amazon Bedrock
• Amazon Bedrock AgentCore
• AI agents, multi‑agent orchestration, tool calling, workflow management

Generative AI & RAG

• Large Language Models (LLMs)
• Retrieval‑Augmented Generation (RAG)
• Embeddings and vector search
• Prompt engineering and context management
• LLM application development

LLM Evaluation & Testing

• DeepEval
• LLM‑as‑a‑Judge techniques
• Evaluation metrics and benchmarking
• Automated evaluation and regression testing
• Assessment of hallucination, faithfulness, relevance, and response quality

AI Observability & Monitoring

• OpenTelemetry
• LangSmith
• Langfuse
• LLM tracing, debugging, and performance monitoring

Production Engineering

• Deploying and operating production‑grade GenAI applications
• API integration and backend engineering
• CI/CD and automated testing integr…