Full-Time (GC/USC only) Role: Senior AI Evaluations Engineer (AWS)_ Remote

Job Title: Senior AI Evaluations Engineer (AWS)

Location: Remote

Job Type: Remote

We are seeking a Senior AI Evaluations Engineer to AWS practice and work embedded with AWS Professional Services (ProServe) teams, designing and operating the evaluation frameworks that determine whether agentic AI systems are accurate, safe and ready for production at large enterprise customers, primarily in financial services. You will define quality rubrics and scoring architectures, build LLM-as-judge and agent-as-judge pipelines, and run offline regression and online, live-traffic evaluation for multi-agent workflows on Amazon Bedrock. The ideal candidate combines expert generative AI engineering skills with a rigorous, metrics-driven approach to quality.

Required Qualifications

• GenAI Expertise: Expert-level (Level 400) generative AI experience with advanced Python.

• Production Systems: Proven experience building and running production-grade systems.

• Agent Evaluation: Hands-on experience with offline/on-demand regression testing and online/continuous live-traffic scoring of agents.

• LLM-as-Judge: Proficiency in rubric design and scoring at session, response and tool-call granularity.

• Evaluation Metrics: Task completion, tool-call accuracy, trajectory quality, groundedness and safety.

• Agentic Frameworks: Hands-on experience with Strands, LangGraph, CrewAI or similar.

• AWS Stack: Amazon Bedrock (models, Guardrails) and CloudWatch, plus Lambda, S3 and IAM fundamentals.

• AWS Certification: Active AWS Certified Generative AI Developer – Professional or AWS Certified Machine Learning Engineer – Associate (required).

Preferred Qualifications

• Bedrock AgentCore Evaluations: On-demand SDK runner, online monitoring and A/B testing.

• Advanced Evaluation Methods: Agent-as-a-judge frameworks and human-in-the-loop (HITL) evaluation pipelines.

• Benchmarking: Benchmarking frameworks (tau-bench, SWE-bench, GAIA) and custom dataset construction.

• RAG Evaluation: RAG quality metrics such as faithfulness, context precision and recall.

• CI/CD & Observability: CI/CD integration for evaluation regression, tracing and structured logging.

• Legacy Modernization: Familiarity with Java/Spring, Sybase, DB2 or COBOL as source context for agentic modernization or integration pattern design.

• Financial Services: Financial services or regulated-industry experience with compliance-aware delivery (trade surveillance, data controls, audit requirements).

• Scale: Large-scale portfolio delivery: reusable patterns or components applied across 50+ applications, including wave planning and throughput measurement.

• Consulting: Prior experience with AWS ProServe or in a consulting delivery role.

Thanks and Regards

NANI

Email: [email protected]