AI/ML Engineer (Contract)

About Us

Deccan AI Experts is a pioneering AI company founded by IIT Bombay and IIM Ahmedabad alumni, with a strong founding team from IITs, NITs, and BITS. We specialize in high-quality human-curated data, AI-first operations, and advanced AI evaluation systems. Our global network of financial crime, compliance, risk, and AI experts helps train and evaluate next-generation AI models through expert AML knowledge, transaction monitoring expertise, and regulatory compliance experience.

Role Overview

We are looking for an AI/ML Engineer to work primarily on managing and evaluating existing AI/ML systems, analyzing AI agents, identifying model failures, and improving overall system performance .

The role requires a strong understanding of LLM/AI fundamentals, model evaluation, AI agent architecture, RAG, model improvement/fine-tuning, and production ML systems .

This is a hands-on role for someone who can not only evaluate AI systems but also understand why models or agents fail and identify practical ways to improve their performance .

Key Responsibilities:

• 2+ years of experience in AI/ML, Manage and monitor existing AI/ML and GenAI systems .
• Conduct structured LLM and model evaluations using defined benchmarks and evaluation criteria.
• Analyze AI/agent outputs and identify accuracy, reasoning, retrieval, tool-use, and behavioral failures .
• Analyze AI agent architecture and workflows to identify failure points and performance bottlenecks.
• Evaluate and improve RAG pipelines , including retrieval quality and response generation.
• Identify opportunities for model improvement, prompt optimization, and fine-tuning .
• Analyze evaluation results and translate findings into actionable improvements.
• Support testing and validation of new model/agent versions.
• Work with production ML systems and help identify reliability, latency, scalability, and performance issues .
• Collaborate with engineering and AI teams to implement and validate improvements.
• Document evaluation results, failure patterns, root causes, and recommended solutions.

Must-Have Skills:

• Strong understanding of LLM and AI fundamentals .
• Experience working with production ML/AI systems .
• Hands-on experience with model and LLM evaluation .
• Understanding of AI agent architecture, agentic workflows, and failure analysis .
• Experience with RAG (Retrieval-Augmented Generation) systems.
• Understanding of model improvement, fine-tuning, and prompt optimization .
• Strong Python programming and analytical skills.
• Ability to investigate model failures and perform root-cause analysis .
• Ability to work with evaluation datasets, benchmarks, metrics, and structured testing.

Who Would Be a Good Fit?

We’re looking for someone who has worked beyond simply building AI models and has hands-on experience with evaluating, debugging, analyzing, and improving AI/LLM systems .

Candidates with experience in LLM evaluation + AI agents + RAG + model improvement + production ML systems would be particularly relevant..

Why Join Us?

• Work on real-world LLM and agentic AI systems .
• Gain hands-on exposure to AI evaluation and model improvement .
• Work on challenging problems involving AI agent failures, RAG, and model performance .
• Opportunity to contribute to production-oriented AI projects.