Senior AI Engineer - Observability

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior AI Engineer - Observability based in United States. This is a hands-on senior engineering opportunity focused on making AI-powered products reliable, measurable, and production-ready.

You’ll build and improve LLM, RAG, semantic search, and agentic experiences while helping define how AI quality is measured.

A major focus will be designing evaluation pipelines, golden datasets, regression testing, and meaningful quality gates.

You’ll also own observability across AI systems, tracking quality, latency, cost, reliability, safety, and real-world user feedback.

Working closely with engineering, product, QA, security, and business teams, you’ll turn production signals into actionable improvements.

You’ll help establish reusable standards, tooling, and engineering practices that enable teams to ship AI faster and with greater confidence.

This role is ideal for an experienced software engineer who combines strong technical depth with product judgment and a rigorous quality mindset.

Accountabilities

• Partner with product engineering teams to design, build, and enhance AI-powered features using LLMs, RAG, semantic search, and agentic workflows.
• Contribute directly to production codebases using Python, C#/.NET, or related technologies, while improving prompts, retrieval, context assembly, ranking, grounding, and response-generation patterns.
• Evaluate architecture and technology trade-offs across AI quality, latency, cost, privacy, maintainability, and operational reliability.
• Design and implement AI evaluation pipelines, scoring methodologies, regression frameworks, and quality gates that establish whether features are ready for production.
• Build and maintain versioned golden datasets covering real-world use cases, edge cases, failure modes, and customer-critical workflows.
• Apply LLM-as-judge, heuristic, human-feedback, and task-specific evaluation approaches, while assessing whether evaluation methods accurately measure meaningful outcomes.
• Instrument LLM interactions, RAG pipelines, tool calls, and agent workflows using observability platforms and establish dashboards, alerts, and production monitoring.
• Track and analyze latency, token usage, cost, retrieval quality, groundedness, safety signals, failure modes, and user feedback to identify regressions and improvement opportunities.
• Develop reusable libraries, SDKs, templates, documentation, and reference implementations for AI instrumentation, evaluation, tracing, cost reporting, and incident response.
• Coach product teams on AI quality, observability, evaluation design, and reliable release practices, helping establish consistent standards across product areas.
• Support responsible AI delivery by ensuring appropriate handling of sensitive data and alignment with security, privacy, SOC 2, ISO 27001, and data-residency requirements.
• Contribute to model and provider evaluations, migrations, fallback strategies, rollout plans, and ongoing optimization of AI-powered systems.

Requirements

• 5+ years of professional software engineering experience building and supporting production systems.
• Hands-on experience developing or operating LLM-powered features, RAG systems, AI workflows, or comparable AI applications.
• Strong programming capabilities in Python, C#/.NET, or both, with the ability to contribute effectively to production engineering environments.
• Practical understanding of prompts, embeddings, vector search, retrieval quality, orchestration patterns, grounding, and modern LLM application architecture.
• Demonstrated experience designing evaluations, quality metrics, regression frameworks, or benchmarking approaches, with strong judgment around whether an evaluation measures what truly matters.
• Strong observability fundamentals covering tracing, logging, metrics, alerting, production debugging, and analysis of system behavior.
• Experience with CI/CD, Git-based workflows, cloud environments, and production release practices; experience with Azure DevOps is advantageous.
• Familiarity with AI-assisted development tools such as Claude Code, PlayerZero, or comparable technologies.
• Experience with AI observability and LLMOps platforms such as Arize, Langfuse, LangSmith, W&B, Humanloop, or Helicone is preferred.
• Knowledge of OpenTelemetry, Azure Monitor,…