AI Solutions and Platforms Engineer

Overview Agentic AI Observability Senior Engineer is responsible for deploying, integrating, and operating a scaled Agentic AI observability platform across both internal and external agent frameworks. This role focuses on production-ready instrumentation and telemetry pipelines that provide end-to-end visibility across multi-step agent workflows—including planner/executor loops, tool/function calls, RAG retrieval, and memory/state—ensuring reliability, safety, performance, and cost governance at scale

Responsibilities

• Agentic AI Observability at Scale (0%)
• Platform Deployment & Operations (Agentic AI Observability at Scale)
• Deploy and run the Agentic AI observability platform across dev/uat/prod with HA, resiliency, and controlled rollouts
• Implement release automation (CI/CD), canary deployments, rollback strategies, and configuration management for platform components
• Own operational readiness: on-call runbooks, incident response, and production support for agent observability services
• End-to-End Agent Workflow Tracing (Planner → Tools → Retrieval → Response)
• Implement distributed tracing for full agent execution graphs, including correlation across: prompts, intermediate reasoning steps (where permitted), tool calls, external APIs, retrieval queries, and final responses
• Enforce consistent trace context propagation, correlation IDs, and semantic conventions across agent services
• Build instrumentation patterns to represent agent flows as spans (e.g., plan span, tool span, retrieval span, memory span, response span)
• Agent Framework Integrations & Standardized Instrumentation
• Deploy and maintain integrations for internal agent frameworks and external ecosystems such as Crew.ai, LangChain, Semantic Kernel, AutoGen, and custom orchestrators
• Create reusable SDKs/middleware/sidecar patterns for teams to instrument agents with minimal effort
• Define and implement tagging standards for: agent name/version, tool name, model provider, prompt template ID, retrieval source, tenant/app, and environment
• Agentic AI Telemetry Pipelines & AI-Specific Signals
• Build scalable pipelines for agent telemetry (logs/metrics/traces) using OpenTelemetry and platform observability tooling
• Capture AI-specific metrics including: token usage, cost per task, tool-call latency, retrieval latency, grounding score proxies, error rates, and agent loop iterations

• Implement sampling and redaction strategies for sensitive agent payloads (prompts, responses, retrieved content) aligned to governance requirements

• Collaboration with Teams (10%)

• Collaborate with transformation teams and business stakeholders to understand requirements and tailor AI agents to specific domains.

• Work closely with AI platform teams to build scalable and cross-domain AI agents while ensuring end-to-end observability.

• Integration & Deployment (10%)

• Build and maintain CI/CD pipelines for agent services and operations center components, including automated testing and deployment
• Automate onboarding for new agent use cases (templates, scaffolding, configuration checks)

• Drive best practices for secure, scalable, and cost-effective agent deployments

• Continuous Learning (10%)

• Stay updated with the latest advancements in AI and machine learning technologies and integrate these into existing or new AI agents.
• Conduct thorough testing and validation to ensure the reliability and accuracy of AI agents and solutions.

Qualifications

• Education: Bachelor’s or Masters in Computer Science, AI/ML, Data Science, or a related field.
• Experience: 4-8+ years of software engineering experience; 2-3+ years building and observe AI/ML or GenAI applications preferred

• Required Expertise:

• Strong hands-on experience deploying observability solutions (Prometheus/Grafana/Elastic/Splunk/Datadog or equivalent)
• Deep working knowledge of OpenTelemetry instrumentation and telemetry pipeline operations
• Experience observing agentic AI systems: tool/function calls, orchestration, routing, memory/state, and RAG pipelines
• Familiarity with Crew.ai, LangChain, Semantic Kernel, AutoGen, or similar agent frameworks
• Experience with evaluation/quality monitoring and safe logging strategies for LLM systems
• FinOps experience for tracking token and GPU spend, chargeback/showback, and cost anomaly detection
• Experience implementing data governance controls for AI telemetry (PII redaction, retention, auditability)
• Strong Kubernete…