Senior Product Manager (AI Observability)
- Source: Workday
- London Stock Exchange
- On-site
- London
- Full-Time
- Senior
Job Description
- As part of the LSEG AI, we are hiring a Senior Product Manager – AI Observability to own the strategy, design and rollout of telemetry systems that monitor, measure and analyse how AI models, MCPs, and AI-enabled features behave across all LSEG products
- This role is responsible for defining how we collect, structure, analyse and act on AI-specific telemetry signals—including prompt patterns, model performance, MCP call usage, latency, error conditions, cost metrics, user behaviour signals, and system reliability
- You will develop a telemetry foundation that supports:
- AI governance & risk
- Model performance tracking
- Cost efficiency
- User experience optimisation
- Operational reliability
- Auditability
- And the long-term evolution of our AI platform
- Telemetry & Observability Strategy:
- Define the end-to-end telemetry vision and roadmap for LLMs, MCPs, vector stores, embeddings, inference layers and AI-powered user experiences
- Establish a standardised telemetry schema for capturing prompts, tool calls, model responses, durations, errors, confidence signals, and quality indicators across business lines
- Partner with platform engineering to ensure instrumentation is consistent, scalable and compliant
- Signal Design & Data Architecture:
- Identify key signals required for:
- Quality & reliability
- Latency & throughput
- Cost tracking & optimisation
- MCP usage patterns
- User workflow insights
- Failure pattern detection
- Guardrail and safety event monitoring
- Define retention rules, PII considerations, anonymisation and usage policies in partnership with AI Governance and Compliance
- Platform & Tooling:
- Own requirements for dashboards, monitoring tools, model comparison views, anomaly detection alerts, and performance scorecards
- Design systems that allow product and engineering teams to self-serve insights about AI model behavior and MCP interactions
- Partner with engineering to build pipelines that support real-time and batch analytics
- Cross-Functional Collaboration:
- Collaborate with product owners across LSEG to instrument their AI features consistently
- Partner with AI Evaluation PM (previous role), Model Risk, GSSR, Legal and Compliance to align telemetry with governance frameworks
- Work closely with Engineering and SRE teams to drive observability improvements and reliability engineering for AI systems
- Optimisation & Insights:
- Identify cost inefficiencies across model and MCP usage, and drive recommendations to improve ROI
- Surface workflow-level insights on how customers use AI features—informing product roadmaps
- Partner with product leaders across divisions to understand how telemetry can improve customer experience, reliability and performance
- Governance & Standards:
- Define and maintain AI telemetry standards and best practices across all LSEG divisions
- Contribute to LSEG’s AI governance and Responsible AI initiatives by providing data-driven insights on model behavior and user impact
- Ensure telemetry systems support auditability, compliance and explainability requirements
- Measures of Success:
- Platform Reliability & Performance:
- Reduction in AI system error rates, latency spikes and operational incidents
- Stability and transparency of MCP usage across product lines
- Quality and coverage of telemetry signals across all AI-enabled products
- Insight Generation & Adoption:
- Adoption of telemetry dashboards and self-service tools across divisions
- Number of identified and resolved model or MCP reliability issues driven by telemetry
- Level of insight generated to influence product prioritisation and AI model choices
- Operational Efficiency & Cost:
- Improved cost transparency and cost-saving impact via telemetry-driven optimisation
- Reduction in unnecessary model invocation or inefficient tool usage
- Governance & Compliance:
- Contribution to Responsible AI frameworks through robust, auditable telemetry
- Improved explainability and traceability of AI features for internal/external stakeholders- Experience in product management with a strong foundation in observability, telemetry, data platforms, monitoring, or SRE / DevOps-driven products
- Experience working with senior engineering, data science, risk and governance stakeholders
- Ability to define data schemas, signal taxonomies, aggregation strategies and data contracts
- Strong analytical skills and ability to derive insights from large-scale system telemetry
- Understanding of LLMs, embeddings, vec…
Skills
- serverless
- workday
