Director, Engineering AI Evaluation & Quality

FactSet creates flexible, open data and software solutions for over 200,000 investment professionals worldwide, providing instant access to financial data and analytics that investors use to make crucial decisions.

At FactSet, our values are the foundation of everything we do. They express how we act and operate, serve as a compass in our decision-making, and play a big role in how we treat each other, our clients, and our communities. We believe that the best ideas can come from anyone, anywhere, at any time, and that curiosity is the key to anticipating our clients’ needs and exceeding their expectations.

Your Team’s Impact AI has evolved from an isolated feature into a capability that spans our entire platform. The engineering infrastructure required to measure, monitor, and validate it must now evolve as well.

Today, teams instrument AI systems independently, with no shared visibility when something behaves unexpectedly in production and no common framework for validating changes before they ship. You will build that foundation.

You will deploy and operate LangSmith in our own infrastructure, develop the evaluation tooling on top of it, and own the observability layer for AI systems running across FactSet. When an AI workflow fails in a client-facing context, your platform is how the organization detects it — and proves it has been resolved.

This is a founding engineering role. You will inherit a small team and be responsible for consolidating organically developed evaluation solutions into a single, authoritative capability. Adoption will be earned, not mandated — your platform’s influence across the organization will grow through the value it demonstrably delivers.

What You Will Own LangSmith (Self-Hosted): Own the full lifecycle — deployment, operations, upgrades, and security. This includes Kubernetes infrastructure, datastores, SSO, access control, data isolation, and capacity management. Lead the vendor relationship and partner with Cloud and Central Technology to ensure compliant, secure hosting of client-adjacent evaluation data.

Evaluation Tooling: Build the tooling that makes the platform practical — eval harnesses for offline and production traffic, LLM-as-judge scoring pipelines, versioned golden datasets, and regression suites that integrate into existing CI pipelines. Your product counterpart defines what gets measured; you build what does the measuring.

AI Production Observability: Own tracing, metrics, and alerting across AI systems in production covering latency, cost, model and prompt versioning, retrieval behavior, and drift detection. This layer transforms a client escalation into a diagnosable, reproducible event.

Production Reliability: The team owns the pager for what it builds. You set the operational standard with a first-year objective of building a sustainable on-call rotation as adoption scales.

Technical Leadership: You are stepping into a hands-on technical leadership role, not a pure management position. You will inherit a team of 4 engineers today, with the expectation of growing to 8–10 as the platform scales and proves its value. Beyond people leadership, you will drive architectural decisions, review critical code and design changes, and hire deliberately -bringing on the right people at the right time rather than growing for the sake of it.

Minimum Requirements What We’re Looking For :

• 10+ years of production software experience, with significant time on platform or infrastructure teams that other engineers depended on
• Hands-on technical leadership: you have led small teams while staying in the code, and you prefer it that way
• Strong Python proficiency is required. The LLM evaluation and observability ecosystem is Python-native.

Critical Skills

• Platform track record: you have built internal platforms that teams chose to adopt, and you understand why the ones that failed were slower than what they replaced
• LLM production experience: you can reason about tracing, retrieval behavior, prompt and model versioning, and regression analysis; familiarity with tooling such as LangSmith, LangFuse, Braintrust, or Weights & Biases is a plus
• Kubernetes and cloud infrastructure depth, including operating relational and analytical datastores under real load
• Self-hosted platform operations: you have run stateful third-party platforms in production, including upgrades, da…