Senior Applied AI Engineer

Senior Applied AI Engineer

Document Intelligence, Retrieval & AI Evaluation

Experience: Senior Level

Position Type: Individual Contributor

Core Technologies: Python, Multimodal LLMs, Hybrid Retrieval, AI Evaluation, Cloud AI

About the Opportunity

We are seeking a highly experienced Senior Applied AI Engineer to own and enhance the applied AI and data science capabilities of a production document-intelligence platform.

The platform processes high volumes of unstructured documents using multimodal large language models, hybrid retrieval systems, and human-in-the-loop review workflows.

The selected engineer will take end-to-end ownership of extraction modeling, retrieval quality, and evaluation infrastructure.

This is a hands-on production engineering role, not a research or proof-of-concept position.

Key Responsibilities

1. Multimodal Document Intelligence

• Design and optimize production-scale multimodal LLM document extraction pipelines.
• Implement schema-constrained decoding, tool calling, structured JSON outputs, and automated validation and repair.
• Develop long-document chunking and stitching strategies.
• Implement field-level confidence scoring and probability calibration.
• Define decision thresholds based on business impact and error costs.

2. Hybrid Retrieval & Search Optimization

• Build and optimize dense, learned-sparse, and lexical retrieval systems.
• Design retrieval strategies for large reference catalogs.
• Evaluate approximate nearest-neighbor indexing approaches.
• Implement rank fusion and reranking techniques.
• Optimize retrieval quality against latency and infrastructure costs.

3. AI Evaluation & Experimentation — Critical Requirement

• Own the design and implementation of evaluation harnesses for non-deterministic AI systems.
• Develop labeled datasets, annotation standards, and reproducible evaluation pipelines.
• Apply statistical hypothesis testing, confidence intervals, and appropriate sample sizing.
• Evaluate precision, recall, F1, field-level accuracy, and end-to-end system performance.
• Conduct detailed error analysis and identify performance gaps across document types and customer populations.
• Establish automated regression checks and production release quality gates.
• Use evaluation evidence to guide architectural and model selection decisions.

4. Production AI Engineering

• Develop reliable Python-based AI services and processing pipelines.
• Implement asynchronous processing, typed application models, and robust error handling.
• Build event-driven processing services with idempotency, deduplication, retries, and dead-letter handling.
• Implement safe deployment, rollback, replay, and backfill mechanisms.
• Integrate experiment tracking, observability, and automated CI/CD practices.

5. Prompt Engineering & AI Lifecycle Management

• Treat prompts as version-controlled engineering artifacts.
• Implement prompt testing, review, and automated validation.
• Support document-class and tenant-specific prompt configurations.
• Track changes to prompts, models, datasets, and evaluation results.

6. Security, Privacy & Regulatory Compliance

• Implement data classification, minimization, and redaction.
• Enforce tenant isolation across data storage, search indexes, prompts, and telemetry.
• Implement retention policies, least-privilege access, and secure secrets management.
• Ensure AI processing aligns with applicable regulatory requirements.

Required Qualifications

• 8+ years of hands-on production Python experience , including operating production services.
• Proven experience deploying and supporting production AI/ML systems.
• Strong expertise in multimodal LLM-based document extraction.
• Hands-on experience with production hybrid retrieval systems.
• Demonstrated ownership of AI/ML evaluation frameworks.
• Strong statistical foundations, including hypothesis testing, confidence intervals, sampling, and calibration.
• Experience with dataset creation, annotation protocols, and error analysis.
• Advanced Python proficiency, including Asyncio, type annotations, Pydantic v2, packaging, and profiling.
• Experience with event-driven architectures and reliable distributed processing.
• Working knowledge of data privacy, security, and multi-tenant architectures.

Mandatory: Candidates must demonstrate that they personally built an evaluation harness for a non-deterministic system and used its results to make …