Principal AI-Data Engineer

Description:

Principal Engineer – AI Data Control Plane & Lakehouse Federation

Role Overview
We are looking for a Principal Engineer to lead the architectural design and execution of our enterprise AI Data Control Plane. In this role, you will architect a multi-cloud data virtualization layer using Databricks Unity Catalog (Lakehouse Federation) across disparate systems (BigQuery, Snowflake, PostgreSQL) and expose governed, semantic data access to AI agents via the Model Context Protocol (MCP).
You will act as the technical authority bridging data engineering, cloud infrastructure, and AI systems—enabling next-generation AI agents and applications to query enterprise data safely, performantly, and without credential leakage.

Requirements:

Experience: 8+ years of distributed data systems architecture, with 3+ years specifically in Databricks, Unity Catalog, and multi-cloud data environments.
Core Technical Mastery: Deep expertise in Databricks Lakehouse Federation, SQL pushdown mechanics, and query optimization across BigQuery, Snowflake, and PostgreSQL.
Programming & SDKs: Advanced Python proficiency (databricks-sdk, databricks-sql-connector, pydantic, asyncio, FastAPI).
AI & Agentic Systems: Hands-on experience with LLM tool calling, Model Context Protocol (MCP), Databricks Genie, or building custom agentic data wrappers.
Cloud & Networking: Strong understanding of multi-cloud network topologies (AWS PrivateLink, GCP PSC, Azure Private Endpoints) and credential management (OAuth 2.0, Key Vault, IAM).
Infrastructure as Code: Proficiency with Terraform and automated CI/CD pipelines for data platform deployment.
++++
Prior experience building an internal developer platform or API control plane for AI agents.
Active contributions to open-source data or AI tools (e.g., Anthropic MCP SDKs, LangChain, LlamaIndex, PySpark).
Databricks Certified Data Engineer Professional or Databricks Certified Machine Learning Professional.

Job Responsibilities:

Multi-Cloud Data Virtualization Architecture:
Design and deploy regional Databricks Lakehouse Federation topologies across GCP, AWS, and Azure (connecting BigQuery, Snowflake, and PostgreSQL).
Optimize query pushdown execution, parallel JDBC read slicing, and network routing to minimize cross-cloud egress costs and query latency.
MCP & AI Middleware Platform:
Architect and build high-performance, Python-based MCP Servers (API Faade, Router, and Metadata Registry) that expose Unity Catalog primitives as AI tools to external LLM agents.
Implement semantic catalog indexing using Databricks Vector Search and DatabricksIQ (ai_query) to auto-enrich table/column metadata dictionaries.
Data Governance & Security:
Implement row-level security (RLS) filters and column masking rules directly inside Unity Catalog to secure federated data.
Enforce zero-credential leakage using machine-to-machine OAuth, Databricks Secret Scopes, and system-level audit logs (system.access.audit).
Platform Performance & Reliability:
Configure and tune Serverless SQL Warehouses and Materialized Views for high-concurrency, low-latency agentic workloads.
Establish CI/CD, GitOps, and IaC frameworks (Terraform, Databricks Asset Bundles) for automated change management across Dev, QA, and Prod environments.

What We Offer:

Exciting Projects: We focus on industries like High-Tech, communication, media, healthcare, retail and telecom. Our customer list is full of fantastic global brands and leaders who love what we build for them.

Collaborative Environment: You Can expand your skills by collaborating with a diverse team of highly talented people in an open, laidback environment — or even abroad in one of our global centers or client facilities!

Work-Life Balance: GlobalLogic prioritizes work-life balance, which is why we offer flexible work schedules, opportunities to work from home, and paid time off and holidays.

Professional Development: Our dedicated Learning & Development team regularly organizes Communication skills training(GL Vantage, Toast Master),Stress Management program, professional certifications, and technical and soft skill trainings.

Excellent Benefits: We provide our employees with competitive salaries, family medical insurance, Group Term Life Insurance, Group Personal Accident Insurance , NPS(National Pension Scheme ), Periodic health awareness program, extended maternity leave, annual per…