Sr Data Engineer - Databricks & AI Solutions

Client: Founded in 1990, headquartered in Troy, Michigan, they are a global provider of Information Technology and Business Process Services and Solutions. Their expertise in consulting, Client: design, engineering, and operations helps them provide innovative technology services, positioning them as a technology partner of choice.
With over 10k+ employees they leverage their domain expertise in multiple verticals and harness emerging technologies (Robotic Process Automation (RPA), Internet of Things (IoT), Cybersecurity, Artificial Intelligence, Machine Learning, and much more) to enable their clients build future-ready and sustainable businesses.

Position: Senior Data Engineer (Databricks AI Solutions)
Total Experience: 7y - 11y
Relevant Experience: 6y+
Location: Hyderabad/Chennai/Bangalore (candidate must from this location only)
Work Mode: Hybrid - 3 days work from office
CTC Offered: As per experience and skillsets
Joining : Immediate to 20 days
Interview process: 3 rounds, 1 in-person round may be required

Role Overview

Seeking a highly skilled Senior Data Engineer with deep expertise in Databricks, Real-Time Data Processing, Lakehouse Architecture, and AI-driven solutions. The role involves designing, developing, and supporting scalable data platforms and streaming pipelines, leveraging modern Databricks capabilities such as Lakeflow, Unity Catalog, and Generative AI technologies.

Key Responsibilities
a) Design, develop, and maintain end-to-end streaming and batch data pipelines using Databricks and PySpark.
b) Build, optimize, and manage Delta Lake tables within Unity Catalog.
c) Develop and support Lakeflow pipelines with incremental processing, CDC, and data quality validations.
d) Integrate enterprise systems (e.g., SharePoint, external data sources) into the Databricks Lakehouse.
e) Ensure data quality, governance, security, and operational excellence across platforms.
f) Monitor, troubleshoot, and optimize workflows for reliability and performance.
g) Collaborate with stakeholders to deliver scalable, business-aligned solutions.
h) Build and support AI-powered data solutions leveraging LLMs, RAG, and vector search.
i) Implement CI/CD best practices and maintain deployment pipelines.
j) Drive deliverables independently, proactively resolving challenges with minimal supervision.

Requirements
Experience: 6y+ experience relevant experience in the following:

Databricks Data Engineering
- Hands-on experience with Databricks and modern Lakehouse architecture.
- Expertise in PySpark, Spark Structured Streaming, SQL, NoSQL databases, Delta Lake.
- Proficiency with Unity Catalog for governance and security.
- Experience orchestrating workflows with Databricks Workflows Job Orchestration.
- Proven ability to design and manage bronze, silver, and gold layer architectures.
- Databricks Lakeflow - Practical experience with Lakeflow declarative pipelines, Lakeflow Connect, CDC, data quality validation frameworks, and incremental data processing.

AI Generative AI
- Experience building AI agents, copilots, and generative AI solutions using:
- Large Language Models (LLMs)
- Retrieval-Augmented Generation (RAG)
- Vector search technologies
- Prompt engineering
- Understanding of enterprise data platform integration with AI applications.

Data Governance Security
- Strong knowledge of Unity Catalog governance including:
- Role-Based Access Controls (RBAC)
- Data masking protection policies
- Audit logging
- Data lineage compliance

DevOps Deployment
- Experience with Git-based source control, CI/CD pipelines, Databricks Repos, Databricks Asset Bundles (DABs), and automated deployment processes.

Mandatory Skills (Must Have)
- Strong hands-on experience with Databricks and Lakehouse architecture.
- Proven ability to integrate enterprise data platforms with AI-driven applications.
- Expertise in automated deployment and release processes.
- Excellent problem-solving, communication, and stakeholder management skills.

Requirements - Good to have
- Experience in large-scale enterprise data platforms.
- Strong understanding of Lakehouse and Data Mesh principles.
- Exposure to Azure cloud services and enterprise integration patterns.

**Educational…