Senior Data Engineer – Databricks & AI Solutions

Experience: 6+ Years

Location: Hyderabad / Chennai / Bangalore

Work Mode: Hybrid – 3 Days Work From Office

Shift: Day Shift

Notice Period: 0–30 Days / Serving Notice Period

Role Overview

We are looking for an experienced Senior Data Engineer with strong hands-on expertise in Databricks, Lakehouse Architecture, PySpark, Real-Time Data Processing, AI/GenAI solutions, and automated deployment processes .

The ideal candidate will design, develop, and support scalable batch and streaming data platforms while leveraging modern Databricks capabilities such as Lakeflow, Unity Catalog, Delta Lake, and AI/GenAI technologies .

The candidate should be able to work independently, communicate effectively with business and technical stakeholders, and deliver high-quality solutions with minimal supervision.

Mandatory Skills

Candidates Must Have Strong Hands-on Experience In:

• Databricks and modern Lakehouse Architecture
• Integration of enterprise data platforms with AI-driven applications
• Automated deployment and release processes / CI/CD

Technical SkillsDatabricks & Data Engineering

• Strong hands-on experience with Databricks
• Modern Lakehouse Architecture
• PySpark
• Spark Structured Streaming
• SQL
• NoSQL databases
• Delta Lake
• Unity Catalog
• Databricks Workflows and Job Orchestration
• Bronze, Silver, and Gold layer architecture
• Batch and real-time data processing

Databricks Lakeflow

Hands-on Experience With:

• Lakeflow
• Declarative Pipelines
• Lakeflow Connect
• Change Data Capture (CDC)
• Incremental Data Processing
• Data Quality Validation
• Pipeline monitoring and optimization

AI / Generative AI

Experience Building Or Integrating:

• AI Agents
• Copilots
• Generative AI solutions
• Large Language Models (LLMs)
• Retrieval-Augmented Generation (RAG)
• Vector Search
• Prompt Engineering

Candidates should understand how enterprise data platforms can be integrated with AI-powered applications.

Data Governance & Security

Experience With:

• Unity Catalog
• Role-Based Access Control (RBAC)
• Data Masking
• Data Protection
• Audit Logging
• Data Lineage
• Data Governance
• Compliance

DevOps & Deployment

Hands-on Experience With:

• Git-based source control
• CI/CD
• Databricks Repos
• Databricks Asset Bundles (DABs)
• Automated deployment and release processes

Key Responsibilities

• Design, develop, and maintain end-to-end batch and streaming data pipelines using Databricks and PySpark.
• Build and optimize Delta Lake tables within Unity Catalog.
• Develop and support Lakeflow pipelines using CDC, incremental processing, and data quality validations.
• Integrate enterprise systems such as SharePoint and other external data sources into the Databricks Lakehouse.
• Implement data governance, security, data quality, lineage, and compliance standards.
• Monitor, troubleshoot, and optimize data workflows for reliability and performance.
• Work with technical and business stakeholders to understand requirements and deliver scalable data solutions.
• Build and support AI-powered data solutions, copilots, and intelligent agents using LLMs, RAG, and vector search.
• Implement CI/CD best practices and automated deployments using Databricks Repos and Asset Bundles.
• Independently drive deliverables and proactively identify and resolve technical challenges.

Preferred Qualifications

• Experience working on large-scale enterprise data platforms.
• Strong understanding of Lakehouse and Data Mesh concepts.
• Exposure to Azure cloud services.
• Experience with enterprise integration patterns.
• Strong problem-solving and analytical skills.
• Excellent stakeholder management and communication skills.

Candidate Requirements

• Minimum 6+ years of relevant experience.
• Candidate must currently be located in Hyderabad, Chennai, or Bangalore.
• Must be comfortable working 3 days from office.
• Must be willing to move to onsite work in the future if required.
• Notice period must be 0–30 days or candidate should already be serving notice.
• Strong employment stability is required. Candidates with frequent job changes should not apply.
• Prefer candidates with at least 2 years of stability in an organization.
• Excellent communication skills are mandatory.
• Candidates should be available for interview coordination within 24–72 hours.

Skills: enterprise,data,uni…