Azure Databricks Data Engineer / Senior Data Engineer

Role & responsibilities

We are looking for an experienced Azure Databricks Data Engineer to build and optimize scalable cloud-based data engineering solutions on Microsoft Azure. The ideal candidate should have strong hands-on experience in Azure Databricks, PySpark, Azure Data Factory (ADF), Spark SQL, Delta Lake, and ADLS Gen2 with expertise in designing enterprise-grade ETL/ELT pipelines and modern Lakehouse architectures.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks, PySpark, Spark SQL, and Azure Data Factory (ADF).
• Build and implement Lakehouse/Medallion Architecture (Bronze, Silver, Gold layers).
• Develop ETL/ELT pipelines for batch data processing and transformation.
• Perform data cleansing, deduplication, incremental loads, and implement SCD Type 1 & Type 2 using Delta Lake.
• Optimize Spark applications by tuning partitions, caching, memory management, and Adaptive Query Execution (AQE).
• Handle performance optimization using Broadcast Joins, Data Skew handling, Partitioning, and File Optimization techniques.
• Monitor and troubleshoot production jobs using Spark UI, Azure Monitor, and Databricks job logs.
• Build CI/CD pipelines using Git, Azure DevOps, and Databricks Asset Bundles (DABs).
• Collaborate with Data Architects, Analysts, and Business teams to deliver high-quality data solutions.
• Follow best practices in coding standards, testing, deployment, and documentation.

Required Skills

• 3+ years (Data Engineer) / 5+ years (Senior Data Engineer) in Azure Data Engineering.

• Strong experience with:

• Azure Databricks
• PySpark
• Spark SQL
• Azure Data Factory (ADF)
• Azure Data Lake Storage Gen2 (ADLS)

• Delta Lake

• Hands-on experience with:

• Delta MERGE
• Time Travel
• VACUUM

• Managed & External Tables

• Strong SQL skills including:

• Window Functions
• ROW_NUMBER
• RANK
• DENSE_RANK
• LEAD

• LAG

• Experience in:

• ETL/ELT Development
• Data Modeling
• Performance Tuning
• Data Quality

• Schema Evolution

• Good understanding of Git, Azure DevOps, CI/CD, and production deployments.

Preferred Skills

• Structured Streaming
• Unity Catalog
• Infrastructure as Code (Terraform/Bicep)
• Azure Monitor
• Data Governance
• Python
• pytest / Unit Testing
• Cost Optimization in Databricks
• Data Quality Frameworks

Mandatory Skills

• Azure Databricks
• PySpark
• Azure Data Factory (ADF)
• Spark SQL
• ADLS Gen2
• Delta Lake
• Azure Data Engineering
• SQL
• ETL
• Azure Cloud

Good to Have

• Unity Catalog
• Structured Streaming
• Azure DevOps
• Git
• Databricks Asset Bundles
• Lakehouse Architecture
• Medallion Architecture
• Data Modeling

Preferred candidate profile