Senior Databricks / Azure Data Lake Engineer

Position Senior Databricks / Azure Data Lake Engineer Experience 6+ Years Location Hyderabad, Telangana Employment Type Full-TimeRole Overview We are looking for an experienced Senior Databricks / Azure Data Lake Engineer with 6+ years of experience in designing, developing, implementing and supporting scalable data engineering solutions on Microsoft Azure. The candidate will be responsible for building robust data pipelines, data lake architectures and transformation frameworks using Azure Databricks, Azure Data Lake Storage (ADLS Gen2), PySpark and SQL. The role requires close collaboration with Data Engineering, Application Engineering, Business Intelligence, Architecture, DevOps and Data Governance teams to deliver secure, scalable and high-quality data solutions. The ideal candidate should have strong hands-on experience with Databricks, Delta Lake, PySpark, Azure Data Lake and Azure data services, along with good understanding of data architecture and performance optimisation.Key Responsibilities Azure Databricks Design, develop and maintain scalable data engineering solutions using Azure Databricks. Develop PySpark notebooks and production-grade data processing jobs. Build reusable frameworks for batch and incremental data processing. Configure and optimise Databricks clusters and compute resources. Implement appropriate cluster policies, job configurations and access controls. Troubleshoot Databricks jobs, performance issues and production failures. Implement Databricks Workflows/Jobs for scheduling and orchestration. Azure Data Lake Design and manage data solutions using Azure Data Lake Storage Gen2 (ADLS). Implement appropriate data lake folder structures and data organisation strategies. Develop ingestion frameworks for structured and semi-structured data. Implement incremental, full-load and CDC-based ingestion patterns.Work with data formats such as: Parquet Delta JSON CSV Avro Implement data partitioning and lifecycle strategies. Ensure data quality, availability and reliability across data lake environments. Delta Lake Strong understanding of Delta Lake architecture and capabilities. Implement ACID-compliant data processing using Delta tables. Work with: MERGE UPDATE DELETE Time Travel Schema Evolution Optimistic Concurrency Implement incremental processing and CDC pipelines. Optimise Delta tables using appropriate partitioning and optimisation techniques. Troubleshoot data consistency and performance issues. PySpark & Data Engineering Develop scalable data transformation pipelines using PySpark. Write efficient Spark SQL and DataFrame transformations. Optimise Spark jobs for large-volume datasets. Analyse and resolve: Data skew Shuffle issues Partitioning problems Memory issues Long-running jobs Implement appropriate caching, repartitioning and broadcast strategies. Develop reusable PySpark libraries and frameworks. Perform data validation and reconciliation across source and target systems. Data Ingestion & Integration Build data pipelines integrating data from multiple sources, including: REST APIs Relational databases Kafka Files Cloud storage Enterprise applications Implement batch and near-real-time data ingestion.Work with CDC technologies and incremental data processing. Design resilient ingestion frameworks with appropriate error handling and retry mechanisms. Azure Data Services Hands-on experience with relevant Azure services such as: Azure Data Lake Storage Gen2 Azure Databricks Azure Data Factory Azure Key Vault Azure Synapse Analytics Azure Event Hubs Azure Monitor Microsoft Entra ID Experience integrating these services into enterprise data platforms is highly desirable. Azure Data Factory / Orchestration Develop and maintain Azure Data Factory pipelines. Implement pipeline orchestration between ADF and Databricks. Develop parameterised and reusable pipelines. Implement dependency management and error handling. Configure monitoring, alerts and retry mechanisms. Manage production scheduling and operational support. Data Quality & Governance Implement data validation and reconciliation frameworks. Identify and resolve data quality issues. Implement data quality checks at ingestion and transformation stages. Follow enterprise data governance and security standards. Maintain metadata, data lineage and technical documentation. Implement appropriate access controls and data protection mechanisms. Performance Optimisation Analyse and optimise Databricks/Spark workloads. Optimise SQL queries …