Databricks & Azure Synapse / MS Fabric Data Engineer

Role

Looking for a skilled Databricks / Azure Synapse / MS Fabric Senior Data Engineer with at least 6 to 8 years of experience to join our vibrant team. The ideal candidate should have strong background in Azure data services, including Databricks, Synapse Analytics, Microsoft Fabric, PySpark and SQL. He/She will be responsible for designing, building, and maintaining scalable metadata driven data pipeline solutions that meet business requirements and align with industry best practices.

• At least 6 years of experience as a Databricks & Data Engineer, with demonstrated experience designing and implementing data solutions using Azure services such as Azure Data Factory, Synapse Pipelines, Data Flows, Azure SQL Database, Azure Databricks, Azure Cosmos DB, and Microsoft Fabric Lakehouse / Warehouse solutions.
• Strong experience with Python programming and have a good understanding of relational database concepts.
• Proficient in SQL and experience with Dimensional data modeling, Data warehousing, and ETL processes.
• Develop and optimize data pipelines using Azure Databricks and Synapse Pipelines, ensuring efficient processing of large-scale datasets.
• Utilize Apache Spark on Databricks to perform real-time and batch data processing, enabling robust and scalable data analytics.
• Integrate diverse data sources into Fabric One-Lake / Databricks Lakehouse / Azure Data Lake Storage using Azure Data Factory and Data Flows, ensuring data is readily available for analysis.
• Architect and maintain data solutions with Delta Lake on Databricks or Delta tables in Fabric One Lake, ensuring high data quality, consistency, and accessibility.
• Implement and manage Lakehouse architecture to unify data warehousing and big data analytics on a single platform.
• Leverage Microsoft Fabric to streamline data integration and transformation, connecting various data services for seamless workflows.
• Collaborate with data scientists/ business users to operationalize machine learning models within Databricks, ensuring they are effectively integrated into data pipelines.
• Automate and manage CI/CD pipelines using Azure DevOps, focusing on Databricks notebooks, Synapse artifacts, and Fabric workflows.
• Establish data governance and security using Azure Purview and Databricks Unity Catalog to ensure compliance and data integrity across platforms.
• Monitor, optimize, and fine-tune performance in Databricks, Synapse, and Fabric environments to maintain high efficiency and cost-effectiveness.
• Develop and maintain innovative, cost effective yet scalable and highly performing Azure Data solutions (leveraging server-less SQL pool, data flows, Apache Spark pool, Dedicated SQL Pools)
• Expert level understanding on Azure Data Factory, Azure Synapse Pipelines, Data Flows, Azure SQL, Azure Data Lake Gen2 and other relevant Azure App Service is required.
• Knowledge in Azure IoT, Azure HDInsight + Spark, Azure Stream Analytics, Power BI is desirable
• Familiarity with other cloud data warehousing solutions such as Google BigQuery or AWS Redshift.
• Familiarity with Azure Machine Learning services like Azure Machine Learning Studio, Azure Databricks, or Azure Machine Learning Compute.
• Ability to create and maintain Data flow, ETL Specifications documentation, including data dictionaries and technical specifications.
• Ability to work collaboratively in a team environment and to communicate effectively with stakeholders.
• Experience in reconciliation of data with source systems, data accuracy validation and testing.
• Knowledge of code migration, Change management – version control systems
• Understanding of ERP (EBS/ERP/PSFT/JDE/SFDC etc…) source system is a plus if not, Willingness to understand source system business process flow and learn the technical ER models