Senior Azure Expert

Given your work experience, we believe it would be a great opportunity for you to progress in your career at Capgemini.

Senior Azure Databricks Expert

Location: PAN India

Experience: 5 - 9 Years

Job Summary:

We are seeking a highly skilled Senior Azure Databricks Data Engineer to design, build, and optimize scalable data pipelines and data platforms on Azure. The ideal candidate will have deep expertise in Azure Databricks, Airflow orchestration, data modeling, ETL/ELT frameworks, and data integration across diverse source systems, along with strong experience in modern data tooling such as DBT and Azure Data Lake Storage (ADLS).

Key Responsibilities:

• Design, develop, and maintain robust, scalable, and high-performance data pipelines using Azure Databricks.
• Build and optimize ETL/ELT workflows leveraging Spark, Python, and SQL.
• Develop and manage data orchestration workflows using Apache Airflow.
• Implement and maintain data models (dimensional, normalized, and Lakehouse architectures) to support analytics and reporting needs.
• Architect and manage data solutions on Azure Data Lake Storage (ADLS Gen2).
• Utilize DBT (Data Build Tool) for data transformation, testing, and documentation.
• Design and implement data integration solutions across multiple source systems (e.g., APIs, RDBMS, SaaS platforms, streaming sources).
• Ensure data quality, consistency, governance, and lineage across the data ecosystem.
• Collaborate with data scientists, analysts, and business stakeholders to translate requirements into scalable data solutions.
• Optimize performance and cost of data workloads in Azure.
• Implement monitoring, logging, and alerting mechanisms for data pipelines.
• Follow best practices for CI/CD in data engineering workflows.

Required Skills & Qualifications:

• 7-12 years of hands-on experience with Azure Databricks (PySpark, Spark SQL).
• Proven experience with Apache Airflow for orchestration.
• Solid understanding of ETL/ELT methodologies and best practices.
• Expertise in data modeling techniques (star schema, snowflake, data vault, Lakehouse).
• Experience with Azure Data Lake Storage (ADLS Gen2).
• Hands-on experience with DBT for transformations and testing.
• Strong SQL and Python programming skills
• Strong problem-solving and communication skills
• Leadership experience or ability to mentor junior engineers

Additional Skills:

• Good understanding of Data Governance and Security
• Experience with different source data integrations.
• Databricks certification
• Strong problem-solving and communication skills