Data Engineer - Azure(Exp: 4-9 yrs)

Job Summary

We are looking for an experienced Data Engineer with 4–9 years of hands-on experience in designing, developing, and maintaining scalable data pipelines and data platforms on Microsoft Azure.

The ideal candidate should have strong expertise in Python, SQL, Azure Data Factory, Azure Data Lake Storage, Databricks, data warehousing, ETL/ELT, and big data technologies. The candidate will be responsible for building reliable data solutions, processing large datasets, optimizing data pipelines, and collaborating with Data Scientists, Data Analysts, BI teams, and software engineers.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL/ELT workflows on Azure.
• Build data ingestion pipelines from databases, APIs, applications, files, and other data sources.
• Develop data transformation and processing workflows using Azure Data Factory and Databricks.
• Work with large volumes of structured and unstructured data.
• Develop and optimize SQL queries, stored procedures, and data transformations.
• Build and maintain Azure Data Lake and cloud-based data warehouse solutions.
• Implement batch and incremental data processing pipelines.
• Perform data cleansing, validation, transformation, and quality checks.
• Optimize pipelines for performance, scalability, reliability, and cost efficiency.
• Monitor data pipelines and troubleshoot production failures and data-quality issues.
• Implement appropriate logging, alerting, error handling, and recovery mechanisms.
• Collaborate with Data Scientists, Data Analysts, BI developers, software engineers, and business stakeholders.
• Follow best practices for data security, governance, access control, and compliance.
• Participate in architecture and technical design discussions.
• Develop reusable data engineering frameworks and components.
• Maintain technical documentation for data pipelines, workflows, and data architecture.
• Participate in code reviews and follow Agile development methodologies.

Required Technical SkillsProgramming

• Strong hands-on experience with Python.
• Strong proficiency in SQL.
• Good understanding of data structures, algorithms, and software engineering principles.
• Ability to write clean, reusable, maintainable, and production-ready code.

Azure Data Engineering

Strong hands-on experience with multiple Azure services, preferably:

• Azure Data Factory (ADF) – ETL/ELT and pipeline orchestration
• Azure Data Lake Storage Gen2 (ADLS) – Data Lake
• Azure Databricks – Data processing and transformation
• Azure Synapse Analytics – Data warehousing and analytics
• Azure SQL Database – Relational data storage
• Azure Functions – Serverless processing
• Azure Event Hubs – Real-time data ingestion
• Azure Key Vault – Secrets and credential management
• Azure Monitor / Log Analytics – Monitoring and logging
• Microsoft Entra ID (Azure AD) – Identity and access management

Candidates should have practical experience working with Azure data services in production environments.

ETL / ELT & Data Pipelines

• Strong experience designing and developing ETL/ELT pipelines.
• Experience with batch and incremental data processing.
• Experience handling data ingestion from multiple sources.
• Knowledge of pipeline scheduling, dependencies, retries, error handling, and monitoring.
• Experience implementing data validation and quality checks.
• Understanding of CDC and incremental-load strategies is preferred.

Azure Databricks / Apache Spark

• Strong hands-on experience with Azure Databricks.
• Experience with Apache Spark / PySpark.
• Understanding of distributed data processing.
• Experience working with large datasets.
• Knowledge of Spark optimization techniques including:
• Partitioning
• Joins
• Caching
• Data skew
• File optimization
• Experience working with Delta Lake / Delta tables is preferred.

Data Warehousing

• Strong understanding of data warehouse concepts and architecture.
• Experience with Azure Synapse Analytics or similar cloud data warehouses.
• Knowledge of:
• Fact and Dimension Tables
• Star Schema
• Snowflake Schema
• Slowly Changing Dimensions (SCD)
• Data Modeling
• Experience optimizing warehouse queries and data models.

Data Lake & Modern Data Architecture

• Hands-on experience with Azure Data Lake Storage Gen2.
• Understanding of Data Lake and Lakehouse architecture.
• Experience working wit…