Azure Data Engineer_6+ Years

Job Summary We are seeking an experienced Azure Data Engineer with 6–14 years of experience in designing, developing, and maintaining scalable cloud-based data engineering solutions on Microsoft Azure. The ideal candidate should have strong expertise in Python, PySpark, Azure Data Factory (ADF), Azure Databricks, SQL, Pandas, and NoSQL databases , along with hands-on experience in building high-performance ETL/ELT pipelines for large-scale data processing.

The candidate will collaborate with cross-functional teams to develop robust, secure, and efficient data solutions that support analytics, reporting, and business intelligence initiatives.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Azure Data Factory (ADF).
• Develop data transformation and processing workflows using Python and PySpark.
• Build and optimize data pipelines using Azure Databricks and Delta Lake.
• Process large-scale structured, semi-structured, and unstructured datasets efficiently.
• Develop reusable data ingestion frameworks and transformation logic.
• Implement incremental data loading, Change Data Capture (CDC), and data validation processes.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Design and manage data storage solutions using Azure Data Lake Storage (ADLS Gen2) and Blob Storage.
• Work with PostgreSQL, MongoDB, and SQL databases for data extraction, transformation, and loading.
• Develop complex SQL queries, stored procedures, and performance tuning strategies.
• Implement data quality checks, logging, monitoring, and error-handling mechanisms.
• Collaborate with Data Scientists, Software Engineers, Business Analysts, and DevOps teams.
• Participate in code reviews, sprint planning, technical discussions, and solution design.
• Support CI/CD deployments using Azure DevOps or Git.
• Troubleshoot production issues and optimize existing data pipelines.

Required Skills Mandatory Skills

• Python
• PySpark
• Azure Data Factory (ADF)
• Azure Databricks
• SQL
• Pandas
• NoSQL (MongoDB or PostgreSQL)
• Azure Data Lake Storage (ADLS)
• Delta Lake
• ETL/ELT Development
• Data Pipeline Development
• Git
• Agile Methodology

Good to Have

• NumPy
• Azure Functions
• Azure DevOps
• Azure Event Hub / Kafka
• Delta Live Tables
• Unity Catalog
• Docker
• Power BI
• Azure Synapse Analytics

Technical Skills

• Strong Python programming and object-oriented concepts.
• Experience in developing scalable PySpark applications.
• Hands-on expertise in Azure Data Factory pipelines, triggers, datasets, and linked services.
• Strong knowledge of Azure Databricks notebooks, clusters, jobs, and workflows.
• Experience with Delta Lake, Medallion Architecture (Bronze, Silver, Gold).
• Knowledge of data partitioning, caching, broadcast joins, and Spark optimization techniques.
• Experience working with relational and NoSQL databases.
• Strong SQL development and query optimization skills.
• Experience handling large datasets and performance tuning.
• Familiarity with CI/CD pipelines and version control systems.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a related field.
• 6–14 years of overall IT experience.
• At least 4+ years of hands-on experience in Azure Data Engineering.

Preferred Certifications

• Microsoft Certified: Azure Data Engineer Associate (DP-203)
• Microsoft Azure Fundamentals (AZ-900) – Preferred

Soft Skills

• Excellent communication and interpersonal skills.
• Strong analytical and problem-solving abilities.
• Ability to work independently and within Agile teams.
• Strong collaboration and stakeholder management skills.
• Good documentation and knowledge-sharing practices.
• Ability to manage multiple priorities in a fast-paced environment.

Skills: python,pandas,pyspark,sql,nosql