Azure Data Engineer (Python | PySpark | Azure Databricks)

Experience: 5–8 Years

Location: Bangalore / Hyderabad / Chennai / Pune

We are looking for an experienced Azure Data Engineer with strong expertise in Python, PySpark, Azure Databricks, and Azure Cloud Services . The ideal candidate should have hands-on experience designing and developing scalable data pipelines, processing large datasets, and supporting enterprise-grade data platforms. The role involves working closely with business stakeholders, building robust ETL solutions, and maintaining production data pipelines in a DevOps environment.

Key Responsibilities

• Design, develop, and optimize scalable data pipelines using Python, PySpark, and Azure Databricks.
• Build and maintain batch and streaming data processing solutions.
• Develop ETL/ELT workflows to ingest data from multiple structured and unstructured data sources.
• Work with business and technical teams to gather requirements and translate them into data engineering solutions.
• Develop data lake and data warehouse solutions using Azure services.
• Monitor, troubleshoot, and support production data pipelines.
• Collaborate with DevOps teams to automate deployments using CI/CD pipelines.
• Implement logging, monitoring, and performance tuning for data processing jobs.
• Ensure data quality, security, and governance best practices.
• Participate in code reviews, documentation, and technical design discussions.

Required Skills

• 5+ years of experience in Data Engineering.
• Strong programming skills in Python.
• Hands-on experience with PySpark and Azure Databricks.
• Experience developing both Batch and Structured Streaming pipelines.
• Strong knowledge of SQL and exposure to NoSQL databases.

• Hands-on experience with Azure Cloud services, including:

• Azure Data Lake Storage (ADLS Gen2)
• Azure Databricks
• Azure Event Hub
• Azure Event Grid
• Azure SQL Data Warehouse / Azure Synapse Analytics
• Azure Functions

• Azure Key Vault

• Good understanding of Serverless Architecture.
• Knowledge of Infrastructure as Code using Terraform.
• Experience with Azure DevOps and CI/CD pipelines.
• Strong debugging, troubleshooting, and performance optimization skills.

Preferred Skills

• Azure Data Factory (ADF)
• ARM Templates
• Kubernetes (AKS)
• Google Cloud Platform (GCP)
• Machine Learning or AI exposure
• Experience with data lake and modern cloud data platform architectures

Preferred Candidate Profile

• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management abilities.
• Experience working in Agile/Scrum environments.
• Comfortable working in a DevOps model involving both development and production support.
• Ability to independently design and deliver scalable data engineering solutions.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a related field.

Skills: pyspark,azure,nosql,python