Hiring For Azure Data Engineer

We are looking for an experienced AI / Data Engineer with strong hands-on expertise in Python, PySpark, Azure and Databricks. The candidate will be responsible for designing, developing and optimizing scalable data pipelines and data processing solutions on Azure.

The ideal candidate should have strong experience in PySpark and Databricks, along with good knowledge of data engineering, ETL/ELT, SQL and cloud data platforms. Exposure to AI/ML workloads and Azure Synapse Analytics will be an added advantage.

Key Responsibilities

• Design, develop and maintain scalable ETL/ELT data pipelines using Azure Databricks and PySpark.
• Develop data transformation and processing solutions using Python and PySpark.
• Work with large-scale structured and semi-structured datasets using Apache Spark.
• Build and optimize data pipelines for performance, scalability and reliability.
• Develop Databricks notebooks, jobs and workflows for batch data processing.
• Implement data cleansing, transformation, validation and reconciliation processes.
• Work with Azure data services to build cloud-based data engineering solutions.
• Support development of data platforms for AI/ML and advanced analytics use cases.
• Prepare high-quality datasets/features for AI and Machine Learning models.
• Optimize Spark jobs, queries, partitions and resource utilization.
• Implement data quality, error handling, monitoring and troubleshooting mechanisms.
• Work closely with Data Scientists, Data Analysts, Architects and other engineering teams.
• Participate in code reviews, technical discussions and Agile delivery processes.
• Maintain technical documentation related to data pipelines and data processing solutions.