Azure Databricks Data Engineer

Job Summary:
- We are seeking a skilled and experienced Data Engineer with a focus on Azure Databricks to build and optimize our data pipelines. - In this role, you will be responsible for designing and developing robust data ingestion, transformation, and storage solutions on the Microsoft Azure platform. - You will leverage Azure Databricks and other Azure services to build data lakes and warehouses that power our analytics and AI initiatives.

**Key Responsibilities:** - - Design, build, and maintain scalable data pipelines for batch and real-time processing using Azure Databricks. - - Develop and maintain ETL/ELT processes using Apache Spark and other big data technologies. - - Optimize data storage using various Azure services (Data Lake Storage, Blob Storage) and data formats (Delta Lake, Parquet). - - Build and maintain data models and data warehouses for business intelligence and reporting. - - Collaborate with data scientists and analysts to provide clean, well-structured data for their use cases. - - Implement data quality and governance policies within data pipelines. - - Troubleshoot and resolve complex data pipeline issues.   
 
**Required Qualifications:** - - Bachelor's degree in Computer Science, Information Technology, or a related field. - - 4-6 years of experience in data engineering, with a strong focus on Microsoft Azure. - - Deep expertise in Azure Databricks, Apache Spark (PySpark or Scala). - - Strong experience with data modeling, data warehousing, and ETL/ELT processes. - - Proficiency in SQL and experience with cloud databases (e.g., Azure SQL, Synapse). - - Experience with CI/CD pipelines and version control.   
 
**Preferred Qualifications:** - - Experience with Azure Data Factory and Azure DevOps. - - Experience with data governance and data quality frameworks. - - Azure Data Engineer or Azure Databricks certification.   
 
**Key Competencies:** - - Technical Expertise: Deep knowledge of Azure Databricks and the Spark ecosystem. - - Problem-Solving: Strong analytical and troubleshooting skills. - - Performance Focus: Commitment to building high-performance data pipelines. - - Collaboration: Excellent communication and teamwork skills.