Principal Azure Cloud Platform Engineer - Databricks & CI/CD Pipeline

Position Overview:

We are seeking a highly skilled and experienced Principal Azure Cloud Engineer to lead the design, implementation, and optimization of Azure-based cloud solutions, with a focus on data solutions, infrastructure automation, and migration. The ideal candidate will possess a strong technical background in cloud architecture, especially in building Infrastructure as Code (IaC), designing scalable Azure Data Lake and database solutions, and driving migration strategies. As a technical leader, you will guide cross-functional teams to ensure that our cloud architecture aligns with business goals, follows best practices, and optimizes performance, security, and compliance.

Key Responsibilities:

1. Technical Leadership & Strategic Guidance:
   Provide technical leadership and mentorship to cross-functional teams (cloud engineers, data engineers, business stakeholders) to architect robust Azure-based data solutions.
   Ensure alignment of cloud solutions with business objectives, considering current and future growth.
   Stay updated with the latest Azure innovations, trends, and best practices to recommend the best solutions for complex business needs.

2. Azure Data Lake and Database Solutions:
   Design and deploy scalable and high-performance Azure Data Lake storage solutions to handle large volumes of unstructured and structured data.
   Architect, configure, and manage Azure SQL Database, Cosmos DB, and other database offerings to ensure high availability, disaster recovery, and optimized performance.

3. Databricks Administration & Integration:
   Manage, configure, and administer Azure Databricks environments, ensuring optimal performance, scalability, and security of data processing workloads.
   Collaborate with data scientists, data engineers, and analysts to integrate Azure Databricks with other Azure services (e.g., Azure Data Lake, Azure Synapse) to build scalable data processing and analytics solutions.
   Implement best practices for managing Databricks clusters, notebooks, libraries, and jobs to ensure efficiency and cost optimization.
   Perform regular monitoring, troubleshooting, and performance tuning of Databricks workloads, ensuring high reliability and uptime.

4. Cloud Migration Strategy & Execution:
   Develop and implement comprehensive strategies for migrating on-premises data and applications to Azure with minimal disruption to business operations.
   Use Azure-native tools (e.g., Azure Migrate, Data Migration Assistant) to facilitate seamless migration of data assets to the cloud.

5. Azure Data Factory (ADF) Expertise:
   Design, implement, and maintain efficient data pipelines using Azure Data Factory(ADF) to orchestrate and automate ETL workflows across various data sources and destinations.
   o Optimize pipeline performance, manage dependencies, and ensure seamless data movement across hybrid cloud environments.

6. Database Services Optimization:
   Administer and optimize Azure Database offerings such as Azure SQL Database, SQL Managed Instances, and other relational/non-relational databases.
   Design high-availability, fault-tolerant architectures, and ensure efficient data replication, backup, and recovery.

7. Performance Tuning & Optimization:
   Identify and address performance bottlenecks within Azure-based data solutions, applying performance tuning strategies to maximize system efficiency.
   Ensure that systems are optimized for cost efficiency, scalability, and reliability by monitoring usage patterns and implementing proactive optimization strategies.

8. Security & Compliance:
   Implement strong security measures across Azure data solutions to ensure confidentiality, integrity, and availability of data.
   Leverage Azure’s security features, including encryption, role-based access control(RBAC), firewalls, and auditing tools to meet compliance requirements (e.g., GDPR,HIPAA, SOC 2).
   Conduct regular security assessments and work with teams to enforce security best practices across all cloud solutions.

9. Collaboration & Communication:
   Engage with stakeholders from business units, IT, and data engineering to understand business requirements, design appropriate solutions, and ensure smooth project delivery.
   Present project updates, technical challenges, and solutions to executive leadership and other stakeholders, ensuring transparency and alignment with organizational goals.
   Facilitate collaborative decision-making and knowledge sharing among cross-fun…