Azure Devops Engineer

We are looking for an experienced Azure Databricks Administrator to manage, monitor, and optimize our Databricks environment on Microsoft Azure. The ideal candidate will be responsible for platform administration, security, cost governance, and ensuring high availability of the Databricks workspace for data engineering and analytics teams.

Key Responsibilities

• Administer and manage Azure Databricks workspaces, clusters, and pools across environments (Dev, QA, Prod).
• Configure and manage Unity Catalog, metastores, catalogs, schemas, and access controls.
• Set up and maintain cluster policies, instance pools, and autoscaling for cost and performance optimization.
• Manage user access, groups, service principals, and role-based access control (RBAC) via Azure AD / Entra ID.
• Integrate Databricks with Azure services such as ADLS Gen2, Key Vault, Azure DevOps, and Azure Data Factory.
• Monitor platform health, job performance, and resource utilization; troubleshoot cluster and job failures.
• Implement security best practices including network configuration (VNet injection, Private Link), secret management, and encryption.
• Manage workspace-level and account-level configurations and governance policies.
• Handle cost management, chargeback reporting, and DBU consumption optimization.
• Automate administrative tasks using Databricks REST APIs, CLI, Terraform, and PowerShell/Python scripts.
• Coordinate with data engineering teams to support CI/CD pipelines and deployment workflows.
• Manage platform upgrades, runtime versions, and library/dependency governance.

Required Skills

• Strong hands-on experience administering Azure Databricks.
• Solid understanding of Apache Spark architecture and cluster management.
• Experience with Unity Catalog and data governance.
• Proficiency in Azure ecosystem: ADLS Gen2, Azure AD/Entra ID, Key Vault, Networking (VNet, Private Link).
• Experience with Infrastructure as Code (Terraform) and Databricks CLI/REST APIs.
• Scripting skills in Python, PowerShell, or Bash.
• Knowledge of RBAC, security, and compliance frameworks.
• Familiarity with CI/CD and Azure DevOps.