Azure Databricks Platform Administrator

Job Title: Azure Databricks Platform Administrator

Job Location: Raritan, NJ

Job Description

We are looking for an experienced Azure Databricks Platform Admin with 57 years of experience in cloud data platform administration operations and support. The role is responsible for administering, securing, monitoring and optimising Azure Databricks environments across development, test, and production landscapes. The ideal candidate should have strong hands-on experience with workspace administration, cluster management, Unity Catalogue security controls, automation and Azure platform services

Key Responsibilities

• Administer and maintain Azure Databricks workspaces, clusters, jobs, notebooks and platform configurations
• Configure and manage Unity Catalogue metastores, catalogues, schemas, external locations, and storage credentials
• Implement and enforce security controls including access management, role-based permissions, service principal secret scopes, and integration with Microsoft Entra ID
• Manage cluster policies, autoscaling runtime versions, job clusters, and cost optimisation measures to improve platform efficiency
• Integrate Azure Databricks with Azure services such as ADLS Gen2, Azure Key Vault, Azure Monitor, Private Link VNets, and managed identities
• Monitor platform health, audit logs, usage patterns, job performance, and troubleshoot failures, connectivity issues, and performance bottlenecks
• Support CICD and infrastructure automation using tools such as Terraform Azure DevOps, Databricks CLI, or REST APIs
• Maintain environment standards across Dev, QA and Prod, including deployment controls, documentation and operational runbooks
• Collaborate with data engineering, architecture security, and support teams to ensure platform reliability, governance and compliance
• Drive continuous improvement in platform operations, observability security posture, and cost management
• 5-7 years of overall experience in cloud platform administration, data platform operations, or infrastructure support
• Hands-on experience administering Azure Databricks in enterprise environments
• Strong knowledge of Unity Catalogue, Delta Lake workspace governance, access controls, and Databricks administration concepts
• Good understanding of Azure services including ADLS Gen2, Key Vault, Entra ID networking, private endpoints, and monitoring services
• Experience with cluster optimisation, troubleshooting Spark job issues, workload tuning, and platform monitoring
• Exposure to automation and DevOps practices using Terraform, Azure DevOps, GitHub Actions, Python Bash, or PowerShell
• Knowledge of identity and access management, audit logging, compliance and data security best practices
• Working knowledge of SQL and familiarity with Python or scripting for platform automation
• Strong problem-solving, stakeholder communication, and documentation skills