Databricks Platform Engineer (AWS)

About Vanguard Founded in 1975, Vanguard is one of the world’s leading investment management companies. The firm offers investments, advice, and retirement services to tens of millions of individual investors around the globe—directly, through workplace plans, and through financial intermediaries.

**Vanguard’s India Office** Vanguard’s office in India is a significant milestone in our global expansion. We are committed to establishing an enduring technology center in Hyderabad, Telangana and are excited to be adding talent who will focus on Artificial Intelligence (AI), mobile, and cloud-based technologies that drive our business outcomes and deliver a world-class experience for our clients.

Role Summary This role provides advanced level technical platform development and support for complex projects. An ideal candidate for this role is a platform-focused databricks expert (5-8 years of experience) who can build, secure, automate, and operate Databricks on AWS , work directly with clients and stakeholders, solve complex platform challenges, and drive enterprise-wide adoption of Databricks capabilities

Responsibilities

• Provision and configure Databricks Accounts, Workspaces, Unity Catalog, and supporting AWS infrastructure.
• Design, deploy, and manage multi-environment platforms (Dev/Test/Prod) and automate platform provisioning and configuration using Terraform and CloudFormation.
• Configure and manage AWS services including VPC, Subnets, Security Groups, IAM, KMS, S3, Lambda, EC2, ECS, RDS, CloudWatch, resolving WIZ violations and related cloud services.
• Implement and govern Unity Catalog, including metastores, catalogs, schemas, external locations, storage credentials, dashboards, Databricks Apps and fine-grained access controls to enable secure data sharing and enterprise data governance.
• Implement observability and operational monitoring using Databricks System Tables, audit logs, cluster logs, usage metrics, and cloud monitoring solutions to improve platform reliability and user experience.
• Analyze platform usage patterns, cluster utilization, DBU consumption, and infrastructure costs to identify optimization opportunities and implement cost governance controls.
• Support migration, onboarding, and enablement of enterprise applications, data engineering workloads, AI/ML platforms, and GenAI use cases onto Databricks while adhering to architecture and security standards.
• Leverage Databricks Asset Bundles (DAB), Databricks CLI, REST APIs, Git integration, and CI/CD pipelines to automate deployment and lifecycle management of notebooks, jobs, workflows, and platform configurations.
• Enable AI/ML and GenAI capabilities, including MLflow, Model Serving, Feature Store, Vector Search, and Mosaic AI, while supporting governance, monitoring, and operationalization of machine learning workloads.
• Implement and govern Unity Catalog, including metastores, catalogs, schemas, external locations, storage credentials, and fine-grained access controls.
• Enable audit logging, lineage tracking, data access governance, and platform monitoring to meet enterprise standards and analyze DBU consumption, cluster usage patterns, and AWS infrastructure spend to identify optimization opportunities and implement cost governance controls.
• Provide hands-on troubleshooting and operational support for platform, integration challenges and troubleshoot infrastructure, networking, security, authentication, and connectivity issues.
• Partner directly with application teams, data engineers, data scientists, architects, and business stakeholders to understand business requirements and recommend platform solutions.
• Act as a trusted advisor and primary point of contact for Databricks platform capabilities and adoption.

Qualifications And Skills

• Minimum 5 years of experience in cloud engineering, infrastructure management, or cloud operations.
• Bachelor’s degree (B.E./B.Tech in Computer Science/IT) or equivalent experience.
• Strong experience with Databricks administration and platform engineering.
• Deep understanding of Databricks platform components including Unity Catalog, Workflows, SQL Warehouses, Cluster Policies, Delta Lake, Auto Loader, Structured Streaming, and Lakehouse architectures.
• Hands-on experience with Databricks Asset Bundles (DAB), Databricks CLI, REST APIs, Git integration, and platform automation.
• Expertise in AWS services including VPC, IAM, EC2, S3…