Databricks Data Engineer (GCP BigQuery To Azure Databricks Migration

Databricks Data Engineer (GCP BigQuery to Azure Databricks Migration)

Location: Bangalore / Hyderabad / Pune
Experience: 6-12 Years

We are looking for a highly skilled Databricks Data Engineer to join a large-scale enterprise cloud modernization program focused on migrating data platforms from Google Cloud Platform (BigQuery/GDAP) to Azure Databricks Lakehouse.

The ideal candidate will have hands-on experience in BigQuery migration projects, modern data engineering practices, Azure Databricks, PySpark, Delta Lake, and large-scale ETL modernization initiatives.

This role requires strong technical expertise in designing scalable data pipelines, optimizing lakehouse architectures, and ensuring successful end-to-end migration of enterprise data assets.

Key Responsibilities

• Design, develop, and optimize scalable ETL/ELT pipelines using Azure Databricks and PySpark.
• Migrate BigQuery datasets, tables, views, and schemas to Azure Databricks Delta Lake.
• Implement Medallion Architecture (Bronze, Silver, Gold) and Lakehouse best practices.
• Develop robust batch and incremental data processing frameworks.
• Build data validation, reconciliation, and quality assurance frameworks.
• Perform SQL optimization, Spark performance tuning, and workload optimization.
• Support historical data migration, testing, deployment, and production stabilization.
• Collaborate with architects, business stakeholders, and data platform teams during the migration journey.
• Implement CI/CD processes and version control using Git-based repositories.
• Ensure data governance standards through Unity Catalog and enterprise security controls.

Mandatory Skills

Core Databricks & Spark

• Azure Databricks
• PySpark
• Spark SQL
• Delta Lake
• Unity Catalog
• Data Lakehouse Architecture
• Medallion Architecture

Azure Data Engineering

• Azure Data Lake Storage (ADLS Gen2)
• Azure Data Factory (ADF) (Preferred)
• Data Pipeline Development
• ETL/ELT Frameworks

Migration Expertise (Must Have)

• Google BigQuery
• GCP to Azure Migration
• BigQuery Schema Conversion
• Data Migration Strategy
• Data Validation & Reconciliation Frameworks

Development & DevOps

• SQL Performance Tuning
• Git
• CI/CD Pipelines
• Production Deployment & Support

Preferred Skills

• Airflow / Cloud Composer
• Google Dataflow
• DBT (Data Build Tool)
• Azure Key Vault
• Terraform
• Data Governance & Metadata Management
• Agile/Scrum Delivery Model

Qualifications

• 6-12 years of Data Engineering experience.
• Minimum 3+ years of hands-on Azure Databricks experience.
• Proven experience in at least one large-scale cloud migration program involving BigQuery.
• Strong understanding of modern cloud-native data architectures and Lakehouse concepts.

Looking for short Notice joiners, if interested, Pls share your resume to [email protected]

Total Exp -

Current Location -

Current CTC -

Expected CTC -

Notice period -

Current Company -