Azure Data Lead – Databricks & PySpark Modernization

Position: Azure Data Lead – PySpark & Databricks Modernization

Location: New York City, NY (Hybrid - 3 days/week onsite)

Employment Type: W2 Only

Role Overview & Key Responsibilities:

• Lead the modernization and migration of existing Python object-oriented applications into scalable PySpark and Spark SQL data-processing solutions on Azure Databricks.

• Analyze single-node Python OOP processing logic and redesign for distributed Apache Spark execution.

• Design, build, and deploy enterprise-grade data pipelines on Azure Databricks, establishing reusable PySpark frameworks and modules.

• Implement Delta Lake architectures utilizing the Bronze-Silver-Gold (Medallion) pattern.

• Orchestrate ETL/ELT pipelines using Azure Data Factory (ADF), ADLS Gen2, and Azure Synapse Analytics.

• Execute Spark performance tuning (partitioning, bucketing, broadcast joins, Adaptive Query Execution, Delta optimization) to ensure cost efficiency and performance parity against original implementations.

• Implement automated data quality, validation, reconciliation, and monitoring frameworks.

Required Qualifications & Skills:

• Senior-level experience leading large-scale Databricks modernization programs.

• Expert proficiency in Python (OOP and advanced design patterns), PySpark, Spark SQL, and SQL.

• Hands-on expertise with Azure Databricks, Azure Data Factory (ADF), and ADLS Gen2.

• Strong experience with relational databases including MS SQL and Oracle PL/SQL.

• Deep understanding of Apache Spark, Delta Lake, and Data Lakehouse architectures.

• Must be local to the NYC metro area for 3 days/week onsite work on a W2 basis.