Job Closed
Azure Data Lead
- Source: LinkedIn
- Inherent Technologies
- On-site
- New York City
- Full-Time
- Senior
Job Description
Position: Azure Data Lead Location: NYC, NY***3 days onsite*** Duration: 1 Year Job Description
- Azure Data lead - Python, Pyspark, Databricks, ADF, Data Lake,
- Azure Senior Data Lead Leads modernization of Python applications into scalable PySpark solutions on Azure Databricks
- Lead the modernization and migration of existing Python object-oriented applications into scalable PySpark and Spark SQL data-processing solutions on Azure Databricks - bringing a strong blend of software engineering, data engineering, cloud architecture, and performance optimization.
Key Responsibilities
- Analyze existing Python OOP applications and redesign single-node processing logic for distributed Spark execution.
- Design, develop, and deploy enterprise-scale data pipelines on Azure Databricks; build reusable PySpark frameworks and utility modules.
- Implement Delta Lake solutions using the Bronze Silver Gold architecture.
- Build robust ETL/ELT pipelines with Azure Data Factory, ADLS Gen2, and Azure Synapse Analytics.
- Implement data quality, reconciliation, validation, and monitoring frameworks.
- Optimize Spark jobs (partitioning, bucketing, caching, broadcast joins, Adaptive Query Execution, Delta optimization) and benchmark converted applications against original Python implementations. Core Skills
- Python (expert), OOP, and advanced Python design patterns
- PySpark, Spark SQL, and SQL
- Azure Databricks, Azure Data Factory, ADLS Gen2
- Apache Spark, Delta Lake, Data Lakehouse architecture, distributed computing
Must-have (per Requisition)
- Python, Azure Databricks, Azure Data Factory (ADF), MS SQL, Oracle PL/SQL.
Good To Have
- PySpark; certifications in Azure Data Factory, Azure Databricks, SQL, Oracle, or Python.
- Experience & Expected Outcome Senior data engineering leader with proven delivery of large-scale Databricks modernization programs.
- Expected outcome: existing Python applications converted into scalable, cost-efficient, enterprise-grade data solutions on Azure Databricks with proven performance parity.
Skills
- microsoft-azure
- Python
- pyspark
- azure-databricks
- databricks
- Spark
- apache-spark-sql
- NodeJS
- delta-lake
- azure-data-factory
- azure-synapse
- microsoft-azure-synapse
- azure-synapse-analytics
- benchmark
- oracle
- oracle-pl-sql
This job is no longer accepting applications.
Join the Club — Free ForeverYou missed this one - but you don't have to keep missing out. Try Club members get matched with open opportunities before they're gone, so you're always first in line, never last.
