Data Management AI/Databricks Engineer

I have a full time opportunity working remotely that is looking for an Data Management AI/Databricks Engineer to join the team. Please see the job description below and feel free to reach out with any questions. Please send a word version of your resume for consideration.

This role is onsite 3 days a week in Midtown Manhattan.

It is $150-$160k, plus discretionary bonus.

As a Senior Data Engineer, you will own the design and delivery of complex data engineering solutions that power enterprise analytics, AI, and reporting capabilities. Reporting to the Lead Data Engineer, you will drive technical decisions, set engineering standards, and ensure the reliability and scalability of data platform across 40+ integrated enterprise source systems.

This role demands deep technical mastery in SQL, Python, and Azure cloud data engineering, combined with a product orientation—understanding how the data assets you build translate into decisions, reports, and AI outputs for the business. You will mentor Data Engineers, contribute to architectural direction, and serve as a technical anchor for delivery across the team’s sprint cycles.

Advanced Pipeline Development & Ownership

• Architect, build, and own complex data pipelines for high-volume, high criticality workstreams across enterprise data platform.
• Lead the design and implementation of ELT/ETL frameworks using SQL, Python, Azure Data Factory, Databricks, and Azure Synapse Analytics.
• Establish pipeline reliability standards—monitoring, alerting, error handling, and recovery protocols—and ensure adherence across the team.

Data Architecture & Platform Evolution

• Drive the design of scalable data models supporting dimensional warehousing, data lake architectures on Azure.
• Contribute to architectural decisions on data storage, partitioning, compute optimization, and consumption layer design.
• Lead migrations from legacy data solutions to modern cloud-native platforms, managing risk and business continuity throughout.

AI & Analytics Enablement

• Design and deliver feature pipelines and data preparation frameworks that support machine learning model development and deployment.
• Partner with Data Scientists to translate model requirements into production-grade data assets and feature stores.
• Collaborate with Analytics Engineers to ensure data models are optimized for analytical consumption and reporting performance.

Data Quality & Governance Leadership

• Define and implement data quality frameworks—validation rules, SLAs, anomaly detection, and automated testing for pipeline outputs.
• Lead data governance initiatives including metadata management, lineage tracking, data cataloging (Microsoft Purview), and access control.
• Ensure platform compliance with HIPAA, data policies, and applicable regulatory requirements.

Mentorship & Technical Leadership

• Mentor Data Engineers—providing code reviews, technical guidance, and architectural feedback that elevates team capability.
• Contribute to engineering standards, reusable frameworks, and technical documentation.
• Participate in Agile ceremonies and model strong engineering discipline—clear DevOps hygiene, sprint commitment, and delivery accountability.

Education:

• Bachelor’s degree in computer science, Data Science, Information Technology, or a related quantitative field.

Experience:

• 6+ years of progressive experience in data engineering with demonstrated ownership of complex, production-grade data platforms.
• Expert-level SQL (query optimization, indexing strategy, execution plans) and Python (PySpark, pipeline frameworks, testing).
• Deep hands-on experience with Azure data services: Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Data Lake Storage.
• Proven experience designing dimensional data models and data lake architecture at enterprise scale.
• Experience building data pipelines that directly support machine learning feature engineering and model serving.
• Strong background in data quality engineering—automated validation, SLA enforcement, and lineage tracking.
• Experience with relational databases (SQL Server, Oracle) and migration from legacy to cloud-native platforms.

Certifications & Licenses:

The following are considered favorable:

• Microsoft Certified: Azure Data Engineer Associate
• Databricks Certified Associate Developer for Apache Spark

Knowledge:

• Advanced SQL and Python for enterprise-scale data engineering—opti…