Lead Data Engineer – AWS, Databricks & PySpark

**FULLY REMOTE**

ScolerTec is seeking Lead Data Engineer – AWS, Databricks & PySpark to design, build, and lead enterprise-scale cloud data engineering solutions.

This is a hands-on technical leadership role responsible for scalable ETL/ELT pipelines, batch and near-real-time data integration, lakehouse/data-platform implementation, cloud data services, CI/CD, data quality, governance, performance optimization, and technical direction of data engineers.

The ideal candidate has strong recent experience with AWS, Databricks, Apache Spark/PySpark, Python, SQL, Delta Lake, ETL/ELT, and technical leadership .

Key Responsibilities

• Design and implement scalable cloud-based data platforms, including data lake, lakehouse, and data warehouse architectures.
• Build enterprise-scale ingestion and ETL/ELT pipelines for batch and near-real-time workloads.
• Develop data transformation and integration solutions using Python, SQL, Spark/PySpark, and modern cloud data technologies.
• Implement Databricks/Delta Lake-based data engineering solutions and downstream curated/gold data layers.
• Integrate pipelines with AWS compute, storage, messaging, and analytics services.
• Lead technical design reviews, code reviews, engineering standards, and reusable data-engineering patterns.
• Provide day-to-day technical guidance and mentoring to data engineers.
• Implement data-quality checks, validation rules, error handling, observability, and pipeline monitoring.
• Support streaming and event-driven data processing.
• Implement Infrastructure as Code and CI/CD for repeatable data-platform and pipeline deployment.
• Optimize pipeline performance, scalability, reliability, and cloud cost.
• Support production incident resolution and root-cause analysis.
• Partner with architecture, governance, migration, QA, DevSecOps, and security teams.
• Maintain technical documentation, runbooks, and audit-ready engineering artifacts.

Qualifications

• 7+ years of specialized data-engineering experience.
• Strong hands-on experience building enterprise-scale data pipelines.
• Strong production experience with AWS data and compute services.
• Strong hands-on Databricks experience.
• Strong Apache Spark/PySpark and Delta Lake experience.
• Strong Python and SQL skills.
• Strong ETL/ELT and data-transformation experience.
• Experience with batch and streaming/near-real-time ingestion.
• Experience with data orchestration tools.
• Experience with CI/CD and Infrastructure as Code.
• Experience with data quality, governance, metadata, and classification.
• Proven technical leadership, design review, code review, and mentoring experience.
• Federal, public-sector, or regulated-environment experience preferred.

Preferred Certifications

• Databricks Certified Data Engineer Professional
• AWS Certified Data Engineer
• AWS Certified Solutions Architect – Professional
• AWS Certified Data Analytics – Specialty
• CDMP