SENIOR SOFTWARE ENGINEER - ETL

• Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks and PySpark.
• Develop and optimize data transformation logic using PySpark, Spark SQL, and SQL.
• Build and maintain Delta Lake tables following Bronze, Silver, and Gold/Medallion architecture patterns.

• Implement Delta Lake capabilities such as:

• MERGE operations
• Schema enforcement and schema evolution
• Time Travel
• OPTIMIZE and VACUUM

• Incremental data processing

• Develop and manage Databricks Workflows/Jobs for pipeline orchestration and scheduling.
• Troubleshoot pipeline failures, performance issues, and data quality problems.
• Optimize Spark workloads by understanding partitioning, joins, caching, shuffle operations, and query execution.
• Develop reusable data engineering utilities and frameworks using PySpark and basic Python.
• Write complex SQL queries for data transformation, reconciliation, validation, and analysis.
• Integrate Databricks pipelines with external systems using REST APIs.
• Handle inbound and outbound file integrations using SFTP.
• Work with common file formats such as CSV, JSON, Parquet, and Delta.
• Work with AWS services used within the data engineering ecosystem, particularly Amazon S3, and have basic awareness of services such as IAM, Secrets Manager, CloudWatch, and Lambda.
• Implement proper exception handling, logging, auditing, monitoring, and restart/recovery mechanisms.
• Participate in code reviews and follow coding and data engineering best practices.
• Collaborate with architects, business analysts, source-system teams, QA teams, and other engineering teams.
• Support production deployments and troubleshoot production data pipeline issues.

ETL