Lead AWS Data Engineer

Lead AWS Data Engineer

Experience: 7+ Years

Location: Hyderabad(On-Site)

Notice Period: Immediate Joiner Only

Role Overview

We are looking for a Lead AWS Data Engineer with 8+ years of experience to lead the design and development of scalable, reliable data pipelines on AWS. The candidate should be highly hands-on with PySpark, SQL, Terraform, AWS data services, and Lakehouse/Medallion architecture .

Key Responsibilities

• Lead the design and development of scalable AWS data pipelines using PySpark/Spark, Python, and SQL.
• Implement Bronze/Silver/Gold (Medallion) architecture .
• Design incremental processing, CDC, SCD Type 2, Merge/Upsert , and historical data solutions.
• Handle deduplication, late-arriving/out-of-sequence data, updates, and deletes .
• Build idempotent, resilient, and re-processable pipelines.
• Provision and manage AWS infrastructure using Terraform .
• Work with S3, Glue, EMR, Lambda, Step Functions , and related AWS services.
• Drive data quality, performance optimization, monitoring, and production support.
• Provide technical leadership, conduct code/design reviews, and mentor data engineers.

Required Skills

• 8+ years of Data Engineering experience , with Lead/Senior-level ownership.
• Strong PySpark/Spark, Python, and Advanced SQL .
• Strong AWS data engineering experience.
• Hands-on Terraform / Infrastructure as Code .
• Strong knowledge of Lakehouse/Medallion architecture .
• Experience with CDC, SCD Type 2, incremental processing, deduplication, and historical processing .
• Strong understanding of idempotency, merge keys, ordering columns, event time vs. ingestion time, and reprocessing .

Good communication, leadership, and problem-solving skills.