AWS Data Engineer Lead

Job Title:AWS Data Engineer Lead

Primary Skill: AWS, DynamoDB, Aurora PostgreSQL, AWS Sagemaker, Spark, Scala/Python,Medallion Architecture

Location: Hyderabad/ Pune/ Coimbatore

Experience: 7-10 years of experience

Mode of working - 4 Days work from office

We are looking for an experienced AWS Data Engineer with strong handson expertise in building scalable, reliable data pipelines in a Lakehouse/Medallion architecture. The ideal candidate will have deep knowledge of Spark/PySpark, Terraform, and modern data engineering patterns.

Key Responsibilities

• Build and operate data pipelines using PySpark/Spark, SQL, and AWS services.
• Implement Medallion (Bronze/Silver/Gold) architecture with proper separation of concerns.
• Design incremental processing, SCD Type 2, CDC/merge, and late-arriving/outofsequence data handling.
• Ensure pipelines are idempotent, resilient, and reprocessable.
• Manage AWS data infrastructure using Terraform.
• Apply strong data quality, deduplication, schema validation, and operational best practices.

Required Skills

• Terraform for AWS infrastructure provisioning
• PySpark/Spark, Python, Advanced SQL
• Lakehouse/Medallion Architecture (Bronze/Silver/Gold)
• Type 2 Dimensions, CDC/Merge processing
• Incremental & historical processing, deduplication, late-arriving data handling
• Idempotency and resilient pipeline patterns
• AWS data services (S3, Glue, EMR, Lambda, Step Functions, etc.)
• Data modeling fundamentals & data quality mindset

Candidates Must Be Able to Explain

• Updates/deletes from relational sources
• Deterministic deduplication
• Late-arriving/out-of-sequence data handling
• Idempotent pipeline design
• Merge keys, ordering columns, event time vs ingest time, soft deletes, reprocessing windows