Senior AWS Data Engineer (Spark / EMR / Glue / Iceberg)

We are looking for an experienced Senior AWS Data Engineer with strong expertise in Apache Spark, AWS EMR, AWS Glue, Apache Iceberg, Data Lakehouse Architecture, and AWS Cloud Data Services. The ideal candidate will be responsible for designing, developing, and optimizing scalable data ingestion, transformation, and analytics pipelines while supporting enterprise-wide data engineering and modernization initiatives.

The role requires hands-on experience in building cloud-native data platforms, batch and streaming architectures, data governance frameworks, and high-performance data lakehouse solutions.

Key Responsibilities

Data Engineering & Lakehouse Development

• Design and develop scalable data ingestion and transformation pipelines.
• Build enterprise-grade Data Lakehouse architectures using Apache Iceberg.
• Implement high-performance data processing solutions using Apache Spark.
• Develop reusable and scalable data engineering frameworks.

AWS Data Platform Engineering

• Design and implement data solutions using:

• AWS EMR
• AWS Glue
• Amazon S3
• AWS Lambda

• AWS Analytics Services

• Build secure, scalable, and reliable data platforms on AWS.
• Optimize storage, processing, and data access strategies.

Batch, Streaming & Event-Driven Architectures

• Develop batch and real-time data processing pipelines.

• Implement:

• Streaming Architectures
• Event-Driven Architectures

• CDC (Change Data Capture) Solutions

• Enable near real-time analytics and data delivery.

Data Quality & Governance

• Implement data quality monitoring and validation frameworks.
• Establish data lineage and governance processes.
• Ensure compliance with enterprise data standards and best practices.
• Maintain metadata, cataloging, and auditing capabilities.

Performance Optimization

• Optimize Spark jobs and AWS workloads for performance and cost efficiency.
• Tune data pipelines, storage structures, and processing frameworks.
• Improve throughput, scalability, and reliability of data platforms.

Collaboration & Delivery

• Work closely with data architects, business teams, analytics teams, and platform engineers.
• Participate in Agile delivery processes and sprint activities.
• Contribute to code reviews and engineering best practices.
• Support deployment, testing, and production operations.

Required Skills

• 6-10 years of experience in Data Engineering.

• Strong expertise in:

• Apache Spark
• PySpark
• AWS EMR
• AWS Glue

• Apache Iceberg

• Experience building:

• Data Lakehouses
• Data Pipelines

• Data Integration Solutions

• Hands-on experience with:

• Batch Processing
• Streaming Pipelines
• CDC Frameworks

• Event-Driven Architectures

• Strong SQL and Data Modeling skills.
• Experience working with AWS Cloud services.
• Understanding of Data Governance and Data Lineage concepts.
• Experience with performance tuning and optimization.

Preferred Skills

• Apache Kafka.
• AWS Kinesis.
• Delta Lake.
• Airflow.
• Data Catalog and Metadata Management frameworks.
• Data Quality Frameworks.
• CI/CD for Data Engineering.
• Banking or Financial Services experience.

Desired Candidate Profile

• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management abilities.
• Experience working with enterprise-scale data platforms.
• Ability to work independently and collaboratively in Agile teams.
• Strong ownership mindset and commitment to engineering excellence.