AWS Glue Data Engineer - (Blr/Pune/Hyd/BBSR/Chennai/Kochi)

Key Responsibilities

• Design and develop scalable ETL workflows using AWS Glue.

• Build and optimize Glue components such as:

• Crawlers
• Jobs
• Workflows
• Triggers

• Data Catalog

• Develop PySpark-based transformations and data processing pipelines.

• Process structured and semi-structured data formats including:

• JSON
• XML

• CSV

• Implement metadata management using Glue Data Catalog and Crawlers.
• Develop custom transformations using AWS Glue libraries.
• Design and implement data quality solutions using AWS Glue capabilities.
• Manage AWS Glue costs and perform DPU capacity planning.
• Implement security controls, encryption, access management, and auditing.
• Optimize Spark workloads running in Hadoop-based environments.
• Collaborate with architects, data engineers, and business teams to deliver reliable data solutions.

Must-Have Skills

AWS Glue

• Strong hands-on experience with:

• AWS Glue Jobs
• Glue Crawlers
• Glue Triggers
• Glue Workflows

• Glue Data Catalog

• Experience developing enterprise-scale ETL solutions.

PySpark & Big Data

• Strong experience with:

• PySpark
• Apache Spark

• Hadoop ecosystem

• Experience optimizing Spark jobs and troubleshooting performance issues.

Data Engineering

• Strong understanding of:

• ETL/ELT concepts
• Data integration patterns
• Data transformation frameworks

Data Quality & Governance

• Experience implementing:

• Data quality checks
• Data validation processes
• Metadata management

AWS Security

• Knowledge of:

• Data encryption
• IAM permissions
• Access control
• Audit and compliance frameworks

Good-to-Have Skills

• Experience with:

• AWS Streaming Solutions
• AWS Lambda
• S3
• Athena

• Redshift

• Exposure to:

• Real-time data processing
• Data Lake architectures
• Data governance frameworks