Job Closed
AWS Data Engineer – Open Data Platform
- Source: LinkedIn
- Suncap Technology
- Remote
- New York City
- Mid level
Job Description
Title: AWS Data Engineer – Open Data Platform
Location: NYC, NY Remote ContractWe are seeking an experienced AWS Data Engineer to support the design and implementation of an AWS-based open data platform. The role will focus on building scalable data pipelines and integrating Amazon S3, AWS Glue, Apache Iceberg, and Apache Polaris with Snowflake and other analytics platforms.
Key Responsibilities
- Design and develop data ingestion, transformation, and processing pipelines using AWS Glue, Apache Spark, and Amazon S3.
- Build and manage Apache Iceberg tables supporting open, interoperable access across multiple compute engines.
- Configure and integrate Apache Polaris to Iceberg REST Catalog with AWS and Snowflake.
- Support platform security, access controls, networking, performance optimization, and operational monitoring.
- Participate in workload benchmarking and evaluate the cost and performance of AWS Glue versus Snowflake.
Required Skills
- Strong hands-on experience with Amazon S3, AWS Glue, Apache Spark, Iceberg, IAM, and AWS networking.
- Experience designing and implementing cloud-based data pipelines and large-scale data processing solutions.
- Good understanding of Apache Iceberg architecture, table maintenance, schema evolution, partitioning, and performance optimization.
- Should have Install/configuration experience in Apache Polaris and Iceberg over Amazon S3 files.
Skills
- amazon-web-services
- aws-glue
- Spark
- amazon-s3
- apache-iceberg
- aws-config
- Snowflake
- aws-iam
This job is no longer accepting applications.
Join the Club — Free ForeverYou missed this one - but you don't have to keep missing out. Try Club members get matched with open opportunities before they're gone, so you're always first in line, never last.
