Sr. AWS Data engineer

Hi

Please read the below JD and the send profiles

**Job Title: Senior Data Engineer – Scala to PySpark Conversion AWS**

Location : New Jersey, New York, or Fort Mill SC (Hybrid)

Please share only 14+ years profiles

Note: H4 EAD, USC, L2 EAD Only (W2)

Please don’t share OPT or H1B and Don’t waste your time by share OPT or H1B profiles

Need someone who has started as a Big Data or Scala Developer and now a Senior AWS Data Engineer, with strong PySpark coding with Scala job work.

Key Responsibilities

• Analyze and understand existing Scala/Spark-based ETL pipelines and data processing applications .
• Convert and migrate Scala Spark code to PySpark , ensuring functional and data processing equivalence.
• Refactor existing Spark jobs to improve performance, scalability, maintainability, and code quality .
• Design, develop, and maintain ETL pipelines using AWS Glue, Glue Studio, and Glue Data Catalog .
• Build and optimize PySpark applications for large-scale data processing and complex transformations.
• Work with AWS S3, Redshift, Athena, Lambda, and Step Functions for data storage, processing, querying, and workflow orchestration.
• Migrate existing Scala-based data pipelines to PySpark while maintaining data accuracy and business logic.
• Perform code reviews and identify opportunities to optimize Spark transformations, joins, partitioning, caching, and resource utilization..
• Ensure data solutions comply with security, governance, and regulatory requirements, preferably within the BFSI domain .
• Participate in Agile/Scrum ceremonies and contribute to technical design and solution discussions.

Key Skills

Primary: PySpark, Python, Scala, Apache Spark, AWS Glue

AWS: S3, Glue, Glue Data Catalog, Redshift, Athena, Lambda, Step Functions, CloudWatch

Data: ETL/ELT, Data Lakes, Data Warehousing, Data Modeling, Data Quality

Migration: Scala-to-PySpark Conversion, Spark Modernization, Code Refactoring, Performance Optimization

Database: SQL, Redshift, Relational Databases

Methodology: Agile, Scrum, CI/CD