Big Data Engineer – Scala, PySpark & AWS

Job Description: Big Data Engineer – Scala, PySpark & AWS

Job Title: Big Data Engineer
Experience: 5–6 Years
Employment Type: Contract (C2C)
Work Mode: Remote
Working Hours: 11:00 AM – 9:00 PM IST

Job Summary

We are looking for an experienced Big Data Engineer with strong hands-on expertise in Scala, PySpark, AWS, and AWS Glue to develop, maintain, and optimize scalable data processing pipelines.

The ideal candidate will have experience working with Parquet file formats, data profiling, data quality checks, and rule-based data validation frameworks. The role also requires familiarity with AI-assisted coding tools such as GitHub Copilot or Codex to improve development efficiency and code quality.

Key Responsibilities

• Design, develop, and optimize scalable big data pipelines using Scala, PySpark, and AWS Glue.
• Build and maintain rule-based data validation engines to ensure data accuracy, consistency, and quality.
• Work with Parquet and other big data file formats to optimize data storage and processing.
• Perform data profiling, validation, debugging, and performance tuning.
• Develop clean, reusable, efficient, and well-tested code.
• Implement data quality checks and validation rules across data processing workflows.
• Use enterprise-approved AI coding assistants, such as GitHub Copilot or Codex, to improve development productivity.
• Collaborate with business stakeholders, data architects, and engineering teams to translate requirements into technical solutions.
• Participate in code reviews, testing, troubleshooting, and continuous improvement initiatives.
• Evaluate emerging technologies and AI-assisted development practices relevant to big data engineering.

Required Skills and Qualifications

• Overall Experience: 5–6 years in software engineering or big data engineering.
• Big Data Pipelines: 4+ years of experience developing and maintaining data pipelines.
• Scala and PySpark: 3+ years of hands-on programming experience.
• AWS and AWS Glue: 3+ years of experience building and supporting cloud-based data solutions.
• Parquet and Big Data Formats: 3+ years of experience working with Parquet and similar file formats.
• Data Validation and Profiling: At least 1 year of experience with data quality checks, profiling, or rule-based validation engines.
• AI Coding Assistants: At least 1 year of experience using AI-assisted coding tools or code-generation solutions in daily development workflows.
• Strong knowledge of data processing, transformation, and optimization techniques.
• Good understanding of data quality frameworks and validation strategies.
• Strong analytical, problem-solving, and communication skills.

Preferred Candidate Profile

• Experience building high-performance and scalable data processing solutions.
• Ability to troubleshoot complex data pipeline issues and optimize processing performance.
• Strong focus on code quality, maintainability, and automated testing.
• Experience collaborating with cross-functional teams in an Agile environment.
• Willingness to adopt AI-assisted development tools and emerging data engineering technologies.

Additional Information

• Contract Type: C2C
• Work Location: Remote
• Shift: 11:00 AM – 9:00 PM IST

Pay: ₹90,000.00 - ₹1,110,000.00 per year

Experience:

• software engineering: 5 years (Required)
• Big Data Pipelines: 4 years (Required)
• Scala and PySpark: 3 years (Required)

Work Location: Remote