Position Description:

Job Title: Data Engineer ETL, Python/Scala, Apache Spark
Position: Software Engineer
Experience: 5- 7 Years
Category: Software Development/ Engineering
Shift: 1-10PM (Hybrid)
Main location: Bangalore
Position ID: J0926-0281
Employment Type: Full Time

Education Qualification: Bachelor’s degree in Computer Science or related field or higher with minimum 3 years of relevant experience.

Position Description: We are looking for an experienced Data Engineer to join our team. The ideal candidate should be passionate about coding and developing scalable and high-performance applications.

LI-BK7

Your future duties and responsibilities:

Core Data Engineering
ETL/ELT Architecture: Design and implement complex data pipelines from disparate sources into our centralized data lake/warehouse.
Big Data Processing: Leverage Apache Spark (via Dataproc) and Python/Scala to handle petabyte scale transformations.
Advanced SQL: Write and optimize complex queries for data modeling, ensuring high performance in BigQuery.
Stream Batch: Develop real time processing solutions using Pub/Sub and Dataflow (Apache Beam).
GCP Infrastructure
Orchestration: Manage workflow automation using Cloud Composer (Airflow).
Storage Databases: Architect solutions across Cloud Storage, Cloud SQL, and Spanner, choosing the right tool for the specific latency and consistency requirements.
Environment Management: Ensure security, logging, and monitoring are integrated into every pipeline.

Required qualifications to be successful in this role:

Must-Have Skills:
Programming: Mastery of Python or Scala.
Spark Expertise: Proven experience tuning and debugging Spark jobs at scale.
GCP Professional: Deep hands on experience with the full Google data suite (BigQuery, Dataflow, Dataproc, Pub/Sub).
SQL Mastery: Ability to perform complex window functions, CTEs, and query optimization.
Bonus Points (Good to Have)
AI/ML Integration: Familiarity with Vertex AI and the GCP AI Platform.

Good-to-Have Skills:
GenAI Literacy: A basic understanding of LLMs, prompt engineering, and how data pipelines feed into RAG (Retrieval Augmented Generation) architectures.
Infrastructure as Code: Experience with Terraform for managing GCP resources.

Skills:

• Airflow
• Apache Spark
• Python
• Cloud architecture
• Scala That same commitment to fairness extends to how we use technology. To support our recruitment team, AI tools may be used to help assess applications though they never replace human judgement. All hiring decisions remain entirely in the hands of our recruitment professionals.