Hiring Technical Lead Data Engineering | Kafka, AWS, PySpark, Scala

**We are Hiring Technical Lead Data Engineering Kafka, AWS, Spark (Scala/PySpark), Kubernetes/EKS**

Company: Coforge Ltd.

Job Location: Hyderabad, Pune & Greater Noida Only.

Experience: 8 to 12 Years

Employment Type: Full-Time

Key Skills:- Kafka Must Have, Spark, Scala, PySpark, AWS, S3, Glue, EMR, Lambda, Kinesis, Athena, Redshift, EKS, Kubernetes, Docker, Helm, Python, SQL, Terraform, CI/CD, ETL, ELT, Real-Time Data Pipelines, Data Engineering, Technical Leadership.

Share your CV: [email protected]

If you have any questions regarding this opportunity, please feel free to contact us on WhatsApp at: 9667427662 ( Gaurav Kumar HR, Coforge Ltd )

Role Overview:-

We are seeking an experienced Technical Lead Data Engineering to lead a team of Data Engineers in designing, developing, and supporting large-scale, cloud-native data platforms.

The ideal candidate will possess strong expertise in Kafka, AWS Data Engineering services, Spark (Scala/PySpark), and Kubernetes/EKS, along with proven experience in technical leadership, architecture, and stakeholder management.

The role requires hands-on technical leadership while driving engineering best practices, mentoring team members, and ensuring successful delivery of scalable, secure, and high-performance data solutions.

Key Responsibilities:-

Technical Leadership

• Lead and mentor a team of Data Engineers across design, development, testing, and production support.
• Drive architecture and technical design decisions for enterprise-scale data platforms.
• Establish engineering standards, coding practices, CI/CD processes, and operational excellence.
• Conduct code reviews and provide technical guidance to team members.
• Collaborate with architects, product owners, and business stakeholders to translate requirements into scalable solutions.

Data Engineering & Platform Development

• Design and develop real-time and batch data pipelines using Kafka, Spark, Scala, and PySpark.
• Build scalable data processing frameworks on AWS.
• Develop and optimise ETL/ELT pipelines for high-volume structured and unstructured datasets.
• Ensure data quality, governance, observability, and operational monitoring.
• Troubleshoot performance bottlenecks and optimise data processing workloads.

Cloud & Kubernetes

• Deploy and manage data workloads on Amazon EKS (Elastic Kubernetes Service).
• Implement containerised applications using Docker and Kubernetes.
• Develop infrastructure automation and deployment pipelines.
• Work closely with DevOps teams to implement CI/CD and infrastructure-as-code practices.

Stakeholder & Delivery Management:-

• Participate in sprint planning, estimation, and technical roadmaps.
• Manage technical risks, dependencies, and delivery commitments.
• Provide status updates to leadership and stakeholders.
• Support production issues and critical incident resolution.

Required Skills

Data Engineering:-

• Strong experience with Apache Kafka including producers, consumers, Kafka Streams, schema management, and event-driven architectures.
• Expertise in Apache Spark using Scala and PySpark.
• Experience designing and supporting real-time and batch data processing solutions.
• Strong understanding of distributed systems and big data concepts.

AWS Data Engineering - Hands-on experience with:

• AWS S3
• AWS Glue
• AWS EMR
• AWS Lambda
• AWS Kinesis
• AWS Athena
• AWS Redshift
• AWS IAM
• CloudWatch
• Step Functions

Kubernetes & Containers:

• Strong expertise in Amazon EKS
• Kubernetes administration and deployment strategies
• Docker containerisation
• Helm Charts
• Autoscaling and resource optimisation

Programming:

• Scala
• Python
• SQL
• Shell Scripting

DevOps:

• Git/GitHub/GitLab
• Jenkins/Azure DevOps/GitHub Actions
• Terraform (preferred)
• CI/CD pipelines

Leadership Expectations:-

• Lead distributed teams and provide technical mentorship.
• Drive solution design discussions and architecture reviews.
• Ensure adherence to security, compliance, and operational standards.
• Foster innovation and continuous improvement within the team.
• Act as the primary technical escalation point for critical issues.

Preferred Qualifications:-

• Experience working in Bank…