Hadoop Software Engineer

Job Summary

We are seeking an experienced Software Engineer with a strong background in Spark/Lakehouse table format technologies, responsible to build and optimize our next-generation lakehouse platform.

Key Responsibilities

• Our focus is on enhancing Spark stability/usability/performance and extending Icebergs capabilities. Additionally, we are building internal storage and metadata services to power our lakehouse architecture.
• Troubleshoot and resolve production issues, ensuring minimal downtime. Maintain and improve system monitoring, logging, and alerting.

Job Requirements

• A validated, successful track record of 8+ years with strong programming skills in either Java or Scala.
• Comprehensive knowledge of Apache Spark s internals, including Catalyst optimizer, Tungsten execution engine, shuffle mechanics, AQE, and memory management.
• Solid understanding of file formats like Apache Parquet and table formats such as Apache Iceberg Delta or Apache Hudi internals, including snapshots, manifests, partition specs, and table metadata.
• Familiarity with Linux/Unix systems, including shell scripting and system commands. Understanding of networking principles, as Hadoop often operates in distributed environments. Overall, the candidate is expected to have analytical skills to tackle complex distributed system challenges and optimize solutions.
• Excellent problem solving and analytical skill on both defect and performance issue resolution.
• Good team player and able to work with global team
• Demonstrated proficiency building production-grade software with AI coding agents, improving engineering velocity while maintaining high standards for quality, availability, and maintainability.
• Ability to decompose problems and provide agents with the right context, specifications, constraints, and validation criteria
• Ability to capture and express business, architecture, codebase, and operational knowledge so AI coding agents can execute complex tasks accurately and autonomously.
• Hands-on experience delivering complex, production-quality software using AI coding tools such as Claude Code, Codex, etc.

Plus Points

• Contribution to related open source projects like Apache Spark/Iceberg
• Experience in designing and implementing distributed systems.
• Good understanding of other Hadoop related projects including HDFS/HBase/Yarn

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.