Software Development Engineer III

About the role

We are seeking a highly experienced Data Engineering AI Engineering Lead (SDE3) to architect and drive the development of next-generation data and AI platforms. The ideal candidate will be a technical leader with deep expertise in Databricks, Apache Spark, Scala, large-scale distributed systems, and modern Data Engineering practices, Azure Cloud Architecture and Platform Engineering, combined with strong experience in Agentic AI, Enterprise AI solutions, LLM evaluation frameworks, and Agentic AI orchestration.

This role requires someone who can translate business requirements into scalable technical solutions, lead engineering teams through complex implementations, establish engineering standards and best practices, and drive end-to-end delivery of data and AI products. If you are passionate about solving large-scale data challenges, building enterprise-grade AI systems, and mentoring high-performing teams, wed love to hear from you

You will be responsible for

• Architect, design, and build highly scalable, secure, and resilient Data Engineering platforms using Databricks, Apache Spark, and Scala.
• Lead the design and implementation of large-scale distributed systems, ensuring high availability, performance, and operational excellence.
• Define and drive the technical roadmap for data platforms, lakehouse architectures, and enterprise data products.
• Design and implement advanced ETL/ELT pipelines capable of processing large-scale batch and real-time workloads.
• Build and govern engineering standards, coding guidelines, architecture principles, and operational best practices across teams.
• Lead end-to-end feature development, from requirement gathering and solution design to implementation, deployment, and production support.
• Collaborate with Product Managers, Architects, Platform Teams, and Business Stakeholders to translate business goals into scalable technical solutions.
• Drive technical excellence through code reviews, architecture reviews, performance optimization, and engineering mentorship.
• Design and implement observability frameworks including monitoring, logging, tracing, alerting, and reliability engineering practices.
• Establish and maintain robust CI/CD pipelines and DevOps practices for data and AI platforms.
• Lead the adoption and implementation of Agentic AI and Enterprise AI solutions across business use cases.
• Design, develop, and orchestrate multi-agent AI systems utilizing modern agent frameworks and orchestration platforms.
• Define and implement LLM evaluation frameworks including quality, reliability, accuracy, safety, performance, and cost optimization metrics.
• Guide teams in selecting the right architecture patterns, technology stack, and engineering practices for long-term success.
• Mentor and develop engineers, enabling them to grow technically and professionally while fostering a culture of ownership and innovation.
• Continuously evaluate emerging technologies, AI advancements, and industry best practices to drive innovation and strategic differentiation.

You will need

• 9+ years of professional software engineering experience with a strong focus on Data Engineering and distributed systems.
• Expert-level experience with Databricks, Apache Spark, and Scala/Python.
• Deep expertise in designing and building scalable, high-throughput, low-latency data platforms and distributed systems.
• Strong understanding of Data Structures, Algorithms, System Design, and Software Engineering principles.
• Proven experience architecting and implementing large-scale ETL/ELT pipelines in enterprise environments.
• Strong expertise in modern Data Lake, Lakehouse, and Data Platform architectures.
• Hands-on experience implementing CI/CD pipelines using tools such as GitHub Actions, Azure DevOps, Jenkins, or similar platforms.
• Experience building observability solutions including monitoring, logging, tracing, alerting, and reliability engineering practices.
• Strong knowledge of data governance, security, privacy, compliance, and data quality frameworks.
• Experience gathering product requirements and converting business needs into scalable technical solutions.
• Demonstrated ability to lead technical initiatives, influence architecture decisions, and drive execution across multiple teams.
• Strong stakeholder management and communication skills, with the ability to work effectively across engineering, product, and business teams.
• Pr…