Lead Software Engineer - Data Engineer + Python + Spark/Pyspark+ AWS

JOB DESCRIPTION

We have an opportunity to impact your career and provide an adventure where you can push the limits of what’s possible.

As a Lead Software Engineer - Data Engineering at JPMorgan Chase within Global Banking Technology, you will lead the design and delivery of reliable, scalable data platforms and pipelines that power business-critical use cases, including analytics, search/retrieval, and AI-assisted workflows. You will be accountable for building high-quality curated datasets with clear contracts, lineage expectations, and measurable SLAs/SLOs, while ensuring strong controls across security, privacy, resiliency, and auditability. You will remain hands-on and will set the technical bar for engineering rigor across Java and Python implementations, including batch/stream processing, microservices, and shared libraries. You will also partner with product, UX, and platform teams to enable AI/ML and agentic patterns where they materially improve business outcomes, without compromising governance or operational discipline.

Job Responsibilities

• Design, build, and operate batch and streaming data pipelines that are reliable, observable, and cost-efficient, with clear runbooks and production support ownership.
• Lead data modeling and curation for domain datasets, including schema evolution, data contracts, lineage expectations, and consumer-facing SLAs/SLOs.
  Implement robust ETL/ELT workflows with strong validation controls, including reconciliation, completeness checks, anomaly detection, and automated alerting.
• Engineer high-throughput data processing solutions using a combination of Java and Python, selecting the right tool for performance, maintainability, and platform standards.
• Build and operate orchestration capabilities (for example, Airflow or equivalent), including scheduling, backfills, retries, dependency management, and operational SLAs with end-to-end ownership across architecture, engineering standards, CI/CD, and operational stability in a regulated enterprise context.
• Deliver transformation pipelines using modern transformation frameworks (for example, dbt or equivalent), with strong testing, repeatability, and release discipline.
• Develop and maintain supporting services and APIs (REST and/or gRPC) that expose curated data products and enable downstream consumers, using clean architecture and well-defined contracts.
• Build and maintain search and indexing pipelines (for example, Elasticsearch) that support discovery, retrieval, analytics, and RAG-style experiences, and establish engineering and data quality standards across code review, automated testing, performance tuning, observability (logs/metrics/traces), resiliency patterns, and incident response.
• Partner with security, risk, and controls teams to ensure data solutions meet governance expectations, including access controls, secrets handling, least privilege, and auditability.
• Enable secondary AI/ML capabilities by delivering the data foundations required for evaluation, guardrails, tool/function integration, and traceable AI-assisted workflows, including MCP-style integration patterns where applicable.
• Drives team adoption of enterprise-authorized AI-assisted engineering practices within the work environment to improve code quality, delivery speed, and operational outcomes (e.g., AI-assisted code review/refactoring, test strategy acceleration, incident/root-cause analysis support), while establishing consistent validation standards (secure coding, peer review, automated testing) and promoting reuse of effective patterns across the team.
• Applies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation.

Required qualifications, capabilities, and skills

• Formal training or certification on software engineering concepts and 5+ years applied experience
• Hands-on engineering experience delivering production-grade platforms and data systems, with demonstrated recent experience as a lead Data Engineer building and operating curated datasets and production pipelines end-to-end.
• Strong hands-on proficiency in both Java and Python in production environments, including performance-minded development, design patterns, and maintainable codebases.
• Advanced SQL skills, with strong capability in data modeling, schema design, and schema evolution. with p…