Senior Lead Software Engineer - Java & Databricks

Be an integral part of an agile team that’s constantly pushing the envelope to enhance, build, and deliver top-notch technology products. 

As a Senior Lead Software Engineer at JPMorganChase within the Digital Communications Compliance team, you are an integral part of an agile team that works to enhance, build, and deliver trusted market-leading technology products in a secure, stable, and scalable way. Drive significant business impact through your capabilities and contributions, and apply deep technical expertise and problem-solving methodologies to tackle a diverse array of challenges that span multiple technologies and applications. 

Job responsibilities

• Design and develop scalable, fault-tolerant microservices and APIs that support rule-based and ML-based detection pipelines
  Model and implement supervision and reviewer workflows using state machines
• Build streaming and batch data pipelines that ingest, index, and enrich communications content and alerts
• Design data models on Databricks for surveillance, search, retention, and life cycle management at scale
• Develop and optimize large-scale batch and streaming data pipelines using Databricks, Apache Spark, and Delta Lake
• Build robust unit, integration, and performance tests following Test-Driven Development. Leverage Databricks Workflows, notebooks, and CI/CD pipelines to automate data processing and deployments. Troubleshoot and tune Databricks jobs, clusters, and data pipelines for efficiency and performance
  Drive adoption of Databricks best practices for data engineering, software engineering, observability, and platform operations
• Drives adoption and governance of approved AI-assisted engineering practices across teams to improve code quality, delivery speed, and operational outcomes (e.g., AI-assisted code review/refactoring, test acceleration, release readiness, incident/root-cause analysis), while establishing measurable validation standards (secure coding, peer review, automated testing) and promoting reuse of proven patterns and automation within the SDLC/TLM toolchain.
• Applies knowledge of tools within the Software Development Life Cycle toolchain, including approved AI-assisted development and automation capabilities, to improve the value realized by automation at scale.
• Partner with product management and compliance SMEs to monitor and improve alert accuracy and reliability. Proactively identify hidden problems and patterns in communications data to improve detection quality

Required qualifications, capabilities, and skills

• Formal training or certification on software engineering concepts and 5+ years applied experience
• Hands-on practical experience delivering system design, application development, testing, and operational stability
• Expertise in building resilient, scalable, enterprise-grade cloud-native products, with strong exposure in compliance for the financial industry
• Hands-on Databricks experience including development of production workloads using Spark, Delta Lake, and Databricks Workflows. Expert Java/Kotlin and Python programmer with experience building headless, externally consumable APIs. Experience building cloud-native microservices for streaming and batch architectures using Spark and/or Flink
• Proficient with AWS services including EC2, ECS, EKS, EMR, S3, and Glacier. Hands-on with Semantic search, Kafka, and PostgreSQL. Experience integrating and operationalizing ML/LLM models and pipelines in production. Experience with observability and monitoring tools such as Prometheus, Grafana, and OpenTelemetry
• Experience building CI/CD pipelines using ArgoCD, Helm, Terraform, Jenkins, and GitHub Actions
• Prior experience in Test-Driven Development, delivering products with well-defined SLI/SLO/SLAs
• Strong ownership mentality with excellent communication skills and a collaborative mindset. Experience mentoring engineers and providing technical leadership at a senior level

Preferred qualifications, capabilities, and skills

• Exposure to Unity Catalog, and Databricks Workflows
• Exposure to building and optimizing large-scale batch and streaming data pipelines in cloud-native environments
• Understanding of distributed systems, data engineering best practices, and operationalizing data and ML workloads in production