Principal, Software Engineer

About Team Our team builds and operates a large-scale internal platform that provides real-time streaming, messaging, telemetry, and data-processing capabilities for Walmart technology teams. The platform supports business-critical workloads across event ingestion, stream processing, messaging, observability, large-scale persisted data processing, and data-lake use cases.

We work closely with Product Management, internal customers, and dependent engineering teams to expand platform capabilities, onboard new workloads, improve reliability and performance, and operate the platform efficiently at scale.

What you’ll do

• Serve as a hands-on Principal Engineer, contributing directly to architecture, detailed design, coding, code reviews, debugging, testing, deployment, and production operations.
• Own complex technical initiatives end to end, from requirements clarification and solution design through implementation, rollout, production validation, and ongoing operation.
• Partner with Product Management to translate product direction and customer needs into scalable technical solutions within the existing platform architecture.
• Design and evolve high-throughput event-ingestion, streaming, messaging, and telemetry systems, including efficient batching, routing, serialization, compression, secure data handling, and data delivery.
• Build and improve large-scale real-time and persisted data-processing solutions using technologies such as Apache Spark, Spark Structured Streaming, Apache Flink, Apache Kafka, and comparable distributed technologies.
• Identify and resolve significant issues involving latency, data-processing delays, scalability, reliability, performance, data integrity, and resource utilization, driving important problems from investigation through permanent production improvement.
• Engineer systems for resiliency and failure recovery, including retries, backpressure, fault isolation, failover, fallback strategies, disaster recovery, and prevention of silent data loss.
• Design for efficient use of compute, memory, network, and storage resources, and drive measurable cost improvements while maintaining or improving system performance and reliability.
• Build and improve large-scale data pipelines and data-lake capabilities, including ingestion, transformation, ETL, partitioning, storage, querying, and data-quality diagnostics.
• Drive operational excellence through observability, metrics, distributed telemetry, dashboards, alerting, profiling, capacity planning, peak-traffic readiness, incident response, Root Cause Analysis, and disaster-recovery exercises.
• Improve engineering productivity and production reliability through CI/CD automation, Kubernetes-based deployments, release engineering, configuration management, and secure credential and secret management.
• Collaborate with infrastructure, messaging, platform, data, application, and other dependent teams to resolve technical dependencies and drive cross-team initiatives to completion.
• Mentor engineers and raise the technical bar through design reviews, code reviews, debugging, benchmarking, technical documentation, and hands-on engineering guidance.
• Develop a deep understanding of the existing platform, deliver meaningful improvements within its architecture, and use production experience and data to guide thoughtful architectural evolution over time.
• Participate in the team’s production on-call rotation, including weekend coverage, and provide technical leadership during complex production incidents.

What you’ll bring

• Deep experience designing, building, and operating large-scale distributed systems with demanding throughput, latency, scalability, availability, and reliability requirements.
• Strong hands-on software engineering skills with deep proficiency in Java and at least one additional systems or backend programming language. Hands-on experience with Go is strongly preferred.
• Strong experience with Apache Kafka or comparable messaging technologies, including event-driven architectures, partitioning, producer behavior, delivery semantics, retries, failure handling, and secure connectivity.
• Strong experience with Apache Spark / Structured Streaming, Apache Flink, or comparable large-scale streaming and data-processing technologies.
• Experience building high-volume event-ingestion, telemetry, or data-processing systems, with an understanding of batching, serialization, compression, encryption, rout…