Principal Software Engineer - AI SRE

Job Summary

Harness is seeking a Principal Software Engineer to shape the architecture and direction of AI SRE. You will build scalable platforms and AI investigators that help engineering teams understand incidents, correlate evidence, identify likely causes, and determine the next action.

This role combines hands-on backend and distributed-systems expertise with organization-wide technical leadership, mentorship, and customer engagement.

About AI SRE

Harness AI SRE helps teams investigate and resolve production incidents by reasoning across deployments, source code, pull requests, alerts, telemetry, feature flags, documentation, runbooks, and previous incidents.

We are building systems that investigate, reason, and act-not simply answer questions. This requires reliable distributed systems, intelligent retrieval and agentic workflows, and experiences engineers can trust during high-pressure incidents.

Principal Engineers at Harness

Principal Engineers possess a blend of domain expertise and world-class technical leadership. They influence architecture and product strategy across teams, solve the organization s most difficult technical problems, develop engineers, and translate ambiguous customer needs into scalable solutions.

Technical Competencies

• Strong backend, distributed systems, databases, and reliability expertise.
• Deep understanding of incident management, observability, SLOs, and RCA.
• Experience with AI/agentic systems, retrieval, and engineering data.
• Strong problem-solving and architectural judgment.
• Ability to balance technical decisions with product and customer needs.

Leadership Competencies

• Lead complex, cross-team initiatives with clear technical direction.
• Mentor engineers and raise engineering quality.
• Drive simplification, reusable platforms, and better execution.
• Make strategic decisions around risk, resources, and build-vs-buy.
• Strong communication, collaboration, and organizational impact.

Key Responsibilities

• Define architecture for incident management, on-call, reliability, and AI investigation workflows.
• Build scalable Java services, event-processing systems, data platforms, and cloud infrastructure.
• Develop AI investigators that correlate code, deployments, alerts, telemetry, and organizational knowledge.
• Drive agentic workflows, retrieval pipelines, and evidence-based reasoning.
• Coordinate initiatives across backend, AI/ML, integrations, frontend, Infrastructure, Product, and Design.
• Improve production reliability, scalability, performance, security, and observability.
• Solve complex problems involving real-time correlation, multi-tenancy, AI accuracy, and integration reliability.
• Create reusable platform capabilities and prevent duplicated engineering effort.
• Support roadmap and resource planning while responding to incidents and customer escalations.
• Mentor engineers and represent Harness as an AI SRE and reliability engineering expert.

About You

• 12+ years of professional software development experience.
• Deep experience building large-scale backend systems, preferably in Java.
• Strong knowledge of object-oriented design, algorithms, concurrency, transactions, and distributed systems.
• Experience designing microservices and highly available production systems.
• Production ownership, on-call, incident response, and observability experience.
• Proven ability to lead ambiguous, multi-team initiatives from concept to production.
• Strong technical mentorship, architectural leadership, and product judgment.
• Active use of AI-assisted development tools and understanding of their strengths and limitations.

What Were Looking For

• Strong Backend Engineering - Deep Java expertise, distributed systems, databases, concurrency, scalability, and sound architectural decision-making.
• Production Engineering Mindset - Experience owning production systems, on-call/incident response, troubleshooting complex issues, and building strong observability and reliability.
• Technical Leadership - Strong code reviews, mentoring, driving ambiguous initiatives, and influencing architecture across teams.
• Product Thinking - Customer-focused mindset, challenging assumptions, partnering with Product, and building solutions that deliver real user value.
• AI-Native Engineering - Extensive use of AI in development, familiarity with coding assistants, reasoning models, agents and retrieval, and willing…