Principal Software Engineer

Make Your Mark:

As a Principal Software Engineer, you will play a crucial role in delivering high quality releases to our customers by defining application architecture, designing, developing, troubleshooting, maintaining, optimizing, and scaling BlackLine’s cloud-based backend services and driving innovation into our product suite. In this role, you will work with management to identify strategic directions for system architecture. You will research, test, benchmark and evaluate new technologies, propose scaling solutions, collaborate with other teams to define, and build new features, optimize existing features while writing elegant code, coaching, and mentoring less experienced software engineers and communicating with people at all levels. In this position, every bit of your software design skill be drawn upon in order to drive towards and implement best of breed features in a rapid iterative environment. Your attention to detail, tenacity, and creative problem-solving and thinking ahead will directly impact the company’s success, as well as the success of your peers across the organization. If you have a passion for delivering products that make a difference, are driven to make things better in all that you do and are eager to work in a collaborative agile environment where Developers are dedicated to solving customer problems and innovating, you will be a great fit for the team.

You’ll Get To:

We are looking for an accomplished, hands-on Principal Engineer with 18+ years of software engineering experience and a strong record of setting technical direction in Agentic AI, Generative AI systems and Software/Data engineering. This is the most senior individual contributor role on the Agentic AI platform. You will set the multi-year technical vision and own the architecture across products, while staying close enough to the code to make the critical calls. You will guide Senior Staff Engineers, Staff Engineers and Technical Leads across several teams.

The ideal candidate has led 5+ enterprise-scale AI programs from strategy and solution design through to production at scale, and has shaped platform architecture that several product teams build on. You can define an organization-wide AI architecture strategy and present it to executives, and you can also prototype and write production-grade Python when a problem needs it. You bring the vision to set direction across teams, the judgment to make high-stakes architecture and investment decisions, and the credibility to raise the engineering bar across the company.

The role spans strategy, architecture and execution. You will set the AI platform roadmap, define the governance and trust frameworks that make agents enterprise-ready on a secure, compliant financial platform, and establish the engineering standards that every AI product team follows.

Key Responsibilities

Technical Strategy & Vision

• Define and own the multi-year technical strategy and architecture roadmap for the Agentic AI platform across all product lines.
• Make build-vs-buy, technology selection and platform investment decisions, and present the trade-offs, risks and ROI to executive leadership.
• Track the agentic AI landscape: frameworks, protocols, models, regulation. Turn those shifts into strategic bets and platform direction.
• Identify new opportunities where AI can deliver business value across product areas, and shape them into fundable programs.

Architecture Leadership

• Own the end-to-end architecture of the Agentic AI system: agent registry, policy engine, metrics ingestion pipeline, anomaly detection and HITL orchestration.
• Define the reference architecture for multi-agent systems: trust tier boundaries (T0–T3), separation of concerns, fault isolation and full observability.
• Set organization-wide patterns for data flows, integrations (REST, gRPC, SSE, event-driven) and storage strategies for AI workloads in a multi-tenant FinTech environment.
• Own the design principles for durable DAG execution, pause and resume of long-running financial close workflows, event-sourced audit trails and SLA-tracked HITL approval gates.
• Run the architecture governance process: ADRs, cross-team design reviews and production readiness assessments for critical systems.

Technical Leadership & Organizational Influence

• Split the platform into programs and workstreams. Assign ownership to Senior Staff and Staff Engineers, with clear API contracts, inte…