Advanced Software Architect | Data Engineer

About the Role

We are hiring two Advanced Software Architects to own the full lifecycle of our enterprise Data Lakehouse platform. These are senior, architecture-first roles we are looking for engineers who think in systems, make platform-level design decisions, and can build and operate a production-grade, multi-source lakehouse from the ground up, covering ingestion through consumption across a complex, multi-cloud source landscape.

You will be the technical authority for a platform that consolidates data from 18+ enterprise products (Costpoint, GovWin, Specpoint, Vantagepoint, and others) into a governed, medallion-architected data lake on AWS S3 with Apache Iceberg table format, orchestrated via AWS Step Functions, and queryable through AWS Athena and Trino. This role is end-to-end: you own ingestion, transformation, quality, orchestration, ML data supply, and BI consumption.

Platform Architecture You Will Own

Our current production stack — which you will architect, extend, and operate — spans the following layers:

Ingestion Layer

• Fivetran Managed Data Lake with CDC connectors; 13-minute sync frequency; Iceberg conversion and schema evolution across 500+ Bronze tables
• Multi-cloud source systems: OCI (Costpoint, Maconomy), Azure (Vantagepoint), AWS (GovWin) — all via CDC streams

Storage Layer (AWS S3 + Apache Iceberg)

• Bronze (s3://lakehouse-data/bronze/): Raw CDC data, Fivetran-managed, 500+ tables, 2-year retention
• Silver (s3://lakehouse-data/silver/): Clean, deduplicated, 100+ tables, 5-year retention — your team manages
• Gold (s3://lakehouse-data/gold/): Aggregated, business-optimized, 20+ tables, 7-year retention — your team manages

Processing Layer

• AWS Glue ETL (Serverless PySpark): bronze_to_silver, silver_to_gold, and metadata_capture jobs
• AWS SageMaker: Notebooks and endpoints (ml.c3.medium / ml.m5.xlarge) for ML model training, inference, and Model Registry

Orchestration & Automation

• AWS Step Functions: State machines for customer_360_pipeline, ml_training_pipeline, and metadata_capture
• EventBridge Scheduler for time-based triggers; Lambda functions for check_new_data, check_data_quality, check_model_drift, and send_alerts

Metadata & Governance

• AWS Glue Catalog (fivetran_bronze, lakehouse_silver, lakehouse_gold, lakehouse_metadata databases) — used by all services
• Fivetran REST Catalog (Apache Polaris): auto-generated, Bronze source of truth, read-only
• AWS Lake Formation: row-level security, column masking, and audit logging
• AWS CloudWatch: log aggregation, metrics & alerts, dashboards

Query & Consumption Layer

• AWS Athena (Phase 1): serverless SQL, $5/TB scanned
• Trino on EKS (Phase 2, optional): self-hosted, sub-second queries, 3–10 node cluster
• Consumption: ML Applications (churn alerts, customer insights, usage analytics), BI Dashboards (Tableau, Power BI, Looker), REST APIs (Lambda + API Gateway)

Key Responsibilities

• Architect and evolve the full medallion lakehouse — Bronze, Silver, and Gold layers — on AWS S3 with Apache Iceberg; own schema design, partitioning, compaction, and retention policies.
• Design and implement scalable Glue ETL (PySpark) pipelines for bronze_to_silver and silver_to_gold transformations, incorporating dbt for SQL-layer transformations where appropriate.
• Own and extend CDC ingestion via Fivetran; manage schema evolution, connector health, and sync reliability across 18+ source products.
• Build and maintain AWS Step Functions state machines and EventBridge schedules for end-to-end pipeline orchestration; implement Lambda-based quality and drift monitors.
• Govern the Glue Catalog and Lake Formation policies; enforce column-level security, row-level access controls, and audit logging to meet SOC2 and regulatory requirements.
• Architect the query layer — optimize Athena workgroups and partition pruning; plan and execute Trino-on-EKS deployment for sub-second analytics workloads.
• Partner with data science teams on SageMaker data supply: feature engineering pipelines, training dataset preparation, and model registry integration.
• Implement real-time and near-real-time streaming solutions using Kafka or Kinesis where sub-13-minute latency is required.
• Lead platform modernization initiatives: evaluate emerging formats (Iceberg vs. Delta Lake vs. Hudi), tooling, and cost optimization strategies.
• Establish and enforce data engineering best practices: co…