Aws Observability / Grafana Architect

Role : AWS Observability or Grafana Architect

Location : Warren, NJ (Onsite)

Hire type : Fulltime

Seeking a highly skilled AWS Observability Architect with deep, hands-on expertise in designing and implementing enterprise-grade observability platforms on AWS — with Grafana as the primary observability tool and OpenTelemetry as the instrumentation standard . This is a technical specialist role requiring genuine implementation experience, not platform familiarity.

The ideal candidate has personally architected and delivered large-scale observability solutions for production AWS environments — building telemetry pipelines, designing dashboards that operations teams actually use, and creating alerting frameworks that reduce MTTR rather than add noise. You understand the full observability stack: from application instrumentation with OpenTelemetry SDKs through to Grafana dashboards consumed by SREs, on-call engineers, and engineering leadership.

This role sits at the intersection of cloud infrastructure, software engineering discipline, and operational excellence — requiring someone who can design an enterprise observability architecture in the morning, write a Grafana dashboard query in the afternoon, and advise a development team on OpenTelemetry instrumentation strategy the next day.

Key Responsibilities

Observability Architecture & Strategy

• Define and own the enterprise observability architecture for AWS environments — establishing the target-state design across the four pillars of observability: metrics, logs, traces, and events.
• Design end-to-end telemetry pipelines — from instrumentation at the application and infrastructure layer through collection, processing, storage, and visualisation — with Grafana as the enterprise observability platform.
• Develop observability standards and reference architectures — defining how AWS workloads across compute (EC2, EKS, ECS, Lambda), storage, networking, and managed services should be instrumented, collected, and visualised consistently across the organisation.
• Establish signal-to-noise discipline across the observability platform — designing alerting frameworks that surface actionable signals, eliminate false positives, and ensure on-call engineers are alerted only when human intervention is genuinely required.
• Define observability maturity roadmaps for client environments — assessing current-state coverage, identifying gaps, and building a phased improvement plan from reactive monitoring to proactive, AIOps-ready observability.
• Drive FinOps for observability — optimising telemetry data volumes, retention policies, and Grafana Enterprise licensing costs to ensure the observability platform itself does not become a significant cost centre.

Grafana Enterprise Implementation

• Architect, deploy, and operate Grafana Enterprise or Grafana SaaS as the primary observability platform — including high-availability Grafana deployment on AWS (EKS-based or managed via Grafana Cloud), data source federation, RBAC configuration, and enterprise plugin management.
• Design and implement Grafana data source integrations across the AWS observability ecosystem:
• Amazon CloudWatch — metrics, logs, and alarms as a core AWS data source
• Grafana Mimir — for scalable, long-term Prometheus-compatible metrics storage
• Grafana Loki — for cost-efficient, label-based log aggregation at scale
• Grafana Tempo — for distributed tracing storage and trace-to-log-to-metric correlation
• Amazon Managed Service for Prometheus (AMP) — for AWS-native Prometheus metrics
• Amazon OpenSearch — for log analytics and full-text search use cases
• Elasticsearch / OpenSearch — for existing log infrastructure integration
• Build and maintain a Grafana dashboard library — covering infrastructure health, application performance, SLO/SLA tracking, capacity planning, cost visibility, incident response, and executive reporting — using reusable, variable-driven, and consistently styled templates.
• Implement Grafana alerting at enterprise scale — including alert routing, notification policies, silence management, and integration with PagerDuty, OpsGenie, ServiceNow, and Slack for multi-channel incident notification.
• Configure Grafana RBAC and team structures — designing role hierarchies, folder permissions, and data source access…