AWS Architecture Blog·August 13, 2026Adobe Firefly's Observability Evolution with Amazon Managed Prometheus for GPU Workloads
This article details how Adobe Firefly scaled its observability architecture for GPU-based AI/ML training infrastructure by migrating critical metrics from a self-managed Prometheus deployment to Amazon Managed Service for Prometheus. The shift addressed challenges like high-cardinality telemetry from thousands of GPUs, achieving significant improvements in query performance and reducing operational overhead. This case study highlights architectural decisions for monitoring large-scale, specialized compute environments.
Read original on AWS Architecture BlogAdobe Firefly, a service powering creative AI features, relies on a massive GPU-based training infrastructure built on Amazon EKS. Monitoring this infrastructure, especially for distributed GPU workloads, presented significant challenges due to the high volume and cardinality of metrics generated. Unlike traditional CPU monitoring where a single utilization metric might suffice, GPU training requires observing the interplay of compute, memory, and network layers across thousands of nodes and GPUs, generating billions of data points.
The Challenge: High-Cardinality GPU Observability at Scale
The core problem Adobe faced was that their self-hosted Prometheus infrastructure could not meet the performance requirements for querying metrics from GPU clusters. GPU training clusters produce high-cardinality telemetry across multiple dimensions (GPU health, performance, compute, memory, etc.). For example, monitoring 2,000 nodes with 16,000 GPUs, scraped every 30 seconds, can generate over 1 billion data points in a single query window, leading to slow or timed-out queries.
Architectural Evolution to Amazon Managed Service for Prometheus
Adobe's observability architecture evolved iteratively. They chose Amazon Managed Service for Prometheus as a fully managed alternative to handle horizontal scale without the operational overhead. This wasn't a complete replacement but an incremental adoption using the Amazon Managed Service for Prometheus collector (managed scrapers) to collect critical metrics and forward them directly, operating alongside their existing self-managed Prometheus setup.
- Job-level monitoring: Tracking GPU utilization, memory, and network throughput per training job.
- Pod and node health: Monitoring Kubernetes pod status, node readiness, and resource allocation.
- GPU health: Metrics to determine GPU health and inform replacement decisions.
Incremental Migration Strategy
Instead of a 'big bang' migration, Adobe adopted a phased approach. They used managed scrapers in parallel with their existing Prometheus setup, targeting critical metrics first. This minimized disruption and allowed teams to adapt gradually while demonstrating value early.