Your Infrastructure Now Exists to Keep Prometheus Running

Description

A dark-mode tweet from 'Dev meme' (@devs_memes) with a trollface-style avatar. The text reads: 'Prometheus was supposed to monitor your infrastructure. Now your infrastructure exists primarily to keep Prometheus running'. The joke skewers a real operational inversion: Prometheus's resource appetite (cardinality explosions, TSDB memory usage, federation layers, Thanos/Mimir sidecars) often grows until the monitoring stack consumes more engineering attention and compute than the workloads it was meant to observe

Common Tags

PrometheusMonitoring Observability Monitoring DevOpsIrony Infrastructure ObservabilityTools Devops

Comments

Anonymous ★ Top Pick We added a second Prometheus to monitor the first one; the on-call rotation now exists to acknowledge alerts about the alerting

Anonymous ★ Top Pick

We added a second Prometheus to monitor the first one; the on-call rotation now exists to acknowledge alerts about the alerting
Anonymous

The final SLO: five nines for the system that tells you everything else is down.
@MSYZD 5mo

You can monitor it on another one an keep only exporters on the infrastructure 😄
1. @TheFloofyFloof 5mo
  
  but how will you monitor the second infrastructure that prometheus is on?
  
  @MSYZD 5mo
  
  Via Zabix 😁
  
  @TheFloofyFloof 5mo
  
  but who monitors Zabix
  
  @MSYZD 5mo
  
  Unfortunately Zabix is poor It's can not have another one 😄😄😄
  
  @D13410N3 5mo
  
  Just have small prometheus that is monitoring your main prometheus And your main prometheus should monitor small prometheus There's a very small chance that both of them will die in one moment, i.e. if they are geo-distributed
@Algoinde 5mo

my Loki is taking up as much memory as all my other services