Skip to content
DevMeme
3038 of 7590
Observability Theater: Logging Everything, Monitoring Nothing
Observability Monitoring Post #3353 · source on Telegram

Observability Theater: Logging Everything, Monitoring Nothing

Description

This meme uses the four-panel 'For the Better, Right?' format, featuring Anakin Skywalker and Padmé Amidala from Star Wars. In the first panel, Anakin looks proud and declares, 'WE LOG EVERYTHING!'. In the second, a smiling Padmé follows up with the question, 'AND MONITOR THE LOGS, RIGHT?'. The third panel shows Anakin's confident expression replaced by a silent, blank stare. The final panel shows Padmé's smile has vanished, replaced by a look of dawning horror as she repeats, 'AND MONITOR THE LOGS, RIGHT?'. A watermark for 'imgflip.com' is visible in the bottom-left corner. The humor targets a common organizational anti-pattern in software engineering and operations. Teams often put immense effort into generating vast quantities of logs, creating a false sense of security that their systems are observable. However, the crucial step of setting up effective monitoring, alerting, and analysis on those logs is often neglected. For experienced engineers, this is a painfully familiar scenario where logs exist but provide no real-time insight, rendering them useless during an actual production incident. Anakin's silent panic perfectly captures the moment of realization that the 'observability' is just for show

Categories

Observability Monitoring DevOps SRE OnCall ProductionIssues Bugs Infrastructure

Common Tags

Logging Observability Monitoring Devops DeveloperHumor DevOpsCulture

Comments

37
Anonymous ★ Top Pick Our logging strategy is simple: we write everything to a distributed, highly-available, and infinitely scalable black hole called /dev/null
  1. Anonymous ★ Top Pick

    Our logging strategy is simple: we write everything to a distributed, highly-available, and infinitely scalable black hole called /dev/null

  2. Anonymous

    “We log everything” - translation: our alerting pipeline is the CFO pinging Slack when the S3 bill doubles

  3. Anonymous

    Somewhere there's a 10TB Elasticsearch cluster with perfect log retention, zero dashboards, and one alert: disk space at 95%. The logs will tell us exactly what went wrong... if anyone remembers the Kibana password from 2019

  4. Anonymous

    Ah yes, the classic 'we have observability at home' scenario - petabytes of logs dutifully shipped to S3 at $0.023/GB, zero alerts configured, and a Grafana dashboard that hasn't been opened since the proof-of-concept. When the 3am page comes in, you'll grep through CloudWatch Insights with a rate limit, discover your retention policy deleted last week's data, and realize the only person who knew the log format left six months ago. But hey, at least you're GDPR compliant with that 90-day retention policy, right? Right?

  5. Anonymous

    “We log everything.” Cool - who monitors it, who redacts it, and who pays the ingest bill when cardinality pages Finance before on-call?

  6. Anonymous

    “We log everything” usually means DEBUG in prod, zero sampling, PII confetti, and a CloudWatch invoice that pages finance before SRE

  7. Anonymous

    Logs: our single source of truth, buried under 99.9% cat photos from debug spam

  8. @ZgGPuo8dZef58K6hxxGVj3Z2 5y

    😂😂😂😂😂

  9. @scout_ca11sign 5y

    - We log everything! - And do not erase monthly old logs to make up space? - We dont erase older logs to make up space, don't we?

    1. @RiedleroD 5y

      my school does the opposite - keep courses (including attached images and everything) from every year since 2014 - our servers have run out of space twice this year.

      1. @RiedleroD 5y

        note: we have a lot of teachers that write stuff in latex, parse it to an image and put it into the course like that

        1. @scout_ca11sign 5y

          a w e s o m e

        2. @Roman_Millen 5y

          Not the worst case scenario. Imagine if they actually stuffed the big-size (Full HD-like) pictures into the document shrunk to tiny images, but with original size maintained.

          1. @RiedleroD 5y

            uhm… I don't understand, please elaborate?

            1. @Roman_Millen 5y

              Here's the deal. You take the big size image that takes some big space. You insert it into a document and shrink it on the page to some tiny size. You save the document. The document won't actually shrink the image per se — instead it will maintain its original size, just displaying it minified. In other words, put such big images into a document and it will bloat drastically, no matter how these images are shrunk visually.

              1. @RiedleroD 5y

                ahh I get what you meant. Well, they already do this to some degree. The images have like four times the resolution they have to be, and then insert them into moodle at (for my screen size) ~200% of the screen's resolution.

                1. @RiedleroD 5y

                  although my screen size is only 1366x768, so there's that

                2. @Roman_Millen 5y

                  Oh dear. I'm afraid to even ask how much server space did it all take, if those "eggheads" were actually doing this all the way since 2014.

                  1. @RiedleroD 5y

                    I have no idea, that info isn't accessible to students.

                    1. @RiedleroD 5y

                      but I imagine it's at least several terabytes

        3. @prirai 5y

          🤣🤣 Why not directly in pdf?

          1. @RiedleroD 5y

            because they usually embed it on our website - you can't embed pdfs (afaik)

            1. @sylfn 5y

              But you can use js-based pdf viewer like Mozilla's pdf.js

              1. @RiedleroD 5y

                you can also use js to embed latex directly into html, but the point is that they're incapable

                1. @p4vook 5y

                  mathml

                  1. @RiedleroD 5y

                    yes, that

            2. @prirai 5y

              For embedding, they could use mediawiki or mathjax or mathml. There are many more nowaday.

              1. @RiedleroD 5y

                yes, if they had like 2 minutes of IT experience, they could do that

                1. @prirai 5y

                  Why? It's the same as latex editing.

                  1. @RiedleroD 5y

                    I believe they have a word-like frontend to latex, and you do need to include some javascript libraries (which they can't)

              2. @sylfn 5y

                mathml is supported by Opera Presto (partial) One of old Chromes (partial) Firefox Safari So using mathml for everyone is not good at all

  10. @prirai 5y

    If you know either, you can easily do the other.

  11. @prirai 5y

    Also, some latex editors support exporting as html

    1. @RiedleroD 5y

      look, can we just agree that my teachers are fucking stupid? Thanks

      1. @prirai 5y

        You in HS?

        1. @RiedleroD 5y

          the austrian of equivalent of it, yes

  12. @VlP_AI_TG 5y

    I be like:

More like this

More like this in Observability Monitoring →
Observability Monitoring Logging Observability
Watching Kubernetes Self-Heal Into a Node Outage

Watching Kubernetes Self-Heal Into a Node Outage

Containerization
POV: You Sent Claude the Wrong Logs

POV: You Sent Claude the Wrong Logs

AI ML
Production Stack Traces Chase Dev

Production Stack Traces Chase Dev

OnCall ProductionIssues
Use J and K for navigation