Skip to content
DevMeme
3981 of 7590
OnCall ProductionIssues Post #4334 · source on Telegram

When you silently crashed prod and hope nobody notices the smoke

Description

Two identical horizontal panels show an older man with short white hair sitting at a bright, minimalist desk. He holds a light-pink coffee mug in his right hand while looking at an open silver laptop. Behind him are small potted plants and simple white decor against a sunlit window. Across the bottom of the top panel, bold white capital letters with a black outline read: "TELL ME YOU TOOK DOWN PRODUCTION". Across the bottom of the second panel the caption continues: "WITHOUT TELLING ME YOU TOOK DOWN PRODUCTION". The meme riffs on the social-media phrase "tell me without telling me," poking fun at engineers who bring the live environment down and then speak in vague terms during incident channels or stand-ups. It resonates with DevOps and on-call veterans who recognize the dread of an unspoken production outage and the scramble for rapid incident response

Comments

25
Anonymous ★ Top Pick “Applied a one-line Helm patch, dashboards look quiet.” - senior-speak for “I just sent all prod traffic to /dev/null and I’m praying Prometheus hasn’t figured it out yet.”
  1. Anonymous ★ Top Pick

    “Applied a one-line Helm patch, dashboards look quiet.” - senior-speak for “I just sent all prod traffic to /dev/null and I’m praying Prometheus hasn’t figured it out yet.”

  2. Anonymous

    You know someone took down prod when they suddenly become very interested in reading the disaster recovery documentation they ignored during onboarding, their Slack status mysteriously changes to 'in a meeting,' and they're googling 'how to update LinkedIn quietly.'

  3. Anonymous

    You never have to tell anyone you took down production - Grafana, PagerDuty, and the sudden silence in Slack will handle the announcement for you

  4. Anonymous

    The beauty of modern observability tooling is that your Slack alerts, PagerDuty notifications, and Datadog dashboards will tell your entire organization you took down production approximately 47 seconds before you finish typing 'I think we have a problem' in the incident channel - giving management the plausible deniability they crave while you frantically roll back that 'minor config change' you deployed at 4:45 PM on Friday

  5. Anonymous

    Translation: “We’ll roll forward” = autoscaler at zero, a migration holding a global lock, and the CDN heroically serving 200s on the error page

  6. Anonymous

    Not an outage - I just rotated a TLS cert and our service mesh interpreted zero‑trust as zero traffic

  7. Anonymous

    That post-merge smile when 'kubectl delete namespace/prod --all' was just a 'quick cache clear'

  8. @TERASKULL 4y

    my management is a bunch of morons and somehow I have access to prod despite being a newhire

  9. dev_meme 4y

    guys, i need you to calm down in 3, 2, 1....

  10. @insan3d 4y

    Well... I will remember how the mtime flag of find command works while purging old logs on the legacy server and I will definitely double-check the working directory next time.

  11. @jMdZzZ 4y

    /var/www$ rm -rf html/ folder 😓

  12. @PeGa041 4y

    "it seems that we need to improve our alerting systems"

    1. dev_meme 4y

      😭

    2. @SamsonovAnton 4y

      "$(uname -s) really needs to be more fool-proof..."

  13. @cptnBoku 4y

    Hi manager, you might get a call or two

  14. @cptnBoku 4y

    I swear I was connected to test db

  15. @dsmagikswsa 4y

    devops and you commit wrong codes…

  16. @darlan256 4y

    "I have good news and bad news."

    1. @zohnannor 4y

      "Start with the good one." "Our server insurance is still valid"

      1. @ceterre 4y

        "Good one: the last backup was on Saturday and completed successfully"

  17. @K3pp1 4y

    We either had an earthquake at the data centre, or the delete_user endpoint I exposed yesterday has to be tweaked a little🌚

  18. @denisndenis 4y

    Oops

  19. @Infinitelineman 4y

    ...im sorry honey, i've been on call with 20+ people last 4 hours, couldn't answer

  20. @MrZarei 4y

    a little development was needed so . . .

  21. @Algoinde 4y

    "Hey do you know why grafana stopped showing request rates?"

Use J and K for navigation