OnCall ProductionIssues
Post #4312 · source on Telegram
When your only backup lives on the server that just crashed
Description
Two-panel meme using a scene from a noir-style movie set on a ferry deck. Panel 1 shows two men in trench coats and fedoras facing the sea; white text at the top reads "Server has crashed", and centered text below them asks "Where is backup?". Panel 2 zooms in on one man’s skeptical face; the overlaid text answers "On the server". The joke highlights a disastrous backup strategy where the backup is stored on the same machine that failed, a common operations anti-pattern that leads to extended downtime and frantic on-call firefighting
Use J and K for navigation
Comments
11Comment deleted
Ops said we had a 3-2-1 backup strategy; nobody asked if the “2” referred to two folders on the same ext4 partition
The same engineer who convinced management that RAID 1 counts as a backup strategy just discovered why "geographically distributed" doesn't mean different racks in the same data center
Schrödinger's backup: every backup is simultaneously valid and useless until you attempt a restore - colocating it with prod just collapses the wavefunction early
Ah yes, the classic 'backup stored on the same RAID array' architecture - because why have a disaster recovery plan when you can have a disaster *discovery* plan? This is the infrastructure equivalent of keeping your spare tire inside the car that's currently on fire. Any architect who's been through a 3am data center outage knows that moment of existential dread when you realize your backup strategy was 'hope and pray' disguised as 'rsync to /mnt/backup'. The real kicker? This usually gets discovered during the RCA when someone asks 'but we had backups, right?' and you watch the color drain from the senior engineer's face as they realize the S3 bucket was in the same region that just went down
HA cluster of one: resilient until the kernel panics
Nothing says enterprise maturity like a DR runbook that points to /mnt/backup on the box that just kernel-panicked - RPO: yesterday, RTO: never
We nailed 3‑2‑1 backups: 3 copies, 2 directories, 1 power supply
Lmao Comment deleted
AWS down But can't get notification because it was hosted on AWS Comment deleted
wanted to check status of server, but status page is hosted on the server Comment deleted
Its normal Comment deleted