Networking
Post #1989, on Aug 31, 2020 in TG
The Network Engineer's Hierarchy of Blame
Description
A screenshot of a tweet from user Philippe_Scorsolini@home (@Phisc0). The tweet's text reads, 'When it's not DNS fault...'. Below this, it embeds another tweet from the account 'Bad Packets' (@bad_packets) which simply states, 'It was BGP.'. The image captures a classic piece of insider humor for network engineers and SREs. It plays on the long-running industry joke that DNS is the root cause of almost all internet-related problems ('It's always DNS'). This meme provides the corollary: on the rare occasion that DNS is innocent, the next most likely culprit for a catastrophic, widespread outage is a BGP (Border Gateway Protocol) misconfiguration. The humor lies in the resigned acknowledgment of this hierarchy of probable failures
Related Memes
Use J and K for navigation
Comments
7Comment deleted
The Sysadmin's Prayer: 'Our packets, who art in heaven, routed be thy name. Thy BGP routes come, thy DNS resolved, on-prem as it is in the cloud. Give us this day our daily uptime, and forgive us our typos in zone files, as we forgive those who misconfigure their AS paths. And lead us not into network partitions, but deliver us from flapping. For thine is the subnet, the port, and the protocol, forever. Amen.'
Runbook update: 1) Blame DNS. 2) If dig answers, diff yesterday’s BGP tables, blame the “harmless” route optimization, and schedule a retro to promise (for the fourth time) we’ll actually roll out RPKI
After 20 years in the industry, you develop a troubleshooting flowchart: Step 1: Blame DNS. Step 2: Actually check DNS. Step 3: Fix DNS. Step 4: Realize it's still broken. Step 5: Discover someone fat-fingered a BGP route advertisement and took down half the internet. Step 6: Update runbook to include 'Check BGP before blaming DNS.' Step 7: Still blame DNS next time
The eternal troubleshooting hierarchy: It's always DNS, except when it's not - then it's BGP, the protocol that can take down entire swaths of the internet because someone fat-fingered a route announcement at 3 AM. At least with DNS you can flush the cache and pretend you fixed something; with BGP, you're just waiting for route convergence while explaining to executives why half the internet thinks your AS doesn't exist
When it’s not DNS, it’s BGP - the bug where a fat‑fingered more‑specific prefix blackholes your traffic and the SSH session you needed to fix it
DNS takes the heat for every lookup fail, but BGP's the one hijacking routes to your outage
We spent hours tuning resolvers before realizing a missing no-export leaked a more-specific - turning our anycast into blackhole‑as‑a‑service