Skip to content
DevMeme
6099 of 7590
When the Traffic Spike is a Security Scan
Security Post #6679 · source on Telegram

When the Traffic Spike is a Security Scan

Description

A two-panel meme using the 'Disappointed Black Guy' format to illustrate a common production incident scenario. The top panel shows a developer's happy, smiling face next to a monitoring graph labeled 'Logs volume'. The graph displays a sudden, sharp spike in activity, which is initially interpreted as a positive sign, like a surge in user engagement. The bottom panel shows the developer's expression changing to one of shock and horror. Next to him is a screenshot of server logs revealing the true cause of the spike: a series of GET requests from a malicious actor or security scanner probing for sensitive files like '/core/.env', '/.aws/credentials', '/admin/config', and '/.env.old'. The meme perfectly captures the sinking feeling a developer, SRE, or on-call engineer experiences when they realize an exciting traffic increase is actually a security threat, specifically a vulnerability scan for exposed configuration and credential files

Categories

Security Observability Monitoring OnCall ProductionIssues DevOps SRE Backend

Common Tags

Logging Observability Security ProductionIncidents SensitiveDataExposure SecretsManagement OnCall Debugging Devops

Comments

44
Anonymous ★ Top Pick That's not user engagement, that's a free, unsolicited penetration test from a script kiddie who thinks '/.env' is the new 'hello world'
  1. Anonymous ★ Top Pick

    That's not user engagement, that's a free, unsolicited penetration test from a script kiddie who thinks '/.env' is the new 'hello world'

  2. Anonymous

    Nothing like a 19:00 ‘harmless’ spike to remind you that `.aws/credentials` isn’t meant to double as a public REST endpoint

  3. Anonymous

    The real horror isn't the 75 requests per second spike - it's realizing your junior dev's 'temporary fix' of chmod 777 on the project root from six months ago is still in production, and now someone's downloading your AWS keys faster than you can rotate them

  4. Anonymous

    That moment when your log volume graph looks like a hockey stick and you realize someone's running a directory enumeration script against your production API - and your .env files are responding with 200 OK. Nothing says 'Friday evening' quite like watching attackers systematically request every possible secrets file path while your monitoring dashboard lights up like a Christmas tree. At least the observability stack is working perfectly to document your impending PagerDuty incident and inevitable post-mortem titled 'Why Our AWS Credentials Were Briefly Public.'

  5. Anonymous

    Config management so distributed across /env paths, it's a CAP theorem violation - consistent leaks, available everywhere, but partitioned by dev negligence

  6. Anonymous

    That 'Logs volume' spike wasn’t growth; it was the internet’s nightly cron GETting /.env and /.aws/credentials - why are we alerting on volume instead of 2xx to sensitive paths?

  7. Anonymous

    If your WAF lets GET /.aws/credentials through, you don’t need chaos engineering - you’re already running adversary-driven failure injection

  8. @Algoinde 1y

    HTTP 404 GET /phpinfo HTTP 404 GET /.env HTTP 404 GET /.git 🥱 HTTP 200 GET /phpinfo HTTP 200 GET /.env HTTP 200 GET /.git 💀

  9. @qtsmolcat 1y

    WAF idea: return fake but seemingly legitimate responses for those paths. That way you trick the attackers!

    1. @SamsonovAnton 1y

      I once configured my home server to respond with multi-gigabyte iso file (Linux or BSD install DVD) to such requests. Did not notice any effect, but I guess some users unconsiously participating in a botnet probably went out of prepaid traffic limit, or had to pay extra.

      1. @qtsmolcat 1y

        Cloudflare AI labyrinth kinda be like:

      2. @TheRamenDutchman 1y

        Seems like you'd make yourself very vulnerable to getting steamrolled by visitors

        1. @SamsonovAnton 1y

          My home server that serves nearly nothing and never had any traffic quota? I doubt so. The real fun happened last autumn when an army of smart speakers with a poorly self-written SNTP client started to query public servers each second, causing the entire country zone pool to collapse over a short period of time. 😅

          1. @Algoinde 1y

            we put libcurl into too many small things and are now suffering the consequences

          2. @TheRamenDutchman 1y

            I'm just surprised that they even download and process the file And that your server can upload that much p2p to an entire country zone How did you even find out that happened through your server and how do you have an entire country zone of upload through your provider?

            1. @chupasaurus 1y

              the zone thing is about NTP servers in a country pool, pool.ntp.org is using GeoDNS for CNAME to country-code pool (i.e. us.pool.ntp.org) which answers with a list of servers in the country.

            2. @SamsonovAnton 1y

              You got it all wrong. Those were 2 disconnected stories. I had fun with my web server being probed, say, around 20 years ago, when having an Internet connection without traffic quota (truly unlimited — no "small font" tricks) was not common. Since I didn't put much load on my connection otherwise, I could afford replying with an ISO file to anyone who made "strange" HTTP requests to my server. IIRC, many probers had safety limits on HTTP response (whether a time limit or a size limit), but there were those who tried to download the entire file, perhaps only being stopped by network errors. The NTP zone collapse happened last autumn (see the attached chart). In my case, the bottleneck was my router: despite being a full-fledged x86-64 computer controlled by pfSense, it was based on entry-level CPU and desktop-class NIC, so was overwhelmed when incoming traffic reached just 50 Mbit/s — roughly 1 Mpps. And although NTP pool allows to set load quota, its basically a weight only, which does not really matter when only 4 servers are left for the entire huge country (because other servers already excluded from the pool for the same reasons of unreachability or long response times).

    2. @erizpl 1y

      it's called honeypot

  10. @ZgGPuo8dZef58K6hxxGVj3Z2 1y

    They just wanted to get ready to add a few lines to your source code 🥰

  11. @viktorrozenko 1y

    A couple years back I got into algotrading crypto, made a bot, rented a VM and deployed it. I don’t remember exactly why it needed an API, but it had one, so, naturally, it was accessible through the web. I deployed it on DigitalOcean and saw lots of these types of logs of people trying to find vulnerabilities. What’s interesting is when I deploy anything on Russian cloud providers, I don’t have any such traffic 🤪

    1. @ZgGPuo8dZef58K6hxxGVj3Z2 1y

      Not people, bot nets

    2. @qtsmolcat 1y

      Probably because a lot of those bots originate in Russia or China

    3. @ZgGPuo8dZef58K6hxxGVj3Z2 1y

      There are only 2 types of Russian servers, the ones that have good security and the ones that have terrible security but they dont care

      1. @ZgGPuo8dZef58K6hxxGVj3Z2 1y

        Guess which group is Putin's mail server

        1. @sylfn 1y

          "dont care"

    4. @slnt_opp 1y

      Can't relate with the server location, but for me it always seemingly coincides with getting Let's Encrypt certs🌚

      1. @viktorrozenko 1y

        Hmmmmm

      2. @viktorrozenko 1y

        Maaaaybe

      3. @viktorrozenko 1y

        Now that I think about it…

    5. @drbogar 1y

      Are there Russian cloud providers?

      1. @viktorrozenko 1y

        Yandex Cloud for example

    6. @azizhakberdiev 1y

      hackers avoid friendly fire

    7. @pyrothefuck 1y

      those are botnets, see that on every server I deploy, access logs are always filled with attempts to ssh into my server with usernames like "admin" or "nginx" and password auth

  12. @viktorrozenko 1y

    Even though I didn’t advertise my bot to anyone while the stuff I have on Russian clouds is a literal SaaS with thousands of users

  13. @viktorrozenko 1y

    Nah, most of it was Turkey, Portugal and South America

    1. @qtsmolcat 1y

      Their exit IPs were there sure

  14. @viktorrozenko 1y

    Turkey was surprisingly big one

  15. @sysoevyarik 1y

    /restore.php

  16. @zepyr 1y

    or... put fake .env files

  17. @zepyr 1y

    and enter in loophole

  18. @mpolovnev 1y

    Does it still work anywhere?

  19. @azizhakberdiev 1y

    what hackers gonna do if they steal my source code? Debug it?

  20. @adhdj_ch 1y

    😂

  21. @qtsmolcat 1y

    Jokes on you, my env is actually titled index.html

More like this

More like this in Security →
Security Logging Observability
Office turns live SQL-injection honeypot logs into an impromptu spectator sport

Office turns live SQL-injection honeypot logs into an impromptu spectator sport

Security
The Art of Spinning a Security Incident When You Have No Logs

The Art of Spinning a Security Incident When You Have No Logs

Security
The Sisyphean Task of Cybersecurity Advice

The Sisyphean Task of Cybersecurity Advice

Security
Use J and K for navigation