Author here. This attack was extremely broad. I saw parts of this attack come from my own home ISP's ASN, though not my IP thankfully. If we just "blocked those places" there would be a lot of collateral damage. As it stood, we did temporarily bump up rate limiting for the biggest attack ASNs and we absolutely heard from real, regular users about it (Sorry to those affected).
Either it's coming from certain places and you can block those places, or it's coming from everywhere including some that are within your legal jurisdiction.
> The description/approach seems static and limited? Why not maintain a leaky bucket that counts each request (tickets/points) with higher cost for expensive requests (404, redirects). As the IP's reputation deteriorates (IPv4/32), it begins to spill over to a broader subnet like IPv4/31 then /30 and so on. Fight adaptivity with adaptivity. // maybe I describe something totally obvious, I'm not involved in the web ddos protection side of things.
The penalty box strategy I described in the post is along these lines. It penalizes excessive expensive requests directly. Specifically, it does add those to a score and will rate limit more broadly as necessary.
* JA3s are mostly useless. JA4s supersede them entirely.
* Using JA4s in rate limits is pretty useful and helps a lot against proxy scraping. It was not very helpful in this attack.
* Bot detections are somewhat helpful but they don't solve scrapers/attacks by themselves. They're useful as a 2nd/3rd data point (eg. low bot score + bot detection + something else)
My assumption would be that it would drastically reduce the attack to be borderline irrelevant. I've never turned on Under Attack so somebody else may have more insight and the docs[1] don't describe precisely what happens besides a JS interstitial.
I know that JS challenges, both interactive and non-interactive, can be solved by bots. I've seen it. However, I suspect that the challenges just get harder and harder until the attack levels drop.
It changes the economics of the attack because it requires the attacker to do compute before they can make requests. Depending on how good the fingerprinting is, it can also get very expensive (eg. requiring you to run a full browser, rather than merely computing a few sha256 hashes)
Author here. This was not a misconfigured data scraper. We see those every week[1]. This attack wasn't scraping useful content. It was almost entirely 404s and 302s and pulled virtually zero real docs. It specifically looked for URLs not served by the CDN and when it found a pattern, did millions of variations of it. Whether built by an AI or not, it was designed to cause outages and financial damage from autoscaling. However, as others have suggested, we may have been a test run for a real target.
Do you have a list of the addresses that were hammering your site? Have you tried any of the techniques I list here? [1] Do sets of the IP's show up in here [2]? Are the bots mostly residential, VPS, Tor? What is the HTTP protocol breakdown? HTTP/1.1, 2.0, 3.0? Are they missing any expected client headers? Have you tried blackhole routing any of them from an out of band management console?
# only useful if not behind a CDN
for Ip in $(cat /dev/shm/list-of-attackers.ipset);do ip route add blackhole "${Ip}" 2>/dev/null;done
[Edit] appears you are behind Cloudflare so the blackhole would be up to them. One could still return a 429 or 525 to the attackers.
> Do you have a list of the addresses that were hammering your site?
I could get a list of IPs, but it wouldn't be particularly helpful. It was hundreds of millions of unique IPs. Most IPs made fewer than 100 requests and then disappeared.
> Have you tried any of the techniques I list here?
We were already doing essentially all of the techniques there and they were not helpful in this attack. This wasn't some spider run amok. This was a dedicated attack with intelligence behind it.
> Do sets of the IP's show up in here [2]?
If we're looking at individual IPs, we've already lost.
> Are the bots mostly residential, VPS, Tor? What is the HTTP protocol breakdown? HTTP/1.1, 2.0, 3.0? Are they missing any expected client headers?
The article outlines most of this. It was a mix of residential and major/minor hosting providers and some corporates. If I were to say which was the largest, it was minor hosting providers. However, the attack pivoted between coming from different sources. At first it was coming mostly from minor hosting providers/VPS. By the end, it was entirely residential and corporate IPs. Tor was not involved at all (RTD supports accessing through Tor, but you are more likely to get a challenge). All the expected headers were there and UAs cycled between a very large set of standard UAs for normal browsers/devices. The attack was 95%+ HTTP2/HTTP3.
> [Edit] appears you are behind Cloudflare so the blackhole would be up to them. One could still return a 429 or 525 to the attackers.
It's trivial to setup an IP list[1] in CF and then you can apply all sorts of rules (lower rate limits, outright challenges, etc.) to it. It can be managed through Terraform as well. You can also use CIDR notation. This along with more classifications to specific ASNs are something we are looking at.
Inside this .bz2 [1] there is a directory called bh_routes/bh_vps_120/ and in that there are many AS files named for many of the VPS/Server providers. If you site does not need inbound connections from such places that is a starting point to reduce the traffic a bit. Each file has a comment with the AS#/whois name if you want to double check who/what you may be blocking. Not perfect but it may make a dent in the traffic. Or ask Cloudflare if they have an option to block anything that is not residential or LTE wireless. Even if you want to allow bots, maybe have a "shields up" mode where bots are sent away until the attack passes.
Another potential option would be to configure nginx IP limits using the Cloudflare header that represents the IP, set the shared memory size rather high and return 525 to the IP's exceeding a limit to avoid them hitting your redirect rules though it feels like CF should be able to create something custom for this assuming its a paid account.
Any IP that is not VPS/server should be archived for the feds as they are working on shutting down residential proxy providers including apps that are turning peoples cell phones into proxies.
I talked about that directly in the post. We didn't want to just challenge everyone. We use JS challenges but we try to use them sparingly. The rest of the ops team and I were fighting to stay up but it never got so bad that it was a choice between complete outage and using the Under Attack mode.
I'm the author of the blog. I don't know. Internally, we were half joking that we were going to get ransom notice, but we never did.
The only thing that sort of correlates with this attack is that before it started, we began rolling out some slightly more aggressive rate limits one by one. This was mostly because anytime any new "company" thinks they're going to catchup with Claude/OpenAI, they scrape us very aggressively (and they're not respectful about it). My guess is that the attackers behind this attack were already probing us (they were) and they thought the window of opportunity might be closing.
I definitely fall into that, on linux it's just a lot of extensions, scripts etc. Thing is, it's also brittle and not really useful for anybody to talk about it? Not sure I get what you mean with the third sentence.
Just curious, if you're tolerant of scraping, do you make an archive of all your content available so that scraping is unnecessary, and if so do the scrapers prefer that?
For somebody who knows a bit how things are set up, or is willing to spend 10 minutes researching, it's a no-brainer that you can just "git clone" entire linux kernel development history, or download entire wikipedia [0].
Alas, large number of scrapers are not willing to spend those 10 minutes, it would appear. So, here we are.
It's terabytes of content and other than we're the host not really related to each other. However, for most projects, it's possible to download a zip file of all the HTML docs for that project. We have a lower rate limit to pull these, but a scraper can pull thousands of docs at once. We only host a few hundred thousand projects so pulling a zip of the latest docs for all of them could be done in a day or two at a very reasonable rate.
It's also possible to request the docs already processed into markdown[1]. Lastly, basically all of the docs come from Git. A smart scraper could just clone a project's repo.
I use DDG on Android and it works pretty well there. I also saw this line in the article and it's somewhat of a stretch to call DDG either a "major" browser (as you point out) or a "non-Chromium browser". My understanding is it uses the browser engine bundled with the OS on Android/iOS/Mac/Windows and it isn't available for Linux.
The wikipedia article on it says:
> The core browser functionality is the WebView component provided by the operating system. This means the browser engine is Blink on Android and Windows, and WebKit on iOS and macOS.
Interesting, thanks for the context! I hadn't realized it was mobile-only, which is probably part of why I hadn't realized it existed. And yeah, it does seem like they maybe didn't know what they were talking about if the engine is literally Blink.
Nowadays, somebody can just ask claude to build them a scraper/bot that hooks into a proxy network and all of a sudden they can easily send 20k+ reqs/min from hundreds or thousands of IPs cycling them as they get rate limited or banned. In my work, the scrapers have gotten way more aggressive in the last 2 years or so. Frankly, I'm happy there is a solution.
There may be things to criticize Cloudflare for, but the problem of bots and scrapers destroying the open web was getting worse no matter what.
No tracking. No cookies. No behavioral targeting (targeting based on stuff you've previously done). Every website where our ads appear AND every advertiser is hand approved. No JS from advertisers: just a plain JPG/PNG and text.
We're small but on track to pay out $500k to publishers this year.
reply