That's like what, one major incident per month now, Nov 18, Dec 5, and now this one?
I'll bet JGC can write his own ticket by now, but unretiring would be really bad optics. He's on the board though and still keeping a watchful eye. But a couple more of these and CFs reputation will be in the gutter.
My understanding of Cloudflare's history is that they built their reputation and their client base on some high quality products.
And instead on focusing on maintaining those, they decided to go for more money, first adding new features on their products (at the risk of breaking them) and then adding new products altogether in a move to start being an actual cloud provider.
Priorities shifted from the quality products to pushing features daily, and the person who built and maintained the good products probably left or have been assigned to shinier products, leaving the base to decay.
As a daily user, its quite frustrating to have a console that is getting far worse than AWS/Azure, and features that are more a POC than actual production-ready features.
Hm. Mixed feelings, I would like a more rigorous approach to this a lot better. CF is really too big to fail now, I've had absolutely no qualms about recommending CF but after the last couple of months I'm revising that until things are measurably better.
Your legacy is one of showing how to apply good engineering principles to complex problems at scale and I think CF is risking that reputation right now.
That’s what I also thought when I saw this incident. I wonder if there’s something up internally at Cloudflare or that it was always like this.
I feel like something such as a route leak should not be something that happens to Cloudflare. I’m surprised they set their systems up to allow this human error.
John left in April last year I think so it probably isn't directly related, so please take my comment in jest, but still it is worrisome, CF is in many ways 'too big to fail' and if this really becomes a regular thing it is going to cause a lot of people focused on their 'nines' to be pissed off.
One thing to their credit though: BGP is full of complexity and it definitely isn't the first time that something like this goes wrong, it is just that at CF scale the impact is massive so there is no room for fuckups. But doing this sort of thing right 100% of the time is a really hard problem, and I'm happy I'm not in any way responsible for systems this important.
Whoever is responsible learned a lot of valuable lessons today (you hope).
The older I get, the less I buy into "too big to fail" arguments. I now view it as "can't fail soon enough". The sooner it breaks down, the sooner something better will supplant it.
This last sentiment holds true generally since organizations no longer subject to meaningful competition inevitably squat on their laurels and stop excelling at the things they used to be good at. We've seen it everywhere - Boeing, Google, Microsoft (with OS's), etc.
There was never much of an argument behind "too big to fail", it is generally a euphemism for upper-class welfare. In a more realist world, "too big to fail" is a mis-statement of "too risky to keep". Everything fails eventually and keeping incentives aligned relies on having a mechanism - failure - to flush out incompetents.
> The sooner it breaks down, the sooner something better will supplant it.
That's not always possible, because the counterparty - aka threat actors - is always growing bigger, and you practically need to be the size of Cloudflare, Akamai or the Big 3 cloud providers to be able to weather attacks. You need to have big enough pipes to data centers and exchange points worldwide, otherwise any sufficiently motivated attacker can just go and swamp them, but big pipes are helluvalot expensive so you need to have enough large and financially capable customers.
That's also why Cloudflare has expanded their offerings so much (e.g. Zero Trust), they need to have their infrastructure at some base load to economically justify it.
And that's also why Cloudflare will not be kicked off the throne any time soon. First of all, the initial costs to set up a competitor are absurdly high, second, how is a competitor supposed to lure large long term customers away from CF?
Any case, the real "fix" to Cloudflare being too-big-to-fail isn't building up competitors, it's getting the bad actors off of the Internet. Obviously that means holding both enemy (NK, Russia, China) and frenemy (India, Turkey) nations accountable, but it also means cleaning up shop at home - the aforementioned nation states and their botnet operators rely on an armada of hacked servers, ordinary computers and IoT devices in Western countries to carry out the actual work. And we clearly don't do anywhere near enough to get rid of these. I 'member a time when writing an abuse@ mail report that this would be taken seriously and the offender being disconnected by their ISP. These days, no one gives a fuck.
"Threat actor" is a relative definition, because for Italy the Cloudflare CEO was a "threat actor" who openly threatened availability of their systems.
Cloudflare knows they are just a glorified firewall + CDN that's why they desperately push into edge computing and getting these dozens of features.
They made themselves 'Guardians of The Internet' then gave up. If they cared, these things wouldn't happen. How many more outages, accidents, incidents that effect millions of customers and millions of customers for other services are needed before they 'care'?
They don't, because at the end of the day it's not their problem, the money rolls in regardless.
It's sad, but it's how it is. If they cared, these things wouldn't happen. They have a lot of responsibility, but show none whatsoever.
I'll bet JGC can write his own ticket by now, but unretiring would be really bad optics. He's on the board though and still keeping a watchful eye. But a couple more of these and CFs reputation will be in the gutter.