To me the analogy here is that the new boss rolled in and sold all the fire extinguishers. That by itself doesn't set the building on fire - it doesn't even increase the chances of a fire occurring on any given day. But when one does...
Every SRE knows that the leading cause of outages by far is someone making a change to the system. Twitter isn’t shipping many new features right now or even doing much maintenance. But eventually they will have to.
So the analogy becomes, the new boss sold all the fire extinguishers and also placed a short temporary ban on cooking in the building. But eventually people are going to start turning on stoves again… and then…
This is correct, but also: a sudden decline in maintenance is a kind of change in its own right. Even automated processes have humans in the loop and manual sign-offs; there's always some cronjob or short-lived certificate somewhere that a human was dutifully maintaining.
Those things aren't going to fail any sooner than they would have anyways, but they're going to fail a lot harder due to the loss of institutional knowledge.
Except that he at the same time demanded that people invent an entirely new dish by the end of the week, and now they are scrambling to try to figure it out. Already the DMCA auto-takedown bot is apparently broken and people are posting entire movies on Twitter. I would expect other peripheral systems to start breaking down as nobody is maintaining them even as other parts of the system are being changed.
I interpreted the GP's comment less as a moral claim ("the DMCA bot is good") and more as a claim that the DMCA bot's failure is a strong indicator of internal instability (given that it sits directly at the intersection between Twitter's profit interests and microservices architecture).
Put another way: being unable keep a little bot running, one that keeps an entire industry happy, doesn't bode well for other components of the service.
It seems self-evident that the bot was considered low priority, since it isn’t working anymore. But nobody is disputing that: they’re saying that the fact that it is low priority does not bode well.
If it was a prerequisite to land $100M ARR from all the media properties’ marketing budgets to advertise the multi-billion dollar pipelines of the movie and entertainment industry, that lil’ bot was the gate to $11,415 per hour of revenue at risk if its uptime failed to sufficiently please the attorneys and auditors from those customers.
I mean, does Twitter want to be a party to a copyright lawsuit? If not, following legitimate looking DMCA notifications (and legitimate looking DMCA counter-notifications) and responding to suponeas as necessary gets you an affirmative defense for copyright infringement.
You may not like it, but having a bot do that probably saves a lot of legal hassle.
Hm? Intellectual property is explicitly carved out of 230, and even if it wasn’t: it isn’t user generated. Content providers are regularly found liable for infringement on their platforms, especially when the plaintiff can demonstrate willful negligence (which in this case would include discontinuing a seemingly effective scanning system.)
> Nothing in this section shall be construed to limit or expand any law pertaining to intellectual property.
If section 230 from the CDA of 1996 provided immunity from copyright claims, there would be no reason to include procedural requirements for processing claims in the DMCA of 1998.
Many jurisdictions take an even harsher line when it comes to being complicit in intellectual property abuse. We saw this famously with The Pirate Bay, Napster etc.
Well, Disney won’t care why their copyrighted material is publicly available, noone likes this sort of copyright, but if Elon wants to avoid huge fines he better (make someone) fix it ASAP.
Given that Twitter already offered premium API access, they've got billing in place, so now they add a new form that, once your credit card is verified, flicks a boolean on an account that was previously flicked by another process.
It’s more than that. For one it’s not an existing boolean, there are now two different kinds of blue tick that are presumably stored separately. Blue is also supposed to give the user fewer ads (while making them more relevant) as well as additional weight in feed ranking algorithms. It’s also intended to be offered worldwide which adds a lot of complication to things like payment flows.
I’m not saying it’s going to bring the site down tomorrow but that one feature touches on a lot of services.
Clearly worldwide payments is an issue otherwise they’d have rolled it out worldwide day one, and they didn’t. There must be something holding that back.
Plus I really don’t think you can compare B2B payments for premium API access to end user payments. Not least because they aren’t going to be going the same route: a huge number of them will be via Apple or Google in-app purchasing. Ask anyone who works with those systems, it isn’t a quick plug and play job.
In general though, a new subscription tier, feed algorithm changes, UI changes… if these aren’t, what is a big change in your book?
Definitely stealing this as the right way to frame the issue. What would normally be a small kitchen mistake turns into no longer having an apartment complex.
Perhaps a closer analogy would be that the new boss rolled in and threw away 80% of the fire extinguishers.
Whether that will spell disaster when there's a fire depends on whether the building had too many fire extinguishers to begin with and whether the boss can buy new, better fire extinguishers to replace some of them before there's a fire.
> Perhaps a closer analogy would be that the new boss rolled in and threw away 80% of the fire extinguishers.
If we're deep-diving it'd be closer to say that he rolled in and sold 80% of the stuff, largely sight-unseen, and if a fire breaks out he'll find out how much of that stuff was fire extinguishers.
He rolled in, sold most of the fire extinguishers, and made a big show of trying to make cherries jubilee while shoving his dick in one of the few remaining fire extinguishers. Let's be clear, it isn't just the erratic layoffs it's Musk's incessant meddling that's going to be Twitter's downfall. He literally took down SMS based 2FA because "microservices bad". He fired the payroll and tax departments (HR too?). He's scared off Twitter's main source of income while saddling it with significant debt.
As an SRE I would have been shocked if Twitter failed catastrophically (well moreso than broadly disabling authentication) in short order. However failure is pretty much inevitable at this point given the damage that E-Lon is actively doing.
Well, define 'failure'. Minor outages like the one you are talking about were happening from time to time long before Musk bought Twitter, and it even suffered long outages frequently - remember all those fail whales?
I meant that there will be no catastrophic failure that will permanently (or even for a few days) stop Twitter from working at all.
failure. noun. with no security team, hackers are able to get in easily.
everyone's DMs leak, all the anonymous accounts have their identity revealed, and all Twitter's clients (advertisers) have their bank account info made public.
Yeah, I don't think anything remotely close to it will happen, unless some of the fired developers have left themselves some backdoors which they'll give (sell) access to it to someone.
does this stuff happen automatically? is there a robot that goes out and reads about all the new zero-day exploits and patches all the software without human intervention?
On Hacker News, everyone's a comedian! And, yeah, as the sibling comment pointed out there've already been failures as a result of some musky action. While Twitter isn't likely to fail on its own, E-Lon is actively causing problems. You need people to deal with that, and even if he had motivated, relevant, and competent engineers… how long will they stay motivated without a paycheck?
Let's not forget that whatever code monkeys are left are now personally liable for running afoul of the FTC. Whatever motivation they may have now will run out pretty damn quick once they stop getting paid.
Pretty sure that the developers that are left will not be liable for anything unless they are knowingly participating in criminal activity such as criminal negligence that is the direct cause of someone getting seriously injured or killed.
Generally speaking prosecutors want to target the highest level individuals responsible for directing such activity in the first place, not low level implementers who have little say one way or the other.
For now I'm talking about the FTC consent decree, so administrative penalties not criminal charges. Musky fired the folks who were responsible for ensuring compliance.
How do you propose Musky does that with no payroll department? E-Lon walked back the mandatory return-to-office policy last week. Surely if he could find (or thought he could find) suitable replacements he would be pretty comfortable demanding RTO.
I'm not really familiar with how they do payrolls in USA, but I'm pretty sure it isn't some rocket science (pun intended) and can be done rather cheaply by an outsourced firm.
Also, why are you (and many others here) refer to Musk as "E-Lon"? Is it supposed to be a derogatory nickname?
It's simply an abbreviated version of Elongated Muskrat as far as I'm concerned. Payroll is easily one of (if not the) most complex systems at any company. It's not just statutory stuff but personnel stuff as well. With Twitter you're not just dealing with 50 states and the feds, but with every other company in which Twitter has (had?) employees. There's a cottage industry of payroll firms precisely because payroll is so obscenely complex.
Even if you outsource it you'll still need people within your company to manage your service provider. At one company I worked for they got all of their outsourced HR+payroll for free (indefinitely) because the provider (Gevity) consistently fucked up everything they touched. This was at a company of like thirty people.
If you're suggesting Twitter can simply outsource payroll, sure. But you do that before you fire your whole payroll department. You still need people to handle the transition.
Someone purchased some land for $1. Built a house for say $100. And now spends $100,000 a year making it the perfect place to rent, receiving $100,000 a year in rent.
Someone comes along and borrows $1m to buy that house. They feel ripped off but eventually are force to go ahead with the purchase. As a result they have to pay $100,000 a year in interest. They need this thing to be profitable!
To do this they need to cut back on the $100,000 a year spent. They decide go go in quickly and so email all the services saying "go hard or go home". So the plumbers, tradie, cleaners etc that don't like it leave.
As a side hustle also charge visitors to the house $9 to be allowed to wear their bowtie they used to wear for free.
Some of the people do maintenance jobs and improvements. They keep the termites out, fix subsidence issues, and so on.
And the house didn't fall down within 3 weeks of it being purchased.
