> here are some example ideas that are perfectly true when applied to regular software
Hm, I'm listening, let's see.
> Software vulnerabilities are caused by mistakes in the code
That's not exactly true. In regular software, the code can be fine and you can still end up with vulnerabilities. The platform in which the code is deployed could be vulnerable, or the way it is installed make it vulnerable, and so on.
> Bugs in the code can be found by carefully analysing the code
Once again, not exactly true. Have you ever tried understanding concurrent code just by reading it? Some bugs in regular software hide in places that human minds cannot probe.
> Once a bug is fixed, it won’t come back again
Ok, I'm starting to feel this is a troll post. This guy can't be serious.
> If you give specifications beforehand, you can get software that meets those specifications
You should read the footnote marked [1] after "a note for technical folk" at the beginning of the article. He is very consciously making sweeping generalizations about how software works in order to make things intelligible to non-technical readers.
> I’m also going to be making some sweeping statements about “how software works”, these claims mostly hold, but they break down when applied to distributed systems, parallel code, or complex interactions between software systems and human processes.
I'd argue that this describes most software written since, uh, I hesitate to even commit to a decade here.
For the purposes of the article, which is to demonstrate how developing an LLM is completely different from developing traditional software, I'd say they are true enough. It's a CS 101 understanding of the software development lifecycle, which for non-technical readers is enough to get the point across. An accurate depiction of software development would only obscure the actual point for the lay reader.
At least the 1950’s. That’s when stuff like asynchrony and interrupts were worked out. Dijkstra wrote at length about this in reference to writing code that could drive a teletype (which had fundamentally non-deterministic timings).
If you include analog computers, then there are some WWII targeting computers that definitely qualify (e.g., on aircraft carriers).
He is trying to lax the general public perception around AIs shortcomings. He's giving AI a break, at the expense of regular developers.
This is wrong on two fronts:
First, because many people foresaw the AI shortcomings and warned about them. This "we can't fix a bug like in regular software" theatre hides the fact that we can design better benchmarks, or accountability frameworks. Again, lots of people foresaw this, and they were ignored.
Second, because it puts the strain on non-AI developers. It blamishes all the industry, putting together AI with non-AI in the same bucket, as if AI companies stumbled on this new thing and were not prepared for its problems, when the reality is that many people were anxious about the AI companies practices not being up to standard.
I think it's a disgraceful take, that only serves to sweep things under a carpet.
I don't think he's doing that at all. The article is pointing out to non-technical people how AI is different than traditional software. I'm not sure how you think it's giving AI a break, as it's pointing out that it is essentially impossible to reason about. And it's not at the expense of regular developers because it's showing how regular software development is different than this. It makes two buckets, and puts AI in one and non-AI in the other.
He is. Maybe he's just running with the pack, but that doesn't matter either.
The fact is, we kind of know how to prevent problems in AI systems:
- Good benchmarks. People said several times that LLMs display erratic behavior that could be prevented. Instead of adjusting the benchmarks (which would slow down development), they ignored the issues.
- Accountability frameworks. Who is responsible when an AI fails? How the company responsible for the model is going to make up for it? That was a demand from the very beginning. There are no such accountability systems in place. It's a clown fiesta.
- Slowing down. If you have a buggy product, you don't scale it. First, you try to understand the problem. This was the opposite of what happened, and at the time, they lied that scaling would solve the issues (when in fact many people knew for a fact that scaling wouldn't solve shit).
Yes, it's kind of different. But it's a different we already know. Stop pushing this idea that this stuff is completely new.
'we' is the operative word here. 'We', meaning technical people who have followed this stuff for years. The target audience of this article are not part of this 'we' and this stuff IS completely new _for them_. The target audience are people who, when confronted with a problem with an LLM, think it is perfectly reasonable to just tell someone to 'look at the code' and 'fix the bug'. You are not the target audience and you are arguing something entirely different.
Let's pretend I'm the audience, and imagine that in the past I said those things ("fix the bug" and "look at the code").
What should I say now? "AI works in mysterious ways"? Doesn't sound very useful.
Also, should I start parroting innacurate outdated generalizations about regular software?
The post doesn't teach anything useful for a beginner audience. It's bamboozling them. I am amazed that you used the audience perspective as a defense of some kind. It only made it worse.
Please, please, take a moment to digest my critique properly. Think about what you just said and what that implies. Re-read the thread if needed.
> He is trying to lax the general public perception around AIs shortcomings
This is not at all what I'm trying to do. This same essay is cross-posted on LessWrong[1] because I think ASI is the most dangerous problem of our time.
> This "we can't fix a bug like in regular software" theatre hides the fact that we can design better benchmarks, or accountability frameworks
I'm not sure how I can say "your intuitions are wrong and you should be careful" and have that be misintepreted as "ignore the problems around AI"
The word intuition here have power. Surely you must have noticed that vibe coding and intuition have a strong connection.
There are people (definitely not me) who buy 100% of anything AI they read. To that beginner enthusiastic audience, your text looks like "regular software is old and unintuitive, AI is better because you can just vibe", and the text reinforces that sentiment.
> these claims mostly hold, but they break down when applied to distributed systems, parallel code, or complex interactions between software systems and human processes
The claims the GP quoted DON’T mostly hold, they’re just plain wrong. At least the last two, anyway.
Focusing on the last two, which I called out specifically:
> Once a bug is fixed, it won’t come back again
Regressions are extremely common, which is why regression tests are so important. It is definitely not uncommon for bugs that were fixed once to come back again. This statement doesn't "mostly hold".
> If you give specifications beforehand, you can get software that meets those specifications
In theory, maybe, but in practice its messy. It depends on your acceptance testing. It depends on whether stakeholders change their mind during implementation (not uncommon). It depends on whether the specification was complete and nothing new is learned during development (almost never the case). Providing a specification in advance does not necessarily mean what you get out the other end meets that specification, unless its a relatively small, trivial, non-changing piece and the stakeholders have the discipline to not try change things part-way through. I mean, sure, it may be true that if you throw a spec over the wall, then you will get something thrown back that meets the spec, but the real world isn't so simple.
Hm, I'm listening, let's see.
> Software vulnerabilities are caused by mistakes in the code
That's not exactly true. In regular software, the code can be fine and you can still end up with vulnerabilities. The platform in which the code is deployed could be vulnerable, or the way it is installed make it vulnerable, and so on.
> Bugs in the code can be found by carefully analysing the code
Once again, not exactly true. Have you ever tried understanding concurrent code just by reading it? Some bugs in regular software hide in places that human minds cannot probe.
> Once a bug is fixed, it won’t come back again
Ok, I'm starting to feel this is a troll post. This guy can't be serious.
> If you give specifications beforehand, you can get software that meets those specifications
Have you read The Mythical Man-Month?