Trying to turn it around by once again claiming that ai is an existential threat and will take everyone’s job right before brutally murdering every human.

These companies don’t appear to give a single fuck about public perception. Only their investors and the government (in anticipation of the biggest bailout in human history).

> I've found I can produce 10x more code than I could otherwise, of lower quality than I would otherwise, but the speedup is worth it.

The speedup of slop production being “worth it” is what we, as a society, are having trouble evaluating at this point in time. In all likelihood it’s worth it only in the short term.

There is, however, a question of scale.

What is the question? We can obviously pursue multiple cases simultaneously and we can do so effectively.

Scale affects priority. I don't think we have enough federal employees to actually go after everything Trump has done.

And yet those companies aren't collapsing because it became easier to rewrite their software, but because llms themselves have replaced a core part of their business (such as translation or simple art).

Seems like so far the market is not convinced at all that SaaS companies as a whole are any more disruptable than before.

That and talking to a lot of users, Duolingo reduced the quality of their service moving to LLM generated content -- a lot of people moved to other apps.

They score 99% but then can’t explain _why_ their solution works with any accuracy. Not as useful for learning as one might think.

are you referring to this section? > Verification is different from generation: Models scoring 98 can solve problems but can't always explain why their approach works at the level a human mathematician would. It doesn't say it can't explain why with any accuracy, it just says it can't *always* explain at the level of a mathematician, but most of us don't have such a mathematician at our beck and call to answer our questions anyways (thinking of the perspective of a self-learner outside of formal education)

It’s not just math. Anecdotally, LLMs struggle the same way with software engineering where the code they write is correct (compiles and passes tests), but reasoning is wrong often enough to eliminate most trust in these models’ ability to explain codebases or even features they themselves produce. It’s not about the _level_ of the supposed intelligence where a model struggles to summarize things succinctly or simply enough (responding to the “pocket mathematician” comment) or can’t grasp certain concepts at all (if so, how tf is it able to apply them?). It’s that by their design LLMs have no concept of truth and no concept of causality. They guess with every single inference and it’s very hard as a user to understand which guesses are more or less certain, since, you know, confidence ratings aren’t part of these models’ design either.

It was this way well before vibe coding. Over a decade of zero interest rates combined with talent wars and other anticompetitive behaviors by large tech companies did the industry in.

> Now back to AI for software creation. "Review is the bottleneck because EVERYTHING must be judged by a human." No. It doesn't. We just need to build AI review systems, that will do reviews better than (or at least as good as) humans. The human review quality bar is far below 100%. Far far far. If we can show (likely in the next 12-24 months I think) that AI review quality is consistently above the human review quality - again, based on evidence, based on real data - then that's it, then there's no good reason to have humans review the code.

Let me know once the majority of software engineering organizations start only checking in markdown files and let code be generated non-deterministically from these specs in CI. If this is not happening now, there’s clearly a sufficiently high level of distrust in blind LLM output (both code itself and reviews).

And even then, are you suggesting humans shouldn’t at least review the markdown specs? Why not have LLMs review the specs then? Is there, perhaps, some fundamental quality to human review process that is desirable here?

This is weird I didn't imply or say any of these things that you're arguing against right now.

The challenge with comparing these things to humans, is that humans learn. A newbie might not respect your organization’s set of policies on day one, but what about 3 months in? Or 3 years? Meanwhile there’s still no reasonable mechanism for automatically fine tuning LLMs or adjusting their harnesses to make them better at completing your organization’s objectives more successfully. They’re still overwhelmingly governed by the shared weights and harness policies found to be successful for the average case.

Models learn. It just costs $10B and 1 year to do what a human does every night.

LoRAs or even full fine-tunes would be much cheaper than that, and with some investment in the right infra could be updated regularly. And at least LoRAs can be swapped in and out cheaply, making them usable in large-scale inference providers. But there seems to be limited appetite in offering this. Both Anthropic and OpenAI no longer offer fine tuning for current models

does lora do a good job at teaching the model new things that werent in the training data?

without trillions of examples of following instructions at a million context length, im not convinced the behaviour is in the weights to begin with

No. Lora is for tuning behaviour and how the model applies what it learned in training. Teaching a model new facts is still expensive.

There was a recent-ish paper[0] from Sakana AI about baking facts from a document corpus into a LoRA adapter. Claimed near perfect recall on very large needle-in-haystack testing. Haven’t tried it myself though.

[0]: https://sakana.ai/doc-to-lora/

interesting... thanks.

Is this because of catastrophic forgetting, or is that more or less solved?

Is it, alternatively, because loading your LoRA plus someone else's can't be done in the same batch, so inference is made harder or becomes impossible to do at scale?

Or something else?

This is not about learning only.

It is about context window too so no matter how much money you put to train your model it can still have degraded performance in following instructions but it will be better to execute the instructions they can follow.

It is also about the harness and how that can help drive the model and pick and choose what to include or not in the context.

These LLMs do not understand the project. For them any next word is good as long as it was picked by the token predictor. It does not have any way to understand but only to do. probability distribution over their vocabulary and if that vocabulary is tainted and lost parts of the original context what is a good candidate there will not match the intention of the initial project.

That's not a model learning. It is model replacement.

At least these costs are currently preventing the planets surface from being covered in paperclip maximizers for the moment.

I think the issue is that obtaining empirical proof of AI or manual coding being more efficient is very difficult, since true costs and outcomes aren’t known for months and often years. AI _is_ faster at producing short term results though, so naturally, given the state of the industry, everyone is piling both money and time into AI driven workflows and manual work is met with suspicion, if not outright discouragement. Engineers might hate AI, but they need their paychecks.

Hopefully everyone? Else your job could have been outsourced or replaced by a junior with access to Google and StackOverflow way before LLMs (it just wasn’t due to zero interest rates and proliferation of bullshit jobs in tech companies).

That's not been my experience. Most professional software is just a CRUD app in one form or another.

Yes and it’s probably better for society as a whole that all of those can now be vibe coded by someone who is not a full time developer.