How about the same thing we do when companies leak half-the-nation's personal data twice a month. Nothing.
When companies get hacked and millions lose their personal data, nobody cares. When individuals get hacked, it's a major issue that justifies locking down consumer's hardware to protect them from the burden of controlling their own devices. See how that works?
China reducing their oil imports was a huge boon to the entire world. China basically said "look everyone, we're strong, we'll reduce oil imports to help the rest of the world, and that will benefit all of us together, and also we're manufacturing and selling EVs and solar panels".
China is showing the world they have reached the lofty ideal of being a functional society, and I hate that an authoritarian Communist government was how they did it. I don't want to live under such a government, but I can't deny it's looking like a stronger form of government right now.
For the rest of us, if we want democracy to be better than authoritarian Communism, then we actually have to do the "be better" part.
This is the might our 20 trillion dollars spent on military has bought us.
A month or two ago they sent out an email asking common military members for ideas to deal with Iran. We've paid think tanks millions of dollars for plans like that. Guess they weren't worth anything in practice though.
All that money spent on plans and equipment and it sustains--what?--like 3 weeks of bombing Iranian children? (I mean, we did bomb a children's school on the first day, and nobody has been held accountable, so I'm still bitter about that. Let's be better please.)
The US has a lot of military hardware but questionable leadership just now. At least they've brought in testosterone tests so the military should look the part.
This is what you get when you hire a TV personality who cares more about soldiers looking masculine (and white) than anything else and launch an illegal war for no reason other than "those guys make me angry."
We are governed by people who think the very idea of expertise is dumb. A system of pure id and grievance. Like children playing war in the bath tub.
What about the end game where my own computer running my own code controls the experience? There's a 10 trillion dollar incentive to never allow that reality.
Yeah. We are definitely headed to a world where, more than ever, success is determined simply by whether or not you're allowed through the closed door.
The current state of alignment is that we don't know how to make it only as bad as Pol Pot was made by his parents, teachers, genetics, circumstances, etc.
This remains true regardless of if it is or isn't kept away from the public.
"Helpful, harness, and honest": when used by a bastard, even just helpful and harmless are in direct conflict with each other. you may hate the government, but what about every radical group of extremists that wants to take over your government? Are none of them worse?
None of this denies the problems with governments (if or not they take this tech for themselves and refuse it for others), just that it's a lack of imagination to say:
> All the worst outcomes involve taking this technology away from the public
I agree an argument can be made that not all the worst outcomes include restricting public access, but focusing on niche extremist groups (who might also get access to it through government programs by some of them being government employeed) is not a good argument to protect the public.
Governments having unbalanced power can more easily lead to authoritarianism, and thus risk to the public.
At this point, I worry a lot more of billionaires Musk, Altman, Thiel etc then of goverments in general. Thrir project is to get all power and if they succeed, it will be way worst then what we have seen so far.
How much is China learning from open models? Or the other way around, how much is the US losing worldwide mindshare by refusing to allow non-Americans access to bleeding-edge models and letting China fill in the gap? (Playing catchup with Mythos is still catching up, etc.)
The last batch of open weight models releases by Chinese companies are on par with the performance of US-based frontier models, even though they are designed to run on pretty unimpressive hardware.
This is very misleading. There is no Chinese model currently that is on par with Fable or Astra. Kimi K3 is pretty good, but it's not that good.
Furthermore the Chinese models in that class are appropriately large. K3, for example, is a 2.8T-parameter model. Qwen 3.8 Max, another comparable model, is 2.4T params. Even with MoE, these are not "designed to run on pretty unimpressive hardware". Stuff like Qwen3.7-27B is, but it is also not even in the same ballpark as Opus, never mind frontier.
Across all their labs? Probably more than OpenAI and Anthropic learned from Astra/Mythos combined. The American frontier is being driven by an abundance of compute, which doesn't seem to scale efficiently.
> how much is the US losing worldwide mindshare
They're losing US mindshare. I pay $3/month for a Z.AI subscription and get billions of Opencode tokens. The $20/month price point is insane for the way that Claude and Codex treat their users, and that money doesn't go towards anything good like open-sourcing their models. It's a doomed product.
> The very best outcomes involve us intentionally and collectively turning our attention away from this technology.
To expand on the "pure fantasy" sister comment: There is just no way this will happen. It's in the spirit of "we can just stop all wars" and "we can just end world hunger". Technically it's very easy to do. Socially it's impossible to do. Unless you ignore realities.
We can end world hunger but people would be way more mad if we did, on account of all the military action involved to conquer every non-compliant regiemes and the multiple quagmires created.
The danger lies with someone asking the machine to solve those two questions and it decides to cheat the solution. Launching every nuke in the world stops all wars, just like wiping out 99% of humanity ends world hunger.
I don't see how that is more dangerous than two crazy head of states deciding that it's time for armageddon. The outcome is the same (99% of humans dead), it's equally easy to technically not do it (don't press the button, don't continue with LLMs), but also equally hard to regulate away given the real world. And that was my point.
Why? There are numerous technologies we have ignored or abandoned for numerous reasons. Many of the benefits of AI is "requires less humans", but in an age where we question what work people could possibly do in the future, human labor isn't that hard to find.
If the amish can do what they do for whack religious reasons, people could manage it with ai for cultural and social reasons. And I never saw an amish person starve to death. And that is if people don't get pissed enough to start burning stuff down and instead try to be peaceful hippie homesteader types.
But notably the amish don't preclude the rest of us from existing. Some subset of humans could turn their attention away from AI but it would presumably still exist and continue to be developed.
I can't think of any economically beneficial technologies that we've collectively ignored. If you manage to come up with a counterexample then that's an opportunity to make some money for yourself. It's a fundamentally unstable state given our economic system.
Supersonic passenger aircraft were operated profitably and no longer exist.
There’s levels of R&D required for many technologies where the question goes beyond could this be profitable to what are the risk vs reward that this specific project will succeed.
Until the oil shocks, and until “normal” planes became faster.
I happen to know a couple older and very wealthy people, and they say that even though Concorde was a lot of fun when it was a novelty, they now prefer to have a couple more hours flight and enjoy a 777 premium cabin rather than the cramped Concorde interior.
Anyway, the proof is in the pudding: if no one but national carriers ever bought and operated Concorde or Tu-144 at scale, it's not because of some Amish-like sentiment in the flight industry, but simply because that didn't make sense money-wise.
> Supersonic passenger aircraft were operated profitably
Were they? IIRC joint project by British and French national airlines, they expected to sell loads, hardly anyone wanted to buy the planes, the two airlines kept them flying out of government embarassment.
Sure, the regulation that guarded against externalizing certain costs might well have been the only thing precluding profitability. Importantly it wasn't some shared cultural value leading to voluntarily leaving money on the table. Rather the regulator imposed it.
We are able to effectively regulate things like the operation of massive aircraft, eugenics, the dumping of toxic waste, or the refinement of nuclear material. But there are also plenty of things that we can't effectively regulate for purely practical reasons.
Were they? Did Concorde provide return on investment? When was the break-even expected? Sure, if we look at it as R&D subsidized by governments of UK and France, then yet, it was profitable.
In particular, the problem with it was that it could not get to supersonic speeds over urban areas, which significantly limited routes where it made sense, and the range was not good enough to cover longer distances.
But after accounting for maintenance & etc were they more profitable than sinking the equivalent amount of money into something else, such as slower aircraft? Notably there are currently efforts to develop new supersonic passenger liners.
And even if we did manage to limit/cull hardware development, what would keep people from continuing to find ever more efficient models runnable on contemporary hardware?
Bleeding edge models are putting the desk-job competencies of multiple professional fields into something with as many parameters as a single large rodent has synapses.
Because "we" embodies a hugely diverse set of beings with different wants, needs, goals, and priorities. Not to mention different morality and ethics.
So sure, let's say you get 95% of the world to not use or work on LLMs. 5% is still enough to build something that brings forth the End Times.
That's the old checklist trope of "Your idea won't work because: [x] it requires that everyone in the world agrees to do something, all at the same time."
> There are numerous technologies we have ignored or abandoned for numerous reasons.
Are there? Nukes are a thing. Chemical weapons are a thing. Cluster bombs are a thing. Biological weapons are a thing. What is not a thing that shouldn't be? We say certain things should not be a thing, but then behind the scenes we made them a thing anyway.
On chemical weapons, most large state actors have got rid of them. Not because of any moral reasons, though, but simply because they don't actually work all that well in modern conventional warfare. This is then framed as an ethical issue, but you only need to look at where the same countries stand on e.g. landmines to realize how much bullshit it all is.
Conversely, where you do still see chemical weapons used, it's usually asymmetric conflicts where "just gas the rebels" actually works much of the time and is much cheaper than other options. Big guys can afford the other options though, someone like Assad, not so much.
I'm baffled that people expressing your opinion don't understand what they're advocating for. Current generation smartphones are able to run AI models, people have managed to run very large models (even if slowly) via streaming from disk. The math for the underlying implementation is relatively trivial.
Abandoning this technology means a cult-like extremist shift in culture against computers, or a global surveillance state of unprecedented scale and invasiveness. Not even getting into how much it requires us to give up on science, given the shared computational needs.
Watch, very soon this is going to politically divide. They’re sowing the seeds right now with “antiai” campaigns.
The right will for a change be for it but will be able to be talked into regulation because you know “small government” and all only when convenient. The left will bitch and moan about fairness and copyright and automation and UBI, they’ll be the doomers and worldwar chicken littles. The Uniparty will be for it, but only against the public having anything good.
What will be funny to me is that normal tech people outside the frontier model companies are about to find themselves without a home on this topic.
The human brain operates at better levels of intelligence than the best LLMs, at 20 watts of power. We are a long way to that kind of efficiency, it will probably take both bespoke hardware and algorithmic improvements to catch up to nature.
Only if you think in terms of perceived raw intelligence, but self-update is a form of valuable self-improvement that could benefit current models a lot, if they could commit facts from context into their weights cheaply and reliably.
I think the idea is fundamentally improved architectures. For example, transformer-based models were an incredible stepwise improvement. Self improvement would be a model discovering a stepwise improvement similar to the transformer. And presumably the improved models from that would be more likely to make further advances still.
Learning from training data is technically self-improvement but not the sort that is typically meant in this context.
Have you tried identifying exactly what is unreadable about it and telling it to make it more readable?
I had GTP-5.6 write some shader code recently and it wasn't very clear to me. I spent about an hour chatting with the until I understood the concepts and was able to express them back to the AI using math formulas and variables named in a way that made sense to me. The AI then rendered the code using the formula and variables I was familiar with and it was clear to me.
I could have after the AI explained it all to me, but at that point the AI knew what aspects I valued and wanted to emphasize to make it readable, so it just wrote it for me. By that point, the AI was just typing for me.
I'm not familiar with how Rust's async works. What if Rust used the function argument approach, how would that work? Would people pass around a reference to an async executor as arguments (or similar)?
When companies get hacked and millions lose their personal data, nobody cares. When individuals get hacked, it's a major issue that justifies locking down consumer's hardware to protect them from the burden of controlling their own devices. See how that works?
reply