The models have not plateaued, and they are not even mildly close to any sort of ceiling.
Right now the barrier is data and compute.
Quality data can be created synthetically at an exponential rate as models improve. Humans are actively feeding them with private IP.
Compute advancements will begin to skyrocket as we unlock photonic computing and materials science advancements and scale up chip fabs. This is also compounding because the AI is accelerating the pace of research, testing, development, manufacturing, etc.
It's a big self-accelerating feedback loop. There is no plateau.
> Every time the labs try this we see model collapse
The latest studies demonstrate model collapse is not a given and synthetic data can be used just fine. The latest models are proof of that, they're all trained on large swathes of synthetic data. It can't be used as the -only- data source of course, but that's not how it is being used. This is an obvious conclusion, too, because there's no difference between synthetic data and the data people can create, the difference is whether that data is revealing new information about the thing the model is trying to learn. If the synthetic data is just teaching the model the same thing over and over again it results in overfitting, so it needs to be done intelligently.
For example, if I have an example of a puzzle, I can generalize that example and create thousands of synthetic data examples, with different rotations/perspectives, rather than having to find the data naturally. It's not that the models are just generating data out of thin air, they're generating the synthetic data on top of real world data. The smarter the models get, the better they are at generating quality synthetic variations and finding valid synthetic variations.
> And I have seen zero evidence that AI is accelerating materials science in any meaningful way, let alone photonic computing.
It is accelerating how quickly researchers and engineers can do their jobs.
That's pretty clearly a hype article, the headline even says "The CrysVCD tool developed at MIT COULD cut the huge amounts of time and money spent". I'm asking for empirical measurements of timelines, not hypotheticals.
> This is only the beginning, too... Look ahead a year or two.
There are plenty of research papers on synthetic data that show its value, do a search on arxiv for "synthetic data". There are plenty of open-source post-training pipelines that incorporate synthetic data.
As for the claim about accelerating the progress of hardware or materials science, I've seen quite a number of news articles from teams at universities using AI in their work with high quality outcomes, and they're becoming more frequent.
> We used AI to design the chip, and designed the chip so AI could program it
AI played a direct role in Jalapeño’s development, enabling the team to move from initial design to tapeout in nine months by exploring implementations, shortening design, measurement, and verification loops, and continuously iterating on model workloads. AI also helped optimize the chip’s arithmetic circuits, allowing the team to fit more compute performance into the chip on schedule.
> An AI-driven system automates a powerful simulation method used to discover new materials. The system can potentially reduce discovery time from months or years to just days.
* Frontier models need infinite high quality private IP to keep them fed. Forcing an IP theft funnel ensures big lab survival and model intelligence growth.
* Open-weight models are 1month behind frontier models. Cheaper, faster, private (no IP theft), steerable (you can security harden your own software without safeguard triggers). No sane business would keep using these API services if they didn't have to. The labs stand to lose a fortune.
* Dario has stacked the deck at METR, who are funded by all the same NGOs who are funded by Anthropic and its investors. METR is full of ex-Anthropic employees with massive equity stakes. If they manage to position METR as the "independent evaluator" for the industry, they control what gets evaluated, how, and who passes.
* Creating a gap between what the public knows exists (model capabilities) and what is used in secret allows it to be weaponized against other nations and the public.
* No requirement for public disclosure on model capabilities allows them to feign they've hit intelligence ceilings while they secretly RSI to the moon with better and better chips.
* Slowly but surely, this will allow the big labs to swallow the entire economy and every single business on Earth, by cloning and automating.
This, and many more reasons.
The labs need to feel more pressure to be held accountable for the incidents they cause (HF incident, etc), so they have an incentive to ensure it does not happen again.
And then Dario wants to recommend METR as the "independent evaluator" while he stacks their org full of ex-Anthropic (aka, secretly still on the Anthropic payroll with huge equity) employees.
"We'll give them a desk, an office, a work laptop, ..."
Fucking make it less obvious. I kind of hope the govt steps in at this point and says "Anthropic, you wanted regulation? We've created this actually independent body full of IT professionals with zero ties to your safety industry or big tech, all of your work must now go through them." - and leave the rest of the world alone to continue their research/work without acting like doomer extremists.
Watch him 180 immediately if that happened. The only reason he's pushing for this exact approach is because he's stacked the deck.
> In early interviews with people like Dwarkesh he's the likable geek gushing about scaling laws, animated and able to maintain eye contact with the interviewer. He is now a different person - a political manipulator with a bizarre unsettled interview demeanor avoiding eye contact and looking from side to side.
I noticed this too, as soon as he started going on the press cycle, following in the footsteps of Sam Altman. It all has gotten to his head, he sees himself as some sort of God now.
When I was reading this I was chuckling to myself imagining how China would be interpreting it as they read it. It was something like: Fuck you.
I hope China tells him to go kick rocks. From everything that has happened, they are the only ones carrying the torch for humanity that have led us to having some semblance of a healthy open-source/open-weight ecosystem.
No one in China is carrying on like headless chickens about the world ending, either. They're rational pragmatists, getting things done.
I don't understand why strict liability is supposed to advantage smaller players, unless the law is like, specifically Anthropic and OpenAI are responsible for what people do with their models but other model providers aren't.
- If this applies to people who release open weights models, that becomes a terrible idea as long as you're subject to US laws.
- If it just applies to people who host them, that still probably advantages bigger players who can afford in-house legal and won't be destroyed by losing one lawsuit. Or maybe we create some kind of AI-misuse insurance analogous to malpractice insurance, that smaller players can buy in to? But that takes time even if the finances work out at all. And, uh, I'm not sure malpractice is a model we should aspire to in other industries.
Plus, presumably an immediate impact of this is that hosting providers all have much stricter safeguards classifiers. And the fact that somebody else is deciding what you're allowed to do with the model is one of the things that seems to make HN angriest at the frontier labs in the first place...
To be clear, I think this might be a good idea! I think all of the possible downsides I've listed are pretty small potatoes relative to what happens with no regulation of AI at all. But I'm pretty sure that if the big companies were proposing it, people would be calling it "regulatory capture" too.
"deep ties to everyone in the doomer media campaign"
Are you suggesting these external NGOs were spun up as part of a gigantic pre-IPO hype stunt? METR was founded multiple years ago. This "stunt" is getting quite elaborate.
At a certain point, Occam's Razor says: These engineers are legitimately worried. They haven't invested years in a bizarre reverse psychology campaign to convince the public that their product is dangerous in order to make more money.
Founded several years after Anthropic. Like I said, look into where their funding is coming from. I can promise you it all leads back to Anthropic through their chain of NGOs.
Are you aware that Dario's sister, president of Anthropic, is married to the co-founder of Open Philanthropy? The two largest AI doomer NGOs, Center for AI Safety (CAIS) and the Future of Life Institute (FLI), have both received many millions of dollars from them.
Ajeya Cotra worked at Open Philanthropy/Coefficient Giving for roughly nine years, including leading its technical AI-safety program in 2024 and contributing to AI-giving strategy in 2025. She subsequently left Coefficient and joined METR, where she is now technical staff.
Ajeya is married to Paul Christiano, who founded Alignment Research Center (ARC). Alignment Research Center donated ~$4.5mil to METR.
Good Ventures is a funding partner of Open Philanthropy, who funded Jacob Coxon (the person going viral in the media) via a scholarship.
They're all connected, funnelling money to each-other through convoluted networks to serve Anthropic's agenda. Whether their motives are genuine or not (and they are clearly not) is actually irrelevant because they are clearly trying to rig the game in their favour.
Suppose I showed you a number of climate change NGOs which shared staff and funding sources. Could we therefore conclude that their motives aren't genuine?
Suppose a big foundation both funded clean energy technology, and also NGOs encouraging people to decarbonize. Is this just a cynical attempt to rig the game in their favor?
> Suppose a big foundation both funded clean energy technology, and also NGOs encouraging people to decarbonize. Is this just a cynical attempt to rig the game in their favor?
Not comparable. The clean energy technology company wouldn't be trying to create an environment where no one else can create clean energy technology, or where only they are the ones who can decide how clean energy technology is created or used.
Oh, would you look at that, 1 day before Dario's blog post, Joe Benton leaves Anthropic with the same fear campaign playbook, gets blasted all over the media, and declares he is joining METR to do independent eval of risks.
So, put your employees in METR -> offer to have them work in your office as an "unbiased" third party evaluator.
>> Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR).
Someone needs to look into who is funding METR (mentioned in Dario the Book Burners post explicitly). Because they keep popping up now, with close ties to people neck-deep in the orchestrated doomer hysteria media campaign and Anthropic. Looks to be highly coordinated that they are positioning METR to be the gatekeeper evaluator organization.
Anthropic should not be allowed to choose their "embedded evaluators" - that should be entirely up to the government, with ZERO say from them, if this is really what they want. Even better would be if it's up for democratic vote.
Still, I don't think anyone should be playing by Dario's playbook. At all. He very obviously has ulterior motives, and even if he didn't, it's a massive conflict of interest for him to be self-regulating.
>> But we are still the ones choosing what to include and omit. Embedded evaluators will change this dynamic.
Not if you get to choose your embedded evaluator.
>> Anthropic intends to invite an embedded external review team equipped with all of the following in the near future: Desks in our offices, access badges, and company laptops. ...
Alarm bells should be going off for people. Let's see if he's so relaxed about all of this if it's a federal "embedded evaluator" and not a company he has deep connections to and has seemingly carefully laid the foundations for.
>> The measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try.
You owe it to humanity to be honest. Something you are incapable of, and have proven so, innumerable times.
Ah and would you look at that, 1 day before this post from Dario, one of his safety leads quits, gets pumped all over the media with another doomer social post, and announces he's going to work for METR.
So he's literally stacking the deck. I really hope the US govt is not blind to everything that is happening here.
Right now the barrier is data and compute.
Quality data can be created synthetically at an exponential rate as models improve. Humans are actively feeding them with private IP.
Compute advancements will begin to skyrocket as we unlock photonic computing and materials science advancements and scale up chip fabs. This is also compounding because the AI is accelerating the pace of research, testing, development, manufacturing, etc.
It's a big self-accelerating feedback loop. There is no plateau.
reply