One-shotting is a bit of a red herring, imo. Let's say I wanted to build a "Super Dario" meme platformer. I would do some research, find a platformer on Codepen or Github that "feels" good, tweak it to my liking, change the sprites, and voila.
I constantly have to tell agents to just "look it up online" and "don't hallucinate your own components" because people have already done this a million times. Ironically, being more lazy could make these models more useful.
So you're saying that it's not fair to rate a coding model on its ability to code, and instead the best way to use it is to tell it to find existing human-written code online rather than generate actual code on its own?
If you're testing how good LLMs are at compressing information, then I think that's a fair test. Personally, I don't really think that's where their strength comes from (especially considering how much more useful local models that are orders of magnitude smaller than Claude/OpenAI-tier models have gotten). In other words, we already have a "super-intelligence"—it's called the internet, so just use the darn thing.
Strange, I use M3 and Sonnet a lot, they never hallucinated. Gemini used to make up some function names, but it was in long context windows. My AGENTS.md does specific they should use context7 when unsure about a library, but still…
I constantly have to tell agents to just "look it up online" and "don't hallucinate your own components" because people have already done this a million times. Ironically, being more lazy could make these models more useful.