There's an issue with GPT-5.6 Sol where it sometimes starts mixing thinking with output and stops working[1]. Once it starts doing that, the session is essentially cooked and you need to do a bit of gymnastics if you want to recover it.
This happens to me more commonly in large projects (>100k LOC) and in those projects it seems to happen every few sessions. I feel this specific benchmark would be impacted by this more than the smaller contrived benchmarks.
I can't help but imagine agents using caveman speak sometimes start behaving in a stereotypically caveman manner, even if it's subtle. Is there a chance the agent does less reasoning because of it?
Caveman invented fire, the wheel, domesticated wild plants and animals, organised society, survived the Toba catastrophe, cooked food, and was having sex ages before you and me. Don't write him off as stupid.
And I wonder how they actually spoke. Since there was no visual communications medium except for cave art. (Some of which is very excellent. Try drawing 3d curved horns in perspective.) So people would have used verbal communication more. Also no written word. So one would expect there to be quite a lot of oral tradition. Like people reciting poem form epics.
If we assume the time is before farming, population density would have been low and limiting culture. Hunter-gatherers might have travelled a lot more than farmers with a homestead though.
The Pleiades cluster is called the seven sisters in Greek. That's curious, because the human eye under the best conditions can discern only six stars in there. Even more curious, the aboriginal Australians also called this cluster the seven sisters.
Ancient Greeks' and aboriginal Australians' last common ancestors split about 60,000 years ago. And astronomers tell us that 60,000 years ago, there were seven discernable stars in that cluster.
One could this conclude not only is speech likely 60,000 years old, but also that the tale of the seven sisters might be a tale from so long ago.
Interestingly, back in my ill-spent youth, a few of my fellow astronomers and I were out in a very, very dark-sky location in the late 70s/early 80s, and we were able to consistently count and draw between 9 and 11 stars. Although we would tease those who could see 11 stars as using averted imagination. :-) Today, if I can see six stars, it's an okay night in an okay sky.
fwiw, if you can get out to dark skies where you can see fifth- or sixth-magnitude stars with the naked eye, I highly recommend getting out there when it's a low-moisture atmosphere and the Milky Way through Cassiopeia and Perseus is vertical, as it's a rather dramatic sight of this stream of stars heading down to the northern horizon.
The summertime Milky Way overhead down to Sagittarius tends to get all the love, but the wintertime Milky Way is also visually rich and worth spending time on.
It's not exactly a joke, it does reduce the amount of tokens. However, it does not improve performance (fine tunes are finnecky things, hard to get one right).
Personally the only 'enthusiast' modified qwen 3.6 27b or 3.6 35b-a3b I've found useful are the ones that have been run through heretic and adversarial data sets for innocent/dangerous prompts, to produce uncensored LLMs. They have some niche non-coding uses for things that a commercial LLM will never talk about.
I think those are mostly vapor that runs on the small culture of "models should not be censored" thing. But from my experience, they unlock nothing meaningful.
Fine-tuning is great for really small models on specific applications, but it's not something that can essentially improve a more generic model.
That said, there seems to be a fine line in quantization+finetuning that could recover performance. It's just hard to get a hold of it (I feel it in some models, but it's hard to say yet; lots of small labs working on this RN).
The most interesting use I've found for them so far is strictly as a novelty. Give a chat session with one to a completely non technical person, who at least knows that openai and anthropic have some guard rails on stuff, and tell them to wild with something like "give me the precursors and chemical formulas for the precusors for crystal meth" and watch it answer.
But does it answer those queries correctly, or does it just not refuse to not halucinate an incorrect answer? From where would it even have that information?
I don't know enough chemistry to say one way or the other if it's just wildly hallucinating the precursors and processes, but it'll also do things like, write an ISIS press release, or similar. There's a data set of basically a bunch of antisocial or dangerous prompts that some people have got variants of qwen to pass with 0 out of 465 refusals:
Just so we are clear, no "caveman" spoke English. "Caveman speak" is just shortening the vocabulary of english, not a "caveman language". Given this, your concerns for "stereotypical caveman manner" makes very little sense since what caveman are you talking about?
The concern is not that the model was trained on actual caveman artifacts, rather on modern media representations of the stereotypical caveman (that never actually existed).
For those trying to get it to work in OpenCode with a Qwen Cloud Token Plan, this is what worked for me. Note that I've just matched Qwen 3.7 Max for the limits as I don't know exactly what they are.
also, be very careful which API endpoint and API Token you use. make sure you use right one (obseve your quota is used up. if you hit right endpoint quota used almost immediately). so that you do not accidentally burn API endpoint tokens (they are expensive, can easily hit 200 USD / 3 days which do not count towards your membership "Credits", if you say purchased it with 200 USD signup bonus in Alibaba Cloud)
For the vast majority of models I've worked with, 150k tokens of context seems like some magic spot. If you can stay under it then the session will stay on track and the agent won't start getting forgetful and weird.
Just like with real work, it's valuable to break down big tasks into small tasks that you can knock over in a single session. When a session does start getting too big, you just need to ask the agent to reply with a comprehensive handover report and paste it into a new session.
For my quirky workflows I need 100K even to prep Opus 4.8 or higher to begin good work. Not programming, but a complex synthetic architecture for implementing more attentional autonomy for machines.
Nirav from Framework is similar, he speaks openly about compromises they make with their designs and why they make them.
When leaders are both technical and open about these sorts of things it makes me feel like I can trust that they are invested in supporting and improving their products.
I'd say Codex and Claude Code have different strengths and weaknesses. Claude Code is significantly better in terms of their subagent UI for example - being able to see the list of subagents under the input is great.
To be honest though, I've gotten to the point where I prefer the OpenCode UI. A big win for OpenAI is you can log in to your subscription in OpenCode, whereas this is not trivially achievable for a Claude subscription.
I was getting some really impressive cost efficiency today in OpenCode with the following:
* Main session agent: gpt-5.6-sol (high) via OpenAI subscription
* General purpose subagent: deepseek-v4-pro (high) via OpenCode Go subscription
* Using `obra/superpowers` for subagent driven workflows
* The main session only being allowed filesystem read permissions and everything else delegated
It was absolutely crunching through tasks without hitting the limit, and this combination is quite cost effective.
GPT 5.6 was picking up on quality and functional issues from DeepSeek and having it resolve them cleanly, and I didn't even get close to my quotas whereas I can usually blast through them. I feel as people get more comfortable with subagents and mixing and matching models in their daily work, Anthropic's walled garden stance will start to hurt them.
I feel this would be more useful for tasks like "Check website X to see if there are any great deals today". Specifically, tasks that are loosely defined and require some form of intuition.
And even in that case if you give the agent a tool to access website X efficiently it'll work a lot better. Like, say, a CLI tool that can take a website product URL for that specific site and return the opengraph data of the item price on it + any sale price.
The different bindings vs Vim was actually what stopped me using it. I really really wanted to love it and love a lot of the motivation and principles behind it, but unlearning decades of muscle memory is an absolute nightmare.
Many people buy two separate Claude pro subscriptions and that makes the limit become a non-issue. It works surprisingly well when you tend to hit the 5 hourly limit after a few hours, and hit the weekly limit after 4-5 days. $40 vs $100 is significant for a lot of people.
I hit limit of Pro in about 30 minutes, 1 hour max. And only when I use a single session, and when I don't use it extensively, ie waits for my responses, and I read and really understand what it wants, what it does. That's still just 1-2 hours/5 hours.
You're probably having long sessions, i.e. repeated back-and-forth in one conversation. Also check if you pollute context with unneeded info. It can be a problem with large and/or not well structured codebases.
The last time I used pro, it was a brand new Python rest service with about 2000 lines generated, which was solely generated during the session. So how I say to Claude that use less context, when there was 0 at the beginning, just my prompt?
So you had generated 2000 lines in 30 minutes and ran out of tokens? What was your prompt?
I’d use a fast model to create a minimal scaffold like gemini fast.
I’d create strict specs using a separate codex or claude subscription to have a generous remaining coding window and would start implementation + some high level tests feature by feature. Running out in 60 minutes is harder if you validate work. Running out in two hours for me is also hard as I keep breaks. With two subs you should be fine for a solid workday of well designed and reviewed system. If you use coderabbit or a separate review tool and feed back the reviews it is again something which doesn’t burn tokens so fast unless fully autonomous.
Thanks for the tip, didn’t think of using 2 subscriptions at the same company.
When reaching a limits, I switch to GLM 4.7 as part of a subscription GLM Coding Lite offered end 2025 $28/year. Also use it for compaction and the like to save tokens.
This happens to me more commonly in large projects (>100k LOC) and in those projects it seems to happen every few sessions. I feel this specific benchmark would be impacted by this more than the smaller contrived benchmarks.
[1]: https://github.com/openai/codex/issues/37524
reply