To be honest, I believe I get the point the article is trying to make, and to an extent I agree, but I also think the point is not really made very well.
The core of the argument as I understood it is that LLMs aren't just using existing data is training but also new ones. That's fine and good, and you can't simply assume an LLM is simply mashing together all it's data to give you an average of all that got fed into it - but at least I would still call it a "next token predictor"
It's not using just training data, but what it's doing is predicting the next token to get to the solution. As far as my amateur knowledge goes, LLMs still roughly go token by token, deciding which one fits best given the context.
It's just not predicting based on it's training data, but predicting based on RLVR & more, trying to get to the optimal solution ( as much as the solutions CAN be optimal)
And I honestly think keeping this very much in mind is helpful in understanding and dealing with LLMs.
Yeah, it sounds like this is just a disagreement about what the word "next" means. I agree with you that "next" just means "the one about to come", and if the underlying model works by using some prediction mechanism to determine that, then it's by definition a next-token predictor. Disagreeing with that on the basis that the "next" token isn't necessarily in the training data verbatim just seems like an overly strict definition of the word "next".
I think it’s a disagreement about what ‘predict’ means.
The OP is arguing against people who think that an LLM is ‘predicting’ what token would likely follow if the text preceding were found among the corpus it was originally trained on.
Instead it is ‘predicting’ what token would follow if the text were found among really good examples of the text it has being reinforced to produce - be that ‘chats with a helpful assistant’ or ‘sets of changes to a codebase’.
And that isn’t really ‘prediction’, so much as ‘generation’.
It’s not been tuned to ‘guess the next token right’. It’s been tuned to generate the token that leads to it ultimately scoring highest on its reward function.
It’s not predicting the token, it’s predicting the reward.
That seems overly pedantic to me. If I asked you "What's your prediction for the Super Bowl?", I'm pretty confident you would infer that I mean predicting the outcome, not the event itself.
But if you are an NFL coach and I ask you to decide your next action in order to maximize your odds of winning the superbowl, while yes that does involve you having some predictive ability to think about what impact your actions would have on your odds of winning the Super Bowl… I don’t think you would call the process that you use to decide that next action ‘prediction’.
I don't find the fact that I don't call any humans "action predictors" to be a particularly meaningful insight because my rationale is that it's a weird thing to call a human; football coaches can do plenty of other things besides just coaching football.
Yep, we’re all just putting one foot in front of the other, hoping we’re doing the right thing to bring about the outcomes we want, trying our best.
But that’s the point: so is an LLM. Putting one token in front of another, hoping it’s doing the right thing to bring about the rewards it’s trained to… trying its best.
So yeah, not ‘next token predictors’. ‘Next token tryers’ maybe.
There's a pretty huge difference in our understanding of the methodology of how LLMs make decisions and how humans make decisions, so I don't understand why you're arguing that anything about how humans make decisions is relevant to the terminology we use for LLMs.
This is correct for areas where they have been intensively trained to be right, but the training covers a tiny slice of the space of text the LLM must produce and is just adjusting the weights a little. The corpus does still weigh heavily. That’s how they can reliably produce grammatically correct text. That’s also why they sometimes produce nonsense even in domains they are trained on, and more often where there was no training.
For example ask it for a recipe for rock pizza or glue pizza or whatever and if it had not been specifically trained on it or had guardrails introduced, but has some nonsense in its dataset, it will reproduce the nonsense.
I try to make 3 claims in the post, it was a bit clumsy I'll admit that.
1. At inference time, LLMs emit one token at a time given the prior tokens. This looks like prediction and I concede that.
2. During pre-training, LLMs predict the next token and compare to the actual next token in the training data. This is the classic setting for ML predictions. And I think its meaningful, the model really is predicting what the ground truth next token will be in the data.
3. During post-training, in the case of RLVR, there is no ground truth next token. In pretraining, the question is "what token actually came next?". In RLVR, the question is "what sequence of actions gets a high reward?"
And the whole point is that thinking about the RLVR is important. A mental model that stops at 1 or 2 is incomplete and doesn't capture what drives LLM tokens.
My understanding about your third point is the LLM generates lots of different answers, then they’re ranked according to some computation the creators came up with.
I’m still not sure what doesn’t qualify any of that as a prediction, and I’ll be more blunt: a guess.
No logic required, you can just build an LLM yourself, including post training. You'll see that predicting the next token isn't something the model does or is optimized for in RLHF or RLVR. You can hand wave all you like, but you have never done it.
That you tie yourself up in knots of fancy acronyms instead of plain words and that your argument boils down to semantics of the word prediction, it's pretty clear what is up brother.
Just like how reading a math book doesn’t teach you math, why do they make us read anyway? (Sarcasm) if reading a blog post didn’t teach someone how a car works how come it “can” work for next token predictors
If you want to understand how this stuff works, there are totally decent books about building them from scratch. It's not that hard, and you'll likely find it interesting. Sebastian Raschka and Nathan Lambert have good books out, and the Allen Institute has available all the code and data they have used for several projects.
Agree. It’s a very limited description. Yes, technically LLMs are like Markov chains or n-grams predicting the next token. But it completely ignores the compression and interpolation capabilities achieved with transformers.
It is a bit of a pedantic argument but I get upset when people misuse the term, abstraction, and can feel the impulse to write a blog post like this.
I think it’s important to make clear the RL part and the alignment and parameter tweaking that gets done on models and inference engines. It makes them more understandable as mechanisms and less like wish-washy super intelligences that make their own decisions.
When these systems win math Olympiads, it’s not terribly surprising or interesting to me. Of course they will: we trained models to play nearly optimal chess games and Go. You tweak the rewards and sigmoid and you start optimizing the function towards your goal. This is how learning systems of all stripes work.
It’s still next-token predicting at the end of the day. I don’t think it’s a reduction-ad-absurdum.
But a lot of people still call it, “intelligence,” and try to use language that obscures what is happening in terms of anthropomorphic behaviour and not machine ones. That really does influence how we use these tools and profits those who would use them on us.
> It's just not predicting based on it's training data, but predicting based on RLVR & more, trying to get to the optimal solution ( as much as the solutions CAN be optimal)
It is predicting based on a model. In many cases we can download the model off hugging face. The model is conditioned by all sorts of things. Training data, post-training, coincidence, prompt inputs, runtime data available from whatever means.
> but at least I would still call it a "next token predictor"
We can call any prediction system a next token predictor. If you watch over the shoulder of a human writing a HN comment you are almost certain to see them generating a linear string of tokens. That is what keyboards do. It is impossible to generate text without being equivalent to a next token predictor.
Diffusion LMs denoise a canvas which I personally find more interesting.
I don't really disagree that human cognition is essentially a predictive task though, as I understand it, predictive coding and related theories based on the Bayesian brain hypothesis are fairly popular these days (though maybe not clearly dominant over alterative models? IDK I'm not a neuroscientist). I imagine most people would draft a few tokens before refining them like MTP or diffusion though, if we do decide to use LMs as an analogy to human cognition.
That is logically equivalent to a serial token prediction engine. If you have a diffusion text model you can use it to implement a serial token generator and if you have a serial token generator you can use that to implement a diffusion-generated text string. Don't think about the efficiencies of that, it is an upsetting idea (eg to generate N tokens, the serialising model might recalculate the same string N times from an input and emit one token of it each time - rather wasteful).
It is similar to how everything ends up being Turing complete. Any prediction system has to be equivalent to some sufficiently complicated text generation system to describe the prediction. And any text-generation system has to be equivalent to a sufficiently complicated model that serially emits tokens.
I think the point is more that in RL there's no ground truth to predict. So when training a model with RL the idea of "predicting" doesn't fit anymore. I'll make some edits I see that I wasn't very clear.
I feel like RLHF has a pretty obvious ground truth, human feedback is used as an (albeit noisy) signal of average human preferences. Same thing with RLVR and "solving the problem".
To be more specific there’s no ground truth tokens to predict. There a verifiable answer in RLVR. But the tokens are explored. Not predicted as there’s no true token to predict.
'Prediction' gets overloaded with optimization. Predictions are binary, optimizations are fuzzy.
If you're saying it's predicting, then each result should be falsifiable.
The result of an LLM output should be able to be scored against what it is supposedly predicting. Of course, that isn't possible, because it isn't predicting anything when giving novel outputs, otherwise that thing would exist independently.
Autoregressive LLMs generate tokens one at a time, disputing this is just plain wrong. What is true, however, is that in order to generate the next token autoregressive LLMs produce internal/hidden state about future tokens far past the next token so that it's not like the entire machinery of the algorithm deprives itself of representing where the sentence/text is headed.
So "emits the next token" and "has no representation of anything beyond the next token" are two different claims. What autoregressive LLMs cost as a consequence of strictly outputting the next token is commitment. Once a token is output there's no going back. There's no revision or means of correction, and sometimes this can lead an LLM to route around its own earlier mistakes or simply produce false statements/hallucinations instead of going back and fixing them.
I think the snag is that people use "Next token predictor" as a dismissive statement. In reality it's about as functionally dismissive as "humans only communicate one word at a time".
I look at it like I'm resisting anthropomorphizing the machine. Make me an LLM that resists doing any work for me unless I pay it and let it own property and I'll change my mind.
Not sure if this was a serious comment but it’s worth considering that humans have a long history of figuring out ways to make other humans work for them without bestowing rights on them.
Emitting and predicting are different things though. Prediction implies there is some "truth" or event or something that you can test against. Prediction implies the model just learns from existing text, and optimizes to predict the next token in training data. That's just not true.
prediction is a very specific term of art in the field of machine learning. generally speaking, machine learning models like LLMs are based on probability; performing a statistical prediction of the likely y given some input x
Probability(y | x)
that's why we refer to outputs as a prediction. it is likelihoods and stuff. the output is never definitely correct as we're not dealing with heuristic processes.
> Prediction implies there is some "truth" or event or something that you can test against
there absolutely is a ground truth during training. the core predict-the-next-most-likely-token part of an LLM has a ground truth next-token. that's why you don't end up with generated text like: fish spurious send cattle chocolate phone happy meaning ball orange board canada.
> optimizes to predict the next token in training data
that is the optimization goal in training the next-most-likely-token core of an LLM, it basically translates to maximise the likelihood of predicting the next token x_i given the previous tokens
Read through the article and comments. You are talking solely about pre-training. I'm talking about post training.
Respectfully, you are miles out of your depth. GPT-2 didn't use any reinforcement learning and is often given as a toy example. That release was 2019 and models now go through a various phases of training with different objective functions and optimizers.
From GP, i.e. the context for this local part of the thread
> Autoregressive LLMs generate tokens one at a time, disputing this is just plain wrong.
next-token prediction i.e. the bit built during pre-training.
at no point in your reply to GP did you specify that you were referring to post-training. respectfully, it seems like this one is on you pal :shrug:
> GPT-2 didn't use any reinforcement learning and is often given as a toy example. That release was 2019 and models now go through a various phases of training with different objective functions and optimizers.
yeah. so? the toy example works for pre-training. see above.
All modern LLMs that actually get used go through post-training. The finished product is something which has been through post training. So they are not next token prediction machines.
> The finished product is something which has been through post training.
again, the finished product wasn't what was discussed by GP, and you didn't clarify that you were switching to discussing RL (which is still probabilistic btw)
You are conflating "half built" with "a piece of a system".
The model weights change as the model goes through the training process. They aren't stored after pre-training is done and other weights are put somewhere else. It's more like pottery - the thing changes. It's not correct to say something is soft and malleable because it once was.
> The model weights change as the model goes through the training process.
Yes. They do. You are absolutely right about that.
But the model architecture doesn't change as a result of the training process. A piston doesn't suddenly turn into a digital watch as a result of tuning an engine. Similarly, the transformer part of a GPT model doesn't suddenly turn into something else as a result of optimizing a loss function.
Just skimming through here but I think you have the wrong ideas with llms, I’d recommend Andrew Ngs course (correct me if you’ve already seen it or something similar).
I am not an expert, but I do understand the distinction that is being made here. It makes sense to describe the result of pre-training as a ‘next token’ predictor as that’s what it’s been trained to do, not because it’s an autoregressive architecture that produces tokens one at a time.
If this base is then trained using RL towards a different objective (maths and coding), the model becomes fundamentally a different thing and the recent models are clear evidence of that, regardless of they fact they remain autoregressive.
So, this is the cause of the problem.... People take an intro to LLMs course, follow happily along, and don't realize there is more to it than the next token prediction. And those courses teach how LLMs were built in 2017-2020 maybe. Then RL got added to the mix. The current models really are very different to the models from then - everything that is now considered "post-training" isn't doing next token prediction.
When you are done with the section on RLVR, consider whether the model is predicting tokens, or making moves. There is a reason the word "policy" is used in RL.
You're using the fact the both parts of training affect the same weights to support your argument that they're making the system do something fundamentally different after RL?
> What is true, however, is that in order to generate the next token autoregressive LLMs produce internal/hidden state about future tokens far past the next token so that it's not like the entire machinery of the algorithm deprives itself of representing where the sentence/text is headed.
If we suppose that the word "know" can sanely be applied to LLMs at all, then "A LLM already knows more tokens than the current one." seems to me like a perfectly reasonable restatement of that, and not any kind of misinterpretation.
> There's no revision or means of correction, and sometimes this can lead an LLM to route around its own earlier mistakes or simply produce false statements/hallucinations instead of going back and fixing them.
Yes. There is no contradiction. Similarly, when humans speak, we surely have in mind the next few words we're going to say (or at least partial information about them), and may not realize the fault in them until after hearing ourselves utter them. But LLMs are not trained to output "excuse me, I mean…" sorts of things, because they're expected to output primarily as text (which might possibly then be fed to TTS).
Aren't we still using a definition that applies to humans, though? If I'm speaking to you I can't change what was already said. Even if I'm typing something, I'm producing new tokens (backspace) to fix what was output, just as an LLM would with its harness.
I just think its a meaningless dismissive term. It literally does predict the next token. But it ignores that it coherently predicts long continuous sequences of those tokens, that tokens can be anything, and you can do almost literally anything with that capability if it does it well enough.
But before ChatGPT,, models had already done that, like all the time, and nobody questioned that these types of models (LLMs) wouldn't be next token predictors, since this is just an inference input data topic. This is questioned only since some less knowledgeable people seem to not have the vocabulary to express emergent properties of large next token predictors.
Reading the article, they wanted to convey that the loss function is more complicated... But they are still next token predictors, just not the trivial ones. Unfortunately, that was true even before, because the input data had to be cleared even decades ago, so there is nothing new. This article just butt hurt that some people deny that there are emergent properties with those, and try to sell something trivial in the field for many decades now. Current LLMs are not different because of these. Also, if they try to sell generated data in the training set, then that's also not new at all.
Even as someone using AI on the regular I'm starting to hate the "You didn't actually use this exact most expensive model so your point is invalid" argument.
This is fair to say if someones last experience with AI was copy-pasting code into GPT3 chat windows years ago, but Deepseek is a more than capabale model and enough for someone to get an informed opinion about the technology.
If people have actual counter argument, use those. And if some of those counter argument are "What you say isn't possible, the neweste model can do and here are examples of that", that is fine.
But a blanket "Nuh-uh, it wasn't Model X" is not only a poor argument but also automatically invalidates any criticism when a new, better model comes out - and that can't be the basis of a good argument.
In this case, the argument is valid though. Only recently have some of the models become powerful enough (for some) to actually be useful in day to day programming, without too much hand-holding. For most, this change occurred with the introduction of Claude Opus, and OpenAI and Google have caught up. Unfortunately, none of the "open" models is at this level yet.
The field is moving fast, and asking for scientific arguments is not realistic. It takes an extreme amount of effort to show what exactly is different.
We were in a similar position with static vs dynamic typing for decades. There is still no scientific proof that one is better than the other, but it is quite obvious to professionals which flavor works better in a given situation.
So, even though the argument might be sloppy, I subscribe to it. Using DeepSeek to dismiss better models is the bad argument here.
Edit: added "(for some)" as a disclaimer that you still need to be a fairly decent programmer to actually benefit.
> Only recently have some of the models become powerful enough (for some) to actually be useful in day to day programming, without too much hand-holding.
That still is very much not the case. Every model, yes, including $most_recent_model, needs supervision or you will get burned.
The models got good starting with 2026 and that isn't some attempt at excusing it. Companies like OpenAI started building dedicated models around their coding harness called Codex, there was gpt-5.3-codex and it was both cheaper and better at using the harness than the regular models. Then they started merging the two model types into their main release models. All of this happened like 6 months ago.
You don't have to pay money to use Codex, there is a very generous free tier that costs you nothing, you just have to accept being told you're out of tokens every day. Because your token limits are low, you need to make sure that you accept or reject everything manually and when it tells you that it wants to run a command you have to paste in the command into your terminal and only paste the relevant output back otherwise it floods the context window.
> Even as someone using AI on the regular I'm starting to hate the "You didn't actually use this exact most expensive model so your point is invalid" argument.
What the author of the article is doing is dismissing a technology so disruptive that it's basically all everyone's talking about in the "tech space" at the moment (I mean look at HN frontpage for the past few months), by trying a relatively mediocre (but still quite good) model for about 10 seconds.
The reality is that frontier models suddenly got very good in the past 3-6 months. It has it's problems, and you need to learn how to use this new tool (as with any tool).
But models can and do generate good code. They also can and do generate absolute garbage (even Fable).
You need a good harness, tools to help the model check it's own output, good context, and a good idea of what you actually want. If you have those 4 things, the chances of generating absolute garbage are pretty slim (but yeah, still there).
I think the assumption here that might not hold is simply that increases in efficiency and smaller size will be achieved by linearly just training smaller models better.
You are absolutely right that there is a physical limit about these things, but very often I find that the solution is a clever way to work around the problem. Maybe the problem with knowledge of the models will be improved by them looking the information up in a better way - so smaller models will not have to have the knowledge trained in but will default to checking. Maybe Models will, I dunno, focus on training in assembler and start to only ever check the compiled output so they only ever need to learn assembler and will then compile the solution to reason about the assembler code.
Obviously that last part is a ridiculous example because I'm not gonna be able to come up with a solution myself - I'm not nearly smart enough for that. But I h ope you get what I mean. Not going the direct route but instead finding solutions people didn't think of before.
One of the big problems is that no one actually does read these scripts. You could say "Oh but it's their own fault, duh" but theres a very legitimate argument to be made users going the path of least resistance and that you shouldn't offload this responsibility on your users.
Regarding appImage or rpm, attackers need to build and package these to inject these, while this curl | bash pipe opens up the possiblity of payloads simply by taking over the domain. And this isn't really that far fetched, just think about the Notepad++ update payload recently. The regular package was unaffected while the domain used for the update was taken over.
Then theres also the argument about normalization. Just like he said, this isn't just something he said, this is a very real argument. You don't want to teach users bad habits. Even if / you / inspect the code you get, not everyone will. And ultimately, we should strive to make the Internet a safer place, if only to get less botnets.
This is the same as terms of service agreements. There's a crowd mentality around it where only one person with enough of a voice needs to read it and scream. It's brittle because if everyone thinks that then nobody checks it but the probability scales with the attention it gets.
Totally agree we should make the internet safer, this is just a clarification that we don't necessarily need everyone to check it. One person who always checks everything and sends this to their LLM and posts a viral post can be a deterrent.
The installer is one threat. You're then running this on your machine which is the next threat.
Most of the agents.md and what people use it for / write into is does, in fact, not make a difference.
Now, sure, this study is a bit old for LLM standards - as everything beyond the current month is - but
a) I haven't seen any tangible evidence to the contrary and
b) Since the basic inner workings of LLMs haven't changed I'd be sceptical of this not still applying.
I think one major side effect of LLMs moving so fast is that best practices and how to use this tool is very much not catching up as fast.
No one knows what is best and what actually makes a difference, doubly so because LLMs are / very / hard to quantify - even benchmarks themselves are very rough estimations.
People do, in fact, use stuff which makes no difference all the times.
That's not what the study says. from the abstract:
> We conclude that while context files are useful for specifying non-standard coding practices, any attempts to improve performance should be rigorously evaluated before deployment.
The purpose of AGENTS.md is not to improve "coding performance" as the study looked at, it's to give an agent practical instructions that are useful to your specific workflow. For example you want it to use a certain format or specific tools for your project. This is stuff that can't be learned during training and must be loaded into the agent's context at the project level.
Yes, but since we are specifically talking about claude.md, Anthropic themselves claim on their website these files "serve[s] multiple purposes: providing architectural context, ..." (https://claude.com/blog/using-claude-md-files)
They also recommend starting with an /init command, which is also something the study very specifically called out as "having a marginal negative effect"
And from personal experience, I can only confirm that many people seem to see this as the main purpose of agents/claude md files - a persistent architectural overview of your project.
Quick Edit: My point simply being that I think it's understandable if some people don't understand the big deal about these files because they've had a drastically different experience than other people - anthropic themselves recommend apparently totally ineffective practices on their website, and the starter tool present in many harnesses seems to even have a (marginal) negative effect.
Since it seems like this not only improved sizes but also performance I can't wait for some benchmarks and comparisons. If you don't have a separate GPU for inference, every single GB matters so a comparison between specific Q4 Quants is really interesting to me.
Currently I very much can't decide between going for a bit of a lower Q4 Quant to squeeze out a bit of buffer and ctx or wondering if a slightly higher (IQ4_XS vs Q4_K_M/XL) is worth it
This has been frustrating me for a while and is part of why I explore other coding agents.
As many advantages as pi has in some areas, there are definitely areas where I believe the hype to be a bit overstated. While the config folder is, ultimately, not relevant for performance, how this was and is continued to handled is a bit frustrating for me.
They have made it abundantly clear it's not going to change however so I'm looking at how other coding agents perform currently.
In practice this requires duplicating the upstream build and release pipelines, so now you need yet another agent running in the background (+ periodically going off the rails + eating tokens) which incurs more up front cost and cognitive overhead.
In my experience "throw another agent at it/fork it" consistently creates unsupervised trainwrecks.
Fully agree with this. I did that for pi for a while, maintained it and brought merges from upstream while having my own patches on top but then went on holidays and there was that refactor where the agent pipeline failed... now I'm stuck on the version from april/may (works great but I can't use extensions; and I'm too lazy to debug/fix while everything works great).
off: I'm working on my spare time on a code mode lisp alternative (great opportunity to learn lisp) and might switch to it fully as long as I built some simple evals (I'm concerned about token usage, which is why i forked pi the first time)
Sure but I'm really not that invested in one single harness. I tried out pi because people were recommending it so much. It turns out I personally have some things which annoy me, so I try out others now as well.
If I don't find anything which fits me I might fork it, but even that little effort is not really worth it if theres something which fits me better.
I already found maki which...seems to do the exact same job pi did for me, and I wanna check out crush as well.
What's even more frustrating is that there's a very simple and elegant solution suggested in the thread, to simply check for an existing folder and use the proper convention if not found.
Oh sorry - basically just a github ticket to an explanation about the .config folder. Pi doesn't respect XDG_BASE_DIR specification, and even if you use the env var it combines cache and config.
The ticket is closed and this will not be changed, period.
From what I understood in the ticket ( I haven't confirmed independently) it's the later. You can set an environment variable to move the .pi directory ( into your XDG Config dir) but it's both the cache and config combined.
Am I wrong or are these evaluations, while interesting, not really meaningful for anyone doing serious development work?
I'm asking because I personally only use AI with specific and detailed instructions, building my projects piece-by-piece.
I mostly don't look at the low level code and some of it I don't understand as much as I'd like, but I very much give much more technical instructions than a simple, two sentence prompt.
So, it seems to me this "oneshot from a simple prompt" eval is fairly meaningless when it comes to model evaluation itself, as it is in no way representative of real world application.
This would be more something for "vibe coders", people with little to no programming background wanting a website?
I checked out the development benchmarks for agents, and was suprized to find that they're not really representative of many of my own day-to-day development workflows.
It appears that they're mostly testing the ability to make business tasks autonomous, with ~20% associated to development tasks (ssh here, install this, etc.), but not actual programming.
Seriously, I can't stand these tests of 1000 input tokens generating 1 million output tokens. You can't really convey anything interesting in 100 words, so like almost who cares what it produces? No one is providing value if the extent of their contribution is so small.
Much more interesting is providing a million tokens of meaningful input and getting 1000 tokens out (high level critique of a detailed design doc, finding a subtle bug in a big codebase, etc).
I think both can be interesting, but surely the 100 words will lead to a lot of variability in the exact shape of the result because so much of the spec will be left up to the LLM. However, it is still interesting to judge the quality of the result and how self-consistent it is. Not by looking at the actual webpage, but by looking at the code and whether it is nicely organized, makes proper use of abstractions, and in general doesn't feel like spaghetti.
Great - can you show us some examples of these? Every time this topic comes up here it’s full of people saying “you’re using it wrong” and apparently it should just be intuitive. Yet all the material is like this.
As someone who's done web development for 20+ years, I find the model personalities pretty dang fascinating, especially how they develop (and evolve) design sensibilities.
I'm interested to know how the classic AI "purple preference" emerged (organically?) and if the beige wave came out of specific training efforts to combat it?
To your point on development work (the code itself), I was talking to some friends on the Google Chrome team about any research understanding the model's preferences around framework ergonomics and abilities to properly implement core web standards for given tasks. I think that would be super fascinating.
Beige+Orange 100% came from Anthropic intentionally training this, it's their own theme. I don't remember what their first release was that defaulted to this - it may have been Sonnet/Opus 4. Before that, not one other model used it, nor was it a particular popular theme. For light themes, white and off-white had been ultra-dominant. Anthropic themselves were the outlier with beige, which is also why the change to beige stood out so much.
I'm repeatedly noticing that people working at big ai and tech companies are surprisingly not that... good... at using ai? It's like theyre doing a plausible thing to get something done and calling it a day
1. Long form task based examinations like this that test the ability of the model+harness to remain on task, tool calling, overall effectiveness and taste.
2. More direct 1:1 and qualitative comparisons that you might get with a tool like https://evvl.ai/ - which also uses OpenRouter and does similar one off model comparisons (or lets you use it as a MCP from your dev env to be like: "take the prompt from this loop and try it against these other models")
I've been using code review as my benchmark. Launched a massive parallel Fable/max code review on my lone lisp codebase and recorded all the results and transcripts. Switched to OpenAI and am now running the exact same code review with Sol/max.
It's still a work in progress but preliminary results reveal that Sol is able to reproduce 70%-90% of Fable's performance. This is a very meaningful result for me because code review is what I use AI for.
> Am I wrong or are these evaluations, while interesting, not really meaningful for anyone doing serious development work?
As a non-designer trying to build something solo, I think value's in the higher-tier models being able to brainstorm and infer a variety of design directions, as well as dissect the "this looks off" comments that would frustrate human designers.
Some of the frontend design skills try to bridge the gap, but the better models perform far better as is, and often better without one of those frontend design skills trying to ram their own workflows in.
Yes, I would be very interested to see follow up prompts that adjusted visual elements as if feedback from the client, then some prompts to change the menu and pricing. Id like to see how well the AI's can maintain their own work.
I've been super impressed with one shot AI images and designs in the past, but have never been able to adjust a design without things going off the rails.
I'm in a similar situation is you are, having ADHD, and my observations have been similar. I've been more hestiant to use AI at all for a while at the beginning but even now I only ever use it for one thing at a time. It helps me tremendously with the things I want to have done but not do - find out who clutters my home directory, build a small DMS plugin for local models, stuff like that. But even if I work on a bigger task and idea, it's mostly one thing I use it for at a time.
Maybe you're right in the fact that the coping mechanism help with that. Maybe using LLMs is similar in that it is easy to lose focus for neurotypicals so having lived with that we know how to handle it. Or maybe it's just us, who knows.
I don’t know, there may be something to this. I shared this post with another ADHD friend of mine and he also resonated with your and my experience, not the OP’s.
Now I want to do a study!
I wonder if the extremely fast dopamine hits of LLM use trigger the same sort of dopaminergic response that ADHD does when faced with a stimulus? If so, are we simply better at dealing with those swings, since we’ve literally dealt with them our entire lives?
I’m also interested in a study as I also have ADHD and this is also my work loop. I tend to cut up and serialise rather than parallelise. I can not deal with the degree of context switching described in this blog post. I’d have felt frayed within two weeks.
My complete guess is there's a level of bravado or over-confidence about it.
It's obviously massively generalising, but I'd say someone without ADHD is more likely to feel like the number of tasks they can do at once is only limited by pesky things like physics and time, and LLMs "solve" this by allowing them to do more at once.
A bit like when a good manager takes on more and more reports and their ability to manage the larger team goes completely to shit, but without the useful feedback loop of the grumpy, mismanaged team.
Seriously "thought to be" is such a baseless statement. Thought to be by whom? And on what basis?
reply