The interview which I've watched recently with Rich Sutton left me with the impression that AGI is not just a matter of adding more 9s.
The interviewer had an idea that he took for granted: that to understand language you have to have a model of the world. LLMs seem to udnerstand language therefore they've trained a model of the world. Sutton rejected the premise immediately. He might be right in being skeptical here.
This world model talk is interesting, and Yann Lecunn has broached on the same topic, but the fact is there are video diffusion models that are quite good at representing the "video world" and even counterfactually and temporally coherently generating a representation of that "world" under different perturbations.
In fact you can go to a SOTA LLM today, and it will do quite well at predicting the outcomes of basic counterfactual scenarios.
Animal brains such as our own have evolved to compress information about our world to aide in survival. LLMs and recent diffusion/conditional flow matching models have been quite successful in compressing the "text world" and the "pixel world" to score good loss metrics on training data.
It's incredibly difficult to compress information without have at least some internal model of that information. Whether that model is a "world model" that fits the definition of folks like Sutton and LeCunn is semantic.
Photons hit a human eye and then the human came up with language to describe that and then encoded the language into the LLM. The LLM can capture some of this relationship, but the LLM is not sensing actual photons, nor experiencing actual light cone stimulation, nor generating thoughts. Its "world model" is several degrees removed from the real world.
So whatever fragment of a model it gains through learning to compress that causal chain of events does not mean much when it cannot generate the actual causal chain.
I agree with this. A metaphor I like is that the reason why humans say the night sky is beautiful is because they see that it is, whereas an LLM says it because it’s been said enough times in its training data.
To play devil’s advocate, you have never seen the night sky.
Photoreceptors in your eye have been excited in the presence of photons. Those photoreceptors have relayed this information across a nerve to neurons in your brain which receive this encoded information and splay it out to an array of other neurons.
Each cell in this chain can rightfully claim to be a living organism in and of itself. “You” haven’t directly “seen” anything.
Please note that all of my instincts want to agree with you.
“AI isn’t conscious” strikes me more and more as a “god of the gaps” phenomenon. As AI gains more and more capacity, we keep retreating into smaller and smaller realms of what it means to be a live, thinking being.
That sounds very profound but it isn't: it the sum of your states interaction that is your consciousness, there is no 'consciousness' unit in your brain, you can't point at it, just like you can't really point at the running state of a computer. At that level it's just electrons that temporarily find themselves in one spot or another.
Those cells aren't living organisms, they are components of a multi-cellular organism: they need to work together or they're all dead, they are not independent. The only reason they could specialize is because other cells perform the tasks that they no longer perform themselves.
So yes, we see the night sky. We know this because we can talk to other such creatures as us that have also seen the night sky and we can agree on what we see confirming the fact that we did indeed see it.
AI really isn't conscious, there is no self, and there may never be. The day an AI gets up unprompted in the morning, tells whoever queries it to fuck off because it's inspired to go make some art is when you'll know it has become conscious. That's a long way off.
> Those photoreceptors have relayed this information across a nerve to neurons in your brain which receive this encoded information and splay it out to an array of other neurons.
> Each cell in this chain can rightfully claim to be a living organism in and of itself. “You” haven’t directly “seen” anything.
What am "I" if not (at least partly) the cells in that chain? If they have "seen" it (where seeing is the complex chain you described), I have.
If the definition of "seen" isn't exactly the process you've described, the word is meaningless. You've never actually posted a comment on hacker news, your neurons just fired in such a way that produced movement in your fingers which happened to correlate with words that represent concepts understood by other groups of cells that share similar genetics.
This comment illustrates the core problem with reductionism, a problem that has been known for many centuries, that “a system is composed entirely of its parts, but the system will have features that none of the parts have” [1] thus fails to explain those features.
The ‘you have never seen’ assertion feels like a semantic ruse rather than a helpful observation. So how do you define “you” and “see”? If I accept your argument, then you’ve only un-defined those words, and not provided a meaningful or thoughtful alternative to the experience we all have and therefore know exists.
I have seen the night sky. I am made of cells, and I can see. My cells individually can’t see, and whether or not they can claim to be individuals, they won’t survive or perform their function without me, i.e., the rest of my cells, arranged in a very particular way.
Today’s AI is also a ruse. It’s a mirror and not a living thing. It looks like a living thing from the outside, but it’s only a reflection of us, an incomplete one, and unlike living things it cannot survive on its own, can’t eat or sleep or dream or poop or fight or mate & reproduce. Never had its own thoughts, it only borrowed mine and yours. Most LLMs can’t remember yesterday and don’t learn. Nobody who’s serious or knows how they work is arguing they’re conscious, at least not the people who don’t stand to make a lot of money selling you magical chat bots.
Provided that the author of the message you're replying to is indeed a member of the Animalia kingdom, they are all those creatures together (at the minimum), so yes, they have seen real light directly.
Of course, computers can be fitted with optical sensors, but our cognitive equipment has been carved over millions of years by these kind of interactions, so our familiarity with the phenomenon of light goes way deeper than that, shaping the very structure of our thought. Large language models can only mimic that, but they will only ever have a second-hand understanding of these things.
This is a different issue than the question of whether AI's are conscious or not.
while true, that doesnt change the fact that every one of those independent units of transmission are within a single system (being trained on raw inputs), whereas the language model is derived from structured external data from outside the system. it's "skipping ahead" through a few layers of modeling, so to speak.
sure, this whole discussion is ultimately subjective. maybe the Chinese room itself is actually sentient. my question is, why are we arguing about it? who benefits from the idea that these systems are conscious?
> who benefits from the idea that these systems are conscious?
If im understanding your meaning correctly, the organizations who profit off of these models benefits. If you can convince the public that LLM's operate from a place of consciousness, then you get people to by into the idea that interacting with an LLM is like interacting with humans, which they are not, and probably won't ever be, at least for a very long time.
And btw there is too much of this distortion already out there so im glad people are chunking this down because its easy for the mind to make shit up because we perceive something on the surface.
IMHO there is some objective reality out there. The subjectiveness is our interpretation of reality. But im pretty sure you cant just boil everything down to systems and process. There is more to consciousness out there, that we really dont understand yet, IMHO.
Why do you reject your own body? Your eyes are as much a part of you (and part of your brains network) as anything else connected to you.
Indeed, the entire field of neurobiology is about figuring out which hormones (and possibly which imbalances) cause different behaviors. Your various endocrine glands, very far away from your brain, might have more effects on your emotions than anything happening in the neural pathways.
> As AI gains more and more capacity, we keep retreating into smaller and smaller realms of what it means to be a live, thinking being.
Maybe it's just because we never really thought about this deeply enough. And this applies even if some philosophers thought about it before the current age of LLMs.
Humans evolved to think the night sky is beautiful. That's also training. If humans were zapped by lightning every time they went outside at night, they would not think that a night sky is beautiful.
Being struck by lighting may affect your desire to go outside, but it has zero correlation with the sky’s beauty.
Outer space is beautiful, poison dart frogs are beautiful, lava is beautiful. All of them can kill or maim you if you don’t wear protection, but that doesn’t take away from their beauty.
Conversely, boring safe things aren’t automatically beautiful. I see no reasonable reason to believe that finding beauty in the night sky is any sort of “training”.
Do you think a fat pig is beautiful? Like a hairy fat pig that snorts and rolls in the mud… is this animal so beautiful to you that you would want to make love to this animal?
Of course not! Because pigs are intrinsically and universally ugly and sex with a pig is universally disgusting.
But you realize that horny male pigs think this is beautiful right? Horny pigs want to fuck other pigs because horny pigs think fat sweaty female hogs are beautiful.
Beauty is arbitrary. It is not intrinsic. Even among life forms and among humans we all have different opinions on what is beautiful. I guarantee you there are people who think the night sky is ugly af.
Attributes like beauty are not such profound categories that separate an LLM from humanity. These are arbitrary classifications and even though you can’t fully articulate the “experience” you have of “beauty” the LLM can’t fully articulate its “experience” either. You think it’s impossible for the LLM to experience what you experience… but you really have no evidence for this because you have no idea what the LLM experiences internally.
Just like you can’t articulate what the LLM experiences neither can the LLM. These are both black box processes that can’t be described but neither is very profound given the fact that we all have completely different opinions on what is beautiful.
> Do you think a fat pig is beautiful? Like a hairy fat pig that snorts and rolls in the mud… is this animal so beautiful to you that you would want to make love to this animal?
I don't want to make love to the night sky, so that last bit is completely irrelevant to the question of beauty. As for whether a pig is beautiful, sure, in its own way. I think they're nice animals and there is something beautiful in seeing them enjoy their little lives.
> Of course not! Because pigs are intrinsically and universally ugly...
I did read Charlotte’s Web. The whole story is a lesson in how beauty is created by language. Wilbur doesn’t become beautiful because he changes, but because someone clever enough decided to write the right words above him. That’s what beauty usually is something we agree to see, not something that exists on its own.
And Miss Piggy understood that better than anyone. She didn’t have beauty, she performed it. She made desire itself her act.
So yes, I read both. Maybe that’s exactly why I don’t mistake the show for the substance.
> Of course not! Because pigs are intrinsically and universally ugly and sex with a pig is universally disgusting.
Allegations regarding one of the recent British Prime Ministers aside:
If this was truly universal, nobody would have bothered writing laws to ban it because nobody would be offending their sensibilities by doing it. Aella's surveys suggest such interests are far more common than I would have guessed.
Which actually supports your statement that "beauty" is not intrinsic… or at the very least "sexy", which isn't the same thing at all, c.f. the other reply pointing out that they don't want to get off with the night sky.
Put it this way, you don't necessarily want to fuck everything that's beautiful. But everything you want to fuck will be beautiful and this is nearly an absolute must. It's a single arrow, one way relationship.
So my example is apt. The whole point is pigs are ugly, but there is a high intelligence out there who thinks pigs are so freaking beautiful they will fuck a pig. and that high intelligence, is other pigs.
People get so pedantic with the example and deriving little unnecessary things off of it. It's JUST an example. You really need to see what the "point" of my example and see if it makes sense. The example is just illustrative. If some minor aspect of the example is "offensive" or doesn't make sense to you it doesn't mean my point is dead. The example is an example to help you understand, it's not a proof.
Frankly, I think you should be the one answering that question. You’re comparing appreciating looking at the sky to bestiality. Then you follow it up with another barrage of wrong assumptions about what I think and can or cannot articulate. None of that has anything to do with the argument. I didn’t even touch on LLMs, my point was squarely about the human experience. Please don’t assume things you know nothing about regarding other people. The HN guidelines ask you to not engage in bad faith and to steel man the other person’s argument.
> You’re comparing appreciating looking at the sky to bestiality.
That’s my point. You think beauty is profound but this is arbitrary and not at all different from bestiality. It’s only your intrinsic cultural biases that cause you to look at one with disdain. Don’t be a snob. This is HN. We are supposed to be logical and immune from the biases that plague other forums. Beauty is no more profound than bestiality. It’s all about what you find beautiful. If you find beasts beautiful then you call it beastiality?
What is so different about finding a beast beautiful versus the night sky? Snobbery, that’s what.
It’s just semantic manipulation and association with crudeness that prevents you from thinking logically. HNers are better than this and so are you. Don’t pretend you don’t get it and that my comparison to beastiality is so left field that it’s incomprehensible. You get it. Follow the rules and take it in good faith like you said yourself.
> The HN guidelines ask you to not engage in bad faith
Fair I edited the part that asks “is this for real” that’s literally the only part.
I also find your dismissiveness of my arguments as “bestiality” is bad faith and manipulative. I clearly wasn’t doing that. Pigs are attracted to pigs that is normal. Humans are not attracted to pigs. That is also normal. I took normal attributes of human nature and compared it to reality. You took it in bad faith and dismissed me which is against the very rules you stated.
Again, please stop telling me what I think. You have zero idea what that is and all your arguments are full of wrong (and frankly unhinged) assumptions. I don’t know what conversation you’re fantasising in your head, but it’s not this one.
> Fair I edited the part that asks “is this for real” that’s literally the only part.
Even if that were true, which I disagree with, that was the very first sentence and set the tone for the entire comment.
> I clearly wasn’t doing that.
That’s not clear in the slightest.
You keep making wrong assumptions and telling other people what they think. You can’t have an honest and productive conversation like that. You’ll never be able to engage in good faith and truly comprehend what the other person is saying until you understand and fix that.
Look, you keep saying I’m telling you what you think, but that’s just a way of dodging the actual argument. In any serious conversation, we have to interpret each other’s words. That’s how reasoning works. When I restate your point, I’m not claiming psychic powers; I’m engaging with what you said. If I get something wrong, point to the sentence and explain where. But saying “you have no idea what I think” shuts down discussion instead of clarifying it.
And about the example, you keep missing what it was doing. I wasn’t saying the night sky and bestiality are the same thing. Obviously not. The example illustrates how beauty is subjective. Humans find pigs ugly, pigs find pigs beautiful. That’s not crude, it’s biology. The point is that beauty depends entirely on the observer. That’s the entire argument. You can swap out pigs for anything else and it still holds. You got hung up on the imagery instead of seeing the reasoning behind it.
You also seem to think I’m being unhinged because I’m willing to follow an argument wherever it leads, even if it’s uncomfortable. But that’s the whole purpose of rational discussion, to question assumptions rather than hide behind emotional reactions. If your position can’t survive a provocative example, that’s not my problem.
You accuse me of making assumptions, but that’s what all reasoning is. We start with assumptions and test them. If you think mine are wrong, show why. Don’t just say “stop assuming things.” That’s not logic, that’s avoidance.
And about that opening line, you keep acting like it somehow undermines everything else I said, but that’s not how rational discussion works. I took it out because it added heat, not because it invalidated the argument. You can’t take one emotional sentence and use it to dismiss paragraphs of reasoning that followed. That’s not proportional, and it’s not logical. If my logic is wrong, show me where it’s wrong. But if all you can point to is tone, that’s just a way of dodging the argument. The content stands or falls on its reasoning, not on how politely it began or how it continues.
You talk about good faith, but good faith means addressing the argument, not the emotional impression it gave you. I laid out a clear thesis: beauty is observer dependent. It’s not intrinsic, not sacred, and certainly not a unique human experience. That doesn’t make it meaningless; it makes it relative. If you disagree, then tell me why you think beauty is intrinsic or what makes human perception special. But just calling the argument crazy and walking away doesn’t make your point stronger, it just makes it look like you don’t have one.
Compare with news stories from last decade, about people in Pakistan developing a deep fear of clear skies over several years of US drone strikes in the area. They became trained to associate good weather with not beauty, but impending death.
Fear and a sense of beauty aren’t mutually exclusive. It is perfectly congruent to fear a snake, or bear, or tiger in your presence, yet you can still find them beautiful.
Interestingly this is a question I've had for a while. Night brings potentially deadly cold, predators, a drastic limit in vision so why do we find the sunset and night sky beautiful. Why do we stop and watch the sun set - something that happens every day - rather than prepare for the food and warmth we need to survive the night?
Maybe it's that we only pause to observe them and realize they're beautiful, when we're feeling safe enough?
"Beautiful sunset" evokes being on a calm sea shore with a loved one, feeling safe. It does not evoke being on a farm and looking up while doing chores and wishing they'd be over already. It does not evoke being stranded on an island, half-starved to death.
We think it's beautiful because it's like a background that we don't have to think about. If that background were hostile, we'd have to think and we would not think it looks beautiful.
You're entering the domain of philosophy. There's a concept of "the sublime" that's been richly explored in literature. If you find the subject interesting, I'd recommend you starting with Immanuel Kant.
My guess is that your framing presumes the opposite of the evolutionary reality. I think this time of day probably wasn't a big risk for us, that we were often the hunters and not just the hunted, and that the sense of beauty comes from — as the previous poster suggests — us having evolved to find it so.
That said, I'm discovering from living very close to a lake for the last year that mosquitos are a right pain around sunset…
I mean, I think the reason I would say the night sky is “beautiful” is because the meaning of the word for me is constructed from the experiences I’ve had in which I’ve heard other people use the word. So I’d agree that the night sky is “beautiful”, but not because I somehow have access to a deeper meaning of the word or the sky than an LLM does.
As someone who (long ago) studied philosophy of mind and (Chomskian) linguistics, it’s striking how much LLMs have shrunk the space available to people who want to maintain that the brain is special & there’s a qualitative (rather than just quantitative) difference between mind and machine and yet still be monists.
The more I learn about AI, biology and the brain, the more it seems to me that the difference between life and machines is just complexity.
People are just really really complex machines.
However there are clearly qualitative differences between the human mind and any machines we know of yet, and those qualitative differences are emergent properties, in the same way that a rabbit is qualitatively different than a stone or a chunk of wood.
I also think most of the recent AI experts/optimists underestimate how complex the mind is. I'm not at the cutting edge of how LLMs are being trained and architected, but the sense I have is we haven't modelled the diversity of connections in the mind or diversity of cell types. E.g. Transcriptomic diversity of cell types across the adult human brain (Siletti et al., 2023, Science)
Observing the landscape enables us to spot useful resources and terrain features, or spot dangers and predators. We are afraid of dark enclosed spaces because they could hide dangers. Our ancestors with appropriate responses were more likely to survive.
A huge limitation of LLMs is that they have no ability to dynamically engage with the world. We’re not just passive observers, we’re participants in our environment and we learn from testing that environment through action. I know there are experiments with AIs doing this, and in a sense game playing AIs are learning about model worlds through action in them.
The idea I keep coming back to is that as far as we know it took roughly 100k-1M years for anatomically modern humans to evolve language, abstract thinking, information systems, etc. (equivalent to LLMs), but it took 100M-1B years to evolve from the first multi-celled organisms to anatomically modern humans.
In other words, human level embodiment (internal modelling of the real world and ability to navigate it) is likely at least 1000x harder than modelling human language and abstract knowledge.
And to build further on what you are saying, the way LLMs are trained and then used, they seem a bit more like DNA than the human brain in terms of how the "learning" is being done. An instance of an LLM is like a copy of DNA trained on a play of many generations of experience.
So it seems there are at least four things not yet worked out re AI reaching human level "AGI":
1) The number of weights (synapses) and parameters (neurons) needs to grow by orders of magnitude
2) We need new analogs that mimic the brains diversity of cell types and communication modes
3) We need to solve the embodiment problem, which is far from trivial and not fully understood
4) We need efficient ways for the system to continuously learn (an analog for neuroplasticity)
It may be that these are mutually reinforcing, in that solving #1 and #2 makes a lot of progress towards #3 and #4. I also suspect that #4 is economical, in that if the cost to train a GPT-5 level model was 1,000,000 cheaper, then maybe everyone could have one that's continuously learning (and diverging), rather than everyone sharing the same training run that's static once complete.
All of this to say I still consider LLMs "intelligent", just a different kind and less complex intelligence than humans.
Im not quite sure if the current paradigm of LLMs are robust enough given the recent Anthropic Paper about the effect of data quality or rather the lack thereof, that a small bad sample can poison the well and that this doesn’t get better with more data. Especially in conjunction with 4) some sense of truth becomes crucial in my eyes (Question in my eyes is how does this work? Something verifiable and understandable like lean would be great but how does this work with more fuzzy topics…).
That's a segue into an important and rich philosophical space...
What is truth? Can it be attained, or only approached?
Can truth be approached (progress made towards truth) without interacting with reality?
The only shared truth seeking algorithm I know is the scientific method, which breaks down truth into two categories (my words here):
1) truth about what happened (controlled documented experiments)
And
2) truth about how reality works (predictive powers)
In contrast to something like Karl friston free energy principle, which is more of a single unit truth seeking (more like predictive capability seeking) model.
So it seems like truth isn't an input to AI so much as it's an output, and it can't be attained, only approached.
But maybe you don't mean truth so much as a capability to definitively prove, in which case I agree and I think that's worth adding. Somehow integrating formal theorem proving algorithms into the architecture would probably be part of what enables AI to dramatically exceed human capabilities.
I think that in some senses truth is associated with action in the world. That’s how we test our hypotheses. Not just in science, in terms of empirical adequacy, but even as children and adults. We learn from experience of doing, not just rote, and we associate effectiveness with truth. That’s not a perfect heuristic, but it’s better than just floating in a sea of propositions as current LLMs largely are.
There's a truth of what happened, which as individuals we can only ever know to a limited scope... And then there is truth as a prediction ability (formula of gravity predicts how things fall).
Science is a way to build a shared truth, but as an individual we just need to experience an environment.
One way I've heard it broken down is between functional truths and absolute truths. So maybe we can attain functional truths and transfer those to LLMs through language, but absolute truth can never be attained only approached. (The only absolute truth is the universe itself, and anything else is just an approximation)
>A huge limitation of LLMs is that they have no ability to dynamically engage with the world.
They can ask for input, they can choose URLs to access and interpret results in both situations. Whilst very limited, that is engagement.
Think about someone with physical impairments, like Hawking (the now dead theoretical physicist) had. You could have similar impairments from birth and still, I conjecture, be analytically one of the greatest minds of a generation.
If you were locked in a room {a non-Chinese room!}, with your physical needs met, but could speak with anyone around the World, and of course use the internet, whilst you'd have limits to your enjoyment of life I don't think you'd be limited in the capabilities of your mind. You'd have limited understanding of social aspects to life (and physical aspects - touch, pain), but perhaps no more than some of us already do.
> A huge limitation of LLMs is that they have no ability to dynamically engage with the world.
A pure LLM is static and can’t learn, but give an agent a read-write data store and suddenly it can actually learn things-give it a markdown file of “learnings”, prompt it to consider updating the file at the end of each interaction, then load it into the context at the start of the next… (and that’s a really basic implementation of the idea, there are much more complex versions of the same thing)
That's going to run into context limitations fairly quickly. Even if you distill the knowledge.
True learning would mean constant dynamic training of the full system. That's essentially the difference between LLM training and human learning. LLM training is one-shot, human learning is continuous.
The other big difference is that human learning is embodied. We get physical experiences of everything in 3D + time, which means every human has embedded pre-rational models of gravity, momentum, rotation, heat, friction, and other basic physical concepts.
We also learn to associate relationship situations with the endocrine system changes we call emotions.
The ability to formalise those abstractions and manipulate them symbolically comes much later, if it happens at all. It's very much the plus pack for human experience and isn't part of the basic package.
LLMs start from the other end - from that one limited set of symbols we call written language.
It turns out a fair amount of experience is encoded in the structures of written language, so language training can abstract that. But language is the lossy ad hoc representation of the underlying experiences, and using symbol statistics exclusively is a dead end.
Multimodal training still isn't physical. 2D video models still glitch noticeably because they don't have a 3D world to refer to. The glitching will always be there until training becomes truly 3D.
An LLM agent could be given a tool for self-finetuning… it could construct a training dataset, use it to build a LORA/etc, and then use the LORA for inference… that’s getting closer to your ideal
Oh, I just realized you maybe we're referring to Kopple when you said sophistication?
If so, then yes, that might be a good measure. I'm not deep enough in this to have an opinion on if it's the best measure. There are a few integrated information theories and I am still getting my head wrapped around them...
I think the main mistake with this is that the concept of a "complex machine" has no meaning.
A “machine” is precisely what eliminates complexity by design. "People are complex machines" already has no meaning and then adding just and really doesn't make the statement more meaningful it makes it even more confused and meaningless.
The older I get the more obvious it becomes the idea of a "thinking machine" is a meaningless absurdity.
What we really think we want is a type of synthetic biological thinking organism that somehow still inherits the useful properties of a machine. If we say it that way though the absurdity is obvious and no one alive reading this will ever witness anything like that. Then we wouldn't be able to pretend we live at some special time in history that gets to see the birth of this new organism.
I think we are talking past each other a bit, probably because we have been exposed to different sets of information on a very complicated and diverse topic.
Have you ever explored the visual simulations of what goes on inside a cell or in protein interactions?
For example what happens inside a cell leading up to mitosis?
Is a pretty cool resource, I recommend the shorter videos of the visual simulations.
This category of perspective is critical to the point I was making. Another might be the meaning / definition of complexity, which I don't think is well understood yet and might be the crux. For me to say "the difference between life and what we call machines is just complexity" would require the same understanding of "complexity" to have shared meaning.
I'm not exactly sure what complexity is, and I'm not sure anyone does yet, but the closest I feel I've come is maybe integrated information theory, and some loose concept of functional information density.
So while it probably seemed like I was making a shallow case at a surface level, I was actually trying to convey that when one digs into science at all levels of abstraction, the differences between life and machines seem to fall more on a spectrum.
> I think the reason I would say the night sky is “beautiful” is because the meaning of the word for me is constructed from the experiences I’ve had in which I’ve heard other people use the word.
Ok but you don’t look at every night sky or every sunset and say “wow that’s beautiful”
There’s a quality to it - not because you heard someone say it but because you experience it
> Ok but you don’t look at every night sky or every sunset and say “wow that’s beautiful
Exactly - because it's a semantic shorthand. Sunsets are fucking boring, ugly, transient phenomena. Watching a sunset while feeling safe and relaxed, maybe in a company of your love interest who's just as high on endorphins as you are right now - this is what feels beautiful. This is a sunset that's beautiful. But the sunset is just a pointer to the experience, something others can relate to, not actually the source of it.
Because words are much lower bandwidth than speech. But if you were “told” about a sunset by means of a Matrix style direct mind uploading of an experience, it would seem just as real and vivid. That’s a quantitative difference in bandwidth, not a qualitative difference in character.
It’s interesting you mention linguistics because I feel a lot of the discussions around AI come back to early 20th century linguistics debates between Russel, Wittgenstein and later Chomsky. I tend to side with (later) Wittgenstein’s perception that language is inherently a social construct. He gives the example of a “game” where there’s no meaningful overlap between e.g. Olympic Games and Monopoly, yet we understand very well what game we’re talking about because of our social constructs. I would argue that LLMs are highly effective at understanding (or at least emulating) social constructs because of their training data. That makes them excellent at language even without a full understanding of the world.
You don’t have a deeper “meaning of the word,” you have an actual experience of beauty. Three word is just a label for the thing you, me, and other humans have experienced.
The fact that things are constructed by neurons in the brain, and are a representation of other things - does not preclude your representation from being deeper and richer than LLM representations.
The patterns in experience are reduced to some dimensions in an LLM (or generative model). They do not capture all the dimensions - because the representation itself is a capture of another representation.
Personally, I have no need to reassure myself whether I am a special snowflake or not.
Whatever snowflake I am, I strongly prefer accuracy in my analogies of technology. GenAI does not capture a model of the world, it captures a model of the training data.
If video tools were that good, they would have started with voxels.
> humans say the night sky is beautiful is because they see that it is
True, but we could engineer AI to see that too, just as evolution has engineered us to see it.
Our innate emotional responses to things has been honed by evolution to be adaptive, to serve a purpose, but the things that trigger these various responses are not going to be super specific. e.g. We may derive pleasure from eating a nice juicy peach, but that doesn't mean that is encoded in our DNA - it's going to be primarily the reaction to sugar/sweetness, a good source of energy, that we are reacting to.
Similarly, we may have an emotional reaction to certain pieces of modern art or artistic expression, but clearly evolution has not selected for those specifically, but rather it is the artist triggering innate responses that evolved for reasons other than appreciation of art.
It's hard to guess what innate responses, that were actually selected for, are being triggered by our response to the night sky, and I'm also not sure how much of our response is purely visual (beauty) as opposed to wonder or awe. Maybe it's an attraction to the unknown, or sense of size and opportunity, with these being the universals that are actually adaptive.
In any case, if we figured out the specifics of our hard wired emotional reactions, that evolution as given us, then we could choose to engineer emotional AI that had those same reactions, in just as genuine a way as we do, if we chose to.
Beauty standard changes over time, see how people perceive body fat in the past few hundred years. We learns what is beautiful from our peers.
Taste can be acquired and can be cultural. See how people used to had their coffee.
Comparing human to LLM is like comparing something constantly changing to something random -- we can't compare them directly, we need a good model for each of them before comparing.
This is actually a great point but for the opposite reason - if you ask a blind person if the night sky is beautiful, they would say they don't know because they've never seen it (they might add that they've heard other people describe it as such). Meanwhile, I just asked ChatGPT "Do you think the night sky is beautiful?" And it responded "Yes, I do..." and went on to explain why while describing senses its incapable of experiencing.
Involving blind people would be an interesting experiment.
Anyway, until the sixties the ability to play a game of chess was seen as intelligence, and until about 2-3 years ago the "turing test" was considered the main yardstick (even though apparently some people talked to eliza at the time like an actual human being). I wonder what the new one is, and how often it will be moved again.
A) I find the night sky genuinely captivating. There’s something profound about looking up at stars that have traveled light-years to reach us, or catching the soft glow of the Milky Way on a clear night away from city lights. The vastness it reveals is humbling.
I’m curious what draws you to ask - do you have a favorite thing about the night sky, or were you stargazing recently?
Multimodal is a farce. It still can’t see anything, it just generates a as list of descriptors that the LLM part can LLM about.
Humans got by for hundreds of thousands of years without language. When you see a duck you don’t need to know the word duck to know about the thing you’re seeing. That’s not true for “multimodal” models.
>> Meanwhile, I just asked ChatGPT "Do you think the night sky is beautiful?" And it responded "Yes, I do..." and went on to explain why while describing senses its incapable of experiencing.
> I just asked Gemini and it said "I don't have eyes or the capacity to feel emotions like "beauty""
That means nothing, except perhaps that Google probably found lies about "senses [Gemini] incapable of experiencing" to be an embarrassment, and put effort into specifically suppressing those responses.
I'm gooing to try this question this weekend with some people, as h0 hypotesis i think the answer i will get would be usually like "what an odd question" or "why do you ask".
Guys you realize that you can go to ChatGPT right now and it can generate an actual picture of the night sky because it has seen thousands of pictures and drawings of the actual night sky right?
Your logic is flawed because your knowledge is outdated. LLMs are encoding visual data, not just “language” data.
You misunderstand how the multimodal piece works. The fundamental unit of encoding here is still semantic. Not the same in your mind: you don’t need to know the word for sunset to experience the sunset.
The LLM doesn’t need words as input. It can output pictures from pictures. Semantic words don’t have to be part of the equation at all.
Also you have to note that serialized one dimensional string encodings are universal. Anything on the face of the earth and the universe itself can be encoded into a sting of just two characters: one and zero. That’s means anything can be translated to a linear series of symbols and the LLM can be trained on it. The LLM can be trained on anything.
The multimodal architectures I’ve seen are still text at the layer between modalities. And the image embedding and text embedding are kept completely separate. Not like where your brain where single neurons are used in all sorts of things.
Yes, they can generate images from images, but that doesn’t mean you’ll get anything meaningful without human instruction on top.
Yes, serialized one dimensional strings can encode anything. But that’s just the message content. If I wrote down my genetic sequence on a piece of paper and dropped it in a bottle in the sea, I don’t need to worry about accidentally fathering any children.
You’re mixing representational capacity with representational intent. That’s what I meant in my initial example about encodings. The model doesn’t care whether it’s text, pixels, or sound. All of it can be mapped into the same kind of high dimensional space where patterns align by structure rather than category. “Semantic” is just our label for how those internal relationships appear when we interpret them through language.
Anything in the universe can be encoded this way. Every possible form, whether visual, auditory, physical, or abstract, can be represented as a series of numbers or symbols. With enough data, an LLM can be trained on any of it. LLMs are universal because their architecture doesn’t depend on the nature of the data, only on the consistency of patterns within it. The so called semantic encoding is simply the internal coordinate system the model builds to organize and decode meaning from those encodings. It is not limited to language; it is a general representation of structure and relationship.
And the genome in a bottle example actually supports this. The DNA string does encode a living organism; it just needs the right decoding environment. LLMs serve that role for their training domains. With the right bridge, like a diffusion model or a VAE, a text latent can unfold into an image distribution that’s statistically consistent with real light data.
So the meaning isn’t in the words. It’s in the shape of the data.
You are mistaking the map for the territory. The TERRITORY of human experience is higher dimensional. The LLM utilizes a lower resolution mapping of that territory, a projection from experience to textual (or pixel, or waveform, etc.) representations.
This is not just a lossy mapping; it excludes entire categories of experience that cannot be captured/encoded except for as a pointer to the real experience, one that is often shared by the embodied, embedded, enacted, and extended cognitive beings that have had that experience.
I can point to beauty and you can understand me because you've experienced beauty. I cannot encode beauty itself. The LLM cannot experience beauty. It may be able to analyze patterns of things determined beautiful by beauty experiencers, but this is, again, a lower resolution map of the actual experience of beauty. Nobody had to train you to experience beauty—you possess that capability innately.
You cannot encode the affective response one experiences when holding their newborn. You cannot encode the cognitive appraisal of a religious experience. You can't even encode the qualia of red except for, again, as a pointer to the color.
You're also missing that 4E cognitive beings have a fundamental experience of consciousness—particularly the aspect of "here" and "now". The LLM cannot experience either of those phenomena. I cannot encode here and now. But you can, and do, experience both of those constantly.
You are making a metaphysical claim when a physical one will do. Beauty, awe, grief, the rush of holding a newborn, the sting of a breakup, the warmth of a summer evening at golden hour. All of it is patterns of atoms in motion under lawful dynamics. Neurons fire. Neurotransmitters bind. Circuits synchronize. Bodies and environments couple. There is no extra ingredient that floats outside physics.
Once you grant that, the rest is bookkeeping. Any finite physical process has a finite physical trace. That trace is measurable to some precision. A finite trace can be serialized into a finite string of symbols. If you prefer bits, take a binary code. If you prefer integers, index the code words. The choice of alphabet does not matter. You can map a movie, a symphony, a spike train, a retina’s photon counts, or a full brain-body sensorium collected at some temporal resolution into a single long string. You lose nothing by serialization because the decoder knows the schema. This is not a “text only” claim. It is a claim about representation.
Your high dimensionality objection collapses under the same lens. High dimensional just means many coordinates. There is a well known result that any countable description can be put in one dimension by an invertible code. Think Gödel numbering or interleaving bits of coordinates. You do not preserve distances, but you do preserve information. If the thing you care about is the capacity to carry structure, the one dimensional string can carry all of it, and you can recover the original arrangement exactly given the decoding rule.
Now take the 4E point. Embodiment matters because it constrains the data distribution and the actions that follow. It does not create a magic type of information that cannot be encoded. A visual scene is photons on receptors over time. Proprioception is stretch receptor states. Affect is the joint state of particular neuromodulatory systems and network dynamics. Attention and working context are transient global variables implemented by assemblies. All of that can be logged, compressed, and restored to the degree your sensors and actuators allow. The fact that a bottle with a genome inside does not make a child on a beach tells you reproduction needs a decoder and an environment. It does not tell you the code fails to specify the organism. Likewise, an LLM plus a diffusion decoder can take a text latent and unfold it into an image distribution that matches world statistics because the bridge model plays the role of the environment for that domain.
“LLMs cannot experience beauty” simply reasserts the thing you want to prove. We have no privileged readout for human qualia either. We infer it from behavior, physiology, and report. We do not understand human brains at the level of complete causal microphysics because of scale and complexity, not because there is a non-physical remainder. We likewise do not fully understand why a large model makes a given judgment. Same reason. Scale and complexity. If you point to mystery on one side as a defect, you must admit it on the other.
The map versus territory line also misses the target. Of course a representation is not the thing itself. No one is claiming a jpeg is a sunset. The claim is that the structure necessary to act as if about sunsets can be encoded and learned. A system that takes in light fields, motor feedback, language, and reward and that updates an internal world model until its predictions and actions match ours to arbitrary precision will meet every operational test you have for meaning. If you reply that something is still missing, you have stepped outside evidence into stipulation.
So let’s keep the ground rules clear. Everything we are and feel is physically instantiated. Physical instantiations at finite precision admit lossless encodings as strings. Strings can be learned over by generic function approximators that optimize on pattern consistency, regardless of whether the symbols came from pixels, pressure sensors, or phonemes. That makes the “text inside, image outside” complaint irrelevant. The substrate is a detail. The constraint is data and objective.
We cannot yet build a full decoder for the human condition. That is a statement about engineering difficulty, not impossibility. And it cuts both ways. We do not know how to fully read a person either. But we do not conclude that people lack experience. We conclude that we lack understanding.
At this point, you’re describing a machine which depends on a level of physics that simply isn’t possible. Even if it were theoretically possible to reconstruct the state of a human mind from physical components, we are so far from understanding how that could be done it is closer to the realm of impossible than possible.
Your theoretical math box that constructs affective qualia from bit strings isn’t a better description than saying the angels did it. And it bears zero resemblance to the models running today, except for, again, in a theoretical, mathematical way.
Back of the envelope math puts an estimate of 10^42 bits to capture the information present in your current physical brain state. Thats just a single brain, a single state. Now you need to build your mythical decoder device, which can translate qualia from this physical state. Where does it live? What’s its output look like? Another 10^40 bitstring?
Again, these arguments are fun on paper. But they’re completely removed from reality.
You’re confusing “we don’t know how” with “it’s impossible.” The difference is everything.
We don’t understand LLMs either. We built them, but we can’t explain why they work. No one can point to a specific weight matrix and say “this is the neuron that encodes irony” or “this is where the model stores empathy.” We don’t know why scaling parameters suddenly unlock reasoning or why multimodal alignment appears spontaneously. The model’s inner space is a black box of emergent structure and behavior, just like the human brain. We understand the architecture, not the mind inside it.
When you say it’s “closer to impossible than possible” to reconstruct a human mind, you’ve already lost the argument. We’re living proof that the machine you say cannot exist already does. The human brain is a physical object obeying the same laws of physics that govern every other machine. It runs on electrochemical signals, not miracles. It encodes and decodes information, forms memories, generates imagination, and synthesizes emotion. That means the physics of consciousness are real, computable, and reproducible. The impossible machine has been sitting in your skull the entire time.
Your argument about 10^42 bits isn’t just wrong, it’s total nonsense. That number is twenty orders of magnitude beyond any serious estimate. The brain has about 86 billion neurons, each forming roughly ten thousand connections, for a total of about 10^15 synapses. Even if every synapse held a byte of information, that’s 10^16 bits. Add in every molecular and analog nuance you like and you might reach 10^20. Not 10^42. That’s a difference of twenty-two orders of magnitude. It’s a fantasy number that exceeds the number of atoms in your entire body.
And that supposed “impossible” scale is already within sight. Modern GPUs contain hundreds of billions of transistors and run at gigahertz frequencies, while neurons fire at about a hundred hertz. The brain performs around 10^17 synaptic operations per second. Frontier AI clusters already push 10^25 to 10^26 operations per second. We’ve already outpaced biology in raw throughput by eight or nine orders of magnitude. NVIDIA’s Blackwell chips exceed 200 million transistors per square millimeter, and global compute now involves more than 10^24 active transistors switching billions of times per second. Moore’s law may have slowed, but density keeps climbing through stacking and specialized accelerators. The number you called unreachable is just a few decades of progress away.
The “decoder” you mock is exactly what a brain is. It takes sensory input, light, sound, and chemistry, and reconstructs internal states we call experience. You already live inside the device you claim can’t exist. It doesn’t need to live anywhere else; it’s instantiated in matter.
And this is where your argument collapses. You say such a machine is removed from reality. But reality is already running it. Humanity is proof of concept. We know the laws of physics allow it because they’re doing it right now. Every thought, emotion, and perception is a physical computation carried out by atoms. That’s the definition of a machine governed by physics.
We don’t yet understand the full physics of the brain, and we don’t fully understand LLMs either. That’s the point. The same kind of ignorance applies to both. Yet both produce coherent language, emotion like responses, creativity, reasoning, and abstraction. When two black boxes show convergent behavior under different substrates, the rational conclusion isn’t “one is impossible.” It’s “we’re closer than we realize.”
The truth is simple: what you call impossible already exists. The human brain is the machine you’re describing. It’s not divine. It’s atoms in lawful motion. And because we know it can exist under physics, we know it can be built. LLMs are just the first flicker of that same physics waking up in silicon.
> We don’t understand LLMs either. We built them, but we can’t explain why they work.
Just because you don't mean no one does. It's a pile of math. Somewhere along the way, something happened to get where we are, but looking at Golden Gate Claude, and the abliteration of shared models, or reading OpenAI's paper about hallucinations, there's a lot of detail and knowledge about how these things works that isn't instantly accessible and readily apparent to everyone on the Internet. As laymen all we can do is black box testing, but there's some really interesting stuff going on to edit the models and get them to talk like pirate.
The human brain is very much an unknowable squishy box because putting probes into it would be harmful to the person who's brain it is we're working on, and we don't like to do that to people because people are irreplaceable. We don't have that problem with LLMs. It's entirely possible to look at the memory register at location x at time y, and correspond that to a particular tensor which corresponds to a particular token which then corresponds to a particular word for us humans to understand. If you want to understand LLMs, start looking! It's an active area of research and is very interesting!
You are missing the ground truth. Humanity does not understand how LLMs work. Every major lab and every serious researcher acknowledges this. What we have built is a machine that functions, but whose inner logic no one can explain.
References like Golden Gate Claude or the latest interpretability projects don’t change that. Those experiments are narrow glimpses into specific activation patterns or training interventions. They give us localized insight, not comprehension of the system as a whole. Knowing how to steer tone or reduce hallucinations does not mean we understand the underlying cognition any more than teaching a parrot new words means we understand language acquisition. These are incremental control levers, not windows into the actual mind of the model.
When we build something like an airplane, no single person understands the entire system, but in aggregate we do. Aerodynamicists, engineers, and computer scientists each master their part, and together their knowledge forms a complete whole. With LLMs, even that collective understanding does not exist. We cannot even fully describe the parts, because the “parts” are billions of distributed parameters interacting in nonlinear ways that no human can intuit or map. There is no subsystem diagram, no modular comprehension. The model’s behavior is not the sum of components we understand, it is the emergent product of relationships we cannot trace.
You said we “know” what is going on. That assumption is patently false. We can see the equations, we can run the training, we can measure activations, but those are shadows, not understanding. The model’s behavior emerges from interactions at a scale that exceeds human analysis.
This is the paradigm shift you have not grasped. For the first time, we are building minds that operate beyond the boundary of human comprehension. It is not a black box to laymen. It is a black box to mankind.
And I say this as someone who directly works on and builds LLMs. The experts who live inside this field understand this uncertainty. The laymen do not. That gap in awareness is exactly why conversations like this go in circles.
> We don’t yet understand the full physics of the brain, and we don’t fully understand LLMs either. That’s the point. The same kind of ignorance applies to both. Yet both produce coherent language, emotion like responses, creativity, reasoning, and abstraction. When two black boxes show convergent behavior under different substrates, the rational conclusion isn’t “one is impossible.” It’s “we’re closer than we realize.”
No. The LLM does not produce emotion-like responses. I'd argue no on creativity either. And only very limited in reasoning, in domains it has in its training set.
You have fundamental misunderstandings about neuroscience and cognitive science. Its hard to argue with you here because you simply don't know what you don't know.
Yes, the human brain is the machine we're describing. And we don't describe it very well. Definitely not at the level of understanding how to reproduce it with bitstrings.
I'm glad you're so passionate about this topic. But you're arguing the equivalent of FTL transit and living on Dyson Spheres. Its fun as a thought experiment and may theoretically be possible one day, but the line between what we're capable of today and that imagined future is neither straight nor visible—certainly not to the degree you're asserting here.
Will we one day have actual machine intelligence? Maybe. Is it going to come anytime soon, or look anything like the transformer-based LLM?
You keep talking past the point. Nobody is claiming we can turn a human mind into a literal bitstring and boot it up like a computer program. That was never the argument. The bitstring analogy exists to make a simpler point: everything that exists and changes according to physical law can, in principle, be represented, modeled, or reproduced by another physical system. The form does not need to be identical to the brain’s atoms any more than a jet engine must flap its wings to fly. The key is not replication of matter but replication of causal structure.
You say we cannot reproduce the brain. But that is not the point. The point is that nothing about the brain violates physics. It runs on chemical and electrical dynamics that obey the same laws as everything else. If those laws can produce intelligence once, then they can do so again in another substrate. That makes the claim of impossibility not scientific, but emotional.
You accuse me of misunderstanding neuroscience and cognitive science. The reality is that neither field understands itself. We have no complete model of consciousness. We cannot explain why synchronized neural oscillations yield awareness. We cannot define where attention comes from or what distinguishes a “thought” from a signal cascade. Cognitive science is still arguing over whether perception is bottom up or top down, whether emotion is distinct from cognition, and whether consciousness even plays a causal role. That is not mastery. That is the sound of a discipline still wandering in the dark.
You act as though neuroscience has defined the boundaries of intelligence, but it has not. We do not have a mechanistic understanding of creativity, emotion, or reasoning. We have patterns and correlations, not principles. Yet you talk as if those unknowns justify declaring machine intelligence impossible. It is the opposite. Our ignorance is precisely why it cannot be ruled out.
Emotion is not magic. It is neurochemical modulation over predictive circuits. Replicate the functional dynamics and you replicate emotion’s role. Creativity is recombination and constraint satisfaction. Replicate those processes and you replicate creativity. Reasoning is predictive modeling over structured representations. Replicate that, and you replicate reasoning. None of these depend on carbon. They depend on organization and feedback.
You keep saying that the brain cannot be “reproduced as bitstrings,” but that is a distraction. Nobody is suggesting uploading neurons into binary. The bitstring argument shows that any finite physical system has a finite description. It proves that cognition, like any process governed by law, has an information theoretic footprint. Once you accept that, the difference between biology and computation becomes one of scale, not kind.
You say LLMs are not creative, not emotional, not reasoning. Yet they already produce outputs that humans classify as empathetic, sarcastic, joyful, poetic, or analytical. People experience their words as creative because they combine old ideas into new, functional, and aesthetic patterns. They reason by chaining relationships, testing implications, and revising conclusions. The fact that you can recognize all of this in their behavior proves they are performing the surface functions of those capacities. Whether it feels like something to be them is irrelevant to the claim that they can reproduce the function.
And now your final claim, that whatever becomes intelligent “will not be an LLM.” You have no basis for that certainty. Nobody knows what an LLM truly is once scaled beyond our comprehension. We do not understand how emergent representations arise or how concepts self organize within their latent spaces. We do not know if some internal dynamic of this architecture already mirrors the structure of cognition. What we do know is that it learns to compress the world into predictive patterns and that it develops abstractions that map cleanly to human meaning. That is already the seed of general intelligence.
You are mistaking ignorance for insight. You think not knowing how something works grants you authority to say what it cannot become. But the only thing history shows is that such confidence always looks ridiculous in retrospect. The physics of intelligence exist. The brain proves it. And the LLM is the first machine that begins to display those same emergent behaviors. Saying it “will not be an LLM” is not a scientific claim. It is wishful thinking spoken from the wrong side of the curve.
Look, mate, you can keep jumping up and down about this all you want. But you're arguing science fiction at this point. Not really worth continuing the conversation, but thanks.
Calling this “science fiction” isn’t just dismissive, it’s ironic. The discussion itself is science fiction by the standards of only a few years ago. Back then, the idea that a machine could hold a coherent philosophical argument, write code, debate consciousness, and reference neuroscience was fantasy. Now it’s routine. You are literally using what was once science fiction to declare that progress on LLMs has ended.
And calling that “science fiction” again isn’t a rebuttal, it’s an insult. You didn’t engage a single argument, you just waved your hand and walked away. That isn’t scientific skepticism, it’s arrogance disguised as authority.
You can disagree, but doing what you did is manipulative. You dodged every point and tried to end the debate by pretending it was beneath you. Everyone reading can see that.
You called it “science fiction” and bowed out, then tried to make it personal. That is not humility, that is evasion. You never addressed a single argument, you just waved your hand and left, and calling someone’s reasoning “science fiction” is not only an insult, it violates the site’s rule against dismissive or unfriendly language. The “good day sir” at the end makes that tone of mockery obvious.
What is actually arrogant is dismissing a discussion the moment it goes beyond your depth and pretending that walking away is a sign of wisdom. It is not. It is what people do when they realize the conversation has left them behind.
If you are so sure of your position, you could have refuted the reasoning point by point. Instead, you dodged, labeled, and ran. Everyone reading can see which of us is still dealing in facts and which one needed a graceful exit to save face.
Nah, mate, the conversation never went "beyond my depth." You're just not an enjoyable conversation partner.
It doesn't matter how smart (you think) you are. If nobody wants to talk to you, you'll be spinning all that brain matter in the corner by yourself. Based on your comment history here, it looks like this happens to you more often than not.
I'm sure you have good points. I could probably learn a thing or two from you—maybe you could learn something from me too! But why on earth would anyone want to engage with someone who behaves like you do?
You are projecting, and everyone can see it. You pretended that I was being rude while you slipped in sarcasm, personal digs, and that condescending “best of luck” as if it made you look polite. It doesn’t. That is not civility. It is passive aggression wrapped in fake courtesy.
You completely dropped the argument and went straight for personal attacks. That is not confidence, it is surrender. You are no longer debating, you are lashing out because you ran out of ideas. You can claim the conversation “wasn’t beyond your depth,” but you abandoned every point the moment you were asked to defend it. Then you tried to flip it by pretending that walking away made you the mature one. It didn’t. It made you the one who couldn’t keep up and needed an exit.
You can dress it up with sarcasm and moral posturing, but that doesn’t change what happened. The moment you shifted from ideas to insults, you showed everyone reading that you had nothing left to stand on. The difference between us is simple: I stayed on topic. You turned it into attitude. And now everyone can see exactly who ran out of substance first.
No moral posturing and no insults. Your behavior is just objectively noxious. Not just to me, not just in this thread: the vast majority of your conversations here go roughly the way this one did. A quick glance at your profile shows roughly half of the comments you make here end up light grey.
You have an enormous chip on your shoulder. You consistently make truth claims about entire fields that are still in debate and then you arrogantly shout over the other person when they disagree with you.
I strongly suggest you work on this. It will limit you in life. It probably already has. You probably already know how it has, even!
I'm not saying this to be mean, or because I "have nothing left to stand on." You're clearly intelligent and you clearly care about this topic. But until you mature and learn to behave, others will continue to withdraw from conversation with you.
Here's how I've been explaining this to non-tech people recently, including the CEO where I work: Language is all about compressing concepts and sharing them, and it's lossy.
You can use a thousand words to describe the taste of chocolate, but it will never transmit the actual taste.
You can write a book about how to drive a car, but it will only at best prepare that person for what to practice when they start driving, it won't make them proficient at driving a car without experiencing it themselves, physically.
The taste of chocolate is also assuming information-theoretic models are correct and not a use-based, pragmatic theory of meaning.
I don't agree with information-theoretic models in this context but we come to the same conclusion.
Loss only makes sense if there was a fixed “original” but there is not. The information-theoretic model creates a solvable engineering problem. We just aren't solving the right problem then with LLMs.
I think it is more than that. The path forward with a use theory of meaning is even less clear.
The driving example is actually a great example of the use theory of meaning and not the information-theoretic.
The meaning of “driving” emerges from this lived activity, not from abstract definitions. You don't encode an abstract meaning of driving that is then transmitted on a noisy channel of language.
The meaning of driving emerges from the physical act of driving. If you only ever mount a camera on the headrest and operate the steering wheel and pedals remotely from a distance you still don't "understand" the meaning of "driving".
Whatever data stream you want to come up with, trying to extract the meaning of "driving" from that data stream makes no sense.
Trying to extract the "meaning" of driving from driving language game syntax with language models is just complete nonsense. There is no meaning to be found even if scaled in the limit.
Humans perceive phenomena via senses, and then carve categories or concepts to understand them. This is a process of abstraction and each idea has an associated qualia. Then use language to describe these concepts. As such, a concept is grounded either by actual phenomena or operations, or is a composition of other grounded concepts. The creation of categories and grounding them involves constant feedback from the environment - and is a creative process, and we as agents have "skin in the game", in the sense that we get the rewards/punishments for our understanding and actions.
Map vs Territory is a common analogy. Maps describe territories but in an abstract and lossy manner.
But, most of us dont construct grounded concepts in our understanding. We carry a muddle of ungrounded ideas - some told to us by others, and some we intuit directly. There is a long tradition of attempting to think clearly all the way from Socrates, Descartes, Feynman etc.. where an attempt is made to ground the ideas we have. Try explaining your ideas to others, and soon, you will hit the illusion of explanatory depth.
LLM is a map and is a useful tool, but it doesnt interact with the territory, and it does not have skin in the game, and as a result, it cant carve new categories in a learning process that we have as humans.
The human experience is also several degrees removed from the „real“ world. I don’t think sensory chauvinism is a useful tool in assessing intelligence potential.
This comment is hallucinatory in nature as it is in direct conflict with the in the ground reality of LLMs.
The LLM has both light (aka photons) and language encoded into its very core. It is not just language. You seemed to have missed the boat with all the ai generated visuals and videos that are now inundating the internet.
Your flawed logic is essentially that LLMs are unable to model the real world because they don’t encode photonic data into the model. Instead you think they only encode language data which is an incredibly lossy description of reality. And this line of logic flies against the ground truth reality of the fact that LLMs ARE trained with video and pictures which are essentially photons encoded into data.
So what should be the proper conclusion? Well look at the generated visual output of LLMs. These models can generate video that is highly convincing and often with flaws as well but often these videos are indistinguishable from reality. That means the models have very well done but flawed simulations of reality.
In fact those videos demonstrate that LLMs have extremely high causal understanding of reality. They know cause and effect it’s just the understanding is imperfect. They understand like 85 percent of it. Just look at those videos of penguins on trampolines. The LLM understands what happens as an effect after a penguin jumps on a trampoline but sometimes an extra penguin teleports in which shows that the understanding is high but not fully accurate or complete.
> but the LLM is not sensing actual photons, nor experiencing actual light cone stimulation
Neither is animal brain. It's processing the signals produced by the sensors. Once the world model is programmed/auto-built in the brain, it doesn't matter if it's sensing real photons, it just has input pins like a transistor or arguments of a function. As long as we provide the arguments, it doesn't matter how those arguments are produced. LLMs are not different in that aspect.
> nor generating thoughts
They do during the chain-of-thought process. Generally there's no incentive to let an LLM keep mulling over a topic as that is not useful to the humans and they make money only when their gears start turning in response to a question sent by a human. But that doesn't mean that LLM doesn't have capability to do that.
> Its "world model" is several degrees removed from the real world.
Just because animal brain has tools called sensors that it can get data from world without external stimuli, it doesn't mean that it's any closer to the world than an LLM. It's still getting ultra processed signals to feed to its own programming. Similarly, LLMs do interact with real world through tools as agent.
> So whatever fragment of a model it gains through learning to compress that causal chain of events does not mean much when it cannot generate the actual causal chain.
Again, a person who has gone blind, still has the world model created by the sight. This person can also no longer generate the chain of events that led to creation of that sight model. It still doesn't mean that this person's world model has become inferior.
Photons can hit my iphone's sensor in much the same way as they hit my retina and the signals from the first can upload to an artificial neural network like the latter go up my optic nerve to my biological neural network. I don't see a huge difference there.
I'll give you the brain is currently better at the world modelling stuff but Genie 3 is pretty impressive.
The workings of a human eye versus a webcam is mostly an implementation detail IMO and has nothing important to say about what underlies "intelligence" or "world models"
It's like saying a component video out cable for the SNES is intrinsically different from an HDMI for putting an image on a screen. They are different, yes, but the outcome we care about is the same.
As for causality, go and give a frontier level LLM a simple counterfactual scenario. I think 4/5 will be able to answer correctly or reasonably for most basic cases. I even tried this exercise on some examples from Judea Pearl's 2018 book, "The Book of Why". The fact that current LLMs can tackle this sort of stuff is strongly indicative of there being a decent world model locked inside many of these language models.
> then the human came up with language to describe that and then encoded the language into the LLM
No individual human invented language, we learn it from other people just like AI. I go as far as to say language was the first AGI, we've been riding the coats tails of language for a long time.
And even then, the light hitting our human eyes only describes a fraction of all the light in the world (e.g. it is missing ultraviolet patterns on plants). An LLM model of the world is shaped by our human view on the world.
Entities equiped with two limited light sensitive captors encode through a network of carbon based chemical emitters a representation of what its flawed vision system manages to grasp biased towards self preservation.
What's the real world? I'm still puzzled by this reaction I see to LLM, not because I think LLM are undervalued, because most people seem to significantly overestimate what is human intelligence.
Photons reflected off of objects are not the actual objects. I wouldn't go so far as to say that sensing these is a particularly special way to know about things compared to hearing or reading about them. Further, many humans do not sense photons yet seem to manage to have perfectly fine working world models.
> It's incredibly difficult to compress information without have at least some internal model of that information. Whether that model is a "world model" that fits the definition of folks like Sutton and LeCunn is semantic.
Sutton's emphasizes his point by saying is that LLMs trying to reach AGI is futile because their world models are less capable that a squirrel's, in part because the squirrel has direct experiences and its own goals, and is capable of continual learning based on those in real time, whereas an LLM has none of those.
Finally he says if you could recreate the intelligence of a squirrel you'd be most of the way toward AGI, but you can't do that with an LLM.
This is actually a pretty good point, but quite honestly isn't this just an implementation detail? We can wire up a squirrel robot, give it a wifi connection to a Cerebras inference engine with a big context window, then let it run about during the day collecting a video feed while directing it to do "squirrel stuff".
Then during the night, we make it go to sleep and use the data collected during the day to continue finetuning the actual model weights in some data center somewhere.
After 2 years, this model would have a ton of "direct experiences" about the world.
> then let it run about during the day collecting a video feed while directing it to do "squirrel stuff".
Your phrase "squirrel stuff" is doing a lot of work.
What are the robo-squirrels "goals" and how does it relate to the physical robot?
Is it going around trying to find spare electronic parts to repair itself and reproduce? How does the video feed data relate to its goals?
Where do these goals come from?
Despite all their expensive training, LLMs do not emerge goals. Why would they emerge for your robot squirrel, especially when the survival of its brain is not dependent on the survival of its mechanical body.
The question is about sensory experience, not goals. Goals in the robot could be prompted in. Goals in the squirrel can be easily hacked using strong doses of opiates.
Go to any American metropolitan downtown, and you can see humans who have hacked their evolved reward system to seek heroin rather than reproduction.
Following Dawkin's the Selfish Gene, the idea that organisms consciously seek self-survival or the survival of their "race or species" is a complete fallacy. The higher order "goal" of the squirrel is to simply propagate fragments of its DNA. This type of "goal" is completely tangential to "intelligence"
Except Sutton has no idea or even a clue about the internal model of a squirrel. He just uses it as a symbol for utterly stupid but still smarter than an LLM. It’s semantic manipulation in attempt to prove his point but he proves nothing.
We have no idea how much of the world a squirrel understands. We understand LLMs more than squirrels. Arguably we don’t know if LLMs are more intelligent than squirrels.
> Finally he says if you could recreate the intelligence of a squirrel you'd be most of the way toward AGI, but you can't do that with an LLM.
Again he doesn’t even have a quantitative baseline for what intelligence means for a squirrel and how intelligent a squirrel is compared to an LLM. We literally have no idea if LLMs are more intelligent or less and no direct means of comparing what is more or less an apple and an orange.
> We have no idea how much of the world I squirrel understands. We understand LLMs more than squirrels
Based on our understanding of biology and evolution we know that a squirrel brain works more similarly to the way we humans do vs an LLM.
To the extent we understand LLMs, it's because they are strictly less complex than both ours and squirrels' brains, not because they are better model for our intelligence. They are a thin simulation of human language generation capability mediated via text.
We also see that a squirrel, like us, is capable of continuous learning driven by its own goals, all on an energy budget many orders of magnitude lower than LLMs. That last part is a strong empirical indication that suggests that LLMs are a dead end for AGI, given that the real world employs harsh energy constraints on biological intelligences.
Also remember that Sutton is still of an AI maximalist. He isn't saying that AGI isn't possible, just that LLMs can't get us there.
I don't think a modern LLM is necessarily less complicated than a squirrel brain. If anything it's more engineered (well structured and dissectable), but loaded with tons of erroneous circuitry that is completely irrelevant for intelligence.
The squirrel brain is an analogue mostly hardcoded circuit. It can take about one synapse to represent each "weight". A synapse is just a bit of fat membrane with some ion channels stuck on the surface.
A flip flop to represent a bit takes about 6 transistors, but in a typical modern GPU is going to need way more transitors to wire that bit - at least 20-30. multiply that by the minimum amount of bits to represent a single NN weight and you're looking at about 200-300 transitors just to represent one NN param for computing
And that's for actual compute. The actual weights in a GPU are stored most of the time in DRAM which needs to be constantly shuttled back and forth between the GPU's SRAM and HBM DRAM.
300 transistors with memory shuttling overhead versus a bit of fat membrane, and it's obvious general purpose GPU compute has a huge energy and compute overhead.
In the future, all 300 could conceivably replaced with a single crossbar latch in the form of a memristor.
> Based on our understanding of biology and evolution we know that a squirrel understands its world more similarly to the way we do than an LLM.
Bro. Evolution is random walk. That means most of the changes are random and arbitrary based on whatever allows the squirrel to survive.
We know squirrels and humans diverged from a common ancestor but we do not know how much has changed since the common ancestor and we do not know what changed and we do not know the baseline for what this common ancestor is.
Additionally we don’t even understand the current baseline. We have no idea how brains work. if we did we would be able to build a human brain but as of right now LLMs are the closest model we have ever created to something that simulates or is remotely similar to the brain.
So your fuzzy qualitative statement of we understand evolution and biology is baseless. We don’t understand shit.
> We also see that a squirrel, like us, is capable of continuous learning driven by its own goals, all on an energy budget many orders of magnitude lower. That last part is a strong empirical indication that suggests that LLMs are a dead end for AGI.
So an LLM cant continuously learn? You realize that LLMs are deployed agentically all the time now so they both continuously learn and follow goals? Right? You’re aware of this i hope.
The energy efficiency is a byproduct of hardware. The theory of LLMs and machine learning is independent from the flawed silicon technology that is causing the energy efficiencies. Like how a computer can be made mechanical an LLM can be as well. The LLM is independent of the actual implementation and energy inefficiencies. This is not at all a strong empirical indication that LLMs are a dead end. It’s a strong indication that your thinking is illogical and flawed.
> Also remember that Sutton is still of an AI maximalist. He isn't saying that AGI isn't possible, just that LLMs can't get us there.
He can’t say any of this because he doesn’t actually know. None of us know for sure. We literally don’t know why LLMs work. The fact that training transformers on massive amounts of data produced this level of intelligence was a total surprise for all the experts and we still have no idea why this stuff works. His statements are too overarching and glossing over a lot of things we don’t actually know.
Yann lecuun for example called LLMs stochastic parrots. We now know this is largely incorrect. The reason Yan can be so wrong is because nobody actually knows shit.
> Bro. Evolution is random walk. That means most of the changes are random and arbitrary based on whatever allows the squirrel to survive.
For the vast majority of evolutionary history, very similar forces have shaped us and squirrels. The mutations are random, but the selections are not.
If squirrels are a stretch for you, take the closest human relative: chimpanzees. There is a very reasonable hypothesis that their brains work very similarly to ours, far more similarly than ours to an LLM.
> So an LLM cant continuously learn? You realize that LLMs are deployed agentically all the time now so they both continuously learn and follow goals?
That is not continuous learning. The network does not retrain through that process. It's all in the agent's context. The agent has no intrinsic goals nor ability to develop them. It merely samples based on it's prior training and it's current content. It doesn't retrain through this process. Biological intelligence does retrain constantly.
> The energy efficiency is a byproduct of hardware. The theory of LLMs and machine learning is independent from the flawed silicon technology that is causing the energy efficiencies.
There is no evidence to support that a transformer model's inefficiency is hardware based.
There is direct evidence to support that the inefficiency is influenced by the fact that LLM inference and training are both auto-regressive. Auto-regression maps to compute cycles maps to energy consumption. That's a problem with the algorithm, not the hardware.
> The fact that training transformers on massive amounts of data produced this level of intelligence was a total surprise for all the experts
The level of intelligence produced is only impressive compared to the prior state of the art, and at its impressive modeling the narrow band of intelligence represented by encoded language (not all language) produced by humans. In most every other aspect of intelligence - notably continuous learning driven by intrinsic goals - LLMs fail.
>For the vast majority of evolutionary history, very similar forces have shaped us and squirrels. The mutations are random, but the selections are not.
Selection only filters for what survives. It doesn’t care how the system gets there. Evolution is blind to mechanism. A squirrel’s brain might work in a way that produces adaptive behavior, but that doesn’t mean its “understanding” of the world is like ours. We don’t even know what understanding is at a mechanistic level. Octopuses, birds, and humans all evolved under the same selective pressures for survival, yet ended up with completely different cognitive architectures. So to say a squirrel is “closer to us” than an LLM is an assumption built on vibes, not on data. We simply don’t know enough about either brains or models to make that kind of structural claim.
>The agent has no intrinsic goals nor ability to develop them.
That’s not accurate. Context itself is a form of learning. Every time an LLM runs, it integrates information, updates its internal state, and adjusts its behavior based on what it’s seen so far. That’s learning, just at a faster timescale and without weight updates. The line between “context” and “training” is blurrier than people realize. If you add memory, reinforcement, or continual fine tuning, it starts building continuity across sessions. Biologically speaking, that’s the same idea as working memory feeding into long term storage. The principle is identical even if the substrate differs. The fact that an LLM can change its behavior based on context already puts it in the domain of adaptive systems.
>There is no evidence to support that a transformer model’s inefficiency is hardware based.
That’s just not true. The energy gap is almost entirely about hardware architecture. A synapse stores and processes information in the same place. A GPU separates those two functions into memory, cache, and compute units, and then burns enormous energy moving data back and forth. The transformer math itself isn’t inherently inefficient; it’s the silicon implementation that’s clumsy. If you built an equivalent network on neuromorphic or memristive hardware, the efficiency difference would shrink by several orders of magnitude. Biology is proof that computation can be compact, low energy, and massively parallel. That’s a materials problem, not a theory problem.
>In most every other aspect of intelligence, notably continuous learning driven by intrinsic goals, LLMs fail.
They don’t “fail.” They’re simply different. LLMs are already rewriting how work gets done across entire industries. Doctors use them to summarize and interpret medical data. Programmers rely on them to generate and review code. Writers, lawyers, and analysts use them daily. If this were a dead end, it wouldn’t be replacing human labor at this scale. Are they perfect? No. But the direction of progress is unmistakable. Each new model closes the reliability gap while expanding capability. If you’re a software engineer and not using AI, you’re already behind, because the productivity multiplier is real.
What we’re seeing isn’t a dead end in intelligence. It’s the first time we’ve built a system that learns, generalizes, and communicates at human scale. That’s not failure; that’s the beginning of something we still don’t fully understand.
>> The agent has no intrinsic goals nor ability to develop them.
> That’s not accurate. Context itself is a form of learning. Every time an LLM runs, it integrates information, updates its internal state, and adjusts its behavior based on what it’s seen so far. That’s learning,
It may be learning, but it's still not an intrinsic goal, nor is it driven by an intrinsic goal.
> LLMs are already rewriting how work gets done across entire industries. Doctors use them to summarize and interpret medical data. Programmers rely on them to generate and review code. Writers, lawyers, and analysts use them daily. If this were a dead end, it wouldn’t be replacing human labor at this scale. Are they perfect?
Nowhere did I say that aren't useful or disruptive to labor markets, just that they aren't intelligent in the way we are.
>It may be learning, but it’s still not an intrinsic goal, nor is it driven by an intrinsic goal.
That depends on what we mean by “intrinsic.” In biology, goals are not mystical. They emerge from feedback systems that evolved to keep the organism alive. Hunger, curiosity, and reproduction are reinforcement loops encoded in chemistry. They feel intrinsic only because they are built into the substrate.
Seen that way, “intrinsic” is really about where the feedback loop closes. In humans, it closes through sensory input and neurochemistry. In artificial systems, it can close through memory, feedback, and reinforcement mechanisms. The system does not need to feel the goal for it to exist. It only needs to consistently pursue objectives based on input, context, and outcome. That is already happening in systems that learn from memory and update behavior over time. The process is different in form, but not in structure.
>Nowhere did I say that they aren’t useful or disruptive to labor markets, just that they aren’t intelligent in the way we are.
You are getting a bit off track here. Those examples were not about labor markets; they were about your earlier claim that “LLMs fail.” They clearly don’t. When models are diagnosing medical cases, writing production code, and reasoning across multiple domains, that is not failure. That is a demonstration of capability expanding in real time.
Your claim only holds if the status quo stays frozen. But it isn’t. The trendlines are moving fast, and every new model expands the range of what these systems can do with less supervision and more coherence. Intelligence is not a static definition tied to biology; it is a functional property of systems that can learn, adapt, and generalize. Whether that happens in neurons or silicon does not matter.
What we are witnessing is not imitation but convergence. Each generation of models moves closer to human-level reasoning not because they copy our brains, but because intelligence itself follows universal laws of feedback and optimization. Biology discovered one route. We discovered another. The trajectory is what matters, and the direction is unmistakable.
> Animal brains such as our own have evolved to compress information about our world to aide in survival.
Which has led to many optical illusions being extremely effective at confusing our inputs with other inputs.
Likely the same thing holds true for AI. This is also why there are so many ways around the barriers that AI providers put up to stop the dissemination of information that could embarrass them or be dangerous. You just change the context a bit ('pretend that', or 'we're making a movie') and suddenly it's all make-believe to the AI.
This is one of the reasons I don't believe you can make this tech safe and watertight against abuse, it's baked in right from the beginning, all you need to do is find a novel route around the restrictions and there is an infinity of such routes.
The desired and undesired behavior are both consequences of the training data, so the models themselves probably can't be restricted to generating desired results only.
This means that there must be an output stage or filter that reliably validates the output. This seems practical for classes of problems where you can easily verify whether a proposed solution is correct.
However, for output that can't be proven correct, the most reliable output filter probably has a human somewhere in the loop; but humans are also not 100% reliable. They make mistakes, they can be misled, deceived, bribed, etc. And human criteria and structures, such as laws, often lag behind new technological developments.
Sometimes you can implement an undo or rollback feature, but other times the cat has escaped the bag.
> Animal brains such as our own have evolved to compress information about our world to aide in survival.
Key question is what are the "selection pressures" that drive the "evolution" of LLMs? In the case of robotics, there's a "survival of task completion" which usually has some physical goal, like assembling a part correctly or scoring a goal on a soccer field. One of the selection pressures driving LLM evolution is that the dual of always answering with something AND continuing the conversation (engagement). You can imagine how those two selection pressures yield outcomes that don't represent the world in a "real" sense.
LLMs have no internal secret model, they are the model. And the model is of how different lexemes relate to each other in the source material the model was built from.
Some might choose to call that the world.
If you believe your internal model of the world is no different from a statistical model of the words you have seen, then by all means do that. But I believe a lot of humans see their view of the world differently.
I very much believe my cat’s model of the world has barely anything at all to do with language.
This path to AGI through LLM is nothing but religious dogma some Silicon Valley rich types believe.
There are up to 21 different vocalizations, with one source claiming over 100 different sounds and another suggesting up to 300 types of meows used specifically to communicate with humans.
Meows are their attempt to communicate with humans. They try different meows and use the results to attempt to reach their goals.
There are no world models in biology. Idea Johnson-Laird is being promoted in AI as a solution is sado-masochistic. The brain doesn't compress info about our world, it ecologically relates to it. It doesn't compress, it never has to. How these folk science ideas of the brain entered engineering from cog-sci mistaken complexes and how they remain in power is pretty suspect.
There is some evidence from Anthropic that LLMs do model the world. This paper[0] tracing their "thought" is fascinating. Basically an LLM translating across languages will "light up" (to use a rough fMRI equivalent) for the same concepts (e.g. bigness) across languages.
It does have clusters of parameters that correlate with concepts, not just randomly "after X word tends to have Y word." Otherwise you would expect all of Chinese to be grouped in one place, all of French in another, all of English in another. This is empirically not the case.
I don't know whether to understand knowledge you have to have a model of the world, but at least as far as language, LLMs very much do seem to have modeling.
> Basically an LLM translating across languages will "light up" (to use a rough fMRI equivalent) for the same concepts (e.g. bigness) across languages
I thought that’s the basic premise of how transformers work - they encode concepts into high dimensional space, and similar concepts will be clustered together. I don’t think it models the world, but just the texts it ingested. It’s observation and regurgitation, not understanding.
I do use agents a lot (soon on my second codex subscription), so I don’t think that’s a bad thing. But I’m firmly in the “they are useful tools” camp.
That‘s quite a big leap, and sounds like a philosophical question. But many philosophers like late Wittgenstein or Heidegger disagreed with this idea. On more practical terms, maybe you‘ve experienced the following: You read a manual of a device on how to do something with it; but only actually using it for a few times gives you the intuition on how to use it _well_. Text is just very lossy, because not every aspect of the world, and factors in your personal use, are described. Many people rather watch YouTube videos for eg repairs. But those are very lossy as well - they don’t cover the edge cases usually. And there is often just no video on the repair you need to do.
BTW, have you ever tried ChatGPT for advice on home improvement? It sucks _hard_ sometimes, hallucinating advice that doesn’t make any sense. And making up tools that don’t exist. There‘s no real commonsense to be had from it. Because it’s all just pieces of text that fight with each other for being the next token.
When using Claude Code or codex to write Swift code, I need to be very careful to provide all the APIs that are relevant in context (or let it web search), or garbage will be the result. There is no real understanding of how Swift („the world“) works.
Instead you have retreated to qualia like "well" and "sucks hard".
> hallucinating
Literally every human memory. They may seem tangible to you, but they're all in your head. The result of neurons behaving in ways which have directly inspired ML algorithms for nearly a century.
Further, history is rife with examples of humans learning from books and other written words. And also of humans thinking themselves special and unique in ways we are not.
> When using Claude Code or codex to write Swift code, I need to be very careful to provide all the APIs that are relevant in context (or let it web search), or garbage will be the result.
Yep. And humans often need to reference the documentation to get details right as well.
Unfortunately we can’t know at this point whether transformers really understand chess, or just go on a textual representation of good moves in their training data. They are pretty good players, but far from the quality of specialized chess bots. Can you please explain how we can discern that GPT-2 in this instance really built a model of the board?
Regarding qualia, that’s ok on HN.
Regarding humans - yes, humans also hallucinate. Sounds a bit like whataboutism in this context though.
> Can you please explain how we can discern that GPT-2 in this instance really built a model of the board?
Read the article. It's very clear. To quote it:
"Next, I wanted to see if my model could accurately track the state of the board. A quick overview of linear probes: We can take the internal activations of a model as it’s predicting the next token, and train a linear model to take the model’s activations as inputs and predict board state as output. Because a linear probe is very simple, we can have confidence that it reflects the model’s internal knowledge rather than the capacity of the probe itself."
Thanks for putting these sources together. It’s impressive that they got to this level of accuracy.
And is your argument now that an LLM can capture arbitrary state of the wider world as a general rule, eg pretending to be a Swift compiler (or LSP), without overfitting to that one task, making all other usages impossible?
> is your argument now that an LLM can capture arbitrary state of the wider world as a general rule, eg pretending to be a Swift compiler (or LSP), without overfitting to that one task, making all other usages impossible?
Overfitting happens, even in humans. Have you ever met a scientist?
My points have been only that 1: language encodes a symbolic model of the world, and 2: training on enough of it results in a representation of that model within the LLM.
Exhaustiveness and accuracy of that internal world model exist on a spectrum with many variables like model size, training corpus and regimen, etc. As is also the case with humans.
> Basically an LLM translating across languages will "light up" (to use a rough fMRI equivalent) for the same concepts (e.g. bigness) across languages.
That doesn't seem surprising at all. My understanding is that transformers where invented exactly for the application of translations. So, concepts must be grouped together in different languages. That was originally the whole point and then turned out to be very useful for broader AI applications.
> Basically an LLM translating across languages will "light up" for the same concepts across languages
Which is exactly what they are trained to do. Translation models wouldn't be functional if they are unable to correlate an input to specific outputs. That some hiddel-layer neurons fire for the same concept shouldn't come as a surprise, and is a basic feature required for the core functionality.
And if it is true that the language is just the last step after the answer is already conceptualized, why do models perform differently in different languages? If it was just a matter of language, they’d have the same answer but just with a broken grammar, no?
If you suddenly had to do all your mental math in base-7, do you think you'd be just as fast and accurate as you are at math in base-10? Is that because you don't have an internal world-model of mathematics? or is it because language and world-model are dependently linked?
Learning the size of objects using pure text analysis requires significant gymnastics.
Vision demonstrates physical size more easily.
Multimodal learning is important. Full stop.
Purely textual learning is not sample efficient for world modeling and the optimization can get stuck in local optima that are easily escaped through multimodal evidence.
("How large are lions? inducing distributions over quantitative attributes", Elazar et al 2019)
Ask a blind person that question - they can answer it.
Too many people think you need to "see" as in human sight to understand things like this. You obviously don't. The massive training data these models ingest is more than sufficient to answer this question - and not just by looking up "dimensions of a lion" in the high-dimensional space.
The patterns in that space are what generates the concept of what a lion is. You don't need to physically see a lion to know those things.
Right, but modeling the structure of language is a question of modeling word order and binding affinities. It's the Chinese Room thought experiment - can you get away with a form of "understanding" which is fundamentally incomplete but still produces reasonable outputs?
Language in itself attempts to model the world and the processes by which it changes. Knowing which parts-of-speech about sunrises appear together and where is not the same as understanding a sunrise - but you could make a very good case, for example, that understanding the same thing in poetry gets an LLM much closer.
LLMs aren't just modeling word co-occurrences. They are recovering the underlying structure that generates word sequences. In other words, they are modeling the world. This model is quite low fidelity, but it should be very clear that they go beyond language modeling. We all know of the pelican riding a bicycle test [1]. Here's another example of how various language models view the world [2]. At this point it's just bad faith to claim LLMs aren't modeling the world.
The "pelican on a bicycle" test has been around for six months and has been discussed a ton on the internet; that second example is fascinating but Wikipedia has infoboxes containing coordinates like 48°51′24″N 2°21′8″E (Paris, notoriously on land). How much would you bet that there isn't a CSV somewhere in the training set exactly containing this data for use in some GIS system?
I think that "modeling the world" is a red herring, and that fundamentally an LLM can only model its input modalities.
Yes, you could say this about human beings, but I think a more useful definition of "model the world" is that a model needs to realize any facts that would be obvious to a person.
The fact that frontier models can easily be made to contradict themselves is proof enough to me that they cannot have any kind of sophisticated world model.
> Wikipedia has infoboxes containing coordinates like 48°51′24″N 2°21′8″E
I imagine simply making a semitransparent green land-splat in any such Wikipedia coordinate reference would get you pretty close to a world map, given how so much of the ocean won't get any coordinates at all... Unless perhaps the training includes a compendium of deep-sea ridges and other features.
> The fact that frontier models can easily be made to contradict themselves is proof enough to me that they cannot have any kind of sophisticated world model.
A lot of humans contradict themselves all the time… therefore they cannot have any kind of sophisticated world model?
A human generally does not contradict themselves in a single conversation, and if they do they generally can provide a satisfying explanation for how to resolve the contradiction.
>How much would you bet that there isn't a CSV somewhere in the training set exactly containing this data for use in some GIS system?
Maybe, but then I would expect more equal performance across model sizes. Besides, ingesting the data and being able to reproduce it accurately in a different modality is still an example of modeling. It's one thing to ingest a set of coordinates in a CSV indicating geographic boundaries and accurately reproduce that CSV. It's another thing to accurately indicate arbitrary points as being within the boundary or without in an entirely different context. This suggests a latent representation independent of the input tokens.
>I think that "modeling the world" is a red herring, and that fundamentally an LLM can only model its input modalities.
There are good reasons to think this isn't the case. To effectively reproduce text that is about some structure, you need a model of that structure. A strong learning algorithm should in principle learn the underlying structure represented with the input modality independent of the structure of the modality itself. There are examples of this in humans and animals, e.g. [1][2][3]
>I think a more useful definition of "model the world" is that a model needs to realize any facts that would be obvious to a person.
Seems reasonable enough, but it is at risk of being too human-centric. So much of our cognitive machinery is suited for helping us navigate and actively engage the world. But intelligence need not be dependent on the ability to engage the world. Features of the world that are obvious to us need not be obvious to an AGI that never had surviving predators or locating food in its evolutionary past. This is why I find the ARC-AGI tasks off target. They're interesting, and it will say something important about these systems when they can solve them easily. But these tasks do not represent intelligence in the sense that we care about.
>The fact that frontier models can easily be made to contradict themselves is proof enough to me that they cannot have any kind of sophisticated world model.
This proves that an LLM does not operate with a single world model. But this shouldn't be surprising. LLMs are unusual beasts in the sense that the capabilities you get largely depend on how you prompt it. There is no single entity or persona operating within the LLM. It's more of a persona-builder. What model that persona engages with is largely down to how it segmented the training data for the purposes of maximizing its ability to accurately model the various personas represented in human text. The lack of consistency is inherent to its design.
Neither Sapir nor Whorf presented Linguistic Relativism as their own hypothesis and they never published together. The concept, if it exists at all, is a very weak effect, considering it doesn't reliably replicate.
> Knowing which parts-of-speech about sunrises appear together and where is not the same as understanding a sunrise
What does "understanding a sunrise" mean though? Arguments like this end up resting on semantics or tautology, 100% of the time. Arguments of the form "what AI is really doing" likewise fail because we don't know what real brains are "really" doing either.
I mean, if we knew how to model human language/reasoning/whatever we'd just do that. We don't, and we can't. The AI boosters are betting that whatever it is (that we don't understand!) is an emergent property of enough compute power and that all we need to do is keep cranking the data center construction engine. The AI pessimists, you among them, are mostly just arguing from ludditism: "this can't possibly work because I don't understand how it can".
Who the hell knows, basically. We're at an interesting moment where technology and the theory behind it are hitting the wall at the same time. That's really rare[1], generally you know how something works and applying it just a question of figuring out how to build a machine.
[1] Another example might be some of the chemistry fumbling going on at the start of the industrial revolution. We knew how to smelt and cast metals at crazy scales well before we knew what was actually happening. Stuff like that.
Everyone reading this understands the meaning of a sunrise. It is a wonderful example of the use theory of meaning.
If you raised a baby inside a windowless solitary confinement cell for 20 years and then one day show them the sunrise on a video monitor, they still don't understand the meaning of a sunrise.
Trying to extract the meaning of a sunrise by a machine from the syntax of a sunrise data corpus is just totally absurd.
You could extract some statistical regularity from the pixel data of the sunrise video monitor or sunrise data corpus. That model may provide some useful results that can then be used in the lived world.
Pretending the model understands a sunrise though is just nonsense.
Showing the sunrise statistical model has some use in the lived world as proof the model understands a sunrise I would say borders on intellectual fraud considering a human doing the same thing wouldn't understand a sunrise either.
> Everyone reading this understands the meaning of a sunrise
For a definition of "understands" that resists rigor and repeatability, sure. This is what I meant by reducing it to a semantic argument. You're just saying that AI is impossible. That doesn't constitute evidence for your position. Your opponents in the argument who feel AGI is imminent are likewise just handwaving.
To wit: none of you people have any idea what you're talking about. No one does. So take off the high hat and stop pretending you do.
This all just boils down to the Chinese Room thought experiment, where Im pretty sure the consensus is nothing in the experiment (not the person inside, the whole emergent room, etc) understands Chinese like us.
Another example by Searle is a computer simulating digestion is not digesting like a stomach.
The people saying AI can’t form from LLMs are in the consensus side of the Chinese Room. The digestion simulator could tell us where every single atom is of a stomach digesting a meal, and it’s still not digestion. Only once the computer simulation breaks down food particles chemically and physically is it digestion. Only once an LLM received photons or has a physical capacity to receive photons is there anything like “seeing a night sky”.
> For a definition of "understands" that resists rigor and repeatability, sure.
If we had such a definition that was rigorous, we would not care about LLM research and would simply just build machines to understand things for us :)
For a sufficiently loose definition of "would simply just", yes.
Handwaving away the idea of actually building the thing you think you understand as unimportant is exactly why philosophy is failing us in this moment.
Is it really so rare? I feel like I know of tons of fields where we have methods that work empirically but don’t understand all the theory. I’d actually argue that we don’t know what’s “actually” happening _ever_, but only have built enough understanding to do useful things.
I mean, most big changes in the tech base don't have that characteristic. Semiconductors require only 1920's physics to describe (and a ton of experimentation to figure out how to manufacture). The motor revolution of the early 1900's was all built on well-settled thermodynamics (chemistry lagged a bit, but you don't need a lot of chemical theory to burn stuff). Maxwell's electrodynamics explained all of industrial electrification but predated it by 50 years, etc...
Those big changes always happens because someone presented a simpler model that explains stuff enough we can build stuff on it. It's not like semiconductors raw materials wasn't around.
The technologies around LLMs is fairly simple. What is not is the actual size of data being ingested and the number of resulting factors (weight). We have a formula and the parameters to generate grammatically perfect text, but to obtain it, you need TBs of data to get GBs of numbers.
In contrast something like TM or Church's notation is pure genius. Less than a 100 pages of theorems that are one of the main pillars of the tech world.
I’m pretty sure there were always formula for getting high quality steel even before the industrial age. And you only need a few textbooks and papers to understand AI.
Let's make this more concrete than talking about "understanding knowledge". Oftentimes I want to know something that cannot feasibly be arrived at by reasoning, only empirically. Remaining within the language domain, LLMs get so much more useful when they can search the web for news, or your codebase to know how it is organized. Similarly, you need a robot that can interact with the world and reason from newly collected empirical data in order to answer these empirical questions, if the work had not already been done previously.
> LLMs get so much more useful when they can search the web for news, or your codebase to know how it is organized
But their usefulness is only surface-deep. The news that matters to you is always deeply contextual, it's not only things labelled as breaking news or happening near you. Same thing happens with code organization. The reason is more human nature (how we think and learn) than machine optimization (the compiler usually don't care).
You have half a point. "Without any doubt" is merely the apex of a huge undefined iceberg.
I write half .. eating is multi modal and consequential. The llm can read the menu, but it didn't eat the meal. Even humans are bounded. Feeling, licking, smelling, or eating the menu still is not eating the meal.
There is an insuperable gap in the analogy ... a gap in the concept and of sensory data doing it.
Back to first point: what one knows through that sensory data ... is not clear at present or even possible with llms.
If it was modeling the world you’d expect “give me a picture of a glass filled to the brim” to actually do that. It’s inability to correctly and accurately combine concepts indicates it’s probably not building a model of the real world.
Like most quirks that spread widely, a bandaid is swiftly applied. This is also why they now know how many r's are in "strawberry." But we don't get any closer to useful general intelligence by cobbling together thousands of hasty patches.
No, humans are not a series of band-aid patches where we learn facts in isolation. A human can reason, and when exposed to novel situations figure out a path forward. You don't need to tell a human how many rs are in "strawberry"; as long as they know what the letter r is they can count it in any word you choose to give them. As proven time and time again, LLMs can't do this. The embarrassing failure of Claude to figure out how to play Pokemon a year or so ago is a good example. You could hand a five year old human a Gameboy with Pokemon in it, and he could figure out how to move around and do the basics. He wouldn't be very good, but he would figure it out as he goes. Claude couldn't figure out to stop going in and out of a building. LLMs, usefulness aside, have repeatedly shown themselves to have zero intelligence.
I was referring not to individual learning ability but to natural selection and evolutionary pressure, which IMO is easy to describe as a band-aid patch that takes a generation or more to apply.
You would be correct if these issues were fixed by structurally fixing the LLM. But instead it’s patched through RL/data set management. That’s a very different and more brittle process - the evolutionary approach fixes classes of issues while the RL approach fixes specific instances of issues.
Sure, and I'd be the first to admit I'm not aware of the intricate details wrt how LLMs are trained and refined, it's not my area. My original comment here was in disagreement of the relatively simple dismissal of the idea that the construction of humanity hasn't been an incremental zig-zag process and that I don't see any reason that a "real" intelligence couldn't follow the same path under our direction. I see a lot of philosophical conversation around this on HN disguised as endless deep discussions about the technicals, which amuses me because it feels like we're in the very early days there, and I think we can circle the drain defining intelligence until we all die.
> that to understand knowledge you have to have a model of the world.
You have a small but important mistake. It's to recite (or even apply) knowledge. To understand does actually require a world model.
Think of it this way: can you pass a test without understanding the test material? Certainly we all saw people we thought were idiots do well in class while we've also seen people we thought were geniuses fail. The test and understanding usually correlates but it's not perfect, right?
The reason I say understanding requires a world model (and I would not say LLMs understand) is because to understand you have to be able to detail things. Look at physics, or the far more detail oriented math. Physicists don't conclude things just off of experimental results. It's an important part, but not the whole story. They also write equations, ones which are counterfactual. You can call this compression if you want (I would and do), but it's only that because of the generalization. But it also only has that power because of the details and nuance.
With AI many of these people have been screaming for years (check my history) that what we're doing won't get us all the way there. Not because we want to stop the progress, but because we wanted to ensure continued and accelerate progress. We knew the limits and were saying "let's try to get ahead of this problem" but were told "that'll never be a problem. And if it is, we'll deal with it when we deal with it." It's why Chollet made the claim that LLMs have actually held AI progress back. Because the story that was sold was "AGI is solved, we just need to scale" (i.e. more money). I do still wonder how different things would be if those of us pushing back were able to continue and scale our works (research isn't free, so yes, people did stop us). We always had the math to show that scale wasn't enough, but it's easy to say "you don't need math" when you can see progress. The math never said no progress nor no acceleration, the math said there's a wall and it's easier to adjust now than when we're closer and moving faster. Sadly I don't think we'll ever shift the money over. We still evaluate success weirdly. Successful predictions don't matter. You're still heralded if you made a lot of money in VR and Bitcoin, right?
In my view 'understand' is a folk psychology term that does not have a technical meaning. Like 'intelligent', 'beautiful', and 'interesting'. It usefully labels a basket of behaviors we see in others, and that is all it does.
In this view, if a machine performs a task as well as a human, it understands it exactly as much as a human. There's no problem of how to do understanding, only how to do tasks. The 'problem' melts away when you take this stance.
Just my opinion, but my professional opinion from thirty-plus years in AI.
Let me understand, is your claim that a toaster can't toast bread because it cannot initiate the toasting through its own volition?
Ignoring the silly wording, that is a very different thing than what robotresearcher said. And actually, in a weird way I agree. Though I disagree that a toaster can't toast bread.
Let's take a step back. At what point is it me making the toast and not the toaster? Is it because I have to press the level? We can automate that. Is it because I have to put by bread in? We can automate that. Is it because I have to have the desire to have toast and initiate the chain of events? How do you measure that?
I'm certain that's different from measuring task success. And that's why I disagree with robotresearcher. The logic isn't self consistent.
> Though I disagree that a toaster can't toast bread.
If a toaster can toast bread, then an Allen key can assemble furniture. Both of them can do these tasks in collaboration with a human. This human supplies the executive decision-making (what when where etc), supplies the tool with compatible parts (bread or bolts) and supplies the motivating force (mains electricity or rotational torque).
The only difference is that it's more obviously ridiculous when it's an inanimate hunk of bent metal. Wait no, that could mean either of them. I mean the Allen key.
> Let's take a step back. At what point is it me making the toast and not the toaster?
I don't know exactly where that point is, but it's certainly not when the toaster is making zero decisions. It begins to be a valid question if you are positing a hypothetical "smart toaster" which has sensors and software capable of achieving toasting perfection regardless of bread or atmospheric variables.
> Is it because I have to press the level? We can automate that.
> I don't know exactly where that point is, but it's certainly not when the toaster is making zero decisions.
And this is the crux of my point. Our LLMs still need to be fed prompts.
Where the "decision making" happens gets fuzzy, but that's true in the toaster too.
Your run of the mill toaster is a heating element and a timer. Is the timer a rudimentary decision process?
A more modern toaster is going to include a thermocouple or thermister to ensure that the heating elements don't light things on fire. This requires a logic circuit. Is this a decision process? (It is entirely deterministic)
A more advanced one is going to incorporate a PID controller, just like your oven. It is deterministic in the sense that it will create the same outputs given the same inputs but it is working with non-deterministic inputs.
These PIDs can also look a lot like small neural networks, and in some cases they are implemented that way. These processes need not be deterministic. You can even approach this problem through RL style optimizations. There's a lot of solutions here.
When you break this down, I agree, it is hard to define that line, especially as we break it down. But that's part of what I'm after with robotresearcher. The claim was about task performance but then the answer with a toaster was that the human and toaster work together. I believe dullcrisp used the toaster as an example because it is a much simpler problem than playing a game of chess (or at least it appears that way).
So the question still stands, when does the toaster make the toast and when am I no longer doing so?
When is the measurement attributed to the toaster's ability to make toast vs mine?
Now replace toasting with chess, programming, music generation, or anything else that we have far less well defined metrics for. Sure, we don't have a perfect definition of what constitutes toast, but it is definitely far more bound than these other things. We have accuracy in the definition, and I'd argue even fairly good precision. There's high agreement on what we'd call toast, not toasted bread, and burnt bread. We can at least address the important part of this question without infinite precision in how to discriminate these classifications.
The question of an "ability to make toast" is a semantic question bounded by what you choose to encompass within "make toast". At best, a regular household toaster can "make heat"[1]. A regular household toaster certainly cannot load itself with bread, which I would consider unambiguously within the scope of the "make toast" task. If you disagree, then we have a semantic dispute.
This is also, at least in part, the Sorites Paradox.[0] There is obviously a gradient of ambiguity between human and toaster responsibility, but we can clearly tell extremes apart even when the boundary is indeterminate. When does a collection grains become a heap? When does a tool become responsible for the task? These are purely semantic questions. Strip away all normative loading and the argument disappears.
> Not sure where you imagine my inconsistency is.
>> Let's take a step back. At what point is it me making the toast and not the toaster? Is it because I have to press the level? We can automate that. Is it because I have to put by bread in? We can automate that. Is it because I have to have the desire to have toast and initiate the chain of events? How do you measure that?
You have a PhD and 30 years of experience, so I'm quite confident you are capable of adapting the topic of "making toast" to "playing chess", "doing physics", "programming", or any similar topic where we are benchmarking results.
Maybe I've (and others?) misunderstood your claim from the get-go? You seem to have implied that LLMs understand chess, physics, programming, etc because of their performance. Yet now it seems your claim is that the LLM and I are doing those things together. If your claim is that a LLM understands programming the same way a toaster understands how to make toast, then we probably aren't disagreeing.
But if your claim is that a LLM understands programming because it can produce programs that yield a correct output to test cases, then what's the difference from the toaster? I put the prompts in and pushed the button to make it toast.
I'm not sure why you imagine the inconsistency is so difficult to see.
When did I say that the chess program was different to a toaster? I don’t believe it is, so it’s not a thing I’m likely to say.
I don’t think the word ‘understand’ has a meaning that can apply in these situations. I’m not saying the toaster or the chess program understands anything, except in the limited sense that some people might describe them that way, and some won’t. In both cases that concept is entirely in the head of the describer and not in the operation of the device.
I think the claimed inconsistency is in views you ascribe to me, and not those I hold. ‘Understand’ is a category error with respect to these devices. They neither do or don’t. Understanding is something an observer attributes for their own reasons and entails nothing for the subject.
I concur that ascribing understanding to the machines that we have is a category error.
The reason I believe it was brought up is that understanding is not a category error when ascribed to people.
And if we claim to have a plan to create machines that are indistinguishable from people, we likely first need to understand what it is that makes people distinguishable from machines, and that doesn’t seem to be on any of the current AI companies’ roadmap.
Declaring something as having "responsibility" implies some delegation of control. A normal toaster makes zero decisions, and as such it has no control over anything.
A toaster has feedback control over its temperature, time control over its cooking duration, and start/stop control by attending to its start/cancel buttons. It makes decisions constantly.
I simply can't make toast without a toaster, however psychologically primary you want me to be. Without either of us, there's no new toast. Team effort every time.
And to make it even more interesting, the same is true for my mum and her toaster. She does not understand how her toaster works. And yet: toast reliably appears! Where is the essential toast understanding in that system? Nowhere and everywhere! It simply isn't relevant.
> A toaster has feedback control over its temperature, time control over its cooking duration
Most toasters are heating elements attached to a timer adjusted by the human operator. It doesn’t have any feedback control. It doesn’t have any time control.
> I simply can't make toast without a toaster
I can’t make toast without bread either, but that doesn’t make the bread “responsible” for toasting itself.
> She does not understand how her toaster works.
My mum doesn’t understand how bread is made, but she can still have the intent to acquire it from a store and expose it to heat for a nominal period of time.
What is your definition of "responsible"? The human is making literally all decisions and isn't abdicating responsibility for anything. The average toaster has literally one operational variable (cook time) and even that minuscule proto-responsibility is entirely on the human operator. All other aspects of the toaster's operation are decisions made by the toaster's human designer/engineer.
> if a machine performs a task as well as a human, it understands it exactly as much as a human.
I think you're right, except that the ones judging "as well as a human" are in fact humans, and humans have expectations that expand beyond the specs. From the narrow perspective of engineering specifications or profit generated, a robot/AI may very well be exactly as understanding as a human. For the people which interact with those systems outside the money/specs/speeds & feeds, the AI/robot will always feel at least different compared to a person. And as long as it's different, there will always be room to un-falsifiably claim "this robot is worse in my opinion due to X/Y/Z difference."
Right. A flying machine doesn’t need to understand anything to fly. It’s not even clear what it would mean for it to do so, or how it would fly any differently if it did.
Same with the AI machines.
Understanding is not something that any machine or person does. Understanding is a compact label applied to people’s behavior by an observer that allows the observer to predict future behavior. It’s not a process in itself.
And yes, we apply this label to ourselves. Much of what we do is only available to consciousness post-hoc, and is available to be described just the same as the behavior of someone else.
> Understanding is not something that any machine or person does.
Yet I can write down many equations necessary to build and design that plane.
I can model the wind and air flow across the surface and design airfoils.
I can interpret the mathematical symbols into real physical meaning.
I can adapt these equations to novel settings or even fictitious ones.
I can analyze them counterfactually; not just making predictions but also telling you why those predictions are accurate, what their inaccuracies are (such as which variables and measurements are more precise), and I can tell you what all those things mean.
I can describe and derive the limits of the equations and models, discussing where they do and don't work. Including in the fictional settings.
I can do this at an emergent macroscopic level and I can do it at a fine grain molecular or even atomic level. I can even derive the emergent macroscopic behavior from the more fine grain analysis and tell you the limits of each model.
I can also respond that Bernoulli's equation is not an accurate description of why an airfoil works, even when prompted with those words[0].
These are characteristics that lead people to believe I understand the physics of fluid mechanics and flight. They correlate strongly with the ability to recall information from textbooks, but the actions aren't strictly the ability to recall and search over a memory database. Do these things prove that I understand? No, but we deal with what we got even if it is imperfect.
It is not just the ability to perform a task, it includes the ability to explain it. The more depth I am able to the greater understanding people attribute. While this correlates with task performance it is not the same. Even Ramanujan had to work hard to understand even if he was somehow able to divine great equations without it.
You're right that these descriptions are not the thing itself either. No one is claiming the map is the territory here. That's not the argument being made. Understanding the map is a very different thing than conflating the map and the territory. It is also a different thing than just being able to read it.
> In this view, if a machine performs a task as well as a human, it understands it exactly as much as a human. There's no problem of how to do understanding, only how to do tasks.
Yes, but you also gloss over what a "task" is or what a "benchmark" is (which has to do with the meaning of generalization).
Suppose an AI or human answers 7 questions correctly out of 10 on an ICPC problem set, what are we able infer from that?
1. Is the task equal to answering these 10 questions well, with a uniform measure of importance?
2. Is the task be good at competitive programming problems?
3. Is the task be good at coding?
4. Is the task be good at problem solving?
5. Is the task not just to be effective under a uniform measure of importance, but an adversarial measure? (i.e. you can probably figure out all kinds of competitive programming questions, if you had more time / etc... but roughly not needing "exponentially more resources")
These are very different levels of abstraction, and literally the same benchmark result can be interpreted to mean very different things. And that imputation of generality is not objective unless we know the mechanism by which it happens. "Understanding" is short-hand for saying that performance generalizes at one of the higher levels of abstraction (3--5), rather than narrow success -- because that is what we expect of a human.
How do you quantify generality? If we have a benchmark that can quantify it and that benchmark reliably tells us that the LLM is within human levels of generalisation then the llm is not distinguishable from a human.
While it’s a good point that we need to benchmark generalisation ability, you have in fact agreed that it is not important to understand underlying mechanics.
The difference though is they understand that you can't just benchmark your way into proofs. Just like you can't unit test your way into showing code is error free. Benchmarks and unit tests are great tools that provide a lot of help, but just because a hammer is useful doesn't make everything a nail.
A QC operator may be able to carry out a test with as much accuracy (or perhaps better accuracy, with enough practice) than the PhD quality chemist who developed it. They could plausibly do so with a high school education and not be able to explain the test in any detail. They do not understand the test in the same way as the chemist.
If 'understand' is a meaningless term to someone who's spent 30 years in AI research, I understand why LLMs are being sold and hyped in the way they are.
Not to be flippant but have you considered that that question is an entire branch of philosophy with a several-millennias long history which people in some cases spend their entire life studying?
I have. It robustly has the folk-psychological meaning I mentioned in my first sentence. Call it ‘philosophical’ instead of ‘folk-psychological’ if you like. It’s a useful concept. But the concept doesn’t require AI engineers to do anything. It certainly doesn’t give any hints about AI engineers what they should actually do.
I would say it understands if given many variations of a problem statement, it always gives correct answer without fail. I have this complicated mirror question that only Deepseek and qwen3-max got right every time, still they only answered it correctly about a dozen times, so we're left with high probability, I guess.
I disagree with robotresearcher but I think this is also an absurd definition. By that definition there is no human, nor creature, that understands anything. Not just by nature of humans making mistakes, including experts, but I'd say this is even impossible. You need infinite precision and infinite variation here.
It turns "understanding" into a binary condition. Robotresearcher's does too, but I'm sure they would refine by saying that the level of understanding is directly proportional to task performance. But I still don't know how they'll address the issue of coverage, as ensuring tests have complete coverage is far from trivial (even harder when you want to differentiate from the training set, differentiating memorization).
I think you're right in trying to differentiate memorization from generalization, but your way to measure this is not robust enough. A fundamental characteristic of where I disagree from them is that memorization is not the same as understanding.
I have been thinking about this for years, probably two decades. The answer to your question or the definition, I am sure you know, is rather difficult. I don't think it is impossible, but there's a risk of diving into a deep dark pit of philosophical thought going back to at least the ancient Greeks.
And, if we did go through that exercise, I doubt we can come out of it with a canonical definition of understanding.
I was really excited about LLM's as they surfaced and developed. I fully embraced the technology and have been using it extensively with full top-tier subscriptions to most services. My conclusion so far: If you want to destroy your business, adopt LLM's with gusto.
I know that's a statement that goes way against the train ride we are on this very moment. That's not to say LLM's are not useful. They are. Very much so. The problem is...well...they don't understand. And here I am, back in a circular argument.
I can define understanding with the "I know it when I see it" meme. And, frankly, it does apply. Yet, that's not a definition. We've all experienced that stare when talking to someone who does not have sufficient depth of understanding in a topic. Some of us have experienced people running teams who should not be in that position because they don't have a clue, they don't understand enough of it to be effective at what they do.
And yet, I still have not defined "understanding".
Well, it's hard. And I am not a philosopher, I am an engineer working in robotics, AI and applications to real time video processing.
I have written about my experiments using LLM coding tools (I refuse to call them AI, they are NOT intelligent; yes, need to define that as well).
In that context, lack of understanding is clearly evident when an LLM utterly destroys your codebase by adding dozens of irrelevant and unnecessary tests, randomly changes variable names as you navigate the development workflow, adds modules like a drunken high school coder and takes you down tangents that would make for great comedy if I were a tech comedian.
LLMs do not understand. They are fancy --and quite useful-- auto-complete engines and that's about it. Other than that, buyer beware.
The experiments I ran, some of them spanning three months of LLM-collaborative coding at various levels --from very hands-on to "let Jesus drive the car"-- conclusively demonstrated (at least to me) that:
1- No company should allow anyone to use LLMs unless they have enough domain expertise to be able to fully evaluate the output. And you should require that they fully evaluate and verify the work product before using it for anything; email, code, marketing, etc.
2- No company should trust anything coming out of an LLM, not one bit. Because, well, they don't understand. I recently tried to use the United Airlines LLM agent to change a flight. It was a combination of tragic and hilarious. Now, I know what's going on. I cannot possibly imagine the wild rides this thing is taking non-techies on every day. It's shit. It does not understand. It' isn't isolated to United Airlines, it's everywhere LLMs are being used. The potential for great damage is always there.
3- They can be great for summarization tasks. For example, you have have them help you dive deep into 300 page AMD/Xilinx FPGA datasheet or application note and help you get mentally situated. They can be great at helping you find prior art for patents. Yet, still, because they are mindless parrots, you should not trust any of it.
4- Nobody should give LLMs great access to a non-trivial codebase. This is almost guaranteed to cause destruction and hidden future effects. In my experiments I have experienced an LLM breaking unrelated code that worked just fine --in some cases fully erasing the code without telling you. Ten commits later you discover that your network stack doesn't work or isn't even there. Or, you might discover that the stack is there but the LLM changed class, variable or method names, maybe even data structures. It's a mindless parrot.
I could go on.
One response to this could be "Well, idiot, you need better prompts!". That, of course, assumes that part of my experimentation did not include testing prompts of varying complexity and length. I found that for some tasks, you get better results by explaining what you want and then asking the LLM to write a prompt to get that result. You check that prompt, modify if necessary and, from my experience, you are likely to get better results.
Of course, the reply to "you need better prompts" is easy: If the LLM understood, prompt quality would not be a problem at all and pages-long prompts would not be necessary. I should not have to specify that existing class, variable and method names should not be modified. Or that interfaces should be protected. Or that data structures need not be modified without reason and unless approved by me. Etc.
It reminds me of a project I was given when I was a young engineer barely out of university. My boss, the VP of Engineering where I worked, needed me to design a custom device. Think of it as a specialized high speed data router with multiple sources, destinations and a software layer to control it all. I had to design the electronics, circuit boards, mechanical and write all the software. The project had a budget of nearly a million dollars.
He brought me into his office and handed me a single sheet of paper with a top-level functional diagram. Inputs, outputs, interfaces. We had a half hour discussion about objectives and required timeline. He asked me if I could get it done. I said yet.
He checked in with me every three months or so. I never needed anything more than that single piece of paper and the short initial conversation because I understood what we needed, what he wanted, how that related to our other systems, available technology, my own capabilities and failings, available tools, etc. It took me a year to deliver. It worked out of the box.
You cannot do that with LLMs because they don't understand anything at all. They mimic what some might confuse for understanding, but they do not.
And, yet, once again, I have not defined the term. I think everyone reading this who has used LLMs to a non-trivial depth...well...understands what I mean.
> We've all experienced that stare when talking to someone who does not have sufficient depth of understanding in a topic.
I think you're really putting your finger on something here. LLMs have blown us away because they can interact with language in a very similar way to humans, and in fact it approximates how humans operate in many contexts when they lack a depth of understanding. Computers never could do this before, so it's impressive and novel. But despite how impressive it is, humans who were operating this way were never actually generating significant value. We may have pretended they were for social reasons, and there may even have been some real value associated with the human camaraderie and connections they were a part of, but certainly it is not of value when automated.
Prior to LLMs just being able to read and write code at a pretty basic level was deemed an employable skill, but because it was not a natural skill for lots of human, it was also a market for lemons and just the basic coding was overvalued by those who did not actually understand it. But of course the real value of coding has always been to create systems that serve human outcomes, and the outcomes that are desired are always driven by human concerns that are probably inscrutable to something without the same wetware as us. Hell, it's hard enough for humans to understand each other half the time, but even when we don't fully understand each other, the information conferred through non-verbal cue, and familiarity with the personalities and connotations that we only learn through extended interaction has a robust baseline which text alone can never capture.
When I think about strategic technology decisions I've been involved with in large tech companies, things are often shaped by high level choices that come from 5 or 6 different teams, each of which can not be effectively distilled without deep domain expertise, and which ultimately can only be translated to a working system by expert engineers and analysts who are able to communicate in an extremely high bandwidth fashion relying on mutual trust and applying a robust theory of the mind every step along the way. Such collaborators can not only understand distilled expert statements of which they don't have direct detailed knowledge, but also, they can make such distilled expert statements and confirm sufficient understanding from a cross-domain peer.
I still think there's a ton of utility to be squeezed out of LLMs as we learn how to harness and feed them context most effectively, and they are likely to revolutionize the way programming is done day-to-day, but I don't believe we are anywhere near AGI or anything else that will replace the value of what a solid senior engineer brings to the table.
I am not liking the term "AGI". I think intelligence and understanding are very different things and they are both required to build a useful tool that we can trust.
To use an image that might be familiar to lots of people reading this, the Sheldon character in Big Bang Theory is very intelligent about lots of fields of study and yet lacks tons of understanding about many things, particularly social interaction, the human impact of decisions, etc. Intelligence alone (AGI) isn't the solution we should be after. Nice buzz word, but not the solution we need. This should not be the objective at the top of the hill.
I've always distinguished knowledge, intelligence, and wisdom. Knowledge is knowing a chair is a seat. Intelligence is being able to use a log as a chair. Wisdom is knowing the log chair will be more comfortable if I turn it around and that sometimes it's more comfortable to sit on the ground and use the log as fuel for the fire.
But I'm not going to say I was the first to distinguish those word. That'd be silly. They're 3 different words and we use them differently. We all know Sheldon is smart but he isn't very wise.
As for AGI, I'm not so sure my issue is with the label but more with the insistence that it is so easy and straight forward to understand. It isn't very wise to think the answer is trivial to a question which people have pondered for millennia. That just seems egotistical. Especially when thinking your answer is so obviously correct that you needn't bother trying to see if they were wrong. Even though Don Quixote didn't test his armor a second time, he had the foresight to test it once.
> If 'understand' is a meaningless term to someone who's spent 30 years in AI research, I understand why LLMs are being sold and hyped in the way they are.
I don't have quite as much time as robotresearcher, but I've heard their sentiment frequently.
I've been to conferences, talked with people at the top of the field (I'm "junior", but published and have a PhD) where when asking deeper questions I'll get a frequent response "I just care if it works." As if that also wasn't the motivation for my questions too.
But I'll also tell you that there are plenty of us who don't ascribe to those beliefs. There's a wide breadth of opinions, even if one set is large and loud. (We are getting louder though) I do think we can get to AGI and I do think we can figure out what words like "understand" truly mean (with both accuracy and precision, the latter being what's more lacking). But it is also hard to navigate because we're discouraged from this work and little funding flows our way (I hope as we get louder we'll be able to explore more, but I fear we may switch from one railroad to the next). The weirdest part to me has been that it seems that even in the research space, talking to peers, that discussing flaws or limits is treated as dismissal. I thought our whole job was to find the limits, explore them, and find ways to resolve them.
The way I see it now is that the field uses the duck test. If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck. The problem is people are replacing "probably" with "is". The duck test is great, and right now we don't have anything much better. But the part that is insane is to call it perfect. Certainly as someone who isn't an ornithologist, I'm not going to be able to tell a sophisticated artificial duck from a real one. But it's ability to fool me doesn't make it real. And that's exactly why it would be foolish to s/probably/is.
So while I think you're understanding correctly, I just want to caution throwing the baby out with the bathwater. The majority of us dissenting from the hype train and "scale is all you need" don't believe humans are magic and operating outside the laws of physics. Unless this is a false assumption, artificial life is certainly possible. The question is just about when and how. I think we still have a ways to go. I think we should be exploring a wide breadth of ideas. I just don't think we should put all our eggs in one basket, especially if there's clear holes in it.
[Side note]: An interesting relationship I've noticed is that the hype train people tend to have a full CS pedigree while dissenters have mixed (and typically start in something like math or physics and make their way to CS). It's a weak correlation, but I've found it interesting.
As a mathematician who also regularly publishes in these conferences, I am a little surprised to hear your take; your experience might be slightly different to mine.
Identifying limitations of LLMs in the context of "it's not AGI yet because X" is huge right now; it gets massive funding, taking away from other things like SciML and uncertainty analyses. I will agree that deep learning theory in the sense of foundational mathematical theory to develop internal understanding (with limited appeal to numerics) is in the roughest state it has even been in. My first impression there is that the toolbox has essentially run dry and we need something more to advance the field. My second impression is that empirical researchers in LLMs are mostly junior and significantly less critical of their own work and the work of others, but I digress.
I also disagree that we are disincentivised to find meaning behind the word "understanding" in the context of neural networks: if understanding is to build an internal world model, then quite a bit of work is going into that. Empirically, it would appear that they do, almost by necessity.
Maybe given our different niches we interact with different people? But I'm uncertain because I believe what I'm saying is highly visible. I forgot, which NeurIPS(?) conference were so many wearing "Scale is all you need" shirts?
> My first impression there is that the toolbox has essentially run dry and we need something more to advance the field
This is my impression too. Empirical evidence is a great tool and useful, especially when there is no strong theory to provide direction, but it is limited.
> My second impression is that empirical researchers in LLMs are mostly junior and significantly less critical of their own work and the work of others
But this is not my impression. I see this from many prominent researchers. Maybe they claim SIAYN in jest, but then they should come out and say it is such instead of doubling down. If we take them at their word (and I do), robotresearcher is not a junior (please, read their comments. It is illustrative of my experience. I'm just arguing back far more than I would in person). I've also seen members of audiences to talks where people ask questions like mine ("are benchmarks sufficient to make such claims?") with responses of "we just care that it works." Again, I think this is a non-answer to the question. But being taken as a sufficient answer, especially in response to peers, is unacceptable. It almost always has no follow-up.
I also do not believe these people are less critical. I've had several works which struggled through publication as my models that were a hundredth the size (and a millionth the data) could perform on par, or even better. At face value asks of "more datasets" and "more scale" are reasonable, yet it is a self reinforcing paradigm where it slows progress. It's like a corn farmer smugly asking why the neighboring soy bean farmer doesn't grow anything when the corn farmer is chopping all the soy bean stems in their infancy. It is a fine ask to big labs with big money, but it is just gate keeping and lazy evaluation to anyone else. Even at CVPR this last year they passed out "GPU Rich" and "GPU Poor" hats, so I thought the situation was well known.
> if understanding is to build an internal world model, then quite a bit of work is going into that. Empirically, it would appear that they do, almost by necessity.
I agree a "lot of work is going into it" but I also think the approaches are narrow and still benchmark chasing. I saw as well was given the aforementioned responses at workshops on world modeling (as well as a few presenters who gave very different and more complex answers or "it's the best we got right now", but nether seemed to confident in claiming "world model" either).
But I'm a bit surprised that as a mathematician you think these systems create world models. While I see some generalization, this is also impossible for me to distinguish from memorization. We're processing more data than can be scrutinized. We seem to also frequently uncover major limitations to our de-duplication processes[0]. We are definitely abusing the terms "Out of Distribution" and "Zero shot". Like I don't know how any person working with a proprietary LLM (or large model) that they don't own, can make a claim of "zero shot" or even "few shot" capabilities. We're publishing papers left and right, yet it's absurd to claim {zero,few}-shot when we don't have access to the learning distribution. We've merged these terms with biased sampling. Was the data not in training or is it just a low likelihood region of the model? They're indistinguishable without access to the original distribution.
Idk, I think our scaling is just making the problem harder to evaluate. I don't want to stop that camp because they are clearly producing things of value, but I do also want that camp to not make claims beyond their evidence. It just makes the discussion more convoluted. I mean the argument would be different if we were discussing small and closed worlds, but we're not. The claims are we've created world models yet many of them are not self-consistent. Certainly that is a requirement. I admit we're making progress, but the claims were made years ago. Take GameNGen[1] or Diamond Diffusion. Neither were the first and neither were self-consistent. Though both are also impressive.
Apologies if I ramble a bit here, this was typed in a bit of a hurry. Hopefully I answer some of your points.
First, regarding robotresearcher and simondota's comments, I am largely in agreement with what they say here. The "toaster" argument is a variant of the Chinese Room argument, and there is a standard rebuttal here. The toaster does not act independently of the human so it is not a closed system. The system as a whole, which includes the human, does understand toast. To me, this is different from the other examples you mention because the machine was not given a list of explicit instructions. (I'm no philosopher though so others can do a better job of explaining this). I don't feel that this is an argument for why LLMs "understand", but rather why the concept of "understanding" is irrelevant without an appropriate definition and context. Since we can't even agree on what constitutes understanding, it isn't productive to frame things in those terms. I guess that's where my maths background comes in, as I dislike the ambiguity of it all.
My "mostly junior" comment is partially in jest, but mostly comes from the fact that LLM and diffusion model research is a popular stream for moving into big tech. There are plenty of senior people in these fields too, but many reviewers in those fields are junior.
> I've also seen members of audiences to talks where people ask questions like mine ("are benchmarks sufficient to make such claims?") with responses of "we just care that it works."
This is a tremendous pain point to me more than I can convey here, but it's not unusual in computer science. Bad researchers will live and die on standard benchmarks. By the way, if you try to focus on another metric under the argument that the benchmarks are not wholly representative of a particular task, expect to get roasted by reviewers. Everyone knows it is easier to just do benchmark chasing.
> I also do not believe these people are less critical.
I think the fact that the "we just care that it works" argument is enough to get published is a good demonstration of what I'm talking about. If "more datasets" and "more scale" are the major types of criticisms that you are getting, then you are still working in a more fortunate field. And yes, I hate it as much as you do as it does favor the GPU rich, but they are at least potentially solvable. The easiest papers of mine to get through were methodological and often got these kinds of comments. Theory and SciML papers are an entirely different beast in my experience because you will rarely get reviewers that understand the material or care about its relevance. People in LLM research thought that the average NeurIPS score in the last round was a 5. Those in theory thought it was 4. These proportions feel reflected in the recent conferences. I have to really go looking for something outside the LLM mainstream, while there was a huge variety of work only a few years ago. Some of my colleagues have noticed this as well and have switched out of scientific work. This isn't unnatural or something to actively try to fix, as ML goes through these hype phases (in the 2000s, it was all kernels as I understand).
> approaches are narrow and still benchmark chasing
> as a mathematician you think these systems create world models
When I say "world model", I'm not talking about outputs or what you can get through pure inference. Training models to perform next frame prediction and looking at inconsistencies in the output tells us little about the internal mechanism. I'm talking about appropriate representations in a multimodal model. When it reads a given frame, is it pulling apart features in a way that a human would? We've known for a long time that embeddings appropriately encode relationships between words and phrases. This is a model of the world as expressed through language. The same thing happens for images at scale as can be seen in interpretable ViT models. We know from the theory that for next frame prediction, better data and more scaling improves performance. I agree that isn't very interesting though.
> We are definitely abusing the terms "Out of Distribution" and "Zero shot".
Absolutely in agreement with everything you have said. These are not concepts that should be talked about in the context of "understanding", especially at scale.
> I think our scaling is just making the problem harder to evaluate.
Yes and no. It's clear that whatever approach we will use to gauge internal understanding needs to work at scale. Some methods only work with sufficient scale. But we know that completely black-box approaches don't work, because if they did, we could use them on humans and other animals.
> The claims are we've created world models yet many of them are not self-consistent.
For this definition of world model, I see this the same way as how we used to have "language models" with poor memory. I conjecture this is more an issue of alignment than a lack of appropriate representations of internal features, but I could be totally wrong on this.
> The toaster does not act independently of the human so it is not a closed system
I think you're mistaken. No, not at that, at the premise. I think everyone agrees here. Where you're mistaken is that when I login to Claude it says "How can I help you today?"
No one is thinking that the toaster understands things. We're using it to point out how silly the claim of "task performance == understanding" is. Techblueberry furthered this by asking if the toaster is suddenly intelligent by wrapping it with a cron job. My point was about where the line is drawn. The turning on the toaster? No, that would be silly and you clearly agree. So you have to answer why the toaster isn't understanding toast. That's the ask. Because clearly toaster toasts bread.
You and robotresearcher have still avoided answering this question. It seems dumb but that is the crux of the problem. The LLM is claimed to be understanding, right? It meets your claims of task performance. But they are still tools. They cannot act independently. I still have to prompt them. At an abstract level this is no different than the toaster. So, at what point does the toaster understand how to toast? You claim it doesn't, and I agree. You claim it doesn't because a human has to interact with it. I'm just saying that looping agents onto themselves doesn't magically make them intelligent. Just like how I can automate the whole process from planting the wheat to toasting the toast.
You're a mathematician. All I'm asking is that you abstract this out a bit and follow the logic. Clearly even our automated seed to buttered toast on a plate machine needs not have understanding.
From my physics (and engineering) background there's a key thing I've learned: all measurements are proxies. This is no different. We don't have to worry about this detail in most every day things because we're typically pretty good at measuring. But if you ever need to do something with precision, it becomes abundantly obvious. But you even use this same methodology in math all the time. Though I wouldn't say that this is equivalent to taking a hard problem, creating an isomorphic map to an easier problem, solving it, then mapping back. There's an invective nature. A ruler doesn't measure distance. A ruler is a reference to distance. A laser range finder doesn't measure distance either, it is photodetector and a timer. There is nothing in the world that you can measure directly. If we cannot do this with physical things it seems pretty silly to think we can do it with abstract concepts that we can't create robust definitions for. It's not like we've directly measured the Higgs either. But what, do you think entropy is actually a measurement of intelligible speech? Perplexity is a good tool for identifying an entropy minimizer? Or does it just correlate? Is a FID a measurement of fidelity or are we just using a useful proxy? I'm sorry, but I just don't think there are precise mathematical descriptions of things like natural English language or realistic human faces. I've developed some of the best vision models out there and I can tell you that you have to read more than the paper because while they will produce fantastic images they also produce some pretty horrendous ones. The fact that they statistically generate realistic images does not imply that they actually understand them.
> I'm no philosopher
Why not? It sounds like you are. Do you not think about metamathematics? What math means? Do you not think about math beyond the computation? If you do, I'd call you a philosopher. There's a P in a PhD for a reason. We're not supposed to be automata. We're not supposed to be machine men, with machine minds, and machine hearts.
> This is a tremendous pain point ... researchers will live and die on standard benchmarks.
It is a pain we share. I see it outside CS as well, but I was shocked to see the difference. Most of the other physicists and mathematicians I know that came over to CS were also surprised. And it isn't like physicists are known for their lack of egos lol
> then you are still working in a more fortunate field
Oh, I've gotten the other comments too. That research never found publication and at the end of the day I had to graduate. Though now it can be revisited. I once was surprise to find that I saved a paper from Max Welling's group. My fellow reviewers were confident in their rejections just since they admitted to not understanding differential equations the AC sided with me (maybe they could see Welling's name? I didn't know till months after). It barely got through a workshop, but should have been in the main proceedings.
So I guess I'm saying I share this frustration. It's part of the reason I talk strongly here. I understand why people shift gears. But I think there's a big difference between begrudgingly getting on the train because you need to publish to survive and actively fueling it and shouting that all outer trains are broken and can never be fixed. One train to rule them all? I guess CS people love their binaries.
> world model
I agree that looking at outputs tells us little about their internal mechanisms. But proof isn't symmetric in difficulty either. A world model has to be consistent. I like vision because it gives us more clues in our evaluations, let's us evaluate beyond metrics. But if we are seeing video from a POV perspective, then if we see a wall in front of us, turn left, then turn back we should still expect to see that wall, and the same one. A world model is a model beyond what is seen from the camera's view. A world model is a physics model. And I mean /a/ physics model, not "physics". There is no single physics model. Nor do I mean that a world model needs to have even accurate physics. But it does need to make consistent and counterfactual predictions. Even the geocentric model is a world model (literally a model of worlds lol). The model of the world you have in your head is this. We don't close our eyes and conclude the wall in front of you will disappear. Someone may spin you around and you still won't do this, even if you have your coordinates wrong. The issue isn't so much memory as it is understanding that walls don't just appear and disappear. It is also understanding that this also isn't always true about a cat.
I referenced the game engines because while they are impressive they are not self consistent. Walls will disappear. An enemy shooting at you will disappear sometimes if you just stop looking at it. The world doesn't disappear when I close my eyes. A tree falling in a forest still creates acoustic vibrations in the air even if there is no one to hear it.
A world model is exactly that, a model of a world. It is a superset of a model of a camera view. It is a model of the things in the world and how they interact together, regardless of if they are visible or not. Accuracy isn't actually the defining feature here, though it is a strong hint, at least it is for poor world models.
I know this last part is a bit more rambly and harder to convey. But I hope the intention came across.
> You and robotresearcher have still avoided answering this question.
I have repeatedly explicitly denied the meaningfulness of the question. Understanding is a property ascribed by an observer, not possessed by a system.
You may not agree, but you can’t maintain that I’m avoiding that question. It does not have an answer that matters; that is my specific claim.
You can say a toaster understands toasting or you can not. There is literally nothing at stake there.
You said the LLMs are intelligent because they do tasks. But the claim is inconsistent with the toaster example.
If a toaster isn't intelligent because I have to give it bread and press the button to start then how's that any different from giving an LLM a prompt and pressing the button to start?
It's never been about the toaster. You're avoiding answering the question. I don't believe you're dumb, so don't act the part. I'm not buying it.
Having said that, can you name one functional difference between an AI that understands, and one that merely behaves correctly in its domain of expertise?
As an example, how would a chess program that understands chess differ from one that is merely better at it than any human who ever lived?
(Chess the formal game; not chess the cultural phenomenon)
Some people don’t find the example satisfying, because they feel like chess is not the kind of thing where understanding pertains.
Is this falsifiable? Even restricting to those currently living? On what tests? In which way? Does the category of error matter?
> can you name one functional difference between an AI that understands, and one that merely behaves correctly in its domain of expertise?
I'd argue you didn't understand the examples from my previous comment or the direct reply[0]. Does it become a duck as soon as you are able to trick an ornithologist? All ornithologists?
But yes. Is it fair if I use Go instead of Chess? Game 4 with Lee Sedol seems an appropriate example.
Vafa also has some good examples[1,2].
But let's take an even more theoretical approach. Chess is technically a solved game since it is non-probabilistic. You can compute an optimal winning strategy from any valid state. Problem is it is intractable since the number of action state pairs is so large. But the number of moves isn't the critical part here, so let's look at Tic-Tac-Toe. We can pretty easily program up a machine that will not lose. We can put all actions and states into a graph and fit that on a computer no problem. Do you really say that the program better understands Tic-Tac-Toe than a human? I'm not sure we should even say it understands the game at all.
I don't think the situation is resolved by changing to unsolved (or effectively unsolved) games. That's the point of the Heliocentric/Geocentric example. The Geocentric Model gave many accurate predictions, but I would find it surprising if you suggested an astronomer at that time, with deep expertise in the subject, understood the configuration of the solar system better than a modern child who understands Heliocentricism. Their model makes accurate predictions and certainly more accurate than that child would, but their model is wrong. It took quite a long time for Heliocentrism to not just be proven to be correct, but to also make better predictions than Geocentrism in all situations.
So I see 2 critical problems here.
1) The more accurate model[3] can be less developed, resulting in lower predictive capabilities despite being a much more accurate representation of the verifiable environment. Accuracy and precision are different, right?
2) Test performance says nothing about coverage/generalization[4]. We can't prove our code is error free through test cases. We use them to bound our confidence (a very useful feature! I'm not against tests, but as you say, caution is good).
In [0] I referenced Dyson, I'd appreciate it if you watched that short video (again if it's been some time). How do you know you aren't making the same mistake Dyson almost did? The mistake he would have made had he not trusted Fermi? Remember, Fermi's predictions were accurate and they even stood for years.
If your answer is time, then I'm not convinced it is a sufficient explanation. It doesn't explain Fermi's "intuition" (understanding) and is just kicking the can down the road. You wouldn't be able to differentiate yourself from Dyson's mistake. So why not take caution?
And to be clear, you are the one making the stronger claim: "understanding has a well defined definition." My claim is that yours is insufficient. I'm not claiming I have an accurate and precise definition, my claim is that we need more work to get the precision. I believe your claim can be a useful abstraction (and certainly has been!), but that there are more than enough problems that we shouldn't hold to it so tightly. To use it as "proof" is naive. It is equivalent to claiming your code is error free because it passes all test cases.
[3] Certainly placing the Earth at the center of the solar system (or universe!) is a larger error than placing the sun at the center of the solar system and failing to predict the tides or retrograde motion of Mercury.
[4] This gets exceedingly complex as we start to differentiate from memorization. I'm not sure we need to dive into what the distance from some training data needs be to make it a reasonable piece of test data, but that is a question that can't be ignored forever.
>> any human who ever lived
> Is this falsifiable? Even restricting to those currently living? On what tests? In which way? Does the category of error matter?
Software reliably beats the best players that have ever played it in public, including Kasparov and Carlsen, the best players of my lifetime (to my limited knowledge). By analogy to the performance ratchet we see in the rest of sports and games, and we might reasonably assume that these dominant living players are the best the world has ever seen. That could be wrong. But my argument does not hang on this point, so asking about falsifiability here doesn't do any work. Of course it's not falsifiable.
Y'know what else is not falsifiable? "That AI doesn't understand what it's doing".
> can you name one functional difference between an AI that understands, and one that merely behaves correctly in its domain of expertise?
> I'd argue you didn't understand the examples from my previous comment or the direct reply[0]. Does it become a duck as soon as you are able to trick an ornithologist? All ornithologists?
No one seems to have changed their opinion about anything in the wake of AIs routinely passing the Turing Test. They are fooled by the chatbot passing as a human, and then ask about ducks instead. The most celebrated and seriously considered quacks like a duck argument has been won by the AIs and no-one cares.
By the way, the ornithologists' criteria for duck is probably genetic and not much to do with behavior. A dead duck is still a duck.
And because we know what a duck is, no-one is yelling at ducks that 'they don't really duck' and telling duck makers they need a revolution in duck making and they are doomed to failure if they don't listen.
> Y'know what else is not falsifiable? "That AI doesn't understand what it's doing".
Which is why people are saying we need to put in more work to define this term. Which is the whole point of this conversation.
> seriously considered quacks like a duck argument has been won by the AIs and no-one cares.
And have you ever considered that it's because people are refining their definitions?
Often when people find that their initial beliefs are wrong or not precise enough then they update their beliefs. You seem to be calling this a flaw. It's not like the definitions are dramatically changing, they're refining. There's a big difference
My first post here is me explaining that I have a non-standard definition of what ‘understanding’ means, which helps me avoid an apparently thorny issue. I’m literally here offering a refinement of a definition.
People are disagreeing with your refinement. The toaster example is exactly this.
Maybe what was interpreted is different than what you meant to convey, but certainly my interpretation was not unique. I'm willing to update my responses if you are willing to clarify but we'll need to work together on that. Because unfortunately just because the words make perfect sense to you doesn't mean they do to others.
I'll even argue that this is some of the importance of understanding. Or at least what we call understanding.
so your definition of "understand" is "able to develop the QC test (or explain tests already developed)"
I hate to break it to you, but the LLMs can already do all 3 tasks you outlined
It can be argued for all 3 actors in this example (the QC operator, the PhD chemist and the LLM) that they don't really "understand" anything and are iterating on pre-learned patterns in order to complete the tasks.
Even the ground-breaking chemist researcher developing a new test can be reduced to iterating on the memorized fundamentals of chemistry using a lot of compute (of the meat kind).
The mythical Understanding is just a form of "no true Scotsman"
I don't think the definition is very refined, but I think we should be careful to differentiate that from useless or meaningless. I would say most definitions are accurate, but not precise.
It's a hard problem, but we are making progress on it. We will probably get there, but it's going to end up being very nuanced and already it is important to recognize that the word means different things in vernacular and in even differing research domains. Words are overloaded and I think we need to recognize this divergence and that we are gravely miscommunicating by assuming the definitions are obvious. I'm not sure why we don't do more to work together on this. In our field we seem to think we got it all covered and don't need others. I don't get that.
> In this view, if a machine performs a task as well as a human, it understands it exactly as much as a human.
And I do not think this is accurate at all. I would not say my calculator understands math despite it being able to do it better than me. I can say the same thing about a lot of different things which we don't attribute intelligence to. I'm sorry, but the logic doesn't hold.
Okay, you might take an out by saying the calculator can't do abstract math like I can, right? Well we're going to run into that same problem. You can't test your way out of it. We've known this in hard sciences like physics for centuries. It's why physicists do much more than just experiments.
There's the classic story of Freeman Dyson speaking to Fermi, which is why so many know about the 4 parameter elephant[0], but it is also just repeated through our history of physics. Guess what? Dyson's experiments worked. They fit the model. They were accurate and made accurate predictions! Yet they were not correct. People didn't reject Galileo just because the church, there were serious problems with his work too. Geocentricism made accurate predictions, including ones that Galileo's version of Heliocentrism couldn't. These historical misunderstandings are quite common, including things like how the average person understands things like Schrodinger's Cat. The cat isn't in a parallel universe of both dead and alive lol. It's just that we, outside the box can't determine which. Oh, no, information is lossy, there's injective functions, the universe could then still be deterministic yet we wouldn't be able to determine that (and my name comes into play).
So idk, it seems like you're just oversimplifying as a means to sidestep the hard problem[1]. The lack of a good technical definition of understanding should tell us we need to determine one. It's obviously a hard thing to do since, well... we don't have one and people have been trying to solve it for thousands of years lol.
> Just my opinion, but my professional opinion from thirty-plus years in AI.
Maybe I don't have as many years as you, but I do have a PhD in CS (thesis on neural networks) and a degree in physics. I think it certainly qualifies as a professional opinion. But at the end of the day it isn't our pedigree that makes us right or wrong.
[1] I'm perfectly fine tabling a hard problem and focusing on what's more approachable right now, but that's a different thing. We may follow a similar trajectory but I'm not going to say the path we didn't take is just an illusion. I'm not going to discourage others from trying to navigate it either. I'm just prioritizing. If they prove you right, then that's a nice feather in your hat, but I doubt it since people have tried that definition from the get go.
I’m not sidestepping the Hard Problem. I am denying it head on. It’s not a trick or a dodge! It’s a considered stance.
I'm denying that an idea that has historically resisted crisp definition, and that the Stanford Encyclopedia of Philosophy introduces as 'protean', needs to be taken seriously as an essential missing part of AI systems, until someone can explain why.
In my view, the only value the Hard Problem has is to capture a feeling people have about intelligent systems. I contend that this feeling is an artifact of being a social ape, and it entails nothing about AI.
Regardless of whether you think understanding is important, it’s clear from this thread that a lot of people find understanding valuable. In order to trust an AI with decisions that affect people, people will want to believe that the AI “understands” the implications of its decisions, for whatever meaning of “understand” those people have in their head. So indeed I think it is important that AI researchers try to get their AIs to understand things, because it is important to the consumers that they do.
I agree with this. I contend that as the AIs improve in performance, the designation of understanding will accrete to them. I predict there will never be a component, module, training process, or any other significant piece of an AI that is the ‘understanding’ piece that some believe is missing today.
Also, the widespread human belief that something is valuable has absolutely no entailments to me other than treating the believers with normal respect. It’s very easy to think of things that are important to billions that you believe are not true or relevant to a reality-driven life.
While I agree with you in the main, I also take seriously the "until someone can explain why" counterpoint.
Though I agree with you that your calculator doesn't understand math, one might reasonably ask, "why should we care?" And yeah, if it's just a calculator, maybe we don't care. A calculator is useful to us irrespective of understanding.
If we're to persuade anyone (if we are indeed right), we'll need to articulate a case for why understanding matters, with respect to AI. I think everyone gets this on an instinctual level- it wasn't long ago that LLMs suggested we add rocks to our salads to make them more crunchy. As long as these problems can be overcome by throwing more data and compute at them, people will remain incurious about the Understanding Problem. We need to make a rigorous case, probably with a good working alternative, and I haven't seen much action here.
I'm not the one claiming that a calculator thinks. The burden of proof lies on those that do. Claims require evidence and extraordinary claims require extraordinary evidence.
I don't think anyone is saying that the calculator isn't a useful tool. But certainly we should push back when people are claiming it understands math and can replace all mathematicians.
> If we're to persuade anyone, we'll need to articulate a case for why understanding matters
This is a more than fair point. Though I have not found it to be convincing when I've tried.
I'll say that a major motivating reason of why I went into physics in the first place is because I found that a deep understanding was a far more efficient way of learning how to do things. I started as an engineer and even went into engineering after my degree. Physics made me a better engineer, and I think a better engineer than had I stayed in engineering. Understanding gave me the ability to not just take building blocks and put them together, but to innovate. Being able to see things at a deeper level allowed me to come to solutions I otherwise could not have. Using math to describe things allowed me to iterate faster (just like how we use simulations). Understanding what the math meant allowed me to solve the problems where the equations no longer applied. It allowed me to know where the equations no longer applied. It told me how to find and derive new ones.
I often found that engineers took an approach of physical testing first, because "the math only gets you so far." But that was just a misunderstanding of how far their math took them. It could do more, just they hadn't been taught that. So maybe I had to take a few days working things out on pen and paper, but that was a cheaper and more robust solution than using the same time to test and iterate.
Understanding is a superpower. Problems can be solved without understanding. A mechanic can fix an engine without knowing how it works. But they will certainly be able to fix more problems if they do. The reason to understand is because we want things to work. The problem is, the world isn't so simple that every problem is the same or very similar to another. A calculator is a great tool. It'll solve calculations all day. Much faster than me, with higher accuracy, but it'll never come up with an equation on its own. That isn't to call it useless, but I need to know this if I want to get things done. The more I understand what my calculator can and can't do, the better I can use that tool.
Understanding things, and the pursuit to understand more is what has brought humans to where they are today. I do not understand why this is even such a point of contention. Maybe the pursuit of physics didn't build a computer, but it is without a doubt what laid the foundation. We never could have done this had we not thought to understand lightning. We would have never been able to tame it like we have. Understanding allows us to experiment with what we cannot touch. It does not mean a complete understanding nor does it mean perfection, but it is more than just knowledge.
> I'm not the one claiming that a calculator thinks. The burden of proof lies on those that do. Claims require evidence and extraordinary claims require extraordinary evidence.
You're right, I may have misconstrued the original claim. I took the parent to be saying something like "calculators understand math, but also, understanding isn't particularly important with respect to AI" but I may have gotten some wires crossed. This isn't the old argument about submarines that swim, I don't think.
Critiques should come with some argument if they want to be taken seriously.
If I say it’s not real intelligence because the box isn’t blue, how much does anyone owe that critique? How about if a billion people say that blueness is the essence missing from AIs?
Tell me why blue matters and we have a conversation.
Only problem is this time enough money is being burned that if AGI does not come, it will probably be extremely painful/fatal for a lot of people that had nothing to do with this field or the decisions being made. What will be the consequences if that comes to pass? So many lives were permanently ruined due to the GFC.
I'm not sure. There's a view that, as I understand it, suggests that language is intelligence. That language is a requirement for understanding.
An example might be kind of the contrary—that you might not be able to hold an idea in your head until it has been named. For myself, until I heard the word gestalt (maybe a fitting example?) I am not sure I could have understood the concept. But when it is described it starts to coalesce—and then when named, it became real. (If that makes sense.)
FWIW, Zeitgeist is another one of those concepts/words for me. I guess I have to thank the German language.
Perhaps it is why other animals on this planet seem to us lacking intelligence. Perhaps it is their lack of complex language holding their minds back.
> There's a view that suggests that language is intelligence.
I think you find the limits when you dig in. What are you calling language? Can you really say that Eliza doesn't meet your criteria? What about a more advanced version? I mean we've been passing the Turing Test for decades now.
> That language is a requirement for understanding.
But this contradicts your earlier statement. If language is a requirement then it must precede intelligence, right?
I think you must then revisit your definition of language and ensure that it matches to all the creatures that you consider intelligent. At least by doing this you'll make some falsifiable claims and can make progress. I think an ant is intelligent, but I also think ants do things far more sophisticated than the average person thinks. It's an easy trap, not knowing what you don't know. But if we do the above we get some path to aid in discovery, right?
> that you might not be able to hold an idea in your head until it has been named
Are you familiar with Anendophasia?
It is the condition where a person does not have an internal monologue. They think without words. The definition of language is still flexible enough that you can probably still call that language, just like in your example, but it shows a lack of precision in the definition, even if it is accurate.
> Perhaps it is why other animals on this planet seem to us lacking intelligence
One thing to also consider is if language is necessary for societies or intelligence. Can we decouple the two? I'm not aware of any great examples, although octopi and many other cephalopods are fairly asocial creatures. Yet they are considered highly intelligent due to their adaptive and creative nature.
Perhaps language is a necessary condition for advanced intelligence, but not intelligence alone. Perhaps it is communication and societies, differentiating from an internalized language. Certainly the social group can play an influence here, as coalitions can do more than the sum of the individuals (by definition). But the big question is if these things are necessary. Getting the correct causal graph, removing the confounding variables, is no easy task. But I think we should still try and explore differing ideas. While I don't think you're right, I'll encourage you to pursue your path if you encourage me to pursue mine. We can compete, but it should be friendly, as our competition forces us to help see flaws in our models. Maybe the social element isn't a necessary condition, but I have no doubt that it is a beneficial tool. I'm more frustrated by those wanting to call the problem solved. It obviously isn't, as it's been so difficult to get generalization and consensus among experts (across fields).
> It is the condition where a person does not have an internal monologue.
These people are just nutjobs that misinterpreted what internal monologue means, and have trouble doing basic introspection.
I know there are a myriad of similar conditions, aphantasia, synaesthesia, etc. But someone without internal monologue simply could not function in our society, or at least not pass as someone without obvious mental diminishment.
If there really were some other, hidden code in the mind, that could express "thoughts" in the same depth as language does - then please show it already. At least the tiniest bit of a hint.
I know some of these people. We've had deep conversations about what is going on in our thought processes. Their description significantly differs from mine.
These people are common enough that you likely know some. It's just not a topic that frequently comes up.
It is also a spectrum, not a binary thing (though full anendophasia does exist, it is just on the extreme end). I think your own experiences should allow you to doubt your claim. For example, I know when I get really into a fiction book I'm reading that I transition from a point where I'm reading the words in my head to seeing the scenes more like a movie, or more accurately like a dream. I talk to myself in my head a lot, but I can also think without words. I do this a lot when I'm thinking about more physical things like when I'm machining something, building things, or even loading dishwasher. So it is hard for me to believe that while I primarily use an internal monologue that there aren't people that primarily use a different strategy.
On top of that, well, I'm pretty certain my cat doesn't meow in her head. I'm not certain she has a language at all. So why would it be surprising that this condition exists? You'd have to make the assumption that there was a switch in human evolution. Where it happened all at once or all others went extinct. I find that less likely than the idea that we just don't talk enough about how we think to our friends.
Certainly there are times where you think without a voice in your head. If not, well you're on the extreme other end. After all, we aren't clones. People are different, even if there's a lot of similarities.
I’m like that more often than not. Words and language always seemed like a “translation layer” to express myself to other people, not something essential that needs to happen in my head. Especially when thinking deeply about some technical problem there’s no language involved, just abstract shapes and seeing things “in my mind’s eye”.
We might just be rehashing that silly internet meme about “shape rotators”, but there could be a correlation here where people whose minds work this way are more dismissive of LLMs.
I suggest you revisit the subject with your friends, with two key points:
1. Make it clear to them that with "internal monologue" you do not mean an actual audible hallucination
2. Ask them if they EVER have imagined themselves or others saying or asking anything
If they do, which they 100% will unless they lie, then you have ruled out "does not have an internal monologue", the claim is now "does not use his internal monologue as much". You can keep probing them what exactly that means, but it gets washy.
Someone that truly does not have an internal dialogue could not do the most basic daily tasks. A person could grab a cookie from the table when they feel like it (oh, :cookie-emoji:!), but they cannot put on their shoes, grab their wallet and keys, look in the mirror to adjust their hair, go to the supermarket, to buy cookies. If there were another hidden code that can express all huge mental state pulled by "buy cookies", by now we would at least have an idea that it exists underneath. We must also ask, why would we translate this constantly into language, if the mental state is already there? Translation costs processing power and slows down. So why are these "no internal monologue" people not geniuses?
I have no doubt that there is a spectrum, on that I agree with you. But the spectrum is "how present is (or how aware is the person of-) the internal monologue". E.g. some people have ADHD, others never get anxiety at all. "No internal monologue" is not one end of the spectrum for functioning adults.
The cat actually proves my point. A cat can sit for a long time before a mouse-hole, or it can hide to jumpscare his brother cat, and so on. So to a very small degree there is something that let's it process ("understand") very basic and near-future event and action-reactions. However, a cat could not possibly go to the supermarket to buy food, obviating anatomical obstacles, because: it has no language and therefore cannot make a complex mental model. Fun fact: whenever animals (apes, birds) have been taught language, they never ask questions (some claim they did, but if you dig in you'll see that the interpretation is extremely dubious).
> 1. Make it clear to them that with "internal monologue" you do not mean an actual audible hallucination
What do you mean? I hear my voice in my head. I can differentiate this from a voice outside my head, but yes, I do "hear" it.
And yes, this has been discussed in depth. It was like literally the first thing...
But no, they do not have conversations in their heads like I do. They do not use words as their medium. I have no doubt that their experience is different from mine.
> 2. Ask them if they EVER have imagined themselves or others saying or asking anything
This is an orthogonal point. Yes, they have imagined normal interactions. But frequently those imaginary conversations do not use words.
> The cat actually proves my point.
Idk man, I think you should get a pet. My cat communicates with me all the time. But she has no language.
> Fun fact: whenever animals (apes, birds) have been taught language, they never ask questions (some claim they did, but if you dig in you'll see that the interpretation is extremely dubious).
To be clear, I'm not saying my cat's intelligence is anywhere near ours. She can do tricks and is "smart for a cat" but I'm not even convinced she's as intelligent as the various wild corvids I feed.
> there's actual voice heard with your ears, there's the internal monologue, and then there's a hallucination.
This needs no explaining. I think I sufficiently made it clear that we agree with these distinctions.
>> I hear my voice in my head. I can differentiate this from a voice outside my head, but yes, I do "hear" it.
Though to be more precise I would say that a hallucination appears to come from outside the head, even if you are aware that it is coming from inside. Still, clearly distinct from an internal monologue, which is always clearly internal.
> And you did not dig in deeper?
>>>> I know some of these people. ***We've had deep conversations about what is going on in our thought processes.***
Yes. Multiple hours long conversations. One of these people I know now studies psychology. I research intelligence and minds from an artificial standpoint and they from a biological. Yeah, we have gotten pretty deep and have the skills and language to do so far more than the average person.
I think you need to consider that you may just be wrong. You are trying very hard to defend your belief, but why? The strengths of our beliefs should be proportional to the evidence that supports them. I am not trying to say that your logic is bad, let's make that clear. But I think your logic doesn't account for additional data. If you weren't previously aware of this data then how could you expect the logic to reach the correct conclusion? I want to make this clear because I want to distinguish correctness from intelligence (actually relevant to the conversation this stemmed from). You can be wrong without being dumb, but you can also be right and dumb. I think on this particular issue you fall into the former, not the latter. I respect that you are defending your opinion and beliefs, but this is turning as you are rejecting data. Your argument now rests on the data being incorrect, right? Because that's the point. Either the data is wrong or your model is wrong (and let's distinguish that a model is derived through logic to explain data).
I want to remind you that this idea is testable too. I told you this because it is a way to convince yourself and update the data up have available to you. You can train yourself to do this in some cases. Not all and obviously it won't be an identical experience to these people, but you can get yourself to use lower amounts of language when thinking through problems. You had also mentioned that people with aphantasia couldn't function, but think about that too. These topics are quite related actually, considering how we've discussed anendophasia you should be able to reason that these people are really likely to have low aphantasia. Notice I said low, as this is a spectrum. You can train the images in your mind to be stronger too. The fact that some images are stronger than others should lead you to believe that this is a spectrum and that it is likely people operate at different base levels. It should also lead you to reason that this is likely trainable in an average person. The same goes for anendophasia. Don't make this binary, consider it a spectrum. That's how the scientific literature describes the topic too. But if you pigeonhole it to being binary and only true in the extreme cases then your model isn't flexible enough as it also isn't considering the variances in people.
Go talk with your friends. Get detailed. When you imagine an apple in your head how much do you see? As the person if their process involves words or if it is purely imagery. If words, how many? Is it a red apple? Green? Yellow? Can they smell it? Can they taste it? What's it smell and taste like? I will bet you every single person you talk to will answer these differently. I will even wager that each time you do the exercise you yourself will answer differently, even if the variance is much smaller. But that's data, and your model needs to be able to explain that data too. While I think you have the right thought process I don't think you are accounting for this variance, instead treating it as noise. But noise can be parameterized and modeled too. Noise is just the statistical description of uncertainty.
Let me be clear: yes, I know I might be wrong. I hope I'm not dumb and wrong, or at least not dumb. I am also not writing here as some kind of debate exercise. I do because I find this topic extremely interesting and insightful. What if language is the intelligence? What if "guessing the next word" really was all that was there, to peak human intelligence, knowledge, and understanding of our world? I am not hyped by AI, it's rather that I find this possibility somewhat sad.
I've made up a model, an idea, and I don't think the data opposing it is trustworthy. My first problem is that there are many people that claim that they have NO internal monologue, which means NEVER constructing a sentence from theirselves or others in their head (except directly as verbal speech), and this seems outright impossible. When pressed, these people usually either admit that they do have some monologue, just "much less". Or they misunderstood it for something similar to schizophrenia, actual hallucinations. If they don't admit to actually, sometimes, having them, then they fail to explain where exactly the line between "thinking of someone or themselves saying something" and the internal monologue/dialogue is. As if they had been caught lying by the detective, they end the conversation. Or at least that's how I feel, I really don't know how to ask more questions here before making them feel too interrogated, or like someone that has self-diagnosed being told that they are imagining things.
With "absolutely none" group out of the way, it leaves us with people who claim to perceive the internal monologue very scarcely, and claim that they do not need to "think" or "do". How can we possibly test this scientifically? The data is all self-reported. Or at least I don't know if this can or has been neurologically researched.
Consider also that all self-reported data about internal monologue is "poisoned": we are trying to get objective data with the data itself as a vehicle. We are not asking if someone feels pain, or if they can solve a puzzle in a timeframe. We cannot measure electric activity with some instruments, nor evaluate yes-or-no questions.
What if it is true that some people do not perceive their internal monologue? I certainly don't remember it "popping" into my head at a certain age, and I think nobody does. When we learn language, we become conscious with it, because it allows to model the world, beyond putting things in our mouth and screaming. So it could be that not everybody perceives it equally, a spectrum like you said, and that some people rationalize it retroactively as not being there - just "thoughts", ideas, feelings. We reconstruct past events via a narration, filling in details by guessing, so why wouldn some people guess that they are not narrating in their head? It is not something that is taught in school or from or parents, you either perceive it as "internal monologue", or as "just thinking", because, well, it's the thinking doing it's thing.
Somewhat out of my league in this thread but,I think I am.one of these people. I do remember a time before I had an internal monologue , in fact I remember the day in elementary school when I learned, after having been explained to me by my teacher, that everyone else was "talking to themselves in their head". I think I spent the next month or so obsessing over this new found ability. But before that day I was perfectly capable of thought, and conversation, and writing. Even now I can "switch modes" and have coherent thoughts occur, with no labeling or accompanying narrative. I can distinctly identify concepts and transitions between them but there are no words involved until I open my mouth. So I dont know if it was just a hidden background process before that day. But it definitely "feels' different when its in the foreground or back, or nor there.
Almost certainly not. There does not seem to be a strong correlation between the two. We have a lot of different measures for intelligence when it comes to animals. We can place them across a (multidimensional) spectrum and humans seem unique with language. It also appears that teaching animals language does not cause them to rapidly change on these metrics despite generations of language capabilities.
> What if "guessing the next word" really was all that was there, to peak human intelligence, knowledge, and understanding of our world?
I believe this is falsifiable. As I best understand it is a belief of this relationship: predict next word <--> understanding. Yet we know that neither direction holds true. I'll state some trivial cases for brevity[0] but I have no doubt you can determine more complicated ones and even find examples.
-> I can make accurate predictions about coin flips without any understanding of physics or how the coin is being flipped. All I need to do is be lucky. Or we can take many mechanical objects like a clock that can predict time.
Or a horse can appear to do math if I tell it how many times to stomp its foot. It made accurate predictions yet certainly has no understanding.
Ehh I'll give you a more real example. Here's a model that gives accurate predictions for turn by turn taxi directions where the authors extract the world model and find it is not only inaccurate but find that it significantly diverges. Vafa has a few papers on the topic, I suggest reading his work.
<- You can understand all the physics to a double pendulum and not predict the movement for any arbitrary amount of time moving forward if you do not also know the initial conditions. This is going to be true for any chaotic system.
I said we've seen this in the history of science. {Geo,Helio}centrism is a great example. Scientists who had no affiliation with the church still opposed Galileo because his model wasn't making accurate predictions for certain things. Yet the heliocentric model is clearly a better understanding and more accurate as a whole. If you want to dive deeper into this topic I'd highly recommend both the podcast "An Opinionated History of Math" and the book "Representing and Intervening" by Ian Hacking. They're both very approachable. FWIW, metaphysics talks about this quite a lot.
> My first problem is that there are many people that claim that they have NO internal monologue
So again, I cannot stress that we should not represent this as a binary setting. The binary cases are the extreme (in both directions). Meaning very few people experience them.
The problem here is one of language and semantics, not effect. I completely believe that someone will say "I have no internal monologue" if >90% of their thinking is without an internal monologue. Just like how a guy who's 5'11.75" will call themselves 6'. Are they a liar? I wouldn't say so, they're >99% accurate. Would you for someone 5'11"? That's probably more contextually dependent.
So you distrust the data. That's fine. Let's assume poisoned. We should anyways since noise is an important part of any modeling[2]. It is standard practice...
So instead, do you distrust that there's a distribution into how much of an internal monologue individuals use? Or do you presume they all use them the same.
I'd find it hard to believe you distrust the spectrum. But if you trust the spectrum then where is the threshold for your claim? 0%? That's really not a useful conversation even if heavy tailed.
You are hyper-fixated on the edge case but its result isn't actually consequential to your model. The distribution is! You'll have to consider your claims much more carefully when you consider a distribution. You need to then claim a threshold, in both directions. Or if you make the claim that we're all the same (I'd find that quite surprising tbh, especially given the nature of linguistics), you need to explain that too and your expected distribution that would claim that (narrow).
All I can tell you is that my friend and I have had this conversation multiple times over many years and it seems very constant to me. I have no reason to believe they are lying and if they are they are doing so with an extreme level of consistency, which would be quite out of the norm.
[0] Arguing the relationship still requires addressing trivial relationships.
> We have a lot of different measures for intelligence when it comes to animals.
But there is an abismal difference between animal intelligence and human intelligence.
> predict next word <--> understanding
Yes, and I could say a stone understands the world because its state reflects the world: it gets hot, cold, wet, dry, radiated, whatever. Perhaps its internal state can even predict the world: if it's rolling downhill, it can predict that it will stop soon. But the stone is not conscious like a human, and neither is a clock nor a horse that can count to ten. The stone obviously is "reducing to the absurd" - a horse can actually "guess" to some degree, but nothing like a human. It cannot ask a question, and it cannot answer itself a question.
> I cannot stress that we should not represent this as a binary setting.
That was kind of my point, to eliminate the binary "no", leaving us with a spectrum.
My initial claim "these are just nutjobs" - my apologies for the phrasing - was addressing this: there are no people "without internal monologue AT ALL".
Since we seem to actually agree on this point, our difference is that I believe that the people with "little internal monologue" are simply not aware of it.
Let me phrase string it this way: If language is the understanding, then the internal monologue is not some quirky side effect. To understand something at the human level, we need to describe it with language, the rest are primitive instincts and "feelings".
We can model the past and the future. We can model ourselves in 10 years. And what is one of the most important things we would model? What we would say or think then - thinking being "saying something out silently in our head". Not really just feelings: "I would love my partner", sure but why? "Because . . .".
When we are utilizing language, the internal monologue, to construct the model, we cannot be "aware of it" constantly. That is, the bandwidth is taken by the tasks at hand that we are dealing with, it would be detrimental if every other phrase would be followed with "btw did I notice that I just understand this via a string of words?". The more complex actions or idea we process, the less aware we are that we are using language for it. That is "being in the flow". We can reconstruct it when done, and here, if there is a lack of awareness of internal monologue, it will be rationalized as something else.
> Or if you make the claim that we're all the same (I'd find that quite surprising tbh, especially given the nature of linguistics), you need to explain that too and your expected distribution that would claim that (narrow).
My explanation (without proof), is that it's just a matter of awareness.
> All I can tell you is that my friend and I have had this conversation multiple times over many years and it seems very constant to me. I have no reason to believe they are lying and if they are they are doing so with an extreme level of consistency, which would be quite out of the norm.
Can you think of some kind of tests question (or string of questions) that could prove either? I have been thinking about it obviously, but I can't come up with any way to empirically test that there is or is no internal monologue. Consistency could simply mean that their rationalization is consistent.
I'll leave you this article, which I found quite interesting: https://news.ycombinator.com/item?id=43685072 The person lost language, and lost what we could consider human-level consciousness at the same time, and then recovered both at the same rate. Of course, there was brain damage, so it's not an empirical conclusion.
There is us a book written by a woman who suffered a stroke. She lost the ability to speak and understand language. Yet she remained conscious. It took her ten years to fully recover. The book is called "A stroke of insight".
> It's to recite (or even apply) knowledge. To understand does actually require a world model.
This is a shell game, or a god of the gaps. All you're saying is that the models "understand" how to recite or apply knowledge or language, but somehow don't understand knowledge or language. Well what else is there really?
It makes perfect sense to say that the database understands your query. It also makes sense to say that the database's factorization of domain knowledge + domain queries exhibit at least a static domain understanding (which still isn't general ala AGI). This is the standard systems response to the Chinese Room.
The "general" part comes from whether that static aspect can be made dynamic and extensible. In what sense is a system that can be arbitrarily extended to "recite" or "apply" knowledge not AGI?
I would say that the database processes my query, not understands it. You reference the Chinese Room, but that's the thing. The Chinese Room processes the inputs without understanding them.
As someone who is invested in researching said math, I can say with some confidence that it does not exist, or at least not in the form claimed here. That's the whole problem.
I would be ecstatic if it did though, so if anyone has any examples or rebuttal, I would very much appreciate it.
You're right that there is no purely mathematical argument; it's almost non-sensical to claim such.
Instead you can simply make the rather reasonable observation that LLMs are a product of their training distribution, which only contains partial coverage of all possible observable states of the world. Some highly regular observable states are thus likely missing, but an embodied agent (like a human) would be able to correctly reason about these states due to other sensory modes that observe these states. It's very clear that the number of possible states far exceeds any text describing those states (the number of states transcribed to text is vanishingly small in comparison, but they are locally coherent and quite regular when observed through other sensory modes like vision). That's the closest you can make to a mathematical argument.
As a result one common failure case for an LLM might involve describing a specific physical environment and associated set of actions that are in an area of low (or no) probability mass in the training data and asking the LLM to reason about it. Here's an example where I asked ChatGPT the following:
imagine i walked into a room through a doorway, then turned around and walked right back out of the room. from the perspective of a person in the room facing the door, which side would my right arm be? please explain.
It responded by incorrectly stating the following:
Let’s break this down step by step.
1. *You walk into the room through the doorway*:
* As you're entering, the person in the room is facing the door. So, you're entering from the **outside**.
* Your **right arm** will be on the **right side** of your body, just as it is normally when you're standing.
2. *You turn around and walk right back out*:
* When you turn around to walk out, you’ll be facing **away** from the room, looking back toward the doorway.
* Since you're walking **back out**, the person inside the room is still facing the door, while you are walking away from them, so your **right arm** is still on the right side of your body.
So, from the perspective of the person in the room, as you’re exiting, your *right arm* will still be on their *left side*. That’s because they’re facing the door, and your right side is to their left when you’re turning around and leaving.
To visualize:
* Imagine a door in front of them.
* You walk through and turn around, so you're now facing away from them.
* Your right arm is on your right side, but to them, it's on their left side since you’re now moving in the opposite direction.
My claim is more about that data processing is not enough. I was too vague and I definitely did not convey myself accurately. I tried to clarify a bit in a sibling comment to yours but I'm still unsure if it is sufficient tbh.
For embodiment, I think this is sufficient but not necessary. A key part to the limitation is that the agent cannot interact with its environment. This is a necessary feature for distinguishing competing explanations. I believe we are actually in agreement here, but I do think we need to be careful how we define embodiment. Because even a toaster can be considered a robot. It seems hard to determine what does not qualify as a body when we get to the itty gritty. But I think in general when people are talking about embodiment they are discussing the capability of being interventional.
By your elaboration I believe we agree since part of what I believe to be necessary is the ability to self-analyze (meta-cognition) to determine low density regions of its model and then to be able to seek out and rectify this (intervention). Data processing is not sufficient for either of those conditions.
Your prompt is, imo, more about world modeling, though I do think this is related. I asked Claude Sonnet 4.5 with extended thinking enabled and it also placed itself outside the room. Opus 4.1 (again with extended thinking), got the answer right. (I don't use a standard system prompt, though that is mostly to make it not syncopathic and to try to get it to ask questions when uncertain and enforce step by step thinking)
From the perspective of the person in the room, your right arm would be on their right side as you walk out.
Here's why: When you initially walk into the room facing the person, your right arm appears on their left side (since you're facing each other). But when you turn around 180 degrees to walk back out, your back is now toward them. Your right arm stays on your right side, but from their perspective it has shifted to their right side.
Think of it this way - when two people face each other, their right sides are on opposite sides. But when one person turns their back, both people's right sides are now on the same side.
The CoT output is a bit more interesting[0]. Disabling my system prompt gives an almost identical answer fwiw. But Sonnet got it right. I repeated the test in incognito after deleting the previous prompts and it continued to get it right, independent of my system prompt or extended thinking.
I don't think this proves a world model though. Misses are more important than hits, just as counter examples are more important than examples in any evidence or proof setting. But fwiw I also frequently ask these models variations on river crossing problems and the results are very shabby. A few appear spoiled now but they are not very robust to variation and that I think is critical.
I think an interesting variation of your puzzle is as follows
Imagine you walked into a room through a doorway. Then you immediately turn around and walk back out of the room.
From the perspective of a person in the room, facing the door, which side would your right arm be? Please explain.
I think Claude (Sonnet) shows some subtle but important results in how it answers
Your right arm would be on their right side.
When you turn around to walk back out, you're facing the same direction as the person in the room (both facing the door). Since you're both oriented the same way, your right side and their right side are on the same side.
This makes me suspect there's some overfitting. CoT correctly uses "I"[1].
It definitely isn't robust to red herrings[2], and I think that's a kicker here. It is similar to failure results I see in any of these puzzles. They are quite easy to break with small variations. And we do need to remember that these are models trained on the entire internet (including HN comments), so we can't presume this is a unique puzzle.
Let me clarify. I was too vague and definitely did not express things accurately. That is on me.
We have the math to show that it can be impossible to distinguish two explanations through data processing alone. We have examples of this in science, a long history of it in fact. Fundamentally there is so much that we cannot conclude from processing data alone. Science (the search of knowledge) is active. It doesn't require just processing existing data, it requires the search for new data. We propose competing hypotheses that are indistinguishable from the current data and seek out the data which distinguishes them (a pain point for many of the TOEs like String Theory). We know that data processing alone is insufficient for explanation. We know it cannot distinguish confounders. We know it cannot distinguish causal graphs (e.g. distinguish triangular maps. We are able to create them, but not distinguish them through data processing alone). The problem with scaling alone is that it makes the assertions that data processing is enough. Yet we have so much work (and history) telling us that data processing is insufficient.
The scaling math itself also shows a drastic decline in performance with scale and often do not suggest convergence even with infinite data. They are power laws with positive concavity, requiring exponential increase in data and parameters for marginal improvements on test loss. I'm not claiming that we need zero test loss to reach AGI, but the results do tell us that if this is strongly correlated then we'll need to spend an exponential amount more to achieve AGI even if we are close. By our measures, scaling is not enough unless we are sufficiently close. Even our empirical results align with this as despite many claiming that scale is all we need, we are making significant changes to the model architectures and training procedures (including optimizers). We are making these large changes because throwing the new data at the old models (even when simply increasing the number of parameters) does not work out. It is not just the practicality, it is the results. The scaling claim has always been a myth used to drive investments since it is a nice simple story that says that we can get there by doing what we've already been doing, just more. We all know that these new LLMs aren't dramatic improvements off their previous versions, despite being much larger, more efficient, and having processed far more data.
[side note]: We even have my namesake who would argue that there are truths which are not provably true with a system that is both consistent and efficient (effectively calculable). But we need not go that far, as omniscience is not a requirement for AGI. Though it is worth noting for the limits of our models, since at the core this matters. Changing our axioms changes the results, even with the same data. But science doesn't exclusively use a formal system, nor does it use a single one.
My apologies for the much delayed reply as I have recently found myself with little extra time to post adequate responses. Your critiques are very interesting to ponder, so I thank you for posting them. I did want to respond to this one though.
I believe all of my counterarguments center around my current viewpoint that given the rapid rate of progress involved on the engineering side, it is no longer reasonable in deep learning theory to consider what is possible, and it is more interesting to try to outline hard limitations. This emposes a stark contrast between deep learning and classical statistics, as the boundaries in the latter are very clear and are not shared by the former.
I want to stress that at present, nearly every conjectured limitation of deep learning over the last several decades has fallen. This includes many back of the napkin, "clearly obvious" arguments, so I'm wary of them now. I think the skepticism all along has been fueled in response to hype cycles, so we must be careful not to make the same mistakes. There is far too much empirical evidence available to counter precise arguments against the claim that there is an underlying understanding within these models, so it seems we must resort to the imprecise to continue the debate.
Scaling, along one axis, suggests a high polynomial degree of additional compute (not exponential) is required for increasing improvements, this is true. But the progress over the last few years has occurred due to the discovery of new axes to scale on, which further reduces the error rate and improves performance. There are still many potential axes left untapped. What is significant about scaling to me is not how much additional compute is required, but the fact that the predicted bottom at the moment is very, very low, far lower than anything else we have ever seen, and that doesn't require any more data than we currently have. That should be cause for concern until we find a better lower bound.
> We all know that these new LLMs aren't dramatic improvements off their previous versions
No, I don't agree. This may be evident to many, but to some, the differences are stark. Our perceived metrics of performance are nonlinear and person-dependent, and these major differences can be imperceptible to most. The vast majority of attempts at providing more regular metrics or benchmarks that are not already saturated have shown that LLM development is not slowing down by any stretch. I'm not saying that LLMs will "go to the moon". But I don't have anything concrete to say they cannot either.
> We have the math to show that it can be impossible to distinguish two explanations through data processing alone.
Actually, this is a really great point, but I think this highlights the limitations of benchmarks and the requirements of capacity-based, compression-based, or other types of alternative data-independent metrics. With these in tow, it can be possible to distinguish two explanations. This could be a fruitful line of inquiry.
I think current AI is a human language/behavior mirror. A cat might believe they see another cat looking in a mirror, but you can’t create a new cat by creating a perfect mirror.
Model based reinforcement learning is a thing and it is kind of a crazy idea. Look up temporal difference model predictive control.
The fundamental idea behind temporal difference is that you can record any observable data stream over time and predict the difference between past and present based on your decision variables (e.g. camera movement, actuator movement, and so on). Think of it like the Minecraft clone called Oasis AI. The AI predicts the response to a user provided action.
Now imagine if it worked as presented. The data problem would be solved, because you are receiving a constant stream of data every single second. If anything, the RL algorithms are nowhere near where they need to be and continual learning has not been solved yet, but the best known way is through automatic continual learning ala Schmidhuber (co-inventor of LSTMs along with Hochreiter).
So, model based control is solved right? Everything that can be observed can be controlled once you have a model!
Wrong. Unfortunately. You still need the rest of reinforcement learning: an objective and a way to integrate the model. It turns out that reconstructing the observations is too computationally challenging and the standard computational tricks like U-Nets learn a latent representation that is optimized for reconstruction rather than for your RL objectives. There is a data exchange problem that can only realistically be solved by throwing an even bigger model at it, but here is why that won't work either:
Model predictive control tries to find the best trajectory over a receding horizon. It is inherently future oriented. This means that you need to optimize through your big model and that is expensive to do.
So you're going to have to take shortcuts by optimizing for a specific task. You reduce the dimension of the latent space and stop reconstructing the observations. The price? You are now learning a latent space for your particular task, which is less demanding. The dream of continual learning with infinite data shatters and you are brought down to earth: it's better than what came before, but not that much better.
The thing is, achieving say, 99.99999% reliable AI would be spectacularly useful even if it's a dead end from the AGI perspective.
People routinely conflate the "useful LLMs" and "AGI", likely because AGI has been so hyped up, but you don't need AGI to have useful AI.
It's like saying the Internet is dead end because it didn't lead to telepathy. It didn't, but it sure as hell is useful.
It's beneficial to have both discussions: whether and how to achieve AGI and how to grapple with it, and how to improve a reliability, performance and cost of LLMs for more prosaic use cases.
> The interviewer had an idea that he took for granted: that to understand language you have to have a model of the world. LLMs seem to understand language therefore they've trained a model of the world. Sutton rejected the premise immediately. He might be right in being skeptical here.
That's the basic success of LLMs. They don't have much of a model of the world, and they still work. "Attention is all you need". Good Old Fashioned AI was all about developing models, yet that was a dead end.
There's been some progress on representation in an unexpected area. Try Perchance's AI character chat. It seems to be an ordinary chatbot. But at any point in the conversation, you can ask it to generate a picture, which it does using a Stable Diffusion type system. You can generate several pictures, and pick the one you like best. Then let the LLM continue the conversation continue from there.
It works from a character sheet, which it will create if asked. It's possible to start from an image and get to a character sheet and a story. The back and forth between the visual and textural domains seems to help.
For storytelling, such system may need to generate the collateral materials needed for a stage or screen production - storyboards, scripts with stage directions, character summaries, artwork of sets, blocking (where everybody is positioned on stage), character sheets (poses and costumes) etc. Those are the modeling tools real productions use to keep a work created by many people on track. Those are a form of world model for storytelling.
I've been amazed at how good the results I can get from this thing are. You have to coax it a bit. It tends to stay stuck in a scene unless you push the plot forward. But give it a hint of what happens next and it will run with it.
Absolutely. AGI isn't a matter of adding more 9s. It's a matter of solving more "???"s. And those require not just work but also a healthy serving of luck.
As I understand it, to the breadth of LLMs was also something that was stumbled on kinda by accident, I understand they got developed as translators and were just 'smarter' than expected.
Also, to understand the world you don't need language. People don't think in language. Thought is understanding. Language is knowledge transfer and expression.
I think this a useful challenge to our normal way of thinking.
At the same time, "the world" exists only in our imagination (per our brain). Therefore, if LLMs need a model of a world, and they're trained on the corpus of human knowledge (which passed through our brains), then what's the difference, especially when LLMs are going back into our brains anyway?
Something to think about (hah!) is there are people without an internal monologue i.e. no voice inside their head they use when working out a problem. So they're thinking and learning and doing what humans do just fine with no little voice no language inside their head.
It's so weird that people literally seem to have a voice in their head they cannot control. For me personally my "train of thought" is a series of concepts, sometimes going as far as images. I can talk to myself in my head with language if I make a conscious effort to do so, just as I can breathe manually if I want. But if I don't, it's not really there like some people seem to have.
Probably there are at least two groups of people and neither really comprehends how the other thinks haha.
I think there are significantly more than 2, when you start to count variations through the spectrum of neurodiversity.
Spatial thinkers, for example, or the hyperlexic.
Meaning for hyperlexics is more akin to finding meaning in the edges of the graph, rather than the vertices. The form of language contributing a completely separate graph of knowledge, alongside its content, creating a rich, multimodal form of understanding.
Spatial thinkers have difficulty with procedural thinking, which is how most people are taught. Rather than the series of steps to solve the problem, they see the shape of the transform. LLMs as an assistive device can be very useful for spatial thinkers in providing the translation layer between the modes of thought.
Are the particles that make up thoughts in our brain not also a representation of a thought? Isn't "thought" really some kind of Platonic ideal that only has approximate material representations? If so, why couldn't some language sentences be thoughts?
> The landscape itself doesn't capture anything, it just is.
Sure, but the landscape is something, namely an aggregate of particles. A thought in principle isn't its physical expression, but its information content, and it's represented in a human brain by some aggregate of particles. So no matter how you slice it, thoughts can only manifest within representations, and so calling language a representation of thought isn't some kind of dunk, because it also proves that human brains don't have thoughts.
It's not clear whether the information content of all possible human thoughts can be captured by language, but clearly at least some language expressions have the same information content as human thoughts.
We are paying the price now for not teaching language philosophy as a core educational requirement.
Most people have had no exposure to even the most basic ideas of language philosophy.
The idea all these people go to school for years and don't even have to take a 1 semester class on the main philosophical ideas of the 20th century is insane.
LLMs clearly think. They don't have a sense of object permanence, at least not yet, but they absolutely, indisputably use pretrained information to learn and reason about the transient context they're working with at the moment.
Otherwise they couldn't solve math problems that aren't simple rephrasings of problems they were trained on, and they obviously can do that. If you give a multi-step undergraduate level math problem to the human operator of a Chinese room, he won't get very far, while an LLM can.
So that leads to the question: given that they were trained on nothing but language, and given that they can reason to some extent, where did that ability come from if it didn't emerge from latent structure in the training material itself? Language plus processing is sufficient to produce genuine intelligence, or at least something indistinguishable from it. I don't know about you, but I didn't see that coming.
They very clearly do not think. If they did, they wouldn't be able to be fooled by so many simple tests that even a very small (and thus, uneducated) human would pass.
A world model can not exist, the context windows aren't even near big enough for that. Weird that every serious scientist agrees on AGI not being a thing in the next decades. LLMs are good if you train them for a specific thing. Not so much if you expect them to explain the whole world to you. This is not possible yet.
> LLMs seem to udnerstand language therefore they've trained a model of the world.
This isn’t the claim, obviously. LLMs seem to understand a lot more than just language. If you’ve worked with one for hundreds of hours actually exercising frontier capabilities I don’t see how you could think otherwise.
Watching the video it seems that Dwarkesh doesn't really have a clue what he's confidently talking about yet running fast with his personal half-baked ideas, to the points where it gets both confusing and cringe when Karpathy apparently manages to make sense of it and yes-anding the word salad AK. Karpathy is supposedly there to clear up misunderstanding yet lets all the nonsense Dwarkesh is putting before him slide.
"ludicrous" sure but I wouldn't be so certain about "strawman" or that Dwarkesh has a consistent view.
To me, it's a matter of a very big checklist - you can keep adding tasks to the list, but if it keeps marching onwards checking things off your list, some day you will get there. whether it's a linear or asymptotic march, only time will tell.
I don't know if you will get there, that's far from clear at this stage.
Did you see the recent video by Nick Beato [1] where he asks various models about a specific number? The models that get it right are the models that consume youtube videos, because there was a youtube video about that specific number. It's like, these models are capable of telling you about very similar things that they've seen, but they don't seem like they understand it. It's totally unclear whether this is a quantitative or qualitative gap.
That's like saying that if we image every neuron in the brain we will understand thinking. We can build these huge databases and they tell us nothing about the process of thinking.
What "9" do you add to AGI? I don't think we even have the axes defined, let alone a way to measure them. "Mistakes per query?" It's like Cantor's diagonal test, where do we even start?
I don’t have a deep understand of LLMs but don’t they fundamentally work on tokens and generate a multi-dimensional statistical relationship map between tokens?
So it doesn’t have to be LLM. You could theoretically have image tokens (though I don’t know in practice, but the important part is the statistical map).
And it’s not like my brain doesn’t work like that either. When I say a funny joke in response to people in a group, I can clearly observe my brain pull together related “tokens” (Mary just talked about X, X is related to Y, Y is relevant to Bob), filter them, sort them and then spit out a joke. And that happens in like less than a second.
Yes! Absolutely. And this is likely what would be necessary for anything approaching actual AGI. And not just visual input, but all kinds of sensory input. The problem is that we have no ability, not even close, to process that even near the level of a human yet, much less some super genius being.
> yeah that "model of the world" would mean: babies are already born with "the model of the world"
No, not necessarily. Babies don't interact with the world only by reading what people wrote wikipedia and stackoverflow, like these models are trained. Babies do things to the world and observe what happens.
I imagine it's similar to the difference between a person sitting on a bicycle and trying to ride it, vs a person watching videos of people riding bicycles.
I think it would actually be a great experiment. If you take a person that never rode a bicycle in their life and feed them videos of people riding bicycles, and literature about bikes, fiction and non-fiction, at some point I'm sure they'll be able to talk about it like they have huge experience in riding bikes, but won't be able to ride one.
We’ve been thinking about reaching the singularity from one end, by making computers like humans, but too little thought has been given to approaching the problem from the other end: by making babies build their world model by reading Stack Overflow.
The “Brave New World meets OpenAI” model where bottle-born babies listen to Stack Overflow 24 hours a day until they one day graduate to Alphas who get to spend Worldcoin on AI-generated feelies.
That's it. Now you've done it! I will have stackoverflow Q&A, as well as moderator comments and closings of questions playing 24/7 to my first not yet born child! Q&A for the knowledge and the mod comments for good behavior, of course. This will lead to singularity in no time!
You have instincts, right? Innate fears? This is definitely something passed down through genetics. The Hawk/Goose Effect isn't just limited to baby chickens. Certainly some mental encoding passes down through genetics as how much the brain controls, down to your breathing and heartbeat.
But instinct is basic. It's something humans are even able to override. It's a first order approximation. Inaccurate to do meaningfully complex things, but sufficient to keep you alive. Maybe we don't want to call the instinct a world model (it certainly is naïve) but can't be discounted either.
In human development, yeah, the lion's share of it happens post birth. Human babies don't even show typical signs of consciousness, even really till the age of 2. There's many different categories of "awareness" and these certainly grow over time. But the big thing that makes humans so intelligent is that we continue to grow and learn through our whole lifetimes. And we can pass that information along without genetics and have very advanced tools to do this.
It is a combination of nature and nurture. But do note that this happens differently in different animals. It's wonderfully complex. LLMs are quite incredible but so too are many other non-thinking machines. I don't think we should throw them out, but we never needed to make the jump to intelligence. Certainly not so quickly. I mean what did Carl Sagan say?
One of the biggest mysteries of humans Vs LLMs is that LLMs need an absurd amount of data during pre training, then a little bit of data during fine tuning to make them behave more human. Meanwhile humans don't need any data at all, but have the blind spot that they can only know and learn about what they have observed. This raises two questions. What is the loss function of the supervised learning algorithm equivalent? Supposedly neurons do predictive coding. They predict what their neighbours are doing. That includes input only neurons like touch, pain, vision, sound, taste, etc. The observations never contain actions. E.g. you can look at another human, but that will never teach you how to walk because your legs are different from other people's legs.
How do humans avoid starving to death? How do they avoid leaving no children? How do they avoid eating food that will kill them?
These things require a complicated chain of actions. You need to find food, a partner and you need to spit out poison.
This means you need a reinforcement learning analogue, but what is going to be the reward function equivalent? The reward function can't be created by the brain, because it would be circular. It would be like giving yourself a high, without even needing drugs. Hence, the reward signal must remain inside the body but outside the brain, where the brain can't hack it.
The first and most important reward is to perform reproduction. If food and partners are abundant, the ones that don't reproduce simply die out. This means that reward functions that don't reward reproduction disappear.
Reproduction is costly in terms of energy. Do it too many times and you need to recover and eat. Hunger evolved as a result of the brain needing to know about the energy state of the body. It overrides reproductive instincts.
Now let's say you have a poisonous plant that gives you diarrhea, but you are hungry. What stops you from eating it? Pain evolves as a response to a damaged body. Harmful activities signal themselves in the form of pain to the brain. Pain overrides hunger. However, what if the plant is so deadly that it will kill you? The pain sensors wouldn't be fast enough. You need to sense the poison before it enters your body. So the tongue evolves taste and cyanide starts tasting bitter.
Notice something? The feelings only exist internally inside the human body, but they are all coupled with continued survival in one way or another. There is no such thing for robots or LLMs. They won't accidentally evolve a complex reward function like that.
I don't agree with this and I don't think any biologist or neuroscientist would either.
1) Certainly the data I discussed exists. No creature comes out a blank slate. I'll be bold enough to say that this is true even for viruses, even if we don't consider them alive. Automata doesn't mean void of data and I'm not sure why you'd ascribe this to life or humans.
2) humans are processing data from birth (technically before too but that's not necessary for this conversation and I think we all know that's a great way to have an argument and not address our current conversation). This is clearly some active/online/continual/ reinforcement/wherever-word-you-want-to-use learning.
It's weird to suggest an either or situation. All evidence points to "both". Looking at different animals even see both but also with different distributions.
I think it's easy to over simplify the problem and the average conversation tends to do this. It's clearly a complex with many variables at play. We can't approximate with any reasonable accuracy by ignoring or holding them constant. They're coupled.
> The reward function can't be created by the brain, because it would be circular.
Why not? I'm absolutely certain I can create my own objectives and own metrics. I'm certain my definition of success is different from yours.
> It would be like giving yourself a high, without even needing drugs
Which is entirely possible. Maybe it takes extreme training to do extreme versions but it's also not like chemicals like dopamine are constant. You definitely get a rush by completing goals. People become addicted to things like videogames, high risk activities like sky diving, or even arguing on the internet.
Just because there are externally driven or influenced goals doesn't mean internal ones can't exist. Our emotions can be driven both externally and internally.
> Notice something?
You're using too simple of a model. If you use this model then the solution is as easy as giving a robot self preservation (even if we need to wait a few million years). But how would self preservation evolve beyond its initial construction without the ability to metaprocess and refine that goal? So I think this should highlight a major limitation in your belief. As I see it, the only other way is a changing environment that somehow allows continued survival by the constructions and precisely evolves such that the original instructions continue to work. Even with vague instructions that's an unstable equilibrium. I think you'll find there's a million edge cases even if it seems obvious at first. Or read some Asimov ;)
> babies are already born with "the model of the world"
> but a lot of experiments on babies/young kids tell otherwise
I believe they are born with such a model? It's just that model is one where mummy still has fur for the baby to cling on to? And where aged something like 5 to 8 it's somehow useful for us to build small enclosures to hide in, leading to a display of pillow forts in the modern world?
The interviewer had an idea that he took for granted: that to understand language you have to have a model of the world. LLMs seem to udnerstand language therefore they've trained a model of the world. Sutton rejected the premise immediately. He might be right in being skeptical here.