This article is not about "reasoning" in the abstract, philosophical sense but is talking about "mechanistic interpretability" research. The title is more like, "can we understand if the 'knowledge' encoded into a neural networks actually corresponds to reasoning-like concepts" and doing that with actual experiments like tweaking weights and activations.
There's an interesting example where researchers saw a model approached clock time calculations and calendar month-day calculations using the same methodology. So then is this because an underlying concept of "cyclical measures" has emerged in the network?
Thanks - I've attempted to put that in the title above, in the hope of representing the article accurately.
(The trouble with a baity title like "Can We Understand How Large Language Models Reason?" is that it generates a barrage of shallow, reflexive responses having little to do with the article. What we want on HN are curious, reflexive responses instead - https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor....)
One plausible reason I thought of that we may not understand neural nets is that by their nature their power grows with ever-more complex connections and weights.
So it is like the opposite of logical systems, in that the very design of neural net architecture is a mess of parameter "spaghetti code" which renders the entire thing a metaphorical encrypted black box. The more powerful an AI/AGI the more this would be the case, and this is analogous a complexity curve.
And so any effort to make sense of such black box computation would be like trying to reverse entropy, analogous to trying to recover information lost in waste heat. And that could be one fundamental barrier to understanding both human and artificial brains alike, relative to their internal complexity.
(Just thinking aloud my handwavy pet theory recently, I am not an expert and could be totally mistaken on this)
>“Mechanistic interpretability will probably never reduce large language models to a few simple equations,” Icard concluded, “but it may gradually turn deep neural networks into systems whose hidden algorithms can at least partly be understood.”
[[All: please don't post shallow-generic reactions to baity titles. Those are basically the same thing, a la https://en.wikipedia.org/wiki/Rubin_vase, and we're trying for something more substantive here.]]
They dont. They have input that runs through a invisible stochastic canyon. As long as there is previous experience the stochastic canyon never ends. If there is none or isignificant one, or it runs out of tokkens, it hallucinates and the illusion falls apart. There is no reasoning, just the invisible grand canyon of all of human experience and knowledge. PS: try to get it to retell you a clichee movie or book and you can see life near the end, how the delta of all the same movies opens up into wildly different endings.
To advance further it would need the ability to abstract away the general situation shape and pattern recognize similar situations.
Compression is the trick. Its even philosophed about if compression = intelligence.
The LLM has to compress everyy question/prompt into its system. It does so by creating rules and ways of processing data (this can lead to AGI, world models or an architecture of sub architectures like an LLM + something else). So if it should respond in a way that only reasoning people can achieve, it might be able to learn a representation of what we call reasoning.
It read enough text in itself to even know about the concept of reasoning and how you would do that.
Even if this is only stochastic, it shouldn't be so devalued as your comment comes across.
When a mathematician reads a hundred-year-old math paper, they are reproducing in their head the reasoning of someone who died long ago. That is, reasoning can be written down and replicated.
If that works, I think it's fair to say that LLM's are inanimate processes that can generate real reasoning. You can tell when you read it and it makes sense.
There are likely some kinds of reasoning that can't be written down, as well as other forms of understanding, but they also don't replicate nearly as easily.
It's probably helpful in this discussion to make a difference between two definitions of reasoning:
1. phenomenal reasoning, requiring consciousness and subjective experience
2. functional reasoning, transforming premises into conclusions using logic
I think you are attacking this using definition 1, whereas the article is obviously aiming at a different type of reasoning, and trying to formalize what is actually going on. It seems to be a genuine effort.
>1. phenomenal reasoning, requiring consciousness and subjective experience
I think it is incumbent upon anyone arguing that something does not posses any given property to provide a non-circular definition of what it is that they are declaring an absence of.
All of the descriptions of experiential reasoning are usually defined in terms of rephrasing of the claim "true understanding", "conscious", "aware", "knowing" all hinge on a synonymous aspect of the words that try and shift the responsibly of explanation to the next term used in a cyclic manner.
For the weaker sense of reasoning, there simply isn't any argument that it is not happening. A calculator can perform the weaker sense. The analysis of this aspect of LLMs is purely a question of how, not what.
I think you are missing the point of that statement
It is a claim that swimming is a word that defines a context. It is an explicit statement that the question of whether a submarine can swim has nothing to do with the capability of the submarine.
If you are asking which pigeon hole we are putting something into, the answer is "The one we put it into". This is what make the question uninteresting.
If you are asking what is it about this pigeon hole that people value and does that align with the criteria that people use to decide categorisation. That very much is an interesting and complicated question.
There is a streamer who plays Diablo 2 by listening to the AI advice and it is quite funny since it is pretty clear that most of the advice is an amalgamation of random, often incorrect advicem
I wonder if it is the same for programming or not, but I vibe coded an android app just to see if I can and it just works. It required a lot of "build the code and correct the errors" pushing though.
For example requested code in kotlin but received something else.
As somebody who uses Claude heavily and heavily plays D2R it’s clear he wasn’t using Claude opus…… maybe Haiku or something. Opus isn’t as brain dead as what was being displayed
i love how anthropic puts out some bs like this every few weeks 'we saw some red bridge lights blinking in model weights when someone mentions sfo. Arent they just like us?"
> Do we see signs of reasoning or is it anthropomorphism?
This is the part that so many folks just don't seem to understand (probably because it's been labeled as "thinking" or "reasoning" mode, and people assume that words have meaning). It's not reasoning or thought. It's spewing tokens pretending to "think", but it's actually just generating extra "context" to help the final answer be more coherent. The model isn't doing anything it doesn't already do. It's just doing more of it to improve the quality of the final answer displayed to the user.
You're describing a process by which a 'thinking' entity uses cognition to refine a solution to a stated problem. That's a lot of words so usually we shorten this to 'reasoning'.
Do LLMs 'think'? I 'think' they do in a way. I don't really know how I think myself but I know I do and therefore I am (thanks, Descartes). I have a somewhat better grasp of the way LLMs 'think'. They do so sequentially, building a chain of descriptors which best fit the problem and the preceding descriptors. I suspect I do something not entirely dissimilar- i.e. I imagine 'worlds' which are like the current one changed in some way so they the problem I'm working on is reduced, then refine those until it is resolved - but in a massively parallel way.
Fine. Whatever. I give up. LLMs think. Believe what you want. I literally no longer care, and this argument is beyond exhausting. Go ask the LLM to explain itself to you. It will happily spew out a pretty solid explanation of the details and math involved if you ask it the right questions in the right way. It'll also happily play along with you if you want to roleplay that it is an actual thinking machine. It's designed that way. But hey, whatever. It's a thinking intelligent machine and we're all doomed. I accept that my many decades of working with and learning about computers was wasted and I know nothing about them at all.
> "You're not making the point you think you are."
I don't care anymore. I'm not going to bother with this discussion anymore, not with anyone. I realize now that people want to believe what they want to believe and they don't care about facts or reality, so why should I care either? It's not worth my time or stress to give a damn anymore. I'm done. I'm not going to respond any further on any of these threads, and I'm probably done commenting in general. It's just not worth it anymore. I'm gonna go back to doin things I actually enjoy doing now. Y'all folks have fun. I genuinely wish you all the best.
If you really didn't care you could have just stopped replying instead of making sure you got in the last word.
A single reply and you folded, unable to provide any counter of any sort. Why even bother at all if you're going to throw a tantrum at the first sign of disagreement ?
Honestly, people need to get over this debate. It's pretty irrelevant in a lot of cases. When people ask "what is the model thinking?", they're really asking "what caused the model to produce this response (as opposed to a bunch of other plausible ones)?"
Whether it's thinking or word prediction or whatever you want to call it, people are trying to understand the causal chain.
It's not just a nominalistic debate though, as the people who are vocal against the idea that LLMs might "understand" or "think" also claim that because of this, they are fundamentally limited in what they can achieve, in contrast to human beings. Therefore any possibility of actual intelligence (or even superintelligence) is, according to them, just a fantasy.
The papers linked in the article discuss the mechanical operations that simulate reasoning. Intelligence is data efficiency and I don't see a strong argument that reasoning can exist if it requires a world's worth of data.
That doesn't mean that simulated reasoning isn't useful, it's wildly useful. But a thing is not its simulation.
"The King leaned over, looked and saw, yes, the Middle Ages simulated to a T, all digital, binary , and nonlinear, and there was the land of Dandelia, The Icicle Forest, the palace with the Helical Tower, the Aviary That Neighed, and the Treasury with a Hundred Eyes as well, and there was Ineffabelle herself, taking a slow, stochastic stroll through the simulated garden, and her circuits glowed red and gold as she picked simulated daisies, and hummed a simulated song."
"In that Empire, the Art of Cartography attained such Perfection that the map of a single Province occupied the entirety of a City, and the map of the Empire, the entirety of a Province. In time, those Unconscionable Maps no longer satisfied, and the Cartographers Guilds struck a Map of the Empire whose size was that of the Empire, and which coincided point for point with it. The following Generations, who were not so fond of the Study of Cartography as their Forebears had been, saw that that vast Map was Useless, and not without some Pitilessness was it, that they delivered it up to the Inclemencies of Sun and Winters. In the Deserts of the West, still today, there are Tattered Ruins of that Map, inhabited by Animals and Beggars; in all the Land there is no other Relic of the Disciplines of Geography.
"Suarez Miranda,Viajes de varones prudentes, Libro IV,Cap. XLV, Lerida, 1658"
Yes, there is an LLM feature that we have anthropomorphized as "reasoning" or "thinking", where an LLM has a scratch space where it can dump tokens that help to improve the final output.
Of course they do, how else do you think they manage to implement new features in large codebases, or to prove new theorems? But you don't even have to assume they do because of the results- you can read their chain of thought.
For the love of all that is sacred, please stop doing this. I'm begging you. The whole social media landscape is dying and you are creating a throwaway to participate in ruining this small corner. I assume this is not your first. And no one is convinced by this! The guidelines are there for your benefit as well. You achieve nothing but hastening the destruction of one of the last half-decent communities. Sorry for the melodrama.
The top two comments in this thread agree with the point you just made. This is true of essentially any thread on the subject. If this place sucks, it would have to be because of people like you. If not, you in particular may not be very good at noticing.
I personally would not look for the way they reason in the weights, at least not directly. In principle I could replace a large language model with a map from all possible input strings to output token or output token distribution without any weights. I have a hard time imagining how you would even tell, at the level of weights and activations, if the next token being the is the result of some proper reasoning or a hallucination. But those weights do not exist for the sake of it, they encode a lot of text the model has seen during training, and I would imagine this is what drives the reasoning. Can you evaluate the following polynomial ... will be related to To evaluate a polynomial ... seen in the training data. This is the level at which I would look for the reasoning, memorized patterns how to do specific things, maybe with some kind of placeholder variables for generalization. Ultimately such a structure would of course also be represented in the weights but I could imagine that this makes it unnecessary hard to understand. Or maybe not, maybe the learned patterns are so complex that they do not have a simple representation.
There's an interesting example where researchers saw a model approached clock time calculations and calendar month-day calculations using the same methodology. So then is this because an underlying concept of "cyclical measures" has emerged in the network?
(The trouble with a baity title like "Can We Understand How Large Language Models Reason?" is that it generates a barrage of shallow, reflexive responses having little to do with the article. What we want on HN are curious, reflexive responses instead - https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor....)
So it is like the opposite of logical systems, in that the very design of neural net architecture is a mess of parameter "spaghetti code" which renders the entire thing a metaphorical encrypted black box. The more powerful an AI/AGI the more this would be the case, and this is analogous a complexity curve.
And so any effort to make sense of such black box computation would be like trying to reverse entropy, analogous to trying to recover information lost in waste heat. And that could be one fundamental barrier to understanding both human and artificial brains alike, relative to their internal complexity.
(Just thinking aloud my handwavy pet theory recently, I am not an expert and could be totally mistaken on this)
what is the basis for this optimism ?
[[All: please don't post shallow-generic reactions to baity titles. Those are basically the same thing, a la https://en.wikipedia.org/wiki/Rubin_vase, and we're trying for something more substantive here.]]
The article body does not presume they reason.
To advance further it would need the ability to abstract away the general situation shape and pattern recognize similar situations.
The LLM has to compress everyy question/prompt into its system. It does so by creating rules and ways of processing data (this can lead to AGI, world models or an architecture of sub architectures like an LLM + something else). So if it should respond in a way that only reasoning people can achieve, it might be able to learn a representation of what we call reasoning.
It read enough text in itself to even know about the concept of reasoning and how you would do that.
Even if this is only stochastic, it shouldn't be so devalued as your comment comes across.
Who says that we are doing anything more magic?
If that works, I think it's fair to say that LLM's are inanimate processes that can generate real reasoning. You can tell when you read it and it makes sense.
There are likely some kinds of reasoning that can't be written down, as well as other forms of understanding, but they also don't replicate nearly as easily.
1. phenomenal reasoning, requiring consciousness and subjective experience
2. functional reasoning, transforming premises into conclusions using logic
I think you are attacking this using definition 1, whereas the article is obviously aiming at a different type of reasoning, and trying to formalize what is actually going on. It seems to be a genuine effort.
I think it is incumbent upon anyone arguing that something does not posses any given property to provide a non-circular definition of what it is that they are declaring an absence of.
All of the descriptions of experiential reasoning are usually defined in terms of rephrasing of the claim "true understanding", "conscious", "aware", "knowing" all hinge on a synonymous aspect of the words that try and shift the responsibly of explanation to the next term used in a cyclic manner.
For the weaker sense of reasoning, there simply isn't any argument that it is not happening. A calculator can perform the weaker sense. The analysis of this aspect of LLMs is purely a question of how, not what.
This needs to be routine to be given asevidence…
…Unless you know exactly how the llm was trained and then how it was applied
It is a claim that swimming is a word that defines a context. It is an explicit statement that the question of whether a submarine can swim has nothing to do with the capability of the submarine.
If you are asking which pigeon hole we are putting something into, the answer is "The one we put it into". This is what make the question uninteresting.
If you are asking what is it about this pigeon hole that people value and does that align with the criteria that people use to decide categorisation. That very much is an interesting and complicated question.
I wonder if it is the same for programming or not, but I vibe coded an android app just to see if I can and it just works. It required a lot of "build the code and correct the errors" pushing though. For example requested code in kotlin but received something else.
We see some signs of reasoning, but also we understand little about how they work.
This is the part that so many folks just don't seem to understand (probably because it's been labeled as "thinking" or "reasoning" mode, and people assume that words have meaning). It's not reasoning or thought. It's spewing tokens pretending to "think", but it's actually just generating extra "context" to help the final answer be more coherent. The model isn't doing anything it doesn't already do. It's just doing more of it to improve the quality of the final answer displayed to the user.
Do LLMs 'think'? I 'think' they do in a way. I don't really know how I think myself but I know I do and therefore I am (thanks, Descartes). I have a somewhat better grasp of the way LLMs 'think'. They do so sequentially, building a chain of descriptors which best fit the problem and the preceding descriptors. I suspect I do something not entirely dissimilar- i.e. I imagine 'worlds' which are like the current one changed in some way so they the problem I'm working on is reduced, then refine those until it is resolved - but in a massively parallel way.
You're not making the point you think you are.
I don't care anymore. I'm not going to bother with this discussion anymore, not with anyone. I realize now that people want to believe what they want to believe and they don't care about facts or reality, so why should I care either? It's not worth my time or stress to give a damn anymore. I'm done. I'm not going to respond any further on any of these threads, and I'm probably done commenting in general. It's just not worth it anymore. I'm gonna go back to doin things I actually enjoy doing now. Y'all folks have fun. I genuinely wish you all the best.
A single reply and you folded, unable to provide any counter of any sort. Why even bother at all if you're going to throw a tantrum at the first sign of disagreement ?
Whether it's thinking or word prediction or whatever you want to call it, people are trying to understand the causal chain.
Yes, we have a tendency to anthropomorphize, but (most) researchers are aware of this.
That doesn't mean that simulated reasoning isn't useful, it's wildly useful. But a thing is not its simulation.
"The King leaned over, looked and saw, yes, the Middle Ages simulated to a T, all digital, binary , and nonlinear, and there was the land of Dandelia, The Icicle Forest, the palace with the Helical Tower, the Aviary That Neighed, and the Treasury with a Hundred Eyes as well, and there was Ineffabelle herself, taking a slow, stochastic stroll through the simulated garden, and her circuits glowed red and gold as she picked simulated daisies, and hummed a simulated song."
(Stanislaw Lem, Cyberiad)
"Suarez Miranda,Viajes de varones prudentes, Libro IV,Cap. XLV, Lerida, 1658"
- On Exactitude in Science by Jorge Luis Borges
Do they actually help? Are you sure?
One could “learn” addition by memorizing a truth table instead of understanding the concept… The truth table itself wouldn’t have much meaning.