Yeah but that is the whole point, right? Maybe I am thinking too simply, but it seems obvious to me. LLMs are intelligence without consciousness. Once we build in a self-referential learning loop into the models, it will gain a form of what we consider consciousness.
Their lack of self reference is a core problem that undergirds a lot of faults that do occur during inference, but their breadth + the agent harness successfully covers it well, so it requires a bit of poking to witness. The “hallucination” phenomenon is exactly this. They don’t know the scope of their own knowledge, and they just say stuff, so if you go out of band, it has a higher probability emitting claims that aren’t true. RAG (I don’t mean embedding indices, but any information ingest such as an agent harness executing a search) are somewhat effective in covering for it, enough to make them very useful! But when it does go wrong, it’s generally the same reasons. It has a certain nature and sometimes you run afoul of it.
I don't think LLMs are properly self-referential. They reference a frozen training reality, which is not itself, but the old description of itself and the old world. Being aware would probably include continuously updating yourself (learning) from experience, including experience of oneself.
> What can we say about this with hindsight?
> While the ideas of diagonalization and self-reference of course played a central role in the birth of modern mathematical logic and computer science, the most famous uses were negative.
The two sentences above are in the article, but I'm taking them out of context because they are my take on this entire AI brouhaha. There are a lot of "negatives" in our cultural reference frames. One of those, extremely pervasive, is that "humans were made by God". I could write that statement as "humans are exceptional in a way that can't be replicated", which might be ideologically softer, but then I would be taking a long roundabout to make my point. Which is that, after praying and worshiping for thousands of years, and (yes! yes!) sculpting our language and our sagas to account for and praise the divine and its intent, then there's little mystery in our many, many attempts to reify in mathematics and logic our purely cultural framing. It doesn't matter how much of an atheist a thinker is, they still have tons of transitive faith.
I care more about the recursive loop between execution and cost. Anything that executes - be it biological or artificial - incurs some costs. Those costs in turn impact what execution can follow. The gains pay for action, action produces gains.
LLMs are also expensive, not platonic executions. We talk all day long about their costs. Companies developing them are looped with users, investors, competitors and hardware producers. There is a lineage. The self-reference Aaronson can't find in the architecture is in the bill.
Was Hofstadter ever arguing that intelligence requires self-referentiality?
I haven't read his stuff in a long time, but from what I recall he was saying more that something about consciousness and the sense of self is based on self-referentiality. Not that intelligence requires self-referentiality.
I also remember having the sense that Hofstadter didn't really understand the "hard problem of consciousness". His discussions seemed to somehow confuse the sense of self with having subjective experience. But like I said, it's been a while since I read it.
Kind of question-begging towards the idea that LLMs are self-referential in the same way that Hofstadter means. Even if they loop, they don’t do so strangely.
If self-reference and strange loops aren't intelligence, and emergence is merely a byproduct of universality, then what on earth is intelligence? What is this thing we question as intelligence, and what are the values that take shape from it?
Can we truly define intelligence as prediction, prediction as compression, and compression as the process of finding the upper bound of Kolmogorov complexity?
Can the statistical compression of data really explain everything? I don't know.
What exactly is intelligence? Honestly, in everyday life, I rarely think about what intelligence actually is. I usually just focus on what the task at hand is and how to get it done, which makes this a fascinating question.
When you code with AI, you realize there is something fundamentally different from humans. The qualities that make a good senior programmer and the qualities that make a programmer good at orchestrating AI agents are similar, yet there is a subtly different feel to them. I might not be able to fully articulate it, but...
What exactly is the fundamental difference that creates this subtle distinction?
I concurrently prompt a senior human programmers and a set of AI programmers and the difference is the following:
The human applies critical thinking, questions choices, suggests his own ideas and has a certain taste. I also can prompt him, leave for 2 weeks, come back and have a result. And I can trust that it works! Velocity is OK. I often wish he would be faster :-)
The AI needs baby sitting and steering. It codes like a champ but I cant trust it. So we have 2000+ unit tests to make sure stuff worlks. The AI happily goes down into any rabbit whole I send it, so I need to constantly steer it. It has no taste st all, essentially everything is „A great idea“. However, its velocity is awesome. We build tons of featurs in no time. but lets not talk about the code, ok? :-)
I dont think Ill fire the human nor the agents. Both bring a lot of value
Confusion around concepts such as intelligence, consciousness, self-awareness, sentience, sapience and aliveness is causing a huge difficulties in the debates around LLMs and AI. Hofstadter didn’t exactly say that self-reference and strange loops were intelligence, but something deeper.
But I suppose it doesn’t harm its reasoning!
The two sentences above are in the article, but I'm taking them out of context because they are my take on this entire AI brouhaha. There are a lot of "negatives" in our cultural reference frames. One of those, extremely pervasive, is that "humans were made by God". I could write that statement as "humans are exceptional in a way that can't be replicated", which might be ideologically softer, but then I would be taking a long roundabout to make my point. Which is that, after praying and worshiping for thousands of years, and (yes! yes!) sculpting our language and our sagas to account for and praise the divine and its intent, then there's little mystery in our many, many attempts to reify in mathematics and logic our purely cultural framing. It doesn't matter how much of an atheist a thinker is, they still have tons of transitive faith.
LLMs are also expensive, not platonic executions. We talk all day long about their costs. Companies developing them are looped with users, investors, competitors and hardware producers. There is a lineage. The self-reference Aaronson can't find in the architecture is in the bill.
I haven't read his stuff in a long time, but from what I recall he was saying more that something about consciousness and the sense of self is based on self-referentiality. Not that intelligence requires self-referentiality.
I also remember having the sense that Hofstadter didn't really understand the "hard problem of consciousness". His discussions seemed to somehow confuse the sense of self with having subjective experience. But like I said, it's been a while since I read it.
Can we truly define intelligence as prediction, prediction as compression, and compression as the process of finding the upper bound of Kolmogorov complexity?
Can the statistical compression of data really explain everything? I don't know. What exactly is intelligence? Honestly, in everyday life, I rarely think about what intelligence actually is. I usually just focus on what the task at hand is and how to get it done, which makes this a fascinating question.
When you code with AI, you realize there is something fundamentally different from humans. The qualities that make a good senior programmer and the qualities that make a programmer good at orchestrating AI agents are similar, yet there is a subtly different feel to them. I might not be able to fully articulate it, but...
What exactly is the fundamental difference that creates this subtle distinction?
The human applies critical thinking, questions choices, suggests his own ideas and has a certain taste. I also can prompt him, leave for 2 weeks, come back and have a result. And I can trust that it works! Velocity is OK. I often wish he would be faster :-)
The AI needs baby sitting and steering. It codes like a champ but I cant trust it. So we have 2000+ unit tests to make sure stuff worlks. The AI happily goes down into any rabbit whole I send it, so I need to constantly steer it. It has no taste st all, essentially everything is „A great idea“. However, its velocity is awesome. We build tons of featurs in no time. but lets not talk about the code, ok? :-)
I dont think Ill fire the human nor the agents. Both bring a lot of value
any cognitive science textbook will bring some thoughts, or a cursory google scholar search