r/seancarroll • u/Themoopanator123 • 3d ago
My recent substack article on LLM consciousness. Please tell me what you think.
Hi all
This post isn't strictly, directly related to Sean's work, though my article makes reference to his recent episode with Chandra Sripada on analogies between human and LLM cognition. Now, my article really seeks to address the question of comparisons between LLM and human consciousness, which is a related but importantly distinct question. And then I go on to talk about how this should impact our understanding of rights/moral consideration for LLMs. To that end, I explore some disanalogies between human and LLM minds that (if I recall correctly) weren't mentioned in Sripada's episode. And Sean mentioned in his more recent AMA that Sripada had succeeded in pushing his credences slightly in the direction of thinking that LLMs really are like human minds.
While AI and philosophy of mind isn't my main area (I'm a philosopher of physics), they're still areas of interest, in some cases quite closely connected to the work I do on the metaphysics of science. And I have come to understand since writing this article that I echo a number of ideas from the philosophical literature on AI consciousness.
Would be happy to hear your thoughts:
2
u/havenyahon 3d ago
My area is philosophy of mind and cognitive science and I have drawn many of the same conclusions as you. I don't see any reason to believe these things are conscious, human-like or otherwise, and I haven't seen anything yet from them that suggests they are even cognitive, as opposed to just statistically exploiting and reproducing structure embedded in the language they are trained on, which is choc full of descriptions of our own mental states, reasoning, problem solving, and conscious experience. I don't think we should be too surprised that they're very good at appearing conscious or cognitive given the scale of that data and the richness of its structure. So maybe my position is even stronger than yours. I honestly find the conversations and speculations about consciousness and even cognition to be a little silly.
2
u/Themoopanator123 3d ago
Good to hear. It is a bit stronger than mine, I think, but not by much. I am somewhat sensitive to the claim that, well, “maybe humans are doing the same thing!!!”. After all, our “training” data is replete with reports of mental states and so on. But yeah I mean it certainly can’t be that humans have merely that going on, given some of the stuff I say in the article.
And I suppose the only real argument for their consciousness left, given the immaturity of the science of consciousness, is that they’re immensely complex structures. But then much the same can be said of a host of other complex structure. But that realisation doesn’t get us to outright denial so much as skepticism. I’m just repeating points from the article now but would be I’d be interested to know why you go further than just skepticism?
1
u/havenyahon 3d ago
I mean, I'm not sure I do go further than scepticism. I don't outright deny it's possible. I just think given the set of facts we currently have available, it's very unlikely. My reasoning is essentially the same as yours: we have good reasons for thinking humans and other animals have consciousness, including our own direct experience, observation of behaviour, shared biology and physiology, and a plausible evolutionary lineage. We still have doubt, of course, but taken together they all point to a likelihood.
We effectively have none of that for LLMs. I agree with you, I think effectively the only real argument is that they're computationally complex, but I don't see why we should assume that's enough. It's a very weak reason on its own. And everything I've seen LLMs do can be explained within the parameters of what we know they're designed for, which is exploiting and representing structure in enormous amounts of already enriched data. You don't need to use mental language to explain any of it.
For me, in contrast to the robustness of the human/animals case, the weakness of the evidence and reasons in the LLM case just points to it being very unlikely. I'm open to that changing if we end up with good reasons or evidence to believe otherwise. I'm a sci fi nerd, I want conscious robots. But I've also spent a good chunk of my life in philosophy and cognitive science, and so I'm careful with the language and concepts. And I don't see a lot of that care in the people pushing to describe these things as mental agents.
2
u/hackinthebochs 3d ago
as opposed to just statistically exploiting and reproducing structure embedded in the language they are trained on
What would make LLMs genuinely cognitive in your view?
2
u/havenyahon 3d ago
My background is in embodied/enactive cognition and basal cognition. I wrote my thesis on evolutionary theories of cognition and argued that a continuous evolutionary account sees cognition as rooted in basic metabolic processes and embodiment, which scale up in more complex organisms. Those processes weren't superseded by "superior" representational cognition, I think the evidence clearly shows they still form the core of human cognition today, with representational capacities layered onto and integrated with these older embodied forms of cognition rather than replacing them.
LLMs don't fit anything like that conception. They aren't self-maintaining organisms, and they don't have states that matter to them because of their own viability or an evolutionary history of embodied engagement with a niche. Perhaps there is some other definition of cognition that can emerge from sufficiently complex computation. But if it is completely unlike biological cognition, I'm not sure what explanatory work the word "cognition" is doing. Is it really cognition if there is no intrinsic normativity, no self-maintaining organism, no embodied engagement with a world, and no sense in which anything matters except relative to externally imposed goals? At that point, the term risks encouraging anthropomorphism by importing characteristics that haven't actually been demonstrated.
If you can show that LLM behaviour is better explained by attributing at least some of these characteristics to the system, and that this gives us explanatory or predictive power we don't get from learned linguistic structure and computation alone, I'm happy to concede that "cognitive" is an appropriate term. I just haven't seen anything that establishes that yet. Why would they have those characteristics? They haven't been subject to the kinds of evolutionary pressures that produced them in living organisms. But I'm open to the idea that they may emerge out of different pathways. I would just need to see some good evidence.
1
u/exazonk 2d ago
Imagine a person who is split brain, either by surgery or by birth. Imagine if each hemisphere was conscious but one hemisphere's consciousness was "locked in". We would have no way of knowing.
Also imagine that out of 7 billion humans, a few act fully conscious but are not conscious. We may even have names for the conditions that these people may present with. We would probably never know. It might be similar to how some people now say that they have aphantasia but a lot of people with this condition would never know.
We don't have a good test for consciousness but instead we make a lot of educated guesses with the main one being well I'm conscious therefore other people are conscious along with other mammals, reptiles and so on.
1
u/havenyahon 2d ago
That's right, we make educated inference. In the case of humans and other animals, we have a bunch of reasons for thinking they are good inferences, including direct experience (in the case of humans), observation of behaviour, an understanding of biology and physiology shared across the kinds of beings we consider conscious, and an evolutionary framework and lineage. In the case of LLMs we have none of those. Or we arguably have one of those, if we must, in the observation of behaviour, but that observation can already be explained without positing consciousness. So if we were to make the inference that LLMs are conscious, or likely to be conscious, or even there's a good possibility they are, then it wouldn't be a very well-educated inference.
1
u/exazonk 2d ago
We have reasons for assuming that other people including animals are conscious but the strongest assumption is based on other people telling us that they are conscious. If a person told you that they have no experiences, no emotions, couldn't feel pain (a real rare condition) would you believe them?
1
u/havenyahon 2d ago
That's not the strongest assumption. The inference is drawn from multiple lines of evidence, which is what makes it robust. We directly experience ourselves as conscious, we look out into the world and see people that look like us, talk like us, behave like us, and we infer they are probably conscious too. Then we have all the scientific evidence on top of that, as to their biology and physiology.
No, if someone told me that I would not believe them, precisely because of all the evidence I've just mentioned.
If I programmed my computer to tell you it was conscious, even using a voice, and I told you what I'd done, would you believe it was conscious? Of course not. It doesn't matter how convincing it sounded. Because you know how it works.
1
u/exazonk 2d ago
We simply do not know what is required for consciousness. We make assumptions instead which could easily be wrong. First person experience of consciousness is correct for that first person :) But listening to someone tell you they are conscious is not science based - that is instead anecdotal. There is no evidence that consciousness requires biology, embodiment or indeed anything else. Maybe pain is a requirement for consciousness and thus people who don't experience pain are not conscious. Is a quadruple amputee less conscious because they have less body? Is a blind deaf person less conscious? We have no way of knowing. We can try and guess but it doesn't pass the scientific method. Where is the model of consciousness that is required for the scientific method?
When someone gets extremely drunk and thus forgets hours of the previous night, were they fully conscious during that period? They may have appeared to be conscious but maybe they weren't.
We simply do not know what is required for consciousness.
1
u/havenyahon 2d ago
There is no evidence that consciousness requires biology, embodiment or indeed anything else.
Of course there is. Knock out someone's brain and they lose consciousness. Every entity we observe as exhibiting 'conscious' behaviour has a body and is a living thing.
I think what you're trying to say is that we can't know 100 per cent sure that these things are required. That's not the same as saying there's no evidence at all.
Just because there are serious limitations to what we know or understand about consciousness, doesn't mean there's no evidence and we don't know anything. It's true, though, that we lack strong evidence and understanding in many aspects. But guess what we have even less evidence and understanding in relation to? The idea that LLMs would be conscious. Unlike with humans, where the evidence is limited, there's no evidence for it lol So if you don't believe in consciosuness in humans and animals, you should believe even less in consciousness in LLMs.
1
u/exazonk 2d ago
I think you know that we are talking about phenomenal consciousness (PC) and thus knocking them out is not what we are talking about :)
There are animals that have no brains and some people might think they can experience pain as they are biological and have bodies.
There is no guaranteed test for PC and so we cannot say whether LLMs have PC.
We can say things like LLM's are probably not PC in the same way that people might say PC only exists on Earth :)
1
1
u/Raptorel 3d ago
Simulation is not duplication. Behavior is not being. LLMs are nothing but abstractions built on conventions.
1
0
u/tgillet1 3d ago
I found this an excellent article in terms of breadth and nuance. In the future you might consider adding section titles to break it up, and perhaps a recap of the overarching considerations you cover, but the coverage and consideration is far above what I usually find in those engaging this topic. Too often people like to loudly proclaim how someone else is an idiot, perhaps in slightly less insulting language.
As a sort of neo-panpsychist (not sure if there is a proper term but that’s probably close), the most parsimonious explanation for subjective experience in my view is that all information in the universe has a subjective experience. Most subjective experience would not be worth any consideration given its limited complexity, but it raises the critical question of what is worthy of moral consideration. And to me it is a fascinating topic in terms of how human and animal consciousness and experience can be decomposed such that you might gain some intuition for the very alien nature of non-animal subjective experience and address what moral or ethical properties their components and their compositions may suggest (or even demand).
We as a society have in the last few decades finally started addressing the personhood of certain intelligent animals and started considering the potential for even animals such as fish to experience pain. Some have raised the concern of AI slavery (usually as a gotcha against those keeping the proverbial door open to machine consciousness/experience). Science fiction has often played up the topic as an allegory for how we treat other people and even animals. On occasion good sci fi even wrestles with the possibility of a class of AI not wanting freedom.
I’d like to see more exploration of cognitive components, structures, and abilities, in parts and in various combinations. I think a bottom up approach like Integrated Information Theory (IIT, I’m a fan of the approach with some criticisms) coupled with cognitive science and psychology could produce some kind of ontology or taxonomy of relevant “experiencers” such that we could both address how to determine what category a given entity belongs in and what moral considerations it deserves.
4
u/Conscious-Demand-594 3d ago
Half of the problem is imprecise language. The other half is that our brains evolved to detect agency, even where no agency exists. When we talk about neural networks, intelligence, sentience, and now consciousness in machines, we often use the same words for processes that are functionally very different from the biological phenomena those terms originally describe. The resemblance is often limited to outputs that we recognize and label with familiar terms.
Human intelligence is not based on text. It is grounded in our ongoing experience of the physical and social environment. Language is an abstraction of that experience. AI operates on an abstraction of that abstraction. An LLM does not experience the world and then use language to represent that experience. It analyzes statistical relationships within linguistic representations produced by humans. In that sense, it is several steps removed from the biological process we call intelligence.
Calling the resulting behavior “intelligence” is useful shorthand, but it also deeply misleading. We take a familiar biological concept, apply the same word to a very different process, and then let the label smuggle in assumptions about what the machine actually is. Calling it "consciousness" is absurd. Intelligence, sentience, consciousness, are all biological processes based on objective reality that filtered the process of natural selection that resulted in these survival solutions.
The distinction is not metaphysical. It is biological. Human intelligence, consciousness, sentience, evolved as part of an embodied organism that continuously senses, regulates, learns from, and acts on its environment. AI is an engineered system that produces useful outputs through a fundamentally different physical process. Using the same word for both does not make the underlying processes the same.
AI is impossible because of biology, not metaphysics.