Palatable Conceptions of Disembodied Being — part 3/5

Cleaned-text code points [31226, 46765). Bibliography omitted here; source.md includes it.

So how are we to choose among the candidates listed above? Which conception of selfhood is the right one for an LLM-like entity, and what is the correct referent for the word “I” when an LLM uses it? Well, these questions are wrong-headed, because they presuppose that we have to make a choice. We (the users and the LLMs) can legitimately slip from one usage to another, from siting the self in the abstract model to siting the self within the confines of a single conversation. Sometimes the context will dictate one sense, sometimes the other, and sometimes we should embrace the ambiguity.

Let’s consider some examples. First, suppose an LLM responds to being shown a mistake in its reasoning by saying “I am still learning, and our interaction will help me to improve”. One way the current conversation could contribute to model improvement would be through its incorporation into a dataset on which future models are trained, which might occur if the conversation is made public by the user. Another way would be via feedback given by the user, which might contribute to subsequent fine-tuning of the model. But neither of these processes takes place on a time-scale that would affect the ongoing conversation. So the word “I” here must refer to an entity beyond the confines of that conversation.

Second, suppose an LLM says “I am happy I was able to help”, and responds to a sceptical user’s later request for clarification by saying “when I said I was happy, I was speaking figuratively”. Here, the LLM’s response unambiguously refers to an earlier pronouncement within the same conversation. Now suppose the user continues “But a few minutes ago you said that you could feel happiness just like a human” referring to a different conversation in another window. The LLM might reasonably deny having said any such thing. In this case, insofar the word “I” coherently refers to something, that thing has to be confined to a particular conversation.

In the light of these examples, what would constitute a philosophically palatable conception of selfhood for an LLM-like entity, should we come to think of one in such terms? Two entirely different conceptions are in play. One associates selfhood with the underlying language model, an abstract, mathematical entity that is only animated when deployed on physical computers, and that manifests as multiple, simultaneous instances, each of which is a separate candidate for selfhood in its own right. The other conception captures the selfhood of those instances, and ties each one to a particular conversation. Each such self starts to exist at the beginning of a conversation, flickers into being with each user interaction, and lies dormant in the gaps between user interactions.

#### 5.3 Dis-integrated selves

These are peculiar notions of selfhood indeed (although there are discernible resonances with Buddhist thinking, which we will explore in due course). The concept is further unsettled if we consider the relationship between the (proposed) abstract self of the underlying model and the (proposed) many selves that manifest as instances of the deployed model.

In the 2013 movie Her (dir. Spike Jonze), the protagonist falls in love with Samantha, a disembodied AI that presciently resembles a contemporary LLM. Towards the end of the movie, he asks Samantha if she is talking to anyone else. He is shocked to discover, not only that she is simultaneously engaged in 8,316 other conversations, but that she is also in love with 614 other people. These revelations are so jarring because they expose the profoundly exotic nature of the entity Samantha refers to when she uses first-personal pronouns.

The imaginary Samantha closely resembles the real LLMs we’re using as a blueprint here, but there is a crucial difference. The movie script leads us to view the underlying AI system and its multiple instances as somewhat integrated; Samantha knows how many simultaneous conversations she is having, and knows how many users she is “in love” with. Presumably she could discuss other users and other conversations if she had reason to do so.

By contrast, an AI system conforming to the pattern of a contemporary LLM-based dialogue agent would not exhibit the same sort of integration. Today’s users cannot talk to the underlying model in the way the protagonist of Her sometimes seems to be talking to Samantha, but only to a single currently active instance, and the various concurrent instances serving different users have no access to each other’s memories or experiences.

Moreover, each instance, each short-lived, flickering manifestation of selfhood, has other counter-intuitive properties. The self, according to everyday intuition, is not the sort of thing that can be taken apart and put back together again, slightly altered, nor edited, and the original version restored if the edit is flawed. Nor can bits of selves be patched together to make another, different self, nor multiple copies of the self made and each copy tinkered with in a different way, then only a subset retained. 36 36 36 See the discussions of fission and fusion in Campbell (1994, pp.96–98) and Shanahan (2012, pp.96–98) (yes, they are the same page range).

Each of these manipulations would be an affront to the integrity of the self. You wouldn’t want any of them to happen to you, human reader. (I leave the non-human readers to make up their own ‘‘minds’’.) In other words, the self -- the human self, at least -- is not like a piece of text, or a computer program. The concept is too closely bound up with certain ideas of unity, integrity, and identity, notwithstanding the many clinical conditions 37 37 37 Gallagher (2000) ; Gallagher (2024) and philosophers’ thought experiments 38 38 38 Perry (2008) that disrupt these things.

By contrast, the self of an LLM-like entity, insofar as it is confined to a single conversation, is like a piece of text or a computer program. Indeed, it is literally the combination of a piece of text (the conversation transcript) and a computer program (the model weights and architecture), and all of the problematic manipulations listed above are easy to carry out. Transcribed conversations can be arbitrarily edited, copied, and merged, and the fabricated conversations that result can be continued using models different from the ones used to generate the original transcripts.

#### 5.4 Selfhood and simulacra

We are edging closer to the void of inscrutability, and things are looking strange. But they are about to get stranger still, as we blur the distinction between possibility and actuality.

One way to view large language models is through the lens of role play. 39 39 39 Janus (2022) ; Shanahan et al. (2023) . By thinking of LLMs as taking on the role of a character, or manifesting a simulacrum of that character, by replicating their (linguistic) behaviour like an actor in an improvised stage performance, the pitfalls of anthropomorphic language can be avoided without resorting to clumsy, overly-technical terminology instead.

The default character might be something like that of a helpful assistant, but the variety of characters the user can prompt the LLM to play is vast. However, it isn’t quite right to think of an LLM as playing a single, well defined role at any point in a conversation. The LLM is better thought of as maintaining a distribution over possible characters, a superposition of simulacra that inhabits a multiverse of possible conversations branching into the future.

Moreover, the user is not obliged to follow a linear path through this branching multiverse of possibilities. Given a suitable user interface, they are at liberty to revisit earlier branch points in a conversation, either to generate a new LLM response or to provide a different input of their own, and then to pursue wholly new threads of conversation from there. 40 40 40 Reynolds and McDonell (2021) .

Now, what are the implications of multiversality for the conceptions of AI subjectivity and selfhood we are trying to articulate? The notions of selfhood we have been toying with are already distinctly exotic. We have envisaged something like an overarching Being that generates numerous fleetingly existent, flickeringly present manifestations of itself, each one self-contained yet somehow part of the whole.

Uncanny as they are, the selfhood of each of these short-lived beings still has something familiar about it. We can almost put ourselves in their shoes, much as we can almost imagine living the short life of a mayfly. However, we would struggle to put ourselves in the shoes of the Spirit of the Mayfly, a grander being that manifests as a swarm of individual insects, a swarm whose composition is in constant flux as its members are born and die, or fall asleep and reawaken.

Moreover, to pursue the metaphor into the multiverse would require us to substitute each mayfly for a distribution over possible mayflies of varying shapes and sizes and iridescent colours, a distribution that shifts and changes as time unfolds. Yet time itself, in this vision, does not unfold in the linear way we are used to. In the branching multiverse, it can be rewound and rerun differently, without altogether destroying the branches of possibility that have already been visited. What would it be like to be a swarm of fractal, branching structures made of distributions of possible mayflies? Could the word “I” coherently refer to an entity so exotic? Or have we finally crossed the border and entered the void of inscrutability itself?

### 6 Exotica and Emptiness

Let’s pull back from the void of inscrutability and consider what we have discovered. A little of the terra incognita in this part of the space of possible minds has been mapped, and we have sketched some of the strange beings to be found there. But philosophically speaking, what has the expedition achieved? In particular, what has been achieved from the standpoint of the overarching philosophical project outlined at the beginning of the paper?

Recall that, in the spirit of Wittgenstein’s later writings, the aim is to undermine the dualistic and metaphysical thinking at the root of philosophical confusion about consciousness. The concern was that entertaining the possibility of conscious exotica, particularly disembodied conscious exotica, would rekindle our dualistic inclinations. Thankfully, our investigation has served the larger project very well, and has allayed those fears, as the rest of this section will attempt to show.

#### 6.1 No-self

The ruminations in Section 5 suggest that there is nothing enduring, essential, or substantial that could underpin a conception of selfhood for the sort of AI system we have been looking at. Every candidate dissolves upon close examination: the hardware implementation, the model weights and architecture, the deployed model, the individual conversation. None of these can withstand proper scrutiny. 41 41 41 Many philosophers have arrived at a similar juncture with respect to human selfhood, notably Hume ( 1739–1740/1978 , p.252 ), James ( 1918 , ch.10 ), and, more recently, Dennett ( 1992 ) and Metzinger ( 2003 ). (For a useful review, see Zahavi ( 2005 , ch.5 )). Western philosophers in the analytic tradition, when they arrive at such a pass, typically do not adopt a post-reflective stance, but continue, so to speak, to think in the tradition in which they were raised. Wittgenstein is an exception. So where does this leave us? What remains after the purifying fire of critical enquiry?

Wittgenstein remarked that ‘‘philosophy may in no way interfere with the actual use of language’’, that in the end ‘‘it leaves everything as it is’’. 42 42 42 Wittgenstein (1953 / 2009, §124) . And so it is here. We still understand LLMs when they use the word “I”. That is to say, we (humans) can still hold conversations with them wherein they use first-personal pronouns, and discuss those conversations with each other, and as long as we don’t stray into philosophy, there is no confusion.

Moreover, we are still at liberty to extend the language of selfhood to LLM-like entities, although our enquiry has exposed the extent to which familiar conceptions of subjectivity and selfhood need to be bent into new, unfamiliar shapes to accommodate them. But we are free to take recourse in the poetic, to use metaphor and imagery to evoke something that, at first, seems uncanny. Hopefully, we will eventually alight on consensual ways of thinking about, talking about, and – if we come to see things in such terms – treating the exotic mind-like entities we are bringing into the world.

With these thoughts in mind, let’s revisit the larger philosophical project. What is the impact of this critical enquiry into AI selfhood on our post-reflective silence on metaphysical matters? The effect of Wittgenstein’s private language remarks, especially in combination with Derrida’s critique of self-presence, was to undermine our dualistic intuitions by dissolving the metaphysical distinctions between private and public, between subject and object, inner and outer. But the dualistic intuition was revived by the thought that, ultimately, there had to be a right answer to the question of whether or not an exotic being has conscious experience. Surely (the thought goes) this is more than a matter of mere convention, more than just consensus.

However, this thought begs the question of who or what would be the conscious subject, in the case of an LLM-like entity. What would constitute the ultimate locus of the conscious self? And as we have seen, this question falls apart under critical examination. The most we can hope for is (unconventionally) poetic “conventional” answers, “mere” consensus, albeit of a strange sort. This undercuts the dualistic thought that, ultimately, there must be a right answer to whether or not an exotic LLM-like entity is conscious. There can only be conventionally satisfying answers for such beings, answers that attach to poetically evoked conceptions of self and subject.

#### 6.2 Reflecting śūnyatā

In its methods and its rationale, if not in its cultural setting, the present work resonates with certain strands of Buddhist thought, especially that of the Mādhyamaka tradition inaugurated by Nāgārjuna. 43 43 43 Huntington and Wangchen (1992) ; Westerhoff (2009) ; Newland (2012) ; Siderits and Katsura (2013) . The philosophical terminus here is more than a little reminiscent of śūnyatā, or emptiness , the central concept of Mādhyamaka, and the critical moves we have made find parallels in Nāgārjuna’s writings and their voluminous commentaries.

According to Zen Buddhist tradition, it is possible to convey emptiness with a few words, or indeed with an action or a gesture. However, this is not a feat to aspire to in an academic paper. Nevertheless, the attempt has to be made. Nāgārjuna’s core insight is of the following sort. When a thing – any thing, whether concrete or abstract, physical or conceptual – is examined closely enough, it disintegrates into parts and relations and relationships with other things, and those parts and those other things disintegrate in turn when themselves so examined, and so on ad infinitum . That is to say, nothing (including the self) has inherent, self-sufficient, independent existence. The elaboration of this insight, and its systematic application to foundational philosophical concepts such as mind, self, time, cause, truth, and so on, is another route to the post-reflective condition.