Palatable Conceptions of Disembodied Being — part 2/5

Cleaned-text code points [15582, 31226). Bibliography omitted here; source.md includes it.

However, this way of thinking is wrong-headed, from the post-reflective standpoint. The void of inscrutability, as the name suggests, is empty (in the everyday sense). No entities are to be found within it, because the language of consciousness has no purchase there. If the behaviour of an entity is truly inscrutable then, however fascinating it might be in other respects, our concept of consciousness simply cannot be applied to it. If it belongs in the diagram at all, then it belongs down there with the brick, albeit for a different reason. (If we were to plot complexity on a third axis, it might end up a long way from the brick. But complexity is orthogonal to capacity for consciousness.)

So where does this leave the sort of disembodied, mind-like LLM-based artefact that concerns us here? Does that too belong with the brick? Unlike the inscrutable entity imagined above, an LLM-based dialogue agent is very human-like in one respect, its linguistic prowess. But in other respects, it is very exotic indeed, more so than the bee, the octopus, or the cat, with each of whom we share not only the fundamental fact of our embodiment, but also a common biological heritage. For these reasons, the right place for the kinds of disembodied, mind-like entities we are concerned with is the terra incognita where the region of conscious exotica meets the void of inscrutability. But standing here, at the edge of that void, how are we to think of, to speak about, and to treat these artefacts?

One recourse is to the poetic. What we’re aiming for is a kind of truth, but it is not literal truth. These are not factual matters, although the residue of dualistic thinking might incline us to think otherwise. Rather, we are aiming for poetic truth. What we say must ring true, like a powerful image or metaphor in a work of literature. 30 30 30 The exercise of mapping the space of possible minds onto the two dimensions of human-likeness and capacity for consciousness is itself a poetic exercise. Hopefully it has some ring of truth about it. It will surely not withstand close philosophical scrutiny, but it serves a rhetorical purpose here, which is to legitimise a certain way of thinking about some very difficult subject matter. To ring true, it must stand up to a degree of philosophical, and indeed moral, scrutiny. It cannot be mere nonsense, and it should not undermine human values (whatever that might mean). In this spirit, what follows attempts to evoke a form of consciousness that is compatible with the exotic characteristics of contemporary, disembodied AI systems. 31 31 31 See the Appendix for some poetic evocations by LLMs themselves.

### 4 Fragmented Time

William James gives characteristically eloquent expression to the intuition that ordinary human conscious experience is not “a string of bead-like sensations and images, all separate”. 32 32 32 James (1918, vol.1, ch.15) . Rather than a “knife-edge” it is a “saddle-back, with a certain breadth of its own on which we sit perched, and from which we look in two directions into time” (i.e. backwards into the past and forwards into the future).

Husserl, in a similar vein, claims our lived experience necessarily has the sort of tripartite structure James evokes: primal impression (of what is immediately present), retention (of what has just passed), and protention / expectation (of what is about to come). 33 33 33 Husserl (1964) .

Were this not the case, according to James, our consciousness would be “like a glow-worm spark, illuminating the point it immediately covered, but leaving all beyond in total darkness”. James concedes that such a thing is conceivable, that we might “make appropriate speeches, though unaware of any word except the one just on our lips”, that we might make rational decisions “without ever a glimpse of the total grounds of our choice”, but doubts that “a very highly developed practical life [would] be possible under such conditions”.

Anyone flirting with the thought that the language of consciousness should be extended to contemporary AI systems, or to artefacts that resemble them in structure and operation, is bound to ask what form the consciousness of such a system would take, what it would be like for such an artefact. How, for example, might it experience the passage of time? Would its consciousness be temporally confined to a mere glow-worm spark, or would it have a tripartite, saddle-back structure: primal impression, retention, and protention?

#### 4.1 Discreteness and interruptibility

At a mechanistic level, the temporal dynamics of an LLM-based dialogue agent are very different from those of a living animal and its biological brain. In responding to a user’s input, an LLM issues a sequence of tokens, one by one, each token generated by a separate, discrete computation. 34 34 34 Vaswani et al. (2017) . The generation of this sequence is interruptible. That is to say, having generated the n th n^{\rm th} token in a sequence, the computation can be suspended indefinitely before generating the n + 1 th n+1^{\rm th} token (the underlying computational resources perhaps being used for something entirely different in the mean time). The resulting pause, whatever its duration, would have no effect on the LLM’s eventual output, although it might try the patience of the user.

Similar observations can be made at both coarser and finer temporal scales. At a coarser scale, between the n th n^{\rm th} turn and the n + 1 th n+1^{\rm th} turn of a dialogue, the conversation is interruptible, and at a finer scale, even the computation that produces a single token is, in principle, interruptible.

The temporal dynamics of the brain of a living animal, by contrast, are obliged to unfold in synchrony with the physical world. They are continuous, rather than discrete, and are not, in any practical sense, interruptible. A cook boiling a pan, a child catching a ball, a mouse running from a cat; their brains and bodies cannot be paused mid-action, and if they could, the world would not wait around for them to restart; the pan would boil over, the child would fumble the ball, and the cat would catch the mouse. Even while sat in an armchair, resting, or asleep in bed, a person’s brain is a maelstrom of ongoing electrical activity.

It is tempting, at first, to pursue the thought that the fundamentally discrete, interruptible character of the computations underlying an AI system, if built along the lines of a contemporary LLM, would constrain its phenomenology, should it have any, to conform to James’s glow-worm characterisation. But this temptation should be resisted. Indeed, it should be resisted on two levels.

First, an LLM, in generating the next token, takes account of the context window, which contains the conversation so far. This means that information about the system’s immediate past is available and causally potent when it generates the next token. Moreover, the very raison d’etre of its computations is to predict future tokens. So, insofar as we are inclined to speak of a given system’s “experience of the present”, there is nothing structural or architectural to preclude the possibility that this “experience” encompasses an awareness of both the just-passed and the about-to-occur.

In short, there is no obligation, imposed by architecture, to embrace the glow-worm picture of things. On the other hand, if the inherently discrete, interruptible nature of the system, contrasting with the inherently non-interruptible continuous nature of the biological brain, inclines us to reject the saddle-back picture, the glow-worm picture is available (in part thanks to James’s poetic evocation).

Moreover, there are deeper reasons to resist that initial thought. Its very framing presupposes that there are two competing hypotheses, the glow-worm hypothesis and the saddle-back hypothesis, as if both were within our range of reasonable imaginings, and further exploration, further discovery, would enable us to choose between them.

Perhaps that is indeed the right way to think, for certain kinds of artefact. But perhaps, for other kinds of artefact, there is nothing that is within the reach of further exploration, nothing further to be discovered. Perhaps we will extend the language of consciousness to these LLM-like artefacts, but withhold certain concepts from applicability, in this case the temporal character of experience. We might think of this last possibility as discovering, or perhaps opening up, a miniature void of inscrutability, a hole of inscrutability within our conception of an exotic form of consciousness.

#### 4.2 Discontinuity and change

So, insofar as we came to speak of how some LLM-like artefact might experience the passage of time, it’s doubtful that the architectural facts of discreteness and interruptibility would oblige us to speak as if the glow-worm analogy were more fitting than the saddle-back analogy. However, James’s string-of-beads imagery, with a little elaboration, is helpful for evoking the consequences of another exotic aspect of an LLM’s nature, which relates to continuity.

There is a degree of continuity to an embodied animal’s experience that is inherited from various kinds of continuity inherent in the external world it inhabits. The spatial continuity of paths and surfaces, for instance, and the temporal continuity of embodied motion along paths and across surfaces, ensure that what an animal experiences in any given moment is typically very similar to what it experienced immediately prior to that moment. Change, for an animal, whether gradual or abrupt, is perceived against this backdrop of stability.

For a disembodied LLM, by contrast, there is no such continuity. The input to an LLM is a discontinuous sequence of discrete tokens, none of which resembles the tokens that immediately preceded it, and this fact has consequences for the putative experience of any hypothetical LLM-like entity.

James rejects the picture of human consciousness as a “string of bead-like sensations and images” because it suggests that each moment is temporally isolated from every other. But let us re-appropriate the image, setting aside the attributes that, for James, render it inapplicable to human consciousness. (Perhaps the beads are made of a shiny material and each contains a reflection of its neighbours.)

Human consciousness may or may not suit even the modified analogy. But if it were to be likened to a string of beads, each bead would bear a strong similarity to its immediate predecessors, for the reasons set out above. It would be like a line of pearls, all white but with slight variations. The putative consciousness of an LLM-like entity surely would suit the analogy, as it would be constituted by a sequence of discrete moments, thanks to its underlying computational nature. But the LLM’s string of beads would not be like the human’s. Each bead would be different from its neighbours. The whole thing would be less like a line of pearls and more like a necklace of randomly assorted colours, and insofar as change only shows up against a backdrop of stability, change, as humans experience it, would not feature in its consciousness.

### 5 Fractured Selfhood

To what extent has the preceding exercise captured an aspect of what it might be like to be an exotic, disembodied, LLM-like entity? The question, of course, is misguided, insofar as it legitimises the aim of capturing what it’s really like as if it made sense to speak (in a philosophically earnest voice) of such a thing. The aim, rather, is to sketch out meaningful, coherent ways to think and talk about such exotic entities, ways of thinking and talking that accommodate the temptation to extend the language of consciousness to them.

Working so close to the void of inscrutability, we are obliged to bend this language towards the poetic. The trend towards poetic expression is set to continue in this section, which tackles the question of the identity of the subject, insofar as we decide that concept is, or ever could be, applicable to a disembodied, LLM-like artefact. Put differently, the question at hand is whether it is possible to articulate a conception of selfhood that makes sense, poetically if not quite literally, for such exotic entities. 35 35 35 Human selfhood is a multi-faceted concept, as many authors have noted ( James (1918, vol.1, ch.10) ; Neisser (1988) ; Zahavi (2005, ch.5) ). Here, the words “self” and “subject” are used more-or-less interchangeably. Our concern here is not with autobiographical or narrative dimensions of selfhood, which may or may not be applicable, but with aspects of (putative) AI selfhood that intersect with aspects of (putative) AI consciousness.

#### 5.1 The site of the self

Another way to pose the selfhood question is to ask what the words “I” and “me” refer to when they are used by an LLM. Even if we elect not to treat an LLM-like entity as a conscious subject, we can still take first-personal pronouns to refer to something when it uses them. And whether or not we identify this something, this “self”, with a conscious subject, we can still pin down its constitution, and clarify the conditions under which it appears and disappears.

For humans, notwithstanding centuries of philosophical wrangling, the living body, including the brain, can reasonably be considered the site of the self. For contemporary LLMs that lack anything like a body, the picture is considerably less clear, and we can imagine a number of candidates for the site of the self. Let’s attempt to enumerate them.

When an LLM-based dialogue agent uses the word “I”, perhaps the word refers to the underlying model, that is to say the abstract computational entity that comprises a particular neural network architecture plus the set of model weights obtained through training.

Alternatively, the word “I” might refer, not to an abstract entity, but to the deployed model, specifically to the computational process that generated the text that includes the word “I” in question. This computational process would likely be running on highly parallel hardware, possibly distributed over multiple racks of computers, in one or more data centres geographically remote from the user.

However, each such running process can potentially serve multiple users concurrently, meaning that it is effectively running many instances of the underlying model at once. So the subsidiary question arises of whether the word “I” refers to the set of all concurrent instances of the model, or just to the instance serving the specific user in question.

Moreover, each user typically has several conversations ongoing, and even past conversations that the user considers finished can be restarted. So perhaps the word “I” refers to the (somewhat abstract) computational entity comprising the underlying model (its architecture and weights) plus the suspended computational state of an instance of this model representing a single, specific, ongoing conversation.

#### 5.2 Fleeting, flickering selves

To reiterate – at the risk of repetition – the aim here is to articulate a conception of selfhood for an LLM-like entity that rings true poetically. The conception we arrive at should also be philosophically coherent, and should be consistent with the way the LLM works. But we are not trying to discover what the LLM itself actually means by the word “I”, as if the answer to that question could somehow be found inside the model (if only we understood how it worked). Nor are we trying to reveal what the word “I” really refers to, as if that were a matter of metaphysics.