Consciousness in Artificial Intelligence: Insights from the Science of Consciousness — part 13/14

Cleaned-text code points [185540, 201302). Bibliography omitted here; source.md includes it.

As many authors have noted, as we develop increasingly sophisticated AI systems we will face increasingly difficult questions about their moral status (Bryson 2010, Gunkel 2012, Schwitzgebel & Garza 2015, 2020, Metzinger 2021, Shulman & Bostrom 2021). Philosophers disagree about the exact relationship between being conscious and moral status, but it is very plausible that any entity which is capable of conscious suffering deserves moral consideration. If we can reduce conscious suffering, other things being equal, we ought to do so. This means that if we fail to recognise the consciousness of conscious AI systems we may risk causing or allowing morally significant harms.

An analogy with non-human animals helps to illustrate the issue. Humans mistreat farmed animals in very large numbers, motivated by powerful economic incentives. Whether or not this mistreatment depends on a failure to attribute consciousness to these animals, it illustrates the potential problem. In the case of AI, there is likely to be considerable resistance to attributions of consciousness, partly because developers of AI may have powerful economic incentives to downplay concerns about welfare. So if we build AI systems that are capable of conscious suffering, it is likely that we will only be able to prevent them from suffering on a large scale if this capacity is clearly recognised and communicated by researchers. However, given the uncertainties about consciousness mentioned above, we may create conscious AI systems long before we recognise we have done so.

An important point in this context is that being conscious is not the same as being capable of conscious suffering. It is at least conceptually possible that there could be conscious systems that have no valenced or affective conscious experiences—that is, no experiences that feel good or bad to them (Carruthers 2018, Barlassina & Hayward 2019). If it is only valenced experiences that have special moral significance, then the key question for establishing the moral status of AI systems is whether they are capable of such experiences (that is, whether they are sentient, as this term is sometimes used). We have not discussed neuroscientific or philosophical theories of valence or otherwise investigated the prospects of specifically valenced conscious experience in AI systems. Theories of valenced conscious experience are less mature than theories of visual experience, suggesting an important priority for future work. However, we suspect that many possible conscious systems which are also agents will have valenced experiences since agents must evaluate options in order to select actions.

In short, the risk of under-attributing consciousness should be taken seriously. Failing to recognise conscious AI systems as such could lead us to cause unwarranted suffering to many conscious beings.

#### 4.1.2 Over-attributing consciousness to AI

There is also a significant chance that we could over-attribute consciousness to AI systems—indeed, this already seems to be happening—and there are also risks associated with errors of this kind. Most straightforwardly, we could wrongly prioritise the perceived interests of AI systems when our efforts would better be directed at improving the lives of humans and non-human animals. As Schwitzgebel and Garza (2015, 2020) argue, uncertainty about the moral status of AI systems is dangerous because both under- and over-attribution can be costly.

Over-attribution is likely because humans have a well-established tendency to anthropomorphise and over-attribute human-like mental states to non-human systems. A growing body of work examines the tendency of people to attribute consciousness and agency to artificial systems, as well as the factors influencing this attribution (Dennett 1987, Gray & Wegner 2012, Kahn et al. 2006, Krach et al. 2008, Sytsma 2014). There may be various reasons for humans to have an “agent bias”—a natural inclination to ascribe agency, intentions, and emotions to non-human entities—stemming from our evolutionary history (Guthrie 1993).

One possibility is that we might anthropomorphise AI systems because it seems to help us to understand and predict their behaviour—although this impression could be false, with anthropomorphism in fact causing us to make incorrect interpretations. Anthropomorphism allows us to understand and anticipate complex systems like AI using the same cognitive frameworks we use for understanding humans, potentially helping us navigate interactions with AI (Epley et al., 2007). Dennett postulates that individuals employ a cognitive strategy, called the “intentional stance”, to predict and explain the behaviour of various entities, including humans, animals, and artificial systems (Dennett 1987). The intentional stance involves attributing mental states, such as beliefs, desires, and intentions, to an entity to decipher and anticipate its behaviour. Observation and interaction with AI systems often lead to the natural adoption of the intentional stance, especially when their behaviours appear purposeful or goal-directed. This tendency is further amplified when AI systems exhibit human-like characteristics, such as natural language processing, facial expressions, or adaptive learning capabilities (Mazor et al. 2021). Researchers have identified several factors that predispose individuals to anthropomorphise AI systems, including their physical appearance, behaviour, and perceived autonomy (Kahn et al. 2006; Złotowski et al. 2015).

Attributions of agency and consciousness to artificial agents may also be driven by an emotional need for social interaction (Mazor et al. 2021). Individuals who seek social interaction and fulfillment from artificial systems may be more prone to attributing consciousness to them. Assigning human-like traits to AI can help people cope with difficult emotions or situations, such as feeling more comfortable confiding in an AI that appears empathetic or understanding (Turkle 2011). The use of artificial systems such as Replika chatbots as sources of social interaction is already evident.

The recent rapid progress in language model capabilities is particularly likely to drive over-attribution of consciousness, as the case of Blake Lemoine arguably illustrates (Lemoine 2022). Modern LLMs can convincingly imitate human discourse, making it difficult to resist the impression that one is interacting with a conscious agent, especially if the model is prompted to play the role of a person in a conversation (Shanahan et al. 2023).

In addition to the risk of misallocation of resources which we mentioned above, over-attributing consciousness to AI systems creates risks of at least three other kinds. First, if consciousness is sometimes attributed to AI systems on weak grounds, these attributions may undermine better-evidenced claims of consciousness in AI. Observers of debates on this topic may recognise that some attributions are weak, and assume that all are. This effect could be particularly powerful if there are also reasonable concerns that attributions of consciousness to AI are distracting us from addressing other pressing problems. Second, if we judge that a class of AI systems are conscious, this should lead us to treat them differently—training them in different ways, for instance. In principle, this could conflict with work to ensure that AI systems are developed in ways that benefit society. And third, overattribution could interfere with valuable human relationships, as individuals increasingly turn to artificial agents for social interaction and emotional support. People who do this could also be particularly vulnerable to manipulation and exploitation.

Whatever one’s views about the relative importance of these various risks, they amount to a powerful case for research on the prospect of consciousness in AI. If the development of AI is to continue, this research will be crucial: it is risky to judge without good grounds either that no AI systems can be conscious, or that those of a particular class are conscious.

### 4.2 Consciousness and Capabilities

In the popular imagination, consciousness is associated with free will, intelligence and the tendency to feel human emotions, including empathy, love, guilt, anger and jealousy. So our suggestion that conscious AI may be possible in the near-term might be taken to imply that we will soon have AI systems akin to the very human-like AIs depicted in science fiction. Whether this in fact follows depends on the relationships between consciousness and other cognitive traits and capacities. Furthermore, conscious AI systems are more likely to be built if consciousness is (or is expected to be) associated with valuable capabilities in AI. So in this subsection, we briefly consider how consciousness might be related to differences in AI systems’ behaviour and capabilities.

One possible argument for the view that we are likely to build conscious AI is that consciousness is associated with greater capabilities in animals, so we will build conscious AI systems in the course of pursuing more capable AI. It is true that scientific theories of consciousness typically claim that conscious experience arises in connection with adaptive traits, selected for the contributions they make to cognitive performance in humans and some other animals. For example, Baars (1988, 1997) claims that the global workspace “optimizes the trade-off between organization and flexibility” (1988, p. 348), a trade-off that advanced AI systems must also presumably manage.

A weakness of this argument is that human and animal minds are not necessarily a good guide to the connection between consciousness and capabilities in artificial systems. This is because the “design” of animal minds is explained not only by the adaptive value of our capabilities but also by the constraints under which we evolved, which include limits on the quantity and form of data from which we can learn; limits on the amount of energy available to power our minds; and the forms of our ancestors’ minds and the availability of relevant mutations. The space of possible designs for AI systems is different from the space of possible mind designs in biology (Summerfield 2022). So we may well find ways to build high-performing AI systems which are not conscious.

However, some influential AI researchers are currently pursuing projects that aim to increase AI capabilities by building systems that are more likely to be conscious. We have already discussed the ongoing work by Bengio and colleagues which uses ideas from GWT. Goyal and Bengio (2022) write that:

Our aim is to take inspiration from (and further develop) research into the cognitive science of conscious processing, to deliver greatly enhanced AI, with abilities observed in humans thanks to high-level reasoning.

The aim here is to build AI systems that implement at least some of the features underlying consciousness in humans, specifically in order to enhance capabilities. Similarly, LeCun’s proposed architecture for autonomous intelligence has features such as a world model for planning and a “configurator”, described as an element that “takes inputs from all other modules and configures [the other modules] for the task at hand” (LeCun 2022, p. 6). LeCun also suggests that the existence of elements like these in the human brain may be responsible for what he calls “the illusion of consciousness”. These are examples of a widespread and long-established practice in AI of drawing on insights from the cognitive sciences (Hassabis et al. 2017, Zador et al. 2022). So regardless of whether building in consciousness-associated features is the only possible route to greater capabilities, it is one that we are likely to take.

Turning now to the issue of how conscious artificial systems are likely to behave, there are both conceptual and empirical reasons to doubt that being conscious implies having human-like motives or emotions. Conceptually, to be conscious is simply to have subjective experiences. In principle, there could be conscious subjects who had experiences very unlike ours. In particular, there is no apparent conceptual incoherence in the idea of motivationally neutral conscious experiences which lack valence or affect—and even if a conscious subject does have valenced or affective states, these may be triggered by different circumstances from those which trigger our emotions, and motivate them to behave in different ways.

Empirically, the theories of consciousness we have discussed do not generally claim that consciousness implies human-like motives or emotions. One exception is the view that consciousness is only possible in the context of self-maintenance, which we discussed in section 2.4.5, since this presumably entails not only that conscious beings engage in self-maintenance, but that they are motivated to do so. But this view is an outlier. Some theories, such as GWT and PRM, suggest that conscious subjects must be agents and perhaps that their desires are likely to enter into conscious experience—in GWT, the workspace maintains representations that provide important context for processing in the modules, and current desires might fit this description. But these theories are neutral about what motivates conscious subjects. AST is a somewhat more complicated case: Graziano (2013) argues that the use of an attention schema is important for social cognition because it allows one to model attention either in oneself or in others. He further claims that this is a reason to build conscious AI because AI systems with attention schemas will be capable of empathy (Graziano 2017). But the claim here seems to be that implementing AST is necessary for attributing conscious states to others, and, therefore, for being motivated by empathy to help them, not that consciousness is sufficient for empathetic motives.

Many current concerns about the impacts of AI do not turn on whether AI systems might be conscious. For example, the concern that AI systems trained on data that reflects the current structure of society could perpetuate or exacerbate injustices does not turn on AI consciousness. Nor does the concern that AI could enable various forms of repression, or that AI systems could replace human workers in most jobs (although the economic value of replacing human workers may motivate capabilities research which leads to conscious AI, as we have noted). Perhaps most surprisingly, arguments that AI could pose an existential risk to humanity do not assume consciousness. A typical argument for this conclusion relies on the premises that (i) we will build AI systems that are very highly capable of making and executing plans to achieve goals and (ii) if we give these systems goals that are not well chosen then the methods that they find to pursue them may be extremely harmful (see e.g. Hilton 2022). Neither these premises nor the ways in which they are typically elaborated and defended rely on AI systems being conscious.

### 4.3 Recommendations

Several authors have made important practical recommendations concerning the possibility of AI consciousness. These include: Bryson’s (2010) case that conscious AI should be avoided; Metzinger’s (2021) call for a moratorium on work that could lead to conscious AI; Graziano’s (2017) arguments in favour of conscious AI; Schwitzgebel and Garza’s (2020) argument that we should only build particular AI systems if we can either be confident that they will be conscious, or be confident that they will not be; and the detailed recommendations of Bostrom and Shulman (2022). We have not assessed the arguments for these recommendations and do not consider them further here.