Consciousness in Artificial Intelligence: Insights from the Science of Consciousness — part 8/14

Cleaned-text code points [108148, 123164). Bibliography omitted here; source.md includes it.

The third condition is that the system must learn to produce goal-conducive outputs. This point is emphasised by Dretske (1988, 1999), who argues that a system’s output is only an action if it is explained by the system’s own sensitivity to the benefit of producing that output when receiving a given input. A system’s being sensitive to benefits in this way is manifested in its learning to produce beneficial outputs. Dretske’s idea is that this distinguishes some of the behaviour of animals, which he thinks of as exhibiting agency, from the behaviour of plants, which have evolved to respond to stimuli in particular ways, rather than learning to do so, and artefacts like thermostats, which have been designed to produce particular responses. In a similar vein, Russell and Norvig say that an agent lacks autonomy if its success depends on its designer’s prior knowledge of the task (2010, p. 39).

Reinforcement learning (RL) research explicitly aims to build artificial agents which pursue goals (Sutton & Barto 2018), and typical RL systems meet all three of these conditions. In RL, the task is to maximise cumulative reward over an episode of interaction with an environment in which the system’s outputs affect its subsequent inputs. So there is a strong case that typical RL systems meet substantive criteria for agency (Butlin 2022, 2023). 10 10 10 A possible exception is so-called “bandit” systems, which learn from reward signals in environments in which outputs do not affect subsequent inputs. However, this is not to say that RL is necessary for agency—there are other methods by which systems can learn from feedback to more effectively pursue goals. It is also important to note that the criteria for agency suggested here are relatively minimal.

According to UAL advocates, there are two senses in which agents’ goals or values can be flexible (Bronfman et al. 2016, Birch et al. 2020). One way is that the agent can be capable of learning new goals, such as through classical conditioning, in which a novel stimulus can come to be valued through association with one that the agent already values. The other is that the agent’s goals and values can be sensitive to its changing needs, as in cases in which animals’ preferences change depending on their homeostatic condition. A requirement for flexibility of either kind would be somewhat stronger than the requirement for agency, although even very simple agents tend to be capable of either classical conditioning or habit learning, in which new actions are reinforced. There are better arguments that the second form of flexibility is connected to consciousness: it may require a centralised architecture in which multiple sources of value-relevant information are integrated, and it would also mean that the motivational significance of states of different kinds, relating to different goals or values, must be compared. One suggestion in the scientific and philosophical literature is that conscious valence—that is, the degree to which conscious experiences are pleasurable or unpleasant—could constitute a “common currency” making such comparisons possible (Cabanac 1992, Carruthers 2018). So it is arguably flexible responsiveness to competing goals that is most compelling.

One way to spell out the idea of intentional agency, in which action depends on rational relations between belief-like and desire-like states, is through the distinction made in animal behaviour research between “goal-directed” and “habitual” behaviour (Heyes & Dickinson 1990). In the goal-directed case, the animal can learn independently about the value of an outcome and about actions that might lead to it. It can then combine this knowledge with information about other actions and outcomes, in instrumental reasoning, to make a choice. Action, therefore, depends holistically on the animal’s “beliefs” and “desires”. Computational neuroscience interprets goal-directed action selection in animals as an implementation of model-based RL (Dolan & Dayan 2013). On this understanding, the intentional agency which Hurley emphasises is similar to the form of agency which is required by PRM. What matters for PRM is that the agent relies on the accuracy of a set of representations that are belief-like in that they are used in instrumental reasoning for action selection. This gives reality-monitoring its function: accurate representations in perceptual systems, caused by sensory stimulation, should be used to update this set, while representations without this connection with reality should not.

## 2.4.5(b) Embodiment

On both the minimal and intentional conceptions of agency, it seems possible for a system to be an agent without being embodied. For example, consider AlphaGo, the first AI system to beat the world’s best human Go players (Silver et al. 2016). AlphaGo was an agent but lacked the features which seem to distinguish embodied systems. Embodied systems are located at particular positions in their environments, and the actions and observations available to them are constrained by their positions. They also typically have relatively complex effectors, which they must continuously control over extended periods in order to perform effective actions. Their outputs are movements, and these have some direct and systematic effects on their inputs—for example, turning one’s head has a systematic effect on visual input—as well as less direct effects. These ideas are described in a philosophical account of embodiment by Clark (2008, p. 207), who writes that:

…the body is … the locus of willed action, the point of sensorimotor confluence, the gateway to intelligent offloading, and the stable (although not permanently fixed) platform whose features and relations can be relied upon (without being represented) in the computations underlying some intelligent performances.

The point about “intelligent offloading” here is a reference to the thesis of embodied and extended cognition: embodied agents can exploit properties of their bodies and environments in many ways to make the cognitive tasks they face more tractable, such as by recording information in the environment.

One aspect of embodiment which is thought to be connected with consciousness is having what Hurley (1998) calls a “perspective”. This also requires agency. According to Hurley,

Having a perspective means in part that what you experience and perceive depends systematically on what you do, as well as vice versa. Moreover, it involves keeping track … of the ways in which what you experience and perceive depends on what you do. (p. 140)

For embodied agents that move through their environments, sensory inputs can change either because the environment changes, or because the agent changes its position in the environment, either actively or passively. To distinguish these cases, agents must keep track of their own active movements, and learn how these affect inputs. This will allow them to predict the sensory consequences of their own actions, which will typically be tightly coupled to movements, and thus distinguish them from exogenous changes in the environment. Passive movements can also be distinguished because these will cause the kinds of changes in input which are characteristic of movement in the absence of corresponding outputs. These functions involve the agent’s implicitly distinguishing between a self, located in a moving body, and an environment in which its movement takes place. Consciousness arguably requires that the subject has a single perspective or point of view on the environment, and this account aims to explain how embodied agency gives rise to such a perspective (Hurley 1998, Merker 2005, Godfrey-Smith 2019).

Relatedly, according to the sensorimotor theory of perceptual consciousness, conscious experiences are activities of interaction with the environment which constitute exercises of implicit, practical sensorimotor knowledge (Hurley 1998, O’Regan & Noë 2001, Noë 2004, Kiverstein 2007). A simple application of sensorimotor knowledge is moving one’s head in order to see an object from a different perspective. Like Hurley’s proposal, this theory implies that learning a model of output-input contingencies and using this model in perception is a necessary condition for consciousness. The midbrain theory also suggests that consciousness is associated with the presence of an integrated model of the embodied self in its environment, used in perception, action selection and control.

However, to capture the idea of embodiment as opposed to mere agency, we need to specify further features of the output-input model—which might also be called a “transition model” or a “forward model”. AlphaGo, our example of a non-embodied agent, used a model of this form in Monte Carlo tree search, a planning algorithm that involves evaluating the expected consequences of possible actions.

One way in which output-input models in embodied systems may be distinctive is by representing the direct and systematic effects that movements have on sensory inputs. This condition is not necessarily satisfied by Go-playing systems, because each of the inputs they receive may be affected by their opponents’ moves. Furthermore, embodied systems may also be distinctive in the way they use such models. The employment of an output-input model in perception to distinguish endogenous from exogenous change is one possible example of a characteristic use because it is required when sensory input depends on the position and orientation of a moving body.

A second way of using output-input models which may be distinctive of embodied systems is in motor control. Embodied systems often have effectors with multiple degrees of freedom which must be controlled in precise and responsive ways, and it has been argued that forward models (as they are called in this literature) have several uses which are specific to this context (Miall & Wolpert 1996, McNamee & Wolpert 2019). In particular, forward models can help embodied agents to estimate and adjust the positions of their effectors in the course of action, by providing a representation of the expected position at each moment. Discrepancies between these expectations and either sensory feedback or internal representations of goal states could be used for online adjustment.

In addition to agency and embodiment, a further possible necessary condition for consciousness is that conscious systems must be self-producing, self-maintaining, “autopoietic” systems (Maturana & Varela 1991, Thompson 2005, 2007, Seth 2021, Aru et al. 2023). That is, they must sustain their existence and organisation through their own ongoing activity. This feature is characteristic of living things, which continually repair themselves and homeostatically regulate their temperatures and the balance of chemicals present in their tissues. Self-maintaining activity usually, perhaps always, involves “proto-cognitive” processes of sensing and responding (Godfrey-Smith 2016). Advocates of this idea refer to concepts such as agency, selfhood and autonomy in arguing that self-maintenance is necessary for consciousness. For instance, Thompson (2005) writes:

This self-producing organization defines the system’s identity and determines a perspective or point of view in relation to the environment. Systems organized in this way enact or bring forth what counts as information for them; they are not transducers or functions for converting input instructions into output products. For these reasons, it is legitimate to invoke the concepts of selfhood and agency to describe them. (p. 418)

Those who are also sympathetic to predictive processing, such as Seth (2021), think of self-maintenance in living tissues as a similar process to prediction error minimisation in the brain—they are both instances of free energy minimisation, or “self-evidencing” (Hohwy 2022). More prosaically, engaging in self-maintenance gives an extra reason for systems to model their own states, is related to having flexible goals, and arguably adds a dimension of valence to representations of the self and environment. On this last point, however, it is not clear why agency in the service of self-maintenance should be distinguished from agency directed at “external” goals.

Building on this line of thought, Godfrey-Smith (2016) argues that self-maintaining activity is only at all readily possible by virtue of the way that molecules behave at the nanometre scale when immersed in water. There is continual random activity at this scale, in this context, which can be marshalled to support metabolic processes. On this basis, Godfrey-Smith suggests that artificial systems can only bear coarse-grained functional similarities to living organisms and that these will not be sufficient for consciousness. So another proposed necessary condition is that a conscious system must undergo metabolic processes realised at the nanoscale. Man and Damasio (2019) also suggest that self-maintenance, and thus consciousness, may depend on systems’ specific material composition.

## 2.4.5(c) Agency and embodiment indicators

The proposal that consciousness depends on material composition is clearly incompatible with computational functionalism, so we can set it aside. However, the compatibility of the other proposals we have discussed with computational functionalism is a more complicated matter. The issue is the same in each case: natural formulations of the putative indicators make reference to conditions external to the system. This is incompatible with computational functionalism if a similar system could perform the same computations in different external conditions.

For example, consider the claim that self-maintenance is necessary for consciousness, and suppose that for a system to be self-maintaining in the relevant sense, it must persist in part because it keeps track of certain inputs or internal states and acts to regulate them. It seems that for any system that does this, there could be another that performs the same computations—keeping track of the same inputs or internal states and using the same outputs to regulate them—but which does not persist for this reason. Perhaps the system works to keep an “energy measure” from dropping too low, but in fact, it would persist even if this measure fell to zero, and its energy is supplied by an external operator in a way that does not depend on its behaviour.

If agency or embodiment are necessary for consciousness, these conditions may be incompatible with computational functionalism for similar reasons. These conditions require that the system’s inputs are sensitive to its outputs. But in principle, it is possible for a system to exist in an environment in which its inputs do not depend on its outputs, and yet receive, by chance, patterns of inputs and outputs which are consistent with this dependency. Such a system might perform the same computations as one that genuinely interacted with its environment.