Robust agents learn causal world models — part 3/8

Cleaned-text code points [28179, 41642). Bibliography omitted here; source.md includes it.

Published as a conference paper at ICLR 2024
4 D ISCUSSION
Here we discuss the consequences for several fields and open questions, as well as limitations.
Causal representation learning. Causal representation learning (CRL) aims to learn representations
of data that capture unknown causal structure (Schölkopf et al., 2021), with the aim of exploiting
causal invariances to achieve better generalisation across domains. Theorems 1 and 2 show that any
method that enables generalisation across many domains necessarily involves learning an (approxi-
mate) causal model of the data generating process—i.e. a causal representation. Hence, our results
provide theoretical justification for CRL by showing it is necessary for strong robustness guarantees.
Causal bounds on transfer learning. As described in Section 3.2, Theorems 1 and 2 imply
fundamental causal constraints on certain transfer learning tasks. For example in the supervised
learning task depicted in Figure 1, identifying regret-bounded policies under covariate and label
shifts requires learning the causal relations between features and labels. Causal discovery problems
such as this are well understood in many settings (V owels et al., 2022), and in general identifying
this causal structure (e.g. that Y → X in Figure 5 a)) is impossible without interventional data
and/or additional assumptions. This connection allows us to convert (im)possibility results for causal
discovery to (im)possibility results for transfer learning. Future work could explore this for smaller
sets of distributional shifts and derive more general causal bounds on transfer learning.
Good regulator theorem. The good regulator theorem is often interpreted as saying that any good
controller of a system must have a model of that system (Conant & Ross Ashby, 1970). However,
some imagination is needed to take this lesson from the actual theorem, which technically only states
that there exists an optimal regulator that is a deterministic function of the state of the system (which
could be trivial, Wentworth (2021)). Our theorem less ambiguously states that any robust agent must
have learned an (approximate) causal model of the environment, as described in Section 3.2. It can
therefore be interpreted as a more precise, causal good regulator theorem.
Emergent capabilities. Causal models enable a kind of general competency—an agent can use a
causal model of its environment to optimise for any given objective functionU(PaU⊆V ) without
additional data (Theorem 3). This could explain how general competence can arise from narrow
training objectives (Brown et al., 2020; Silver et al., 2021). By Theorems 1 and 2, agents trained
to maximise reward across many environments are incentivized to learn a causal world model (as
they cannot generalise without one), which can in turn be used to solve any other decision task in the
same environment. This incentive does not imply that training an agent with a simple reward signal is
sufficient to learn causal world models. E.g. it will still be impossible for an agent to learn a causal
model (and therefore to generalise) if the model is not identifiable from its training data. The question
is then if current methods and training schemes are sufficient for learning causal world models.
Early results suggest that transformer models can learn world models capable of out-of-distribution
prediction (Li et al., 2022, see Section 6 for discussion). While foundation models are capable of
achieving state of the art accuracy on causal reasoning benchmarks (Kıcıman et al., 2023), how they
achieve this (and if it constitutes bona fide causal reasoning) is debated (Zeˇcevi´c et al., 2023).
Causal discovery. Theorems 1 and 2 involve learning the causal structure of the environment
by observing the agent’s policy under interventions. It is perhaps surprising that the response of
this single variable to interventions is sufficient to identify all associative and causal relations in
AncU . Typically, causal discovery algorithms involve measuring the response of many variables to
interventions (V owels et al., 2022). Also, many causal discovery algorithms assume independent
causal mechanisms (Schölkopf et al., 2021), which is equivalent to assuming no agents are present in
the data generating process (Kenton et al., 2023). However, our results suggest that agents could be
powerful resources for causal discovery. In Appendix B we use the proof of Theorem 2 to derive a
causal discovery algorithm for learning causal structure over latents, and test it on synthetic data.
Applicability of causal methods. Causal models have been used to formally define concepts such as
intent (Halpern & Kleiman-Weiner, 2018; Ward et al., 2024), harm (Richens et al., 2022), deception
(Ward et al., 2023b), manipulation (Ward et al., 2023a) and incentives (Everitt et al., 2021), and are
required for approaches to explainability (Wachter et al., 2017) and fairness (Kusner et al., 2017).
Methods for designing safe and ethical AI systems that build on these definitions require causal
models of the data generating process, which are typically hard to learn, leading some to doubt their
practicality (Fawkes et al., 2022; Rahmattalabi & Xiang, 2022). However, our results show that
learning a causal model of the data generating process is necessary for training robust agents, and
8

Published as a conference paper at ICLR 2024
demonstrate that these models can be elicited from these agents. These models could in turn be used
to support causal methods for aligning agents.
Limitations. Theorems 1 and 2 require agents to be robust to a large set of domain shifts (local
interventions on all environment variables). Theorem 2 shows that loosening regret bounds results
in some causal relations being unidentifiable from the agent’s policy. Hence, we expect it is still
possible to learn some casual knowledge of the environment from agents that are robust to a smaller
set of domain shifts, albeit less complete that the full underlying CBN. Finally, our results only apply
to unmediated decision tasks (Assumption 1). We expect Theorems 1 and 2 can be extended to active
decision tasks, as Assumption 1 does not play a major role beyond simplifying the proofs.
5 R ELATED WORK
Several recent empirical works have explored if deep learning models learn ‘surface statistics’ (e.g.
correlations between inputs and outputs) or learn internal representations of the world (McGrath
et al., 2022; Abdou et al., 2021; Li et al., 2022; Gurnee & Tegmark, 2023). Our results offer some
theoretical clarity to this discussion, tying an agents performance to the fidelity its world model,
and that going beyond ‘surface statistics’ to learning causal relations is fundamentally necessary
for robustness. One study in particular (Li et al., 2022) found that a GPT model trained to predict
legal next moves in the board game Othello learned a linear representation of the board state (Nanda,
2023). Further, this internal representation of the board state could be changed by intervening on
the intermediate activations, with the model updating its predictions consistent with the intervention,
including interventions that take the board state outside of the training distribution. This indicates
that the network is learning and utilising a representation of the data generating process that can
support out-of-distribution generalisation under interventions—much like a causal model.
The problem of evaluating policies under distributional shifts has been studied extensively in causal
transportability (CT) theory (Bareinboim & Pearl, 2016; Bellot & Bareinboim, 2022). CT aims to
provide necessary and sufficient conditions for policy evaluation under known domain shifts when all
assumptions on the data generating process (i.e. inductive biases) can be expressed as constraints
on causal structure (Bareinboim & Pearl, 2016). However, deep learning algorithms can exploit a
much larger set of inductive biases (Neyshabur et al., 2014; Battaglia et al., 2018; Rahaman et al.,
2019; Goyal & Bengio, 2022) which in many real-world tasks may be sufficient to identify low regret
policies without requiring causal knowledge. Thus, CT does not imply that agents must learn causal
models in order to generalise unless we assume agents only use causal assumptions to begin with,
which would be proof by assumption. See Appendix G for further discussion.
A similar result to Theorems 1 and 2 is the causal hierarchy theorem (CHT) (Bareinboim et al.,
2022; Ibeling & Icard, 2021), which shows that observational data is almost always insufficient for
identifying all causal relations between environment variables, whereas our results state that the set of
optimal policies is almost always sufficient to identify all causal relations. In Section 3.2 we discuss
the similarities between these theorems, and in Appendix G we discuss their differences.
6 C ONCLUSION
Causal reasoning is foundational to human intelligence, and has been conjectured to be necessary for
achieving human level AI (Pearl, 2019). In recent years, this conjecture has been challenged by the
development of artificial agents capable of generalising to new tasks and domains without explicitly
learning or reasoning on causal models. And while the necessity of causal models for solving causal
inference tasks has been established (Bareinboim et al., 2022), their role in decision tasks such as
classification and reinforcement learning is less clear.
We have resolved this conjecture in a model-independent way, showing that any agent capable of
robustly solving a decision task must have learned a causal model of the data generating process,
regardless of how the agent is trained or the details of its architecture. This hints at an even deeper
connection between causality and general intelligence, as this causal model can be used to find
policies that optimise any given objective function over the environment variables. By establishing a
formal connection between causality and generalisation, our results show that causal world models
are a necessary ingredient for robust and general AI.
9

Published as a conference paper at ICLR 2024
Acknowledgements. We would like to thank Alexis Bellot, Damiano Fornasiere, Pietro Greiner,
James Fox, Matt MacDermott, David Reber, David Watson and Philip Bachman for their helpful
discussions and comments on the manuscript.
REFERENCES
Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders
Søgaard. Can language models encode perceptual structure without grounding? a case study in
color. arXiv preprint arXiv:2109.06129, 2021.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization.
arXiv preprint arXiv:1907.02893, 2019.
Elias Bareinboim and Judea Pearl. Controlling selection bias in causal inference. In Artificial
Intelligence and Statistics, pp. 100–108. PMLR, 2012a.
Elias Bareinboim and Judea Pearl. Transportability of causal effects: Completeness results. In
Proceedings of the AAAI Conference on Artificial Intelligence, volume 26, pp. 698–704, 2012b.
Elias Bareinboim and Judea Pearl. Causal inference and the data-fusion problem. Proceedings of the
National Academy of Sciences, 113(27):7345–7352, 2016.
Elias Bareinboim, Juan D Correa, Duligur Ibeling, and Thomas Icard. On pearl’s hierarchy and the
foundations of causal inference. In Probabilistic and Causal Inference: The Works of Judea Pearl,
pp. 507–556. 2022.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi,
Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al.
Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261,
2018.
Alexis Bellot and Elias Bareinboim. Partial transportability for domain generalization. 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are
few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
Daniel C Castro, Ian Walker, and Ben Glocker. Causality matters in medical imaging. Nature
Communications, 11(1):3673, 2020.
Taco Cohen and Max Welling. Group equivariant convolutional networks. InInternational conference
on machine learning, pp. 2990–2999. PMLR, 2016.
Roger C Conant and W Ross Ashby. Every good regulator of a system must be a model of that
system. International journal of systems science, 1(2):89–97, 1970.
Juan Correa and Elias Bareinboim. A calculus for stochastic interventions: Causal effect identification
and surrogate experiments. In Proceedings of the AAAI conference on artificial intelligence ,
volume 34, pp. 10093–10100, 2020.
A P Dawid. Influence diagrams for causal modelling and inference. International Statistical Review /
Revue Internationale de Statistique, 70:161–189, 2002.
Daniel C Dennett. The intentional stance. MIT press, 1989.
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Exploring
the landscape of spatial robustness. In International conference on machine learning, pp. 1802–
1811. PMLR, 2019.
Tom Everitt, Ryan Carey, Eric D Langlois, Pedro A Ortega, and Shane Legg. Agent incentives: A
causal perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35,
pp. 11487–11495, 2021.
10