Robust agents learn causal world models — part 4/8

Cleaned-text code points [41642, 55614). Bibliography omitted here; source.md includes it.

Published as a conference paper at ICLR 2024
Abolfazl Farahani, Sahar V oghoei, Khaled Rasheed, and Hamid R Arabnia. A brief review of domain
adaptation. Advances in Data Science and Information Engineering: Proceedings from ICDATA
2020 and IKE 2020, pp. 877–894, 2021.
Jake Fawkes, Robin Evans, and Dino Sejdinovic. Selection, ignorability and challenges with causal
fairness. In Conference on Causal Learning and Reasoning, pp. 275–289. PMLR, 2022.
Alison Gopnik, Laura Schulz, and Laura Elizabeth Schulz. Causal learning: Psychology, philosophy,
and computation. Oxford University Press, 2007.
Anirudh Goyal and Yoshua Bengio. Inductive biases for deep learning of higher-level cognition.
Proceedings of the Royal Society A, 478(2266):20210068, 2022.
Sharut Gupta, Stefanie Jegelka, David Lopez-Paz, and Kartik Ahuja. Context is environment, 2023.
Wes Gurnee and Max Tegmark. Language models represent space and time. arXiv preprint
arXiv:2310.02207, 2023.
Joseph Halpern and Max Kleiman-Weiner. Towards formal definitions of blameworthiness, intention,
and moral responsibility. In Proceedings of the AAAI Conference on Artificial Intelligence ,
volume 32, 2018.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common
corruptions and perturbations. arXiv preprint arXiv:1903.12261, 2019.
Ronald A Howard and James E Matheson. Influence diagrams. Decision Analysis, 2(3):127–143,
2005.
Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlinear
causal discovery with additive noise models. Advances in neural information processing systems,
21, 2008.
Duligur Ibeling and Thomas Icard. A topological perspective on causal inference. Advances in
Neural Information Processing Systems, 34:5608–5619, 2021.
Zachary Kenton, Ramana Kumar, Sebastian Farquhar, Jonathan Richens, Matt MacDermott, and Tom
Everitt. Discovering agents. Artificial Intelligence, pp. 103963, 2023.
Emre Kıcıman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large language
models: Opening a new frontier for causality. arXiv preprint arXiv:2305.00050, 2023.
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of zero-shot
generalisation in deep reinforcement learning. Journal of Artificial Intelligence Research , 76:
201–264, 2023.
Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. Advances in
neural information processing systems, 30, 2017.
Shane Legg and Marcus Hutter. Universal intelligence: A definition of machine intelligence. Minds
and machines, 17:391–444, 2007.
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Watten-
berg. Emergent world representations: Exploring a sequence model trained on a synthetic task.
arXiv preprint arXiv:2210.13382, 2022.
Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis
Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in
AlphaZero. Proceedings of the National Academy of Sciences, 119(47):e2206625119, 2022.
Christopher Meek. Strong completeness and faithfulness in bayesian networks. arXiv preprint
arXiv:1302.4973, 2013.
Nicolai Meinshausen. Causality from a distributional robustness point of view. In 2018 IEEE Data
Science Workshop (DSW), pp. 6–10. IEEE, 2018.
11

Published as a conference paper at ICLR 2024
Jovana Mitrovic, Dino Sejdinovic, and Yee Whye Teh. Causal inference via kernel deviance measures.
Advances in neural information processing systems, 31, 2018.
Joris M Mooij, Jonas Peters, Dominik Janzing, Jakob Zscheischler, and Bernhard Schölkopf. Distin-
guishing cause from effect using observational data: methods and benchmarks. The Journal of
Machine Learning Research, 17(1):1103–1204, 2016.
Neel Nanda. Actually, othello-gpt has a linear emergent world model, Mar 2023. URL <https:
//neelnanda.io/mechanistic-interpretability/othello>.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the
role of implicit regularization in deep learning. arXiv preprint arXiv:1412.6614, 2014.
Masashi Okamoto. Distinctness of the eigenvalues of a quadratic form in a multivariate sample. The
Annals of Statistics, pp. 763–765, 1973.
Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, 2 edition
edition, 2009. ISBN 9780521895606.
Judea Pearl. Theoretical impediments to machine learning with seven sparks from the causal
revolution. arXiv preprint arXiv:1801.04016, 2018.
Judea Pearl. The seven tools of causal inference, with reflections on machine learning. Communica-
tions of the ACM, 62(3):54–60, 2019.
Judea Pearl and Elias Bareinboim. Transportability of causal and statistical relations: A formal
approach. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 25, pp.
247–254, 2011.
Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of
robotic control with dynamics randomization. In 2018 IEEE international conference on robotics
and automation (ICRA), pp. 3803–3810. IEEE, 2018.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua
Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference
on Machine Learning, pp. 5301–5310. PMLR, 2019.
Aida Rahmattalabi and Alice Xiang. Promises and challenges of causality for ethical machine
learning. arXiv preprint arXiv:2201.10683, 2022.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel
Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist
agent. arXiv preprint arXiv:2205.06175, 2022.
Hans Reichenbach. The direction of time, volume 65. Univ of California Press, 1956.
Jonathan Richens, Rory Beard, and Daniel H Thompson. Counterfactual harm. Advances in Neural
Information Processing Systems, 35:36350–36365, 2022.
Donald B Rubin. Causal inference using potential outcomes: Design, modeling, decisions. Journal
of the American Statistical Association, 100(469):322–331, 2005.
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij.
On causal and anticausal learning. arXiv preprint arXiv:1206.6471, 2012.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner,
Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the
IEEE, 109(5):612–634, 2021.
Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato,
and Zac Kenton. Goal misgeneralization: Why correct specifications aren’t enough for correct
goals. arXiv preprint arXiv:2210.01790, 2022.
12

Published as a conference paper at ICLR 2024
Zheyan Shen, Peng Cui, Kun Kuang, Bo Li, and Peixuan Chen. Causally regularized learning
with agnostic data selection bias. In Proceedings of the 26th ACM international conference on
Multimedia, pp. 411–419, 2018.
David Silver, Satinder Singh, Doina Precup, and Richard S Sutton. Reward is enough. Artificial
Intelligence, 299:103535, 2021.
Steven A Sloman and David Lagnado. Causality in thought. Annual review of psychology , 66:
223–247, 2015.
Peter Spirtes, Clark N Glymour, Richard Scheines, and David Heckerman. Causation, prediction,
and search. MIT press, 2000.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu
Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable
multimodal models. arXiv preprint arXiv:2312.11805, 2023.
Matthew J V owels, Necati Cihan Camgoz, and Richard Bowden. D’ya like dags? a survey on
structure learning and causal discovery. ACM Computing Surveys, 55(4):1–36, 2022.
Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening
the black box: Automated decisions and the GDPR. Harv. JL & Tech., 31:841, 2017.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun
Zeng, and Philip Yu. Generalizing to unseen domains: A survey on domain generalization. IEEE
Transactions on Knowledge and Data Engineering, 2022.
Francis Rhys Ward, Tom Everitt, Francesco Belardinelli, and Francesca Toni. Honesty is the best
policy: defining and mitigating AI deception. In Thirty-seventh Conference on Neural Information
Processing Systems, 2023a.
Francis Rhys Ward, Francesca Toni, and Francesco Belardinelli. Defining deception in structural
causal games. In Proceedings of the 2023 International Conference on Autonomous Agents and
Multiagent Systems, pp. 2902–2904, 2023b.
Francis Rhys Ward, Matt MacDermott, Francesco Belardinelli, Francesca Toni, and Tom Everitt. The
reasons that agents act: Intention and instrumental goals. arXiv preprint arXiv:2402.07221, 2024.
John Wentworth. Fixing the good regulator theorem. https://www.alignmentforum.org/
posts/Dx9LoqsEh3gHNJMDk/fixing-the-good-regulator-theorem , 2021. Ac-
cessed: 2023-10-17.
Garrett Wilson and Diane J Cook. A survey of unsupervised deep domain adaptation. ACM
Transactions on Intelligent Systems and Technology (TIST), 11(5):1–46, 2020.
Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning—a
comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis
and machine intelligence, 41(9):2251–2265, 2018.
Matej Zeˇcevi´c, Moritz Willig, Devendra Singh Dhami, and Kristian Kersting. Causal parrots: Large
language models may talk causality but are not causal. arXiv preprint arXiv:2308.13067, 2023.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and
Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1):43–76,
2020.
13

Published as a conference paper at ICLR 2024
A P RELIMINARIES
A.1 S ETUP AND ASSUMPTIONS
The environment is described by a set of random variables C = {C1,C 2,...,C N}, which in
combination with the decision D and utility nodes U define the state space for the CID V =
C∪{D,U}. In out notation individual variablesCi∈C are given indexes, whereas set of variables
are indexless and bold, and we useV =v as short hand for the joint state of the variables in a set
C. The joint probability distribution P (C =c,D = d,U = u) describes the statistical relations
between environment variables. Bayesian networks factorise joint probability distributions according
to a graphG (Pearl, 2009).
Definition 1 (Bayesian networks). A Bayesian networkM = (G,P ) over a set of variablesV =
{V1,...,V n} is a joint probability distributionP (V ) that factors according to a directed acyclic
graph (DAG)G, i.e.P (V1,...,V n) =Qn
i=1P (Vi| PaVi), where PaVi are the parents ofVi inG.
The distributions and statistical relationships between variables may change as a result of external
interventions applied to a system. Hard interventions set a subset C′ ⊆ C of the variables to
particular values c′, denoted do(C′ = c′) or do(c′). Naively, one joint probability distribution
Pdo(c′) would be needed to describe the updated relationship under each possible intervention do(c′).
Fortunately, all interventional distributions can be derived from a single Bayesian network, if G
matches the causal structure of the environment (i.e. has an edgeVi→Vj whenever an intervention
onVi directly influences the value of another variableY , and lacks unmodeled confounders; Spirtes
et al., 2000; Pearl, 2009). When this holds, we call the Bayesian network causal andG a causal
graph. With respect to the causal graphG we denote the direct causes (parents) ofVi as Pai, the set of
all causes (ancestors) Anci, and the variables thatVi directly causes (children) Chi and descendants
Desci as the set of all downstream variables. Note in particular that Anci and Desci refer to proper
ancestors and descendants, i.e. Vi̸∈ Anci andVi̸∈ Desci. We denote a causal Bayesian network
(CBN) asM = (P,G ) whereP is the joint andG is the directed acyclic graph (DAG) describing the
causal structure of the environment. Further, the interventional distributionPdo(v′) is given by the
truncated factorisation
Pdo(v′)(v) =
(Q
i:vi̸∈v′P (vi| pavi) ifv consistent withv′
0 otherwise.
Equivalently, the effect of interventions can be computed by adding an extra node ˆX and edge
ˆVi→ Vi for each node Vi∈ V (Correa & Bareinboim, 2020; Dawid, 2002). Intervening on Vi
then corresponds to conditioning on ˆVi in the extended graph. More general, soft interventions
σ = P′(Vi| Pa∗
i ) replace the conditional probability distribution for Vi with a new one, possibly
using a new parent set Pa∗
o as long as no cycles are introduced in the graph (Correa & Bareinboim,
2020). The modified environment is denotedM(σ).
General soft interventions cannot be defined without prior knowledge of the causal graph G. For
example, the soft interventionσY =P′(y|x) is incompatible with the causal structureY →X as it
would introduce a causal cycle, and so an agent’s policy may not be well defined with respect to this
intervention. We therefore focus our theoretical analysis on a subset of the soft interventions, local
interventions, that can be implemented without assuming knowledge ofG.
Definition 2 (Local interventions). Local intervention σ onVi∈ V involves applying a map to
the states of Vi that is not conditional on any other endogenous variables, vi7→ f(vi). We use
the notationσ = do(Vi =f(vi)) (variableVi is assigned the statef(vi)). Formally, this is a soft
intervention onVi that transforms the conditional probability distribution as,
P (vi| pai;σ) =
X
v′
i:f(v′
i)=vi
P (v′
i| pai) (1)
Example: Fixing the value of a variable (hard intervention) is a local intervention as do(Vi =v′
i) =
do(Vi =f(vi)) wheref(vi) =v′
i.
Example: Translations are local interventions as do(Vi = vi +k) = do(Vi = f(vi)) where
f(vi) = vi +k. This includes changing the position of objects in RL environments (Shah et al.,
2022).
14