Mastering Diverse Domains through World Models — part 1/4

Cleaned-text code points [0, 15690). Bibliography omitted here; source.md includes it.

HTML conversions sometimes display errors due to content that did not convert correctly from the source. This paper uses the following packages that are not yet supported by the HTML conversion tool. Feedback on these issues are not necessary; they are known and are being worked on.

failed: stackengine

failed: xpatch

failed: cuted

failed: xstring

failed: eqparbox

failed: datetime2

failed: environ

failed: titletoc

failed: trimclip

Authors: achieve the best HTML results from your LaTeX submissions by following these best practices .

L[1]Q[l,#1] \NewColumnType C[1]Q[c,#1] \NewColumnType R[1]Q[r,#1] \NewEnviron mytabular[1] \BODY

# Mastering Diverse Domains through World Models

Abstract

Developing a general algorithm that learns to solve tasks across a wide range of applications has been a fundamental challenge in artificial intelligence. Although current reinforcement learning algorithms can be readily applied to tasks similar to what they have been developed for, configuring them for new application domains requires significant human expertise and experimentation. We present DreamerV3, a general algorithm that outperforms specialized methods across over 150 diverse tasks, with a single configuration. Dreamer learns a model of the environment and improves its behavior by imagining future scenarios. Robustness techniques based on normalization, balancing, and transformations enable stable learning across domains. Applied out of the box, Dreamer is the first algorithm to collect diamonds in Minecraft from scratch without human data or curricula. This achievement has been posed as a significant challenge in artificial intelligence that requires exploring farsighted strategies from pixels and sparse rewards in an open world. Our work allows solving challenging control problems without extensive experimentation, making reinforcement learning broadly applicable.

## Introduction

Reinforcement learning has enabled computers to solve tasks through interaction, such as surpassing humans in the games of Go and Dota 1 , 2 . It is also a key component for improving large language models beyond what is demonstrated in their pretraining data 3 , 4 . While PPO 5 has become a standard algorithm in the field of reinforcement learning, more specialized algorithms are often employed to achieve higher performance. These specialized algorithms target the unique challenges posed by different application domains, such as continuous control 6 , discrete actions 7 , 8 , sparse rewards 9 , image inputs 10 , spatial environments 11 , and board games 12 . However, applying reinforcement learning algorithms to sufficiently new tasks—such as moving from video games to robotics tasks—requires substantial effort, expertise, and computational resources for tweaking the hyperparameters of the algorithm 13 . This brittleness poses a bottleneck in applying reinforcement learning to new problems and also limits the applicability of reinforcement learning to computationally expensive models or tasks where tuning is prohibitive. Creating a general algorithm that learns to master new domains without having to be reconfigured has been a central challenge in artificial intelligence and would open up reinforcement learning to a wide range of practical applications.

We present Dreamer, a general algorithm that outperforms specialized expert algorithms across a wide range of domains while using fixed hyperparameters, making reinforcement learning readily applicable to new problems. The algorithm is based on the idea of learning a world model that equips the agent with rich perception and the ability to imagine the future 14 , 15 , 16 . The world model predicts the outcomes of potential actions, a critic neural network judges the value of each outcome, and an actor neural network chooses actions to reach the best outcomes. Although intuitively appealing, robustly learning and leveraging world models to achieve strong task performance has been an open problem 17 . Dreamer overcomes this challenge through a range of robustness techniques based on normalization, balancing, and transformations. We observe robust learning not only across over 150 tasks from the domains summarized in Figure 2 , but also across model sizes and training budgets, offering a predictable way to increase performance. Notably, larger model sizes not only achieve higher scores but also require less interaction to solve a task.

To push the boundaries of reinforcement learning, we consider the popular video game Minecraft that has become a focal point of research in recent years 18 , 19 , 20 , with international competitions held for developing algorithms that autonomously learn to collect diamonds in Minecraft 1 1 1 The MineRL Diamond Competitions were held in 2019, 2020, and 2021 and provided a dataset of human expert trajectories: https://minerl.io/diamond . Competitions in the following years focused on a wide range of tasks. . Solving this problem without human data has been widely recognized as a substantial challenge for artificial intelligence because of the sparse rewards, exploration difficulty, long time horizons, and the procedural diversity of this open world game 18 . Due to these obstacles, previous approaches resorted to using human expert data and domain-specific curricula 19 , 20 . Applied out of the box, Dreamer is the first algorithm to collect diamonds in Minecraft from scratch.

## Learning algorithm

We present the third generation of the Dreamer algorithm 21 , 22 . The algorithm consists of three neural networks: the world model predicts the outcomes of potential actions, the critic judges the value of each outcome, and the actor chooses actions to reach the most valuable outcomes. The components are trained concurrently from replayed experience while the agent interacts with the environment. To succeed across domains, all three components need to accommodate different signal magnitudes and robustly balance terms in their objectives. This is challenging as we are not only targeting similar tasks within the same domain but aim to learn across diverse domains with fixed hyperparameters. This section introduces the world model, critic, and actor along with their robust loss functions, as well as tools for robustly predicting quantities of unknown orders of magnitude.

### World model learning

The world model learns compact representations of sensory inputs through autoencoding 23 and enables planning by predicting future representations and rewards for potential actions. We implement the world model as a Recurrent State-Space Model (RSSM) 24 , shown in Figure 3 . First, an encoder maps sensory inputs x t subscript 𝑥 𝑡 x_{t} italic_x start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT to stochastic representations z t subscript 𝑧 𝑡 z_{t} italic_z start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT . Then, a sequence model with recurrent state h t subscript ℎ 𝑡 h_{t} italic_h start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT predicts the sequence of these representations given past actions a t − 1 subscript 𝑎 𝑡 1 a_{t-1} italic_a start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT . The concatenation of h t subscript ℎ 𝑡 h_{t} italic_h start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT and z t subscript 𝑧 𝑡 z_{t} italic_z start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT forms the model state from which we predict rewards r t subscript 𝑟 𝑡 r_{t} italic_r start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT and episode continuation flags c t ∈ { 0 , 1 } subscript 𝑐 𝑡 0 1 c_{t}\in\{0,1\} italic_c start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ∈ { 0 , 1 } and reconstruct the inputs to ensure informative representations:

Sequence model: h t = f ϕ ⁢ ( h t − 1 , z t − 1 , a t − 1 ) Encoder: z t ∼ q ϕ ⁢ ( z t | h t , x t ) Dynamics predictor: z ^ t ∼ p ϕ ⁢ ( z ^ t | h t ) Reward predictor: r ^ t ∼ p ϕ ⁢ ( r ^ t | h t , z t ) Continue predictor: c ^ t ∼ p ϕ ⁢ ( c ^ t | h t , z t ) Decoder: x ^ t ∼ p ϕ ⁢ ( x ^ t | h t , z t ) missing-subexpression Sequence model: missing-subexpression subscript ℎ 𝑡 subscript 𝑓 italic-ϕ subscript ℎ 𝑡 1 subscript 𝑧 𝑡 1 subscript 𝑎 𝑡 1 missing-subexpression Encoder: missing-subexpression subscript 𝑧 𝑡 similar-to subscript 𝑞 italic-ϕ conditional subscript 𝑧 𝑡 subscript ℎ 𝑡 subscript 𝑥 𝑡 missing-subexpression Dynamics predictor: missing-subexpression subscript ^ 𝑧 𝑡 similar-to subscript 𝑝 italic-ϕ conditional subscript ^ 𝑧 𝑡 subscript ℎ 𝑡 missing-subexpression Reward predictor: missing-subexpression subscript ^ 𝑟 𝑡 similar-to subscript 𝑝 italic-ϕ conditional subscript ^ 𝑟 𝑡 subscript ℎ 𝑡 subscript 𝑧 𝑡 missing-subexpression Continue predictor: missing-subexpression subscript ^ 𝑐 𝑡 similar-to subscript 𝑝 italic-ϕ conditional subscript ^ 𝑐 𝑡 subscript ℎ 𝑡 subscript 𝑧 𝑡 missing-subexpression Decoder: missing-subexpression subscript ^ 𝑥 𝑡 similar-to subscript 𝑝 italic-ϕ conditional subscript ^ 𝑥 𝑡 subscript ℎ 𝑡 subscript 𝑧 𝑡 \displaystyle\begin{aligned} \begin{aligned} \raisebox{8.39578pt}{\hbox to 0.0% pt{\hss\vbox to 0.0pt{\hbox{$\text{RSSM}\hskip 4.30554pt\begin{cases}\hphantom% {A}\\ \hphantom{A}\\ \hphantom{A}\end{cases}$}\vss}}}&\text{Sequence model:}\hskip 35.00005pt&&h_{t% }&\ =&\ f_{\phi}(h_{t-1},z_{t-1},a_{t-1})\\ &\text{Encoder:}\hskip 35.00005pt&&z_{t}&\ \sim&\ q_{\phi}(z_{t}\;|\;h_{t},x_{% t})\\ &\text{Dynamics predictor:}\hskip 35.00005pt&&\hat{z}_{t}&\ \sim&\ p_{\phi}(% \hat{z}_{t}\;|\;h_{t})\\ &\text{Reward predictor:}\hskip 35.00005pt&&\hat{r}_{t}&\ \sim&\ p_{\phi}(\hat% {r}_{t}\;|\;h_{t},z_{t})\\ &\text{Continue predictor:}\hskip 35.00005pt&&\hat{c}_{t}&\ \sim&\ p_{\phi}(% \hat{c}_{t}\;|\;h_{t},z_{t})\\ &\text{Decoder:}\hskip 35.00005pt&&\hat{x}_{t}&\ \sim&\ p_{\phi}(\hat{x}_{t}\;% |\;h_{t},z_{t})\end{aligned}\end{aligned} start_ROW start_CELL start_ROW start_CELL end_CELL start_CELL Sequence model: end_CELL start_CELL end_CELL start_CELL italic_h start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT end_CELL start_CELL = end_CELL start_CELL italic_f start_POSTSUBSCRIPT italic_ϕ end_POSTSUBSCRIPT ( italic_h start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT , italic_z start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT , italic_a start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT ) end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL Encoder: end_CELL start_CELL end_CELL start_CELL italic_z start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT end_CELL start_CELL ∼ end_CELL start_CELL italic_q start_POSTSUBSCRIPT italic_ϕ end_POSTSUBSCRIPT ( italic_z start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT | italic_h start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT , italic_x start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL Dynamics predictor: end_CELL start_CELL end_CELL start_CELL over^ start_ARG italic_z end_ARG start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT end_CELL start_CELL ∼ end_CELL start_CELL italic_p start_POSTSUBSCRIPT italic_ϕ end_POSTSUBSCRIPT ( over^ start_ARG italic_z end_ARG start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT | italic_h start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL Reward predictor: end_CELL start_CELL end_CELL start_CELL over^ start_ARG italic_r end_ARG start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT end_CELL start_CELL ∼ end_CELL start_CELL italic_p start_POSTSUBSCRIPT italic_ϕ end_POSTSUBSCRIPT ( over^ start_ARG italic_r end_ARG start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT | italic_h start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT , italic_z start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL Continue predictor: end_CELL start_CELL end_CELL start_CELL over^ start_ARG italic_c end_ARG start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT end_CELL start_CELL ∼ end_CELL start_CELL italic_p start_POSTSUBSCRIPT italic_ϕ end_POSTSUBSCRIPT ( over^ start_ARG italic_c end_ARG start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT | italic_h start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT , italic_z start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL Decoder: end_CELL start_CELL end_CELL start_CELL over^ start_ARG italic_x end_ARG start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT end_CELL start_CELL ∼ end_CELL start_CELL italic_p start_POSTSUBSCRIPT italic_ϕ end_POSTSUBSCRIPT ( over^ start_ARG italic_x end_ARG start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT | italic_h start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT , italic_z start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) end_CELL end_ROW end_CELL end_ROW (1) Figure 4 visualizes long-term video predictions of the world world. The encoder and decoder use convolutional neural networks (CNN) for image inputs and multi-layer perceptrons (MLPs) for vector inputs. The dynamics, reward, and continue predictors are also MLPs. The representations are sampled from a vector of softmax distributions and we take straight-through gradients through the sampling step 25 , 22 . Given a sequence batch of inputs x 1 : T subscript 𝑥 : 1 𝑇 x_{1:T} italic_x start_POSTSUBSCRIPT 1 : italic_T end_POSTSUBSCRIPT , actions a 1 : T subscript 𝑎 : 1 𝑇 a_{1:T} italic_a start_POSTSUBSCRIPT 1 : italic_T end_POSTSUBSCRIPT , rewards r 1 : T subscript 𝑟 : 1 𝑇 r_{1:T} italic_r start_POSTSUBSCRIPT 1 : italic_T end_POSTSUBSCRIPT , and continuation flags c 1 : T subscript 𝑐 : 1 𝑇 c_{1:T} italic_c start_POSTSUBSCRIPT 1 : italic_T end_POSTSUBSCRIPT , the world model parameters ϕ italic-ϕ \phi italic_ϕ are optimized end-to-end to minimize the prediction loss ℒ pred subscript ℒ pred \mathcal{L}_{\mathrm{pred}} caligraphic_L start_POSTSUBSCRIPT roman_pred end_POSTSUBSCRIPT , the dynamics loss ℒ dyn subscript ℒ dyn \mathcal{L}_{\mathrm{dyn}} caligraphic_L start_POSTSUBSCRIPT roman_dyn end_POSTSUBSCRIPT , and the representation loss ℒ rep subscript ℒ rep \mathcal{L}_{\mathrm{rep}} caligraphic_L start_POSTSUBSCRIPT roman_rep end_POSTSUBSCRIPT with corresponding loss weights β pred = 1 subscript 𝛽 pred 1 \beta_{\mathrm{pred}}=1 italic_β start_POSTSUBSCRIPT roman_pred end_POSTSUBSCRIPT = 1 , β dyn = 1 subscript 𝛽 dyn 1 \beta_{\mathrm{dyn}}=1 italic_β start_POSTSUBSCRIPT roman_dyn end_POSTSUBSCRIPT = 1 , and β rep = 0.1 subscript 𝛽 rep 0.1 \beta_{\mathrm{rep}}=0.1 italic_β start_POSTSUBSCRIPT roman_rep end_POSTSUBSCRIPT = 0.1 :

ℒ ⁢ ( ϕ ) ≐ E q ϕ ⁡ [ ∑ t = 1 T ( β pred ⁢ ℒ pred ⁢ ( ϕ ) + β dyn ⁢ ℒ dyn ⁢ ( ϕ ) + β rep ⁢ ℒ rep ⁢ ( ϕ ) ) ] . approaches-limit ℒ italic-ϕ subscript E subscript 𝑞 italic-ϕ superscript subscript 𝑡 1 𝑇 subscript 𝛽 pred subscript ℒ pred italic-ϕ subscript 𝛽 dyn subscript ℒ dyn italic-ϕ subscript 𝛽 rep subscript ℒ rep italic-ϕ \displaystyle\begin{aligned} \mathcal{L}(\phi)\doteq\operatorname{E}_{q_{\phi}% }\Big{[}\textstyle\sum_{t=1}^{T}(\beta_{\mathrm{pred}}\mathcal{L}_{\mathrm{% pred}}(\phi)+\beta_{\mathrm{dyn}}\mathcal{L}_{\mathrm{dyn}}(\phi)+\beta_{% \mathrm{rep}}\mathcal{L}_{\mathrm{rep}}(\phi))\Big{]}.\end{aligned} start_ROW start_CELL caligraphic_L ( italic_ϕ ) ≐ roman_E start_POSTSUBSCRIPT italic_q start_POSTSUBSCRIPT italic_ϕ end_POSTSUBSCRIPT end_POSTSUBSCRIPT [ ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT ( italic_β start_POSTSUBSCRIPT roman_pred end_POSTSUBSCRIPT caligraphic_L start_POSTSUBSCRIPT roman_pred end_POSTSUBSCRIPT ( italic_ϕ ) + italic_β start_POSTSUBSCRIPT roman_dyn end_POSTSUBSCRIPT caligraphic_L start_POSTSUBSCRIPT roman_dyn end_POSTSUBSCRIPT ( italic_ϕ ) + italic_β start_POSTSUBSCRIPT roman_rep end_POSTSUBSCRIPT caligraphic_L start_POSTSUBSCRIPT roman_rep end_POSTSUBSCRIPT ( italic_ϕ ) ) ] . end_CELL end_ROW (2)