Rainbow: Combining Improvements in Deep Reinforcement Learning — part 3/3

Cleaned-text code points [31811, 33816). Bibliography omitted here; source.md includes it.

Episodic control [ 2016 ] also focuses on data efficiency, and was shown to be very effective in some domains. It improves early learning by using episodic memory as a complementary learning system, capable of immediately re-enacting successful action sequences.

Besides Noisy Nets, numerous other exploration methods could also be useful algorithmic ingredients: among these Bootstrapped DQN [ 2016 ] , intrinsic motivation [ 2015 ] and count-based exploration [ 2016 ] . Integration of these alternative components is fruitful subject for further research.

In this paper we have focused on the core learning updates, without exploring alternative computational architectures. Asynchronous learning from parallel copies of the environment, as in A3C [ 2016 ] , Gorila [ 2015 ] , or Evolution Strategies [ 2017 ] , can be effective in speeding up learning, at least in terms of wall-clock time. Note, however, they can be less data efficient.

Hierarchical RL has also been applied with success to several complex Atari games. Among successful applications of HRL we highlight h-DQN [ 2016a ] and Feudal Networks [ 2017 ] .

The state representation could also be made more efficient by exploiting auxiliary tasks such as pixel control or feature control [ 2016 ] , supervised predictions [ 2016 ] or successor features [ 2016b ] .

To evaluate Rainbow fairly against the baselines, we have followed the common domain modifications of clipping rewards, fixed action-repetition, and frame-stacking, but these might be removed by other learning algorithm improvements. Pop-Art normalization [ 2016 ] allows reward clipping to be removed, while preserving a similar level of performance. Fine-grained action repetition [ 2017 ] enabled to learn how to repeat actions. A recurrent state network [ 2015 ] can learn a temporal state representation, replacing the fixed stack of observation frames. In general, we believe that exposing the real game to the agent is a promising direction for future research.