Dueling Network Architectures for Deep Reinforcement Learning — part 3/3
Cleaned-text code points [30920, 39183). Bibliography omitted here; source.md includes it.
Duel Clip does better than Single Clip on 75.4% of the games (43 out of 57). It also achieves higher scores compared to the Single baseline on 80.7% ( 46 46 out of 57 57 ) of the games. Of all the games with 18 18 actions, Duel Clip is better 86.6 86.6 % of the time (26 out of 30). This is consistent with the findings of the previous section. Overall, our agent (Duel Clip) achieves human level performance on 42 out of 57 games. Raw scores for all the games, as well as measurements in human performance percentage, are presented in the Appendix.
Robustness to human starts. One shortcoming of the 30 no-ops metric is that an agent does not necessarily have to generalize well to play the Atari games. Due to the deterministic nature of the Atari environment, from an unique starting point, an agent could learn to achieve good performance by simply remembering sequences of actions.
To obtain a more robust measure, we adopt the methodology of Nair et al. (2015) . Specifically, for each game, we use 100 starting points sampled from a human expert’s trajectory. From each of these points, an evaluation episode is launched for up to 108,000 frames. The agents are evaluated only on rewards accrued after the starting point. We refer to this metric as Human Starts .
As shown in Table 1 , under the Human Starts metric, Duel Clip once again outperforms the single stream variants. In particular, our agent does better than the Single baseline on 70.2% (40 out of 57) games and on games of 18 18 actions, Duel Clip is 83.3% better (25 out of 30).
Combining with Prioritized Experience Replay. The dueling architecture can be easily combined with other algorithmic improvements. In particular, prioritization of the experience replay has been shown to significantly improve performance of Atari games ( Schaul et al., 2016 ) . Furthermore, as prioritization and the dueling architecture address very different aspects of the learning process, their combination is promising. So in our final experiment, we investigate the integration of the dueling architecture with prioritized experience replay. We use the prioritized variant of DDQN (Prior. Single) as the new baseline algorithm, which replaces with the uniform sampling of the experience tuples by rank-based prioritized sampling. We keep all the parameters of the prioritized replay as described in ( Schaul et al., 2016 ) , namely a priority exponent of 0.7 0.7 , and an annealing schedule on the importance sampling exponent from 0.5 0.5 to 1 1 . We combine this baseline with our dueling architecture (as above), and again use gradient clipping (Prior. Duel Clip).
Note that, although orthogonal in their objectives, these extensions (prioritization, dueling and gradient clipping) interact in subtle ways. For example, prioritization interacts with gradient clipping, as sampling transitions with high absolute TD-errors more often leads to gradients with higher norms. To avoid adverse interactions, we roughly re-tuned the learning rate and the gradient clipping norm on a subset of 9 9 games. As a result of rough tuning, we settled on 6.25 × 10 − 5 6.25\times 10^{-5} for the learning rate and 10 10 for the gradient clipping norm (the same as in the previous section).
When evaluated on all 57 57 Atari games, our prioritized dueling agent performs significantly better than both the prioritized baseline agent and the dueling agent alone. The full mean and median performance against the human performance percentage is shown in Table 1 . When initializing the games using up to 30 no-ops action, we observe mean and median scores of 591% and 172% respectively. The direct comparison between the prioritized baseline and prioritized dueling versions, using the metric described in Equation 10 , is presented in Figure 5 .
The combination of prioritized replay and the dueling network results in vast improvements over the previous state-of-the-art in the popular ALE benchmark.
Saliency maps. To better understand the roles of the value and the advantage streams, we compute saliency maps ( Simonyan et al., 2013 ) . More specifically, to visualize the salient part of the image as seen by the value stream, we compute the absolute value of the Jacobian of V ^ \widehat{V} with respect to the input frames: | ∇ s V ^ ( s , θ ) | . |\nabla_{s}\widehat{V}(s;\theta)|. Similarly, to visualize the salient part of the image as seen by the advantage stream, we compute | ∇ s A ^ ( s , arg max a ′ A ^ ( s , a ′ ) , θ ) | |\nabla_{s}\widehat{A}(s,\argmax_{a^{\prime}}\widehat{A}(s,a^{\prime});\theta)| . Both quantities are of the same dimensionality as the input frames and therefore can be visualized easily alongside the input frames.
Here, we place the gray scale input frames in the green and blue channel and the saliency maps in the red channel. All three channels together form an RGB image. Figure 2 depicts the value and advantage saliency maps on the Enduro game for two different time steps. As observed in the introduction, the value stream pays attention to the horizon where the appearance of a car could affect future performance. The value stream also pays attention to the score. The advantage stream, on the other hand, cares more about cars that are on an immediate collision course.
## 5 Discussion
The advantage of the dueling architecture lies partly in its ability to learn the state-value function efficiently. With every update of the Q Q values in the dueling architecture, the value stream V {V} is updated – this contrasts with the updates in a single-stream architecture where only the value for one of the actions is updated, the values for all other actions remain untouched. This more frequent updating of the value stream in our approach allocates more resources to V V , and thus allows for better approximation of the state values, which in turn need to be accurate for temporal-difference-based methods like Q-learning to work ( Sutton & Barto, 1998 ) . This phenomenon is reflected in the experiments, where the advantage of the dueling architecture over single-stream Q Q networks grows when the number of actions is large.
Furthermore, the differences between Q Q -values for a given state are often very small relative to the magnitude of Q Q . For example, after training with DDQN on the game of Seaquest, the average action gap (the gap between the Q Q values of the best and the second best action in a given state) across visited states is roughly 0.04 0.04 , whereas the average state value across those states is about 15 15 . This difference in scales can lead to small amounts of noise in the updates can lead to reorderings of the actions, and thus make the nearly greedy policy switch abruptly. The dueling architecture with its separate advantage stream is robust to such effects.
## 6 Conclusions
We introduced a new neural network architecture that decouples value and advantage in deep Q Q -networks, while sharing a common feature learning module. The new dueling architecture, in combination with some algorithmic improvements, leads to dramatic improvements over existing approaches for deep RL in the challenging Atari domain. The results presented in this paper are the new state-of-the-art in this popular domain.
## Appendix A Double DQN Algorithm
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .