Benchmarking Batch Deep Reinforcement Learning Algorithms — part 3/3

Cleaned-text code points [31411, 35865). Bibliography omitted here; source.md includes it.

We denote the output of the Atari environment as frames . These frames are grayscaled and resized to 84 84 by 84 84 pixels. Furthermore, the agent only receives a state and selects an action every 4 4 th frame. The selected action is repeated for the next 4 4 frames. The state is defined by the maximum between the previous two frames. Furthermore, the input to the networks is a concatenation of the previous 4 4 states. This means each network receives a tensor with dimensions ( 4 , 84 , 84 ) (4,84,84) , which considers a history of 16 16 frames ( 4 4 frames over 4 4 states). For the first 3 3 time steps, the input to the networks includes states which are set to all 0 0 s. If the environment terminates before 4 4 frames have passed, the state is defined by the maximum of the final two frames before termination.

In accordance to Machado et al. [2018] , sticky actions are used, such that the action a t a_{t} is set to the previous action a t − 1 a_{t-1} with probability p = 0.25 p=0.25 . No-operations are not applied at the beginning of episodes, and random frame skips are not used.

The reward function is defined by the in-game reward, but clipped to a range of [ − 1 , 1 ] [-1,1] . The environment terminates when the game terminates (rather than on a lost life), or after 27 27 k time steps, corresponding to 108 108 k frames or 30 30 minutes of real time.

### A.2 Architecture and Hyper-parameters.

Unless stated otherwise, all networks use the same architecture and hyper-parameters.

Image inputs are passed through a 3-layered convolutional neural network taking an input size of ( 4 , 84 , 84 ) (4,84,84) . The first layer has a kernel depth of 32 32 , kernel size of 8 × 8 8\times 8 and stride 4 4 . The second layer has a kernel depth of 32 32 , kernel size of 4 × 4 4\times 4 and stride 2 2 . The third layer has a kernel depth of 64 64 , kernel size of 2 × 2 2\times 2 and stride 1 1 . This output is flattened into a vector of 3136 3136 and passed to a full-connected network with one hidden layer of 512 512 . The output of the Q-network is a Q-value for each action. ReLU activation functions are used after layer besides the final fully-connected layer.

For methods which require a generative model, the convolutional neural network is shared between both the Q-network and the generative model. The generative model is a secondary fully-connected network with the same architecture. The final layer uses a softmax activation after the output of the network, to recover probabilities for each action.

Hyper-parameters are held consistent across each algorithm and listed in Table 1 . Hyper-parameters were chosen to match the implementation of Rainbow [ Hessel et al., 2017 ] in the Dopamine framework [ Castro et al., 2018 ] .

Algorithm-specific hyper-parameters are listed in Table 2 . Additionally, we regularize the generative model G ω G_{\omega} used in BCQ and KL-Control by a penalty on the final pre-activation output x x by 0.01 ​ x 2 0.01x^{2} . For KL-Control, dropout is applied before both of the fully-connected layers.

Additionally, we list the hyper-parameters of the online DQN, which served as the behavioral policy, in Table 3 . Training frequency corresponds to how often a training update was performed. Warmup time steps defines the initial period where actions are randomly selected and stored in the replay buffer, before any training occurs. We use ϵ \epsilon -greedy for exploration, where ϵ \epsilon is decayed over time.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .