A Minimalist Approach to Offline Reinforcement Learning — part 3/3

Cleaned-text code points [31056, 35174). Bibliography omitted here; source.md includes it.

Heuristics for selecting λ \lambda . While we find a single setting of λ \lambda works across all datasets, some practitioners may be interested in guidelines in choosing λ \lambda , such as setting it to a fixed constant. Note the aim in our heuristic of normalizing by the average absolute value is to balance the importance of value maximization and behavior cloning. With domain knowledge, this heuristic can be circumvented by selecting λ \lambda roughly equal to α \alpha over the expected average value. We can also choose λ \lambda by considering the value estimate of the agent– if we see divergence in the value function due to extrapolation error [ Fujimoto et al., 2019b ] , then we need to decrease λ \lambda such that the BC term is weighted more highly. Alternatively, if the performance resembles the performance of the behavior agent, then higher values of λ \lambda should be considered.

## Appendix C Additional Experiments

### C.1 Additional Datasets

A concern of TD3+BC is the poor performance on random data. In Table 7 we mix the random and expert datasets from D4RL [ Fu et al., 2020 ] , by randomly selecting half the transitions from each dataset and concatenating. We find that TD3+BC performs comparatively to Fisher-BRC. However, both algorithms underperform CQL on Walker2d. One hypothesis for this performance gap is due to the poor performance of BC on the Walker2d expert dataset (see Table 2 and Figure 4 in the main body). For completeness we also report the performance of TD3+BC on the D4RL AntMaze datasets. For this domain we found that state feature normalization was harmful to performance and was not included. All other hyperparameters remain unchanged.

### C.2 State Feature Normalization with Other Algorithms

To better understand the effectiveness of state feature normalization, we apply it to both CQL and Fisher-BRC. We report the percent difference of including this technique in Figure 7 . We find that this generally has minimal impact on performance.

### C.3 Benchmarking against the Decision Transformer

The Decision Transformer (DT) is concurrent work which examines the use of the transformer architecture in offline RL, by framing the problem as sequence modeling [ Chen et al., 2021 ] . The authors use the D4RL -v2 datasets, which non-trivially affects performance and make direct comparison to results on the -v0 datasets inaccurate. To compare against this approach, we re-run TD3+BC on the -v2 datasets. Results are reported in Table 9 . Although DT uses state of the art techniques from language modeling, and some per-environment hyperparameters, we find TD3+BC achieves a similar performance without any additional hyperparameter tuning. Furthermore, we benchmark the training time of DT against TD3+BC, using the author-provided implementation 9 9 9 https://github.com/kzl/decision-transformer , commit 5fc73c19f1a89cb17e83aa656b6bba1986e9da59 in Figure 8 , using the same experimental setup; a single GeForce GTX 1080 GPU and an Intel Core i7-6700K CPU at 4.00GHz. Unsurprisingly, TD3+BC trains significantly faster as it does not rely on the expensive transformer architecture.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .