Financial Trading as a Game: A Deep Reinforcement Learning Approach — part 2/2
Cleaned-text code points [15918, 26317). Bibliography omitted here; source.md includes it.
After some experiment with the original updating scheme for DRQN, we proposed the following modifications: 1. We discover that using a relatively small replay memory is more effective. It is different from the ”common knowledge” in value-based deep reinforcement learning where large replay memories (often millions in size) are used. This makes intuitive sense since in financial trading recent data points are more important than those from the far past. The performance decreases if we enlarge the replay memory.
2. We sample a longer sequence from the replay memory than the number of steps used in the DRQN paper. The reason behind this modification is that, a successful trading strategy involves opening a position at the right time and holding the position for a sufficiently long period of time then exiting the position. Sampling a short sequence can’t effectively train the network to learn the desired long-term dependency.
3. We find that it is unnecessary to train the network for each step since we are sampling a longer sequence. Hence we only train the network for every T T time steps. This significantly reduces computation since the number of backward passes are reduced by a factor of T T . This is also beneficial for real-time trading since trading decision can be carried out with low latency and training can be deferred after market close.
#### 2.5.2 A Complete Online Learning Algorithm
We adopt the above updating scheme to the original DRQN algorithm and propose a complete online learning algorithm that we will evaluate in the next section. We discard the common forwalk-walk optimization process that involves slicing the dataset into consecutive training and testing sets. Since each training set constructed this way are largely overlapped, strong overfitting is observed in our early stage experiments. We therefore optimize our network in a purely online fashion that most resembles real-time trading. We term the resulting algorithm financial deep recurrent Q-network (Financial DRQN).
In practice, we find it useful to implement a simple OpenAI Gym-like environment Brockman et al. (2016) for training. Since most open source backtesting engine is difficult to work with under the RL paradigm.
## 3 Experiment
In this chapter we present numerical results for the financial DRQN algorithm on 12 currency pairs, test the algorithm against different spread settings and investigate the usefulness of the proposed action augmentation technique.
### 3.1 Hyperparameters
In this section, we validate our approach on the spot foreign exchange market. We believe our method can be extended to other financial markets with minimal modification. Hyperparameters used in Algorithm 1 are listed below which are quite standard in modern deep reinforcement learning literature. Hyperparameters and model architecture are kept fixed across all experiments.
Hyperparameters Value Learning timestep T T 96 Replay memory size N N 480 Learning rate 0.00025 Optimizer Adam 1 1 1 The Adam optimizer Kingma and Ba (2014) Discount factor 0.99 Target network τ \tau 0.001
We did not do an exhaustive search over hyperparameters but stick to ones that shows good empirical results.
### 3.2 Simulation Result
We need additional parameters for trading simulation. The parameters are mainly used to compute trading statistics such as annual return and Sharpe ratio. Parameters used are listed below and kept fixed for every simulation.
Simulation Parameters Value Initial cash 100,000 2 2 2 Initial cash is 100,000 in base currency. Trade size 100,000 Spread (bp 3 3 3 We keep bp to be 0.0001 for non-JPY quoted currencies and 0.01 for JPY-quoted currencies. ) 0.08 Trading days 252 days/year
Below we present numerical results for 12 currency pairs. We consider two baselines: buy-and-hold and ”sell-and-hold” since some of the currency pairs show constant down trend throughout the test period. The one producing larger gain is used as baseline. Profit and loss are reported in terms of cumulative percentage returns. Every experiment is carried out for 5 times. This serves as a ”robustness test” for the proposed approach. Equity curve averaged over 5 runs are plotted in blue curve with one standard deviation range in shaded area.
Table 1 summarizes the performance for each currency pair. Annual return and risk-adjusted metrics are calculated by first calculating the daily return 4 4 4 We group one-step PnL into consecutive 96 steps to form the ”daily” PnL. and then annualized by multiplying by factor 252( 252 \sqrt{252} for Sharpe and Sortino ratios.) Baselines are also provided in the parenthesis for annual return. Maximum drawdown (MDD) and log return correlation between the baseline is also computed using daily return.
Additional statistics on the overall trading activities is summarized in Table 2 . We discover that the agent favors higher win rates (around 60%) while maintaining a roughly equal average profit and loss per trade (about 2 basic points in difference). The trading expectation is calculated using the win rate and average PnL. Trading frequency is calculated by dividing the length of the data by total number of trades.
### 3.3 Effect of the Spread
Since spread is the only source for market friction, it is meaningful to examine algorithm performance under various spread settings. We experiment with spread levels 0.08, 0.1, 0.15 and 0.2 basic points 5 5 5 Spread levels are taken from a leading online breaker Interactive Brokers . and discover the following facts: 1. Generally, wider spread results in worse performance. It fits our intuition since the transaction costs paid are proportional to the width of the spread.
2. The agent stays profitable for most of the currency pairs under 0.15 basic points of spread. Profitable strategies cannot be discovered under 0.2 basic point of spread for currency pair USDCAD and EURGBP.
3. An interesting discovery is that wider spread does not always lead to worse performance. For some of the JPY-quoted currency pairs the performance actually enhanced. We deem that a slightly wider spread forces the agent to locate a more reliable strategy that is more robust under market change.
### 3.4 Effectiveness of Action Augmentation
We investigate how useful is the action augmentation technique by comparing it to a traditional ϵ \epsilon -greedy policy with ϵ = 0.1 \epsilon=0.1 . With action augmentation, performance improves and standard deviation narrows, showing the algorithm is more robust and reliable. Table 4 lists performances for both ϵ \epsilon -greedy policy and action augmentation. We gain an additional 6.4% annual return in average when we use action augmentation.
## 4 Conclusion
We conclude and point future directions for this thesis in this chapter.
### 4.1 Achievements
The achievements of this thesis can be summarized as follows: 1. We propose an MDP model for signal-based trading strategies that is flexible to future extensions with minimal modifications to model architecture and learning algorithm.
2. We modify the existing deep recurrent Q-network learning algorithm to make it more suitable in the financial trading setting. Especially, we propose an action augmentation technique to mitigate the need for random exploration. We also use a substantially smaller replay memory compared to ones used in value-based deep reinforcement learning.
3. We give empirical results on the proposed algorithm for 12 currency pairs and achieve positive results under most simulation settings. To the author’s best knowledge, this is the first positive result achieved by pure deep reinforcement learning algorithm under transaction costs. Strategies discovered by the agent exhibit low or no correlation between baselines.
4. We discover a counter-intuitive fact that a slightly increased spread leads to better overall performance. This phenomena is observed for over half of the currency pairs. We think a slightly higher spread forces the agent to discover more robust and reliable trading strategies over the learning process. However, further widening the spread destroys performance.
### 4.2 Future Work
There are many potentials for future improvements on the proposed method. We list some of them that we believe are most important and interesting:
1. Expand state space and action space. We may augment more input features such as price data from other markets (even ones seemly unrelated at first glance), macro data (released news from politics and economics, fundamental data such as economic indices). For the action space, we may give the agent more freedom when making trading decisions such as deciding how much to invest (i.e. the position size) or even posting limit orders. This would requires a more complex action space and a careful output representation of an action.
2. Apply reinforcement learning to different trading scenarios, e.g. high frequency trading, pair trading or long-term equity investment. This is a further test for robustness of our method. A portfolio combining many different strategies can be created to suit investors’ needs.
3. Make use of distributional reinforcement learning Bellemare et al. (2017) to take risk-adjusted actions. In distributional reinforcement learning, rather than learning the expected return 𝔼 [ Q ( s , a ) ] \mathbb{E}[Q(s,a)] , the entire distribution over Q ( s , a ) Q(s,a) is learned. This is possible due to a distributional variant of the Bellman equation, Q ( s , a ) = 𝐷 R ( s , a ) + γ Q ( S ′ , A ′ ) . Q(s,a)\overset{D}{=}R(s,a)+\gamma Q(S^{\prime},A^{\prime}). In this thesis, we choose actions solely to maximize the expected return. That is, we blindly maximize profit without taking risk into account. This is unsatisfying since it is clear that we would prefer an trading decision that comes with lower variance although it may be less profitable. Since the entire distribution is learned, we are able to choose action with the highest expected Q-value and the lowest standard deviation of the Q-value, i.e. a = arg max a ∈ 𝒜 𝔼 [ Q ] Var [ Q ] . a=\arg\max_{a\in\mathcal{A}}\frac{\mathbb{E}[Q]}{\sqrt{\text{Var}[Q]}}. This way, we choose actions with the highest Sharpe ratio and the strategy would be more suitable for modern investors.
## Appendix
In this appendix we provide equity curves for all experiments done in section 4.