Financial Trading as a Game: A Deep Reinforcement Learning Approach — part 1/2
Cleaned-text code points [0, 15918). Bibliography omitted here; source.md includes it.
# Financial Trading as a Game: A Deep Reinforcement Learning Approach
###### Abstract
An automatic program that generates constant profit from the financial market is lucrative for every market practitioner. Recent advance in deep reinforcement learning provides a framework toward end-to-end training of such trading agent. In this paper, we propose an Markov Decision Process (MDP) model suitable for the financial trading task and solve it with the state-of-the-art deep recurrent Q-network (DRQN) algorithm. We propose several modifications to the existing learning algorithm to make it more suitable under the financial trading setting, namely 1. We employ a substantially small replay memory (only a few hundreds in size) compared to ones used in modern deep reinforcement learning algorithms (often millions in size.) 2. We develop an action augmentation technique to mitigate the need for random exploration by providing extra feedback signals for all actions to the agent. This enables us to use greedy policy over the course of learning and shows strong empirical performance compared to more commonly used ϵ \epsilon -greedy exploration. However, this technique is specific to financial trading under a few market assumptions. 3. We sample a longer sequence for recurrent neural network training. A side product of this mechanism is that we can now train the agent for every T T steps. This greatly reduces training time since the overall computation is down by a factor of T T . We combine all of the above into a complete online learning algorithm and validate our approach on the spot foreign exchange market.
###### keywords
## 1 Introduction
In this paper we investigate the effectiveness of applying deep reinforcement learning algorithms to the financial trading domain. Financial trading, differs from the gameplay domain or robotics, posts some unique challenges. We point out some of them that we believe hold the key to successful application.
### 1.1 The Financial Trading Task
One way to describe the financial trading task is as the following:
”An agent interacts with the market trying to achieve some intrinsic goal.”
Note that the agent needs not to be human; algorithmic trading now accounts for large amount of trading activities in modern financial markets. Common interaction involves agent observing newly coming financial data or submitting new order to the exchange, etc. The intrinsic goal, say, for a hedge fund manager may be a risk-adjusted measure e.g. trying to reach a 15% annual return target under a specified volatility threshold. The goal for a naive trader may be simply to pursuit highest profit without properly take into account the risk incurred. An extreme example is that an individual that trades only for the ”gambling sensation” and doesn’t care about the financial market at all.
Although the above description is quite general, there are some characteristics of such task: 1. The agent interacts with the financial market at discrete time steps even though the time steps may be extremely close, say, in the high frequency trading, trading decisions can be made in the matter of milliseconds.
2. There is a set of legal actions an agent can apply to the market from naively submitting market orders with a fixed position size to submitting a fully specified limit order.
3. The financial market produces new information available to the agent at each time step enables the agent to make trading decisions. However, the agent doesn’t have full clue on how the data is generated.
4. The agent has the potential to alter , although without full control over, the financial market if it is powerful enough. Hence it is not entirely realistic to consider the market to be fully exogenous to the agent.
With these characteristics, we desire a unified framework for training such agent. This is part of the motivation behind this thesis.
### 1.2 Motivation
With many successful stories of deep reinforcement learning, a natural question to ask is:
”Can an artificial agent learn to trade successfully?”
Success is defined in terms of the degree the agent is reaching its intrinsic goal. One of the most fundamental hypotheses of reinforcement learning is that goals of an agent can be expressed through maximizing long-term future rewards. Reward is a single scalar feedback signal that reflects the ”goodness” of an agent’s action in some state. This is called the reward hypothesis .
###### Definition 1 .
(Reward Hypothesis) All goals can be described by maximization of expected future reward.
The four characteristics mentioned above resembles that of reinforcement learning . A branch of machine learning that studies the science of sequential decision-making. Reinforcement learning has recently received a considerable amount of attention due to solving challenging control tasks that are infeasible before. The motivation behind this thesis is therefore to see if the recently proposed techniques migrate to the financial trading task and to see how far we can go with these techniques.
### 1.3 Challenges
We identify four major challenges to applying reinforcement learning to financial trading:
1. Lack of baseline. With a large body of work published on applying deep reinforcement learning to video gameplay and robotics. There is relatively little work on how to apply the same algorithm to financial trading Li (2017) . There is no clear baseline nor a suitable MDP model, network architecture or a set of hyperparameters can be followed on early stage experiments.
2. Data quality and availability. Financial data are difficult to obtain in high resolution. Usually only open, high, low and close prices (OHLC) data are freely accessible which, may not be sufficient to produce successful trading strategy. The financial time series itself is non-stationary, posting challenges to standard, gradient-based learning algorithms.
3. Partially observability of financial markets. No matter how ”complete” our input state is, there will always be a degree of unobservability in the financial market. We are unable to observe, for every market participator, their consensus on current market condition.
4. Exploration and exploitation dilemma. Despite the sophistication of modern deep reinforcement learning algorithms, usually a naive exploration policy is used. For example, ϵ \epsilon -greedy exploration in valued-based methods and Boltzmann exploration in policy-based methods Sutton and Barto (1998) . This is infeasible in financial trading setting since random exploration would inevitably generate huge amount of transaction costs and hurt performance.
### 1.4 Contributions
The contributions of this thesis are three-fold:
1. We propose a Markov decision process (MDP) model for general signal-based financial trading task solvable by state-of-the-art deep reinforcement learning algorithm with publicly accessible data only. The MDP model is easily extendable with more sophisticated input features and more complex action spaces with minimal modifications to model architecture and learning algorithm.
2. We modify the existing deep recurrent Q-network algorithm in a way that it’s more suitable for the financial trading task. This involves using a substantially smaller replay memory and sampling a longer sequence for training. We are surprised by the above two discoveries since in deep reinforcement learning, usually a large replay memory is used and length of the sampled sequences are usually only a few time steps long. We also discover workable hyperparameters for the DRQN algorithm able to solve the financial trading MDP through random search. We also develop a novel action augmentation technique to mitigate the need for random exploration in the financial trading environment.
3. We achieve positive return on 12 different currency pairs including major and cross pairs under transaction costs. To the author’s best knowledge, this is the first successful application on real financial data using pure deep reinforcement learning techniques. Numerical results presented in this paper can serve as benchmarks for future studies.
This thesis is structured as follows: in section 2, we give a detailed description on the proposed method including data preparation, feature extraction, model architecture and learning algorithm. In section 3, we combine all of the proposed techniques into a single online learning algorithm. In section 4, we evaluate our algorithm on the spot foreign exchange market and provide numerical results.
## 2 Method
In this section we provide detailed description on the proposed MDP model, model architecture as well as the learning algorithm.
### 2.1 Data Preparation and Feature Extraction
We download tick-by-tick forex data from TrueFX.com from January 2012 to December 2017. We pick 12 currency pairs, namely AUDJPY, AUDNZD, AUDUSD, CADJPY, CHFJPY, EURGBP, EURJPY, EURUSD, GBPJPY, GBPUSD, NZDUSD and USDCAD. For diversity, both major and cross pairs are included. We then resample the data into 15-minute intervals with open, high, low, close prices and tick volume. The reason for choosing forex over other asset classes is the ease of accessing high resolution data, often at very low or no cost.
### 2.2 Financial Trading MDP
In this section we give definitions of the state space, action space and reward function of the financial trading MDP.
#### 2.2.1 State Space ∈ ℝ 198 \in\mathbb{R}^{198}
The state representation is a 198-dimensional vector consists of the following three parts: • Time feature ∈ ℝ 3 \in\mathbb{R}^{3} Since the foreign exchange market has the longest opening hours among all financial markets. In order for our agent to differentiate different market sessions, we add the minute , hour and day of week of the current time stamp to be part of the state representation. This is encoded via a sinusoidal function sin ( 2 π t T ) \sin\left(2\pi\,\frac{t}{T}\right) where t t is the current value (zero-based numbering) and T T is number of possible values for t t .
• Market feature ∈ ℝ 16 × 12 \in\mathbb{R}^{16\times 12} We extract 16 features from OHLCV data containing 8 most recent log returns on both closing price and tick volume. A running Z-score normalization of period 96 is then applied to each dimension of the 16 input features. We also clip the value by 10 after normalization to eliminate outliers. We utilize price features from all 12 currency pairs in the hope that deep neural networks can extract useful intermarket features from the data.
• Position feature ∈ ℝ 3 \in\mathbb{R}^{3} The agent’s current position is encoded via a 3-dimensional one-hot vector indicates whether the current position is of -1, 0 or +1 unit, e.g. if the current position is of +1 unit, the encoding would be [ 0 , 0 , 1 ] [0,0,1] .
#### 2.2.2 Action Space
We adopt a simple action set of three values {-1, 0, 1}. Position reversal is allowed (results in double amount of transaction costs). Note that when the current position is +1 and the agent again outputs +1 at the next time step, no trading action will be executed. This sometimes refers to as target orders where the output indicates the target position size, not the trading decision itself. This simplifies the action space definition and makes the implementation easier.
#### 2.2.3 Reward Function
We define the reward function as portfolio log returns at each time step, i.e. r t = log ( v t v t − 1 ) \displaystyle r_{t}=\log\left(\frac{v_{t}}{v_{t-1}}\right) (1) where v t v_{t} is the portfolio value (account balance plus unrealized PnL from open positions). With the above definition, the portfolio value v t v_{t} satisfies a simple recursive relation v t = v t − 1 + a t ⋅ c ⋅ ( c t − o t ) − d t \displaystyle v_{t}=v_{t-1}+a_{t}\cdot c\cdot(c_{t}-o_{t})-d_{t} (2) where a t a_{t} is the output action, c c is the (constant) trade size, o t o_{t} , c t c_{t} are the current open, close prices and d t d_{t} is the commission term. The commission d t d_{t} is computed by d t = c ⋅ | a t − a t − 1 | ⋅ spread . \displaystyle d_{t}=c\cdot|a_{t}-a_{t-1}|\cdot\text{spread}. (3)
We use spread as a principled way of measuring the cost for making trading decisions. The spread we consider here, unlike real spreads, is kept fixed along the course of learning. This is for the ease of comparing different currency pairs since the width of the spread varies from pair to pair.
Defining the reward function this way, the return G t G_{t} has a nice interpretation as the future discounted log returns. When facing action selection, the agent is effectively picking actions with the highest log returns. We prefer log returns over arithmetic returns as they are additive which is more natural in the RL setting.
### 2.3 Fully Exploit with Action Augmentation
Random exploration is unsatisfying in the financial trading setting since transaction costs occur with a change of position. We propose a simple technique to mitigate the need for exploration by providing the agent with reward signal for every action. This is possible since the reward is easily computable after the price at the current timestep is observed using equation ( 1 ). For example, if the unrealized PnL for the current step is +10 after we execute action +1, then we immediate know that if we were to execute action -1, we would get a reward of -10 and 0 for action 0. Therefore the portfolio value v t v_{t} can be computed (therefore the reward signal) for all actions.
On the other hand, the only part of the state that would be altered if we were to take other actions is the agent’s position . This is known as the zero market impact hypothesis which states that the action taken from the market participator has no influence on the current market condition. We also assume order issued by the agent always executes at the next opening price. That is, we always know the position for the next step if the output action is determined.
Now we are able to update Q-values for all actions. We write down a novel loss function in vector form called action augmentation loss , ℒ ( θ ) \displaystyle\mathcal{L}(\theta) = 𝔼 ( s , 𝒂 , 𝒓 , 𝒔 ′ ) ∼ 𝒟 [ ‖ r + γ Q θ − ( 𝒔 ′ , arg max a ′ Q θ ( 𝒔 ′ , a ′ ) ) − Q θ ( s , 𝒂 ) ‖ 2 ] \displaystyle=\mathbb{E}_{(s,\bm{a},\bm{r},\bm{s^{\prime}})\sim\mathcal{D}}\left[\|\textbf{r}+\gamma Q_{\theta^{-}}(\bm{s^{\prime}},\arg\max_{a^{\prime}}Q_{\theta}(\bm{s}^{\prime},a^{\prime}))-Q_{\theta}(s,\bm{a})\|^{2}\right] (4) θ \displaystyle\theta ← θ − α ∇ θ ℒ ( θ ) \displaystyle\leftarrow\theta-\alpha\nabla_{\theta}\mathcal{L}(\theta) (5) where Q θ − Q_{\theta^{-}} denotes the target network.
### 2.4 Model Architecture
We use a four-layered neural network as function approximator to represent the optimal action-value function q ∗ q_{*} . The first two are linear layers with 256 hidden units and ELU Clevert et al. (2015) activation. The third layer is an LSTM layer with the same size. The fourth layer is another linear layer with 3 output units. The network is relatively small with approximately 65,000 parameters.
#### 2.4.1 Weight Initialization
Weight initialization is crucial for successful training of deep neural networks. We follow initialization scheme presented in He et al. (2015) for weight matrices in both hidden layers and input-to-hidden layer in LSTM. We follow Le et al. (2015) to initialize all hidden-to-hidden weight matrices to be identity. We set all biases in the network to be zero except for the forget gate in the LSTM which are set to be 1. We sparsely initialize the output layer weight matrix with Gaussian distribution 𝒩 ( 0 , 0.001 ) \mathcal{N}(0,0.001) .
### 2.5 Training Scheme
In this section we combine all of the above and present a complete learning algorithm that we will evaluate in section 5 on the spot foreign exchange market.
#### 2.5.1 Modified Training Scheme