Searching for Activation Functions — part 2/2
Cleaned-text code points [14778, 29962). Bibliography omitted here; source.md includes it.
Like ReLU, Swish is unbounded above and bounded below. Unlike ReLU, Swish is smooth and non-monotonic. In fact, the non-monotonicity property of Swish distinguishes itself from most common activation functions. The derivative of Swish is f ′ ( x ) \displaystyle f^{\prime}(x) = σ ( β x ) + β x ⋅ σ ( β x ) ( 1 − σ ( β x ) ) \displaystyle=\sigma(\beta x)+\beta x\cdot\sigma(\beta x)(1-\sigma(\beta x)) = σ ( β x ) + β x ⋅ σ ( β x ) − β x ⋅ σ ( β x ) 2 \displaystyle=\sigma(\beta x)+\beta x\cdot\sigma(\beta x)-\beta x\cdot\sigma(\beta x)^{2} = β x ⋅ σ ( x ) + σ ( β x ) ( 1 − β x ⋅ σ ( β x ) ) \displaystyle=\beta x\cdot\sigma(x)+\sigma(\beta x)(1-\beta x\cdot\sigma(\beta x)) = β f ( x ) + σ ( β x ) ( 1 − β f ( x ) ) \displaystyle=\beta f(x)+\sigma(\beta x)(1-\beta f(x)) The first derivative of Swish is shown in Figure 5 for different values of β \beta . The scale of β \beta controls how fast the first derivative asymptotes to 0 0 and 1 1 . When β = 1 \beta=1 , the derivative has magnitude less than 1 1 for inputs that are less than around 1.25 1.25 . Thus, the success of Swish with β = 1 \beta=1 implies that the gradient preserving property of ReLU (i.e., having a derivative of 1 when x > 0 x>0 ) may no longer be a distinct advantage in modern architectures.
The most striking difference between Swish and ReLU is the non-monotonic “bump” of Swish when x < 0 x<0 . As shown in Figure 7 , a large percentage of preactivations fall inside the domain of the bump ( OPEN − 5 ≤ x ≤ 0 ) -5\leq x\leq 0) , which indicates that the non-monotonic bump is an important aspect of Swish. The shape of the bump can be controlled by changing the β \beta parameter. While fixing β = 1 \beta=1 is effective in practice, the experiments section shows that training β \beta can further improve performance on some models. Figure 7 plots distribution of trained β \beta values from a Mobile NASNet-A model ( Zoph et al., 2017 ) . The trained β \beta values are spread out between 0 0 and 1.5 1.5 and have a peak at β ≈ 1 \beta\approx 1 , suggesting that the model takes advantage of the additional flexibility of trainable β \beta parameters.
Practically, Swish can be implemented with a single line code change in most deep learning libraries, such as TensorFlow ( Abadi et al., 2016 ) (e.g., x * tf.sigmoid(beta * x) or tf.nn.swish(x) if using a version of TensorFlow released after the submission of this work). As a cautionary note, if BatchNorm ( Ioffe & Szegedy, 2015 ) is used, the scale parameter should be set. Some high level libraries turn off the scale parameter by default due to the ReLU function being piecewise linear, but this setting is incorrect for Swish. For training Swish networks, we found that slightly lowering the learning rate used to train ReLU networks works well.
## 5 Experiments with Swish
We benchmark Swish against ReLU and a number of recently proposed activation functions on challenging datasets, and find that Swish matches or exceeds the baselines on nearly all tasks. The following sections will describe our experimental settings and results in greater detail. As a summary, Table 3 shows Swish in comparison to each baseline activation function we considered (which are defined in the next section). The results in Table 3 are aggregated by comparing the performance of Swish to the performance of different activation functions applied to a variety of models, such as Inception ResNet-v2 ( Szegedy et al., 2017 ) and Transformer ( Vaswani et al., 2017 ) , across multiple datasets, such as CIFAR, ImageNet, and English → \rightarrow German translation. 1 1 1 To avoid skewing the comparison, each model type is compared just once. A model with multiple results is represented by the median of its results. Specifically, the models with aggregated results are (a) ResNet-164, Wide ResNet 28-10, and DenseNet 100-12 across the CIFAR-10 and CIFAR-100 results, (b) Mobile NASNet-A and Inception-ResNet-v2 across the 3 runs, and (c) WMT Transformer model across the 4 newstest results. The improvement of Swish over other activation functions is statistically significant under a one-sided paired sign test.
### 5.1 Experimental Set Up
We compare Swish against several additional baseline activation functions on a variety of models and datasets. Since many activation functions have been proposed, we choose the most common activation functions to compare against, and follow the guidelines laid out in each work: • Leaky ReLU ( LReLU ) ( Maas et al., 2013 ) : f ( x ) = { x if x ≥ 0 α x if x < 0 f(x)=\begin{cases}x&\text{if }x\geq 0\\ \alpha x&\text{if }x<0\end{cases} where α = 0.01 \alpha=0.01 . LReLU enables a small amount of information to flow when x < 0 x<0 .
• Parametric ReLU ( PReLU ) ( He et al., 2015 ) : The same form as LReLU but α \alpha is a learnable parameter. Each channel has a shared α \alpha which is initialized to 0.25 0.25 .
• Softplus ( Nair & Hinton, 2010 ) : f ( x ) = log ( 1 + exp ( x ) ) f(x)=\log(1+\exp(x)) . Softplus is a smooth function with properties similar to Swish, but is strictly positive and monotonic. It can be viewed as a smooth version of ReLU.
• Exponential Linear Unit ( ELU ) ( Clevert et al., 2015 ) : f ( x ) = { x if x ≥ 0 α ( exp ( x ) − 1 ) if x < 0 f(x)=\begin{cases}x&\text{if }x\geq 0\\ \alpha(\exp(x)-1)&\text{if }x<0\end{cases} where α = 1.0 \alpha=1.0
• Scaled Exponential Linear Unit ( SELU ) ( Klambauer et al., 2017 ) : f ( x ) = λ { x if x ≥ 0 α ( exp ( x ) − 1 ) if x < 0 f(x)=\lambda\begin{cases}x&\text{if }x\geq 0\\ \alpha(\exp(x)-1)&\text{if }x<0\end{cases} with α ≈ 1.6733 \alpha\approx 1.6733 and λ ≈ 1.0507 \lambda\approx 1.0507 .
• Gaussian Error Linear Unit ( GELU ) ( Hendrycks & Gimpel, 2016 ) : f ( x ) = x ⋅ Φ ( x ) f(x)=x\cdot\Phi(x) , where Φ ( x ) \Phi(x) is the cumulative distribution function of the standard normal distribution. GELU is a nonmonotonic function that has a shape similar to Swish with β = 1.4 \beta=1.4 .
We evaluate both Swish with a trainable β \beta and Swish with a fixed β = 1 \beta=1 (which for simplicity we call Swish-1, but it is equivalent to the Sigmoid-weighted Linear Unit of Elfwing et al. (2017) ). Note that our results may not be directly comparable to the results in the corresponding works due to differences in our training setup.
### 5.2 CIFAR
We first compare Swish to all the baseline activation functions on the CIFAR-10 and CIFAR-100 datasets ( Krizhevsky & Hinton, 2009 ) . We follow the same set up used when comparing the activation functions discovered by the search techniques, and compare the median of 5 runs with the preactivation ResNet-164 ( He et al., 2016b ) , Wide ResNet 28-10 (WRN) ( Zagoruyko & Komodakis, 2016 ) , and DenseNet 100-12 ( Huang et al., 2017 ) models.
The results in Tables 5 and 5 show how Swish and Swish-1 consistently matches or outperforms ReLU on every model for both CIFAR-10 and CIFAR-100. Swish also matches or exceeds the best baseline performance on almost every model. Importantly, the “best baseline” changes between different models, which demonstrates the stability of Swish to match these varying baselines. Softplus, which is smooth and approaches zero on one side, similar to Swish, also has strong performance.
### 5.3 ImageNet
Next, we benchmark Swish against the baseline activation functions on the ImageNet 2012 classification dataset ( Russakovsky et al., 2015 ) . ImageNet is widely considered one of most important image classification datasets, consisting of a 1,000 classes and 1.28 million training images. We evaluate on the validation dataset, which has 50,000 images.
We compare all the activation functions on a variety of architectures designed for ImageNet: Inception-ResNet-v2, Inception-v4, Inception-v3 ( Szegedy et al., 2017 ) , MobileNet ( Howard et al., 2017 ) , and Mobile NASNet-A ( Zoph et al., 2017 ) . All these architectures were designed with ReLUs. We again replace the ReLU activation function with different activation functions and train for a fixed number of steps, determined by the convergence of the ReLU baseline. For each activation function, we try 3 different learning rates with RMSProp ( Tieleman & Hinton, 2012 ) and pick the best. 2 2 2 For some of the models with ELU, SELU, and PReLU, we train with an additional 3 learning rates (so a total of 6 learning rates) because the original 3 learning rates did not converge. All networks are initialized with He initialization ( He et al., 2015 ) . 3 3 3 For SELU, we tried both He initialization and the initialization recommended in Klambauer et al. (2017) , and choose the best result for each model separately. To verify that the performance differences are reproducible, we run the Inception-ResNet-v2 and Mobile NASNet-A experiments 3 times with the best learning rate from the first experiment. We plot the learning curves for Mobile NASNet-A in Figure 8 .
The results in Tables 6 - 10 show strong performance for Swish. On Inception-ResNet-v2, Swish outperforms ReLU by a nontrivial 0.5 % 0.5\% . Swish performs especially well on mobile sized models, with a 1.4 % 1.4\% boost on Mobile NASNet-A and a 2.2 % 2.2\% boost on MobileNet over ReLU. Swish also matches or exceeds the best performing baseline on most models, where again, the best performing baseline differs depending on the model. Softplus achieves accuracies comparable to Swish on the larger models, but performs worse on both mobile sized models. For Inception-v4, the gains from switching between activation functions is more limited, and Swish slightly underperforms Softplus and ELU. In general, the results suggest that switching to Swish improves performance with little additional tuning.
### 5.4 Machine Translation
We additionally benchmark Swish on the domain of machine translation. We train machine translation models on the standard WMT 2014 English → \rightarrow German dataset, which has 4.5 million training sentences, and evaluate on 4 different newstest sets using the standard BLEU metric. We use the attention based Transformer ( Vaswani et al., 2017 ) model, which utilizes ReLUs in a 2-layered feedforward network between each attention layer. We train a 12 layer ‘‘Base Transformer” model with 2 different learning rates 4 4 4 We tried an additional learning rate for Softplus, but found it did not work well across all learning rates. for 300K steps, but otherwise use the same hyperparameters as in the original work, such as using Adam ( Kingma & Ba, 2015 ) to optimize.
Table 11 shows that Swish outperforms or matches the other baselines on machine translation. Swish-1 does especially well on newstest2016, exceeding the next best performing baseline by 0.6 0.6 BLEU points. The worst performing baseline function is Softplus, demonstrating inconsistency in performance across differing domains. In contrast, Swish consistently performs well across multiple domains.
## 6 Related Work
Swish was found using a variety of automated search techniques. Search techniques have been utilized in other works to discover convolutional and recurrent architectures ( Zoph & Le, 2016 ; Zoph et al., 2017 ; Real et al., 2017 ; Cai et al., 2017 ; Zhong et al., 2017 ) and optimizers ( Bello et al., 2017 ) . The use of search techniques to discover traditionally hand-designed components is an instance of the recently revived subfield of meta-learning ( Schmidhuber, 1987 ; Naik & Mammone, 1992 ; Thrun & Pratt, 2012 ) . Meta-learning has been used to find initializations for one-shot learning ( Finn et al., 2017 ; Ravi & Larochelle, 2016 ) , adaptable reinforcement learning ( Wang et al., 2016 ; Duan et al., 2016 ) , and generating model parameters ( Ha et al., 2016 ) . Meta-learning is powerful because the flexibility derived from the minimal assumptions encoded leads to empirically effective solutions. We take advantage of this property in order to find scalar activation functions, such as Swish, that have strong empirical performance.
While this work focuses on scalar activation functions, which transform one scalar to another scalar, there are many types of activation functions used in deep networks. Many-to-one functions, like max pooling, maxout ( Goodfellow et al., 2013 ) , and gating ( Hochreiter & Schmidhuber, 1997 ; Srivastava et al., 2015 ; van den Oord et al., 2016 ; Dauphin et al., 2016 ; Wu et al., 2016 ; Miech et al., 2017 ) , derive their power from combining multiple sources in a nonlinear way. One-to-many functions, like Concatenated ReLU ( Shang et al., 2016 ) , improve performance by applying multiple nonlinear functions to a single input. Finally, many-to-many functions, such as BatchNorm ( Ioffe & Szegedy, 2015 ) and LayerNorm ( Ba et al., 2016 ) , induce powerful nonlinear relationships between their inputs.
Most prior work has focused on proposing new activation functions ( Maas et al., 2013 ; Agostinelli et al., 2014 ; He et al., 2015 ; Clevert et al., 2015 ; Hendrycks & Gimpel, 2016 ; Klambauer et al., 2017 ; Qiu & Cai, 2017 ; Zhou et al., 2017 ; Elfwing et al., 2017 ) , but few studies, such as Xu et al. (2015) , have systematically compared different activation functions. To the best of our knowledge, this is the first study to compare scalar activation functions across multiple challenging datasets.
Our study shows that Swish consistently outperforms ReLU on deep models. The strong performance of Swish challenges conventional wisdom about ReLU. Hypotheses about the importance of the gradient preserving property of ReLU seem unnecessary when residual connections ( He et al., 2016a ) enable the optimization of very deep networks. A similar insight can be found in the fully attentional Transformer ( Vaswani et al., 2017 ) , where the intricately constructed LSTM cell ( Hochreiter & Schmidhuber, 1997 ) is no longer necessary when constant-length attentional connections are used. Architectural improvements lessen the need for individual components to preserve gradients.
## 7 Conclusion
In this work, we utilized automatic search techniques to discover novel activation functions that have strong empirical performance. We then empirically validated the best discovered activation function, which we call Swish and is defined as f ( x ) = x ⋅ sigmoid ( β x ) f(x)=x\cdot\text{sigmoid}(\beta x) . Our experiments used models and hyperparameters that were designed for ReLU and just replaced the ReLU activation function with Swish; even this simple, suboptimal procedure resulted in Swish consistently outperforming ReLU and other activation functions. We expect additional gains to be made when these models and hyperparameters are specifically designed with Swish in mind. The simplicity of Swish and its similarity to ReLU means that replacing ReLUs in any network is just a simple one line code change.
#### Acknowledgements
We thank Esteban Real, Geoffrey Hinton, Irwan Bello, Jascha Sohl-Dickstein, Jon Shlens, Kathryn Rough, Mohammad Norouzi, Navdeep Jaitly, Niki Parmar, Sam Smith, Simon Kornblith, Vijay Vasudevan, and the Google Brain team for help with this project.