How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks — part 11/11

Cleaned-text code points [155121, 159420). Bibliography omitted here; source.md includes it.

ii) The masses of other stars are sampled from [ 0.04 , 18.0 ] ​ k ​ g [0.04,18.0]kg , compared to [ 0.02 , 9.0 ] ​ k ​ g [0.02,9.0]kg in the training set.

iii) The distances are in-distribution, i.e., same sampling process as training set.

##### Model and hyperparameter settings.

We consider the following one-iteration Graph Neural Network (GNN) architecture, a.k.a. Interaction Networks. Given a collection of stars S = { X i } i = 1 n S=\{X_{i}\}_{i=1}^{n} , our GNN runs on a complete graph with nodes being the stars X i X_{i} . GNN learns the star (node) representations by aggregating and transforming the interactions (forces) of all other node vectors o u = MLP ( 2 ) ​ ( ∑ v ∈ S ∖ { u } MLP ( 1 ) ​ ( h v , h u , w ( u , v ) ) ) . \displaystyle o_{u}=\text{MLP}^{(2)}\left(\sum\limits_{v\in S\setminus\{u\}}\text{MLP}^{(1)}\left(h_{v},h_{u},w_{(u,v)}\right)\right). (81) Here, h v h_{v} is the input feature of node v v , including mass, position and velocity h v = ( m v , 𝒙 v , 𝒗 v ) \displaystyle h_{v}=\left(m_{v},\bm{x}_{v},\bm{v}_{v}\right) w ( u , v ) w_{(u,v)} is the input edge feature of edge ( u , v ) (u,v) . The loss is computed and backpropagated via the MSE loss of ∥ [ o 1 , … , o n ] − [ a n s 1 , . . , a n s n ] ∥ 2 , \displaystyle\|[o_{1},...,o_{n}]-[ans_{1},..,ans_{n}]\|_{2}, where o i o_{i} denotes the output of GNN for node i i , and a ​ n ​ s i ans_{i} denotes the true label for node i i in the next frame.

We search the following hyper-parameters for GNNs a) Number of GNN iterations is set to 1 1 .

b) Width of all MLPs are set to 128 128 .

c) The number of layers for MLP ( 1 ) \text{MLP}^{(1)} is set to 4 4 . The number of layers for MLP ( 2 ) \text{MLP}^{(2)} is set to 2 2 .

d) We consider two representations of edge/relations w ( i , j ) w_{(i,j)} . i) The first one is simply 0 0 .

ii) The better representation, which makes the underlying target function more linear, is w ( i , j ) = m j ‖ 𝒙 i t − 𝒙 j t ‖ 2 3 ⋅ ( 𝒙 j t − 𝒙 i t ) \displaystyle w_{(i,j)}=\frac{m_{j}}{\|\bm{x}_{i}^{t}-\bm{x}_{j}^{t}\|_{2}^{3}}\cdot\left(\bm{x}_{j}^{t}-\bm{x}_{i}^{t}\right)

We train the GNN with the mean squared error (MSE) loss, and Adam optimizer. We search the following hyper-parameters for training a) Initial learning rate is set to 0.005 0.005 . learning rate decays 0.5 0.5 for every 50 50 epochs

b) Batch size is set to 32 32 .

c) Weight decay is set to 1 ​ e − 5 1e-5 .

d) Number of epochs is set to 2,000 2,000 .

## Appendix D Visualization and Additional Experimental Results

### D.1 Visualization Results

In this section, we show additional visualization results of the MLP’s learned function out of training distribution (in black color ) v.s. the underlying true function (in grey color ). We color the predictions in training distribution in blue color .

In general, MLP’s learned functions agree with the underlying true functions in training range (blue). This is explained by in-distribution generalization arguments. When out of distribution, the MLP’s learned functions become linear along directions from the origin. We explain this OOD directional linearity behavior in Theorem 1 .

Finally, we show additional experimental results for graph-based reasoning tasks.

### D.2 Extra Experimental Results

In this section, we show additional experimental results.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .