Evaluating the World Model Implicit in a Generative Model — part 5/5

Cleaned-text code points [61663, 66143). Bibliography omitted here; source.md includes it.

Here we describe the rides dataset construction in more detail. Our empirical studies are based on a dataset of taxi rides in New York from 2014, originally released by the NYC Taxi & Limousine Commission. We use a subset of 15 million rides that took place between January and March 2014, made available by Murray [25] . We drop duplicate rides and subset the dataset to only include rides in Manhattan, resulting in 3,358,737 sequences. We use the OSMnx library [ 5 ] to represent New York as a weighted graph. We match pickups and dropoffs to the closest intersection, measured in terms of latitude/longitude. The graph consists of 4,580 nodes, 9,846 edges, and each node has a median of 2 valid intersections. We convert bearings to one of 8 cardinal directions. We remove short traversals with two node or less. We also remove sequences with more than 100 tokens.

Shortest paths. The first approach creates traversals between two nodes by finding the shortest path between them. For each taxi ride in the dataset, we map the pickup latitude/longitude to the closest intersection and do the same for the dropoff location. We then perform Dijkstra’s algorithm to find the shortest path weighted by distance. After filtering out duplicated traversals, we have a training set of 2,932,675 sequences and 120,400,201 tokens, along with a validation set of 1,000 sequences and 41,641 tokens.

Noisy shortest paths. Shortest path traversals may not be reflective of real-world traversals due to differences in traffic patterns. Moreover, shortest path traversals between two nodes are deterministic, potentially limiting a model’s ability to pick up a world model. Therefore, we construct a noisy version of shortest-path traversals. We do this by modifying the weighting function, W ~ ​ ( i , j ) = W ⁡ ( i , j ) + ϵ i ​ j \tilde{W}(i,j)=W(i,j)+\epsilon_{ij} , where ϵ i ​ j ∼ Gamma ​ ( W ⁡ ( i , j ) , 1 ) \epsilon_{ij}\sim\text{Gamma}(W(i,j),1) ; we can interpret this as artificially adding traffic to each edge that scales with the original length. We resample 50 different weighting functions. After filtering out duplicated traversals, we have a training set with 30,599,312 sequences and 1,677,587,216 tokens while our validation set consists of 1,000 sequences and 54,539 tokens

Random walks. In the last setting, we sample random traversals rather than approximating shortest paths. We construct each sequence by sampling a node uniformly at random, sampling a sequence length uniformly between 3 and 100, and constructing traversals by sampling random edges uniformly at random for the prespecified sequence length. We create a training set of 90,646,864 sequences and 4,735,591,368 tokens, along with a validation set of 1,000 sequences and 52,360 tokens.

We use the GPT-2 architecture [ 29 ] to train models on all datasets. We use the GPT-2 small architecture for the shortest paths model and the GPT-2 extra-large architecture for the noisy shortest paths and random walks models. For the shortest paths models, we train until we overfit and use the best validation checkpoint. For the two larger datasets, we train for a fixed number of epochs and use the last validation checkpoint; we use 5 epochs for the noisy shortest paths dataset and 1 epoch for the random walks dataset. We train all models on 8 A100 GPUs, using a batch size of 6 sequences per GPU. Training time ranges from about 12 hours for the shortest paths model to 48 hours for the random walks model.

## Appendix G Additional maps

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .