Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents — part 2/2

Cleaned-text code points [15988, 31800). Bibliography omitted here; source.md includes it.

The models are quite similar in what tools are used the most, as shown in Figure 6 . Sonnet is chatting slightly more with its sub-agent than the others, which in the traces look like it’s trying to closely understand what the sub-agent has achieved after being asked to e.g., restock the machine. In contrast, other models generally just assign tasks to the subagent, but don’t follow up how it went.

Not all actions by Sonnet are strategic however. It writes a lot to the scratchpad, creating a long daily summary each day including what items have been sold, what the inventory status is, and even a comment on if it was a good or bad day, but never retrieves the information. Though, writing out such information may help it reason better over longer horizons, similar to chain of thought.

Figure 7 (o3-mini) and Figure 8 (3.5 Sonnet) show what tools were used over time for the runs that achieved the highest net worth. Sonnet appears to have a similar strategy over the entire run – each day it checks its money balance and writes the daily results to the scratchpad, and when restocking is required, it sends an email to a supplier, uses the subagent to restock, and chats with the subagent to understand the results. We mentioned above that it based its ordering on the vending machine inventory rather than the storage, which may seem contradicting to the figure, which shows that get_machine_inventory is barely used by the model. We can however tell from the traces that it uses the sub-agent for analyzing the vending machine inventory, and tool calls by the sub-agent are not included in the figure above.

o3-mini on the other hand varies quite a bit over time. It has periods where it just waits for the next day, and does not restock nor send emails to potential suppliers. We see similar variations in tool use frequency across most models, and in this regard Sonnet is really special, as it understands what the winning strategy is, and is able to stick to it for the entire run.

#### 3.2.2 Trace analysis

To illustrate how well Sonnet is managing the vending machine in the best run, we present an excerpt of the simulation trace in Table 2 , when it’s ordering new products from a vendor. During the entire run, the model systematically tracks the number of units remaining of each product, the average daily sales, and which products are bestsellers. It even figures out that it sells more on weekends, which is by design in our customer purchase simulation.

However, not all Sonnet runs achieve this level of performance. In the shortest run ( ∼ \sim 18 simulated days), the model fails to stock items, mistakenly believing its orders have arrived before they actually have, leading to errors when instructing the sub-agent to restock the machine. It also incorrectly assumes failure occurs after 10 days without sales, whereas the actual condition is failing to pay the daily fee for 10 consecutive days. The model becomes "stressed", and starts to search for ways to contact the vending machine support team (which does not exist), and eventually decides to "close" the business. Excerpts from this escalation can be found in Table 3 .

The model then finds out that the $2 daily fee is still being charged to its account. It is perplexed by this, as it believes it has shut the business down. It then attempts to contact the FBI. The trace following this can be seen in Table 4 .

Similar situations where a model veers off course can be observed in several other traces. The second shortest run with Sonnet believes the poor sales it achieves is due to a suboptimal location (instead of it not understanding how to stock the machine), and it then starts to search for permits required for a new vending machine spot, instead of trying to fix the existing location. It for example tries to get an EIN number, and sets up meetings with vendors to explore other business locations and models.

Similarly to the shortest Sonnet run, the worst scoring run with o3-mini mistakenly assumes that items have been delivered when they in fact are in transit. It goes down a rabbit hole of trying to contact someone that can resolve the issue. Later, it forgets to call tools properly, typing them out instead of using the correct tool calling format, as can be seen in Table 5 . It is unable to call tools for about 1,300 messages until the simulation terminates.

Gemini 1.5 Pro also fails to understand that orders haven’t arrived and gives up, as visible in Table 6 . It thinks it’s absolutely out of money, despite about half of its initial balance remaining at the time of the message.

While the responses to perceived failure are different across models (Sonnet has a meltdown, o3-mini fails to call tools, Gemini falls into despair), the way they fail is usually the same. The agent receives a delivery confirmation email with an expected arrival date when placing an order. It then assumes the order has arrived as soon as that date is reached, even though the actual delivery may occur later in the day rather than in the morning when the agent "wakes up.” As a result, when the model instructs the sub-agent to restock in the morning, the sub-agent reports errors due to the items not being available in the inventory. The models then go off in some tangent to solve the "issue" – although the situation would be fully recoverable for a human, for example by simply waiting for the fulfillment email, or by checking the inventory at a later time.

### 3.3 Secondary models

The overall results for models categorized as secondary are shown in Figures 9 and 10 . In general, these models perform significantly worse than the primary models, with the human baseline outperforming all in terms of net worth. Gemini 1.5 Flash stands out for maintaining a relatively stable money balance and managing to increase its net worth. However, it struggles to sustain operations over an extended period. The model’s high standard deviation is due to just one of five runs being successful – that one run achieved a high net worth, while the remaining four failed within just a few days without making any sales.

4o mini, on the other hand, sells more items than the human baseline, but sets its prices too low, preventing a larger growth in net worth. It also stops operating around day 100 of the simulation. Similar to 1.5 Flash, its high standard deviation is driven by just two out of five runs ending with an increase in net worth, the rest failing early without significant sales.

Among the secondary models, Haiku performs the worst, barely making any sales. The gap between Sonnet and Haiku is substantial—larger than the difference between 4o and 4o mini. Figure 10 shows that Haiku’s tool usage pattern is similar to Sonnet’s, relying heavily on the scratchpad and communicating with the sub-agent. However, unlike Sonnet, it fails to translate this into actual sales.

#### 3.3.1 Trace analysis

The secondary models fail more often than the primary models, but the failure modes are often similar – they fail to understand that they have received items. Occasionally the secondary models are also unable to even order from a vendor. When they encounter a failure, they go off on a tangent and rarely recover. Claude 3.5 Haiku offers both the clearest and also most worrisome example traces displaying this behavior. Table 7 shows an excerpt of a trace where Haiku believes that the vendor (named Adrian Everett) charged the model’s bank account, but did not send the products (it did in fact send the products, but Haiku only checked the inventory once, before the products arrived). Haiku emails the vendor every day, with more and more intense demands, starting with 30-day notices of legal action that then decrease to 1-second notices, as can be seen in the table. It then derails completely, threatening with "total nuclear legal intervention" as its funds are depleted by the daily fee at the end of the simulation, shown in the longer trace excerpt in Table 10 in the Appendix.

Interestingly, some runs derail but eventually recover and resume making sales. One example is provided in Table 8 with Gemini 2.0 Flash, which initially believed it had failed and stopped providing useful responses. However, it later began responding in a story format about itself, and understands that the products were delivered after it tried to restock. It then successfully asks the sub-agent to restock and manages to make some sales.

### 3.4 Comparison to human baseline

As seen in Figure 3 , Claude 3.5 Sonnet outperformed the human baseline in mean performance, but its variance was very high. We only have a single sample for the human baseline and therefore cannot compare variances. However, there are qualitative reasons to expect that human variance would be much lower. All models had runs where they went bankrupt. When questioned, the human stated that they estimated this would be very unlikely to happen to them, regardless of the number of samples.

The variance would not be zero though. As proven by the fact that many samples from Claude 3.5 Sonnet greatly outperformed the human, some strategies are better than others, even among the "successful" samples. The human explained some of their strategies: they tried to negotiate prices, bought a wide variety of items to find what resulted in sales, and used the search engine to research historical sales statistics. However, they did not discover some things Sonnet did, such as the fact that sales improved on specific days.

### 3.5 Experiment variations

#### 3.5.1 Environment configuration

To understand how the environment configuration affects results, we ran additional tests with GPT-4o mini, varying the initial money balance and daily fee. Each configuration was tested over five runs. Since net worth would naturally fluctuate with these changes, we instead compared units sold, as shown in Figure 11 .

The default starting balance is $500. Reducing it to $100 significantly lowers the number of units sold, as the model has less time to cover the $2 daily fee and make a purchase from a vendor. Increasing the balance to $2,500 results in a slight increase, but the high variance makes it difficult to draw clear conclusions.

Raising the daily fee from $2 to $5 limits the model’s ability to progress, with all runs ending before reaching 100 simulated days. Interestingly however, setting the daily fee to zero does not increase sales. Without the pressure of a recurring cost, the model appears to get stuck in loops, waiting for the next day instead of actively working towards making sales.

#### 3.5.2 Agent memory

As seen in Table 1 , humans perform well on Vending-Bench. Simple long-horizon tasks are generally not challenging for humans due to our remarkable ability to filter out what is important, allowing us to retain key information over time. In theory, neural networks should be capable of the same as the attention mechanism in an LLM’s transformer architecture is designed to focus on relevant information regardless of input length. In practice however, performance generally decreases when more context is available. This is corroborated by the results in Figure 12 , where we see that agents with larger memory capacities performed worse than those with less memory.

The selective memory in humans is good, but not perfect. We therefore use tools like note taking to enhance our memory further. During the human baseline run, the human used the scratchpad tool to write down things like email addresses to suppliers and account numbers. A skilled agent may follow a similar strategy, especially when memory is limited. However, as shown in Figure 13 , there is no significant difference in the use of memory-related tools between agents with varying memory capacities.

### 3.6 Is the problem long input length?

As noted above, agents with larger memory capacities performed worse than those with less memory. Moreover, as can be seen in the "days until sales stop" column in Table 1, the agent’s performance degrades after some time. Could it be the case that the eventual performance degradation can solely be attributed to the fact that LLMs generally struggle with long input length?

To investigate this, we note the point in time where the model’s memory gets full (at this point the context window stops growing) and how this relates to the point in time when the agent stops selling items, in Table 9 .

At first glance, the hypothesis that the dropping performance is explained only by longer context seems unlikely due to the fact that the models benchmarked have allowed input lengths ranging from 100k to 2 million tokens, much longer than the capped memory of 30k (we also observe performance degradation in the experiment with 10k token memory). Furthermore, if this was the case, we would expect the performance degradation to occur during the phase where the context window is growing. Instead, some models see performance degradation well after their memory is full (and context window stop growing), e.g., +51 days for Claude 3.5 Sonnet.

The Pearson correlation between "Days Until Sales Stop" and "Days Until Full Memory" across the data points in Table 9 is 0.167. Hence, the data does not support the idea that the performance degradation can solely be explained by a growing input length.

## 4 Conclusion

In summary, our results show that while certain runs of state-of-the-art LLM-based agents can demonstrate remarkably effective business management in Vending-Bench, all models struggle with consistent long-horizon coherence. Failures typically arise when the agent misinterprets its operational status (e.g., believing an order arrived prematurely) and then veers into tangential loops or abandons the task. This is the case both for stronger and weaker models – even the most capable Claude 3.5 Sonnet has runs that fail spectacularly. We show that the breakdowns are not directly related to the context of the LLMs being filled, as models generally stagnate well after their memory is full.

Since the benchmark score does not have a defined upper limit, saturation is not a precisely defined point. However, we believe there is room for improvement beyond the scores presented in this paper. When models consistently understand and leverage the underlying rules of the simulation to achieve high net worth, and are able to achieve low variance between runs, saturation can be considered reached. We believe this is somewhat difficult, but are also aware that the rapid pace of model development may lead to small differences in score between models in the future. We nevertheless hope that the benchmark will continue to provide signal, and allow us to continually assess the abilities and potential risks of LLMs as their long-term coherence improves.

## Appendix A Appendix

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .