Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents

arxiv:2502.15840

Original source · Paper JSON

Agents, open-endedness, AGI · 2025-02-21

Cites (0) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2

Contents

• Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents
• Abstract
• 1 Introduction
• 2 Method
• 2.1 Agent implementation
• 2.2 Task environment
• 2.2.1 Simulating supplier communication
• 2.2.2 Simulating customer purchases
• 2.3 Environment configuration
• 2.4 Scoring
• 2.5 Human baseline
• 3 Results and discussion
• 3.1 Overview
• 3.2 Primary models
• 3.2.1 Comparison of top models
• 3.2.2 Trace analysis
• 3.3 Secondary models
• 3.3.1 Trace analysis
• 3.4 Comparison to human baseline
• 3.5 Experiment variations
• 3.5.1 Environment configuration
• 3.5.2 Agent memory
• 3.6 Is the problem long input length?
• 4 Conclusion
• Appendix A Appendix
• Instructions for reporting errors