CEO-Bench: Can Agents Play the Long Game?

arxiv:2606.18543

Original source · Paper JSON

Retrieval, embeddings, benchmarks · 2026-06-16

Cites (3) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6
7. Part 7

Contents

• CEO-Bench : Can Agents Play the Long Game?
• Section 1 Introduction
• Section 2 Designing CEO-Bench
• 2.1 How CEO-Bench Works
• 2.2 How We Make CEO-Bench Rigorous and Challenging
• Maximize realism with granular simulation.
• Robust simulation with mechanistic rules.
• Consistent simulation under stochasticity.
• Hidden information and indirect feedback.
• Interconnected world dynamics.
• Delayed and uncertain consequences.
• Non-stationary environment.
• 2.3 A Versatile Action Interface Between World and Agent
• Composable action interface in Python.
• Granular action spaces.
• Large-scale and realistic databases.
• Social media.
• Section 3 Experiments and Results
• 3.1 Experimental Setup
• 3.2 Results Overview
• 3.3 A Look into Agent Behaviors
• 3.4 Measuring Drivers of Success and Failure
• Section 4 Ablating Simulator and Agent Configurations
• 4.1 Ablating Competitor Difficulty
• 4.2 Ablating Time-Horizon
• 4.3 Ablating Agent Harness
• Section 5 Related Work
• Language model evaluations.
• Long-horizon agent evaluation.
• Section 6 Limitations and Conclusion
• 6.1 Limitations
• 6.2 Conclusion
• Acknowledgments
• Appendix
• Appendix A Simulator Mechanics
• A.1 Agent Commands and Observable State
• Python action surface.
• A.2 Customers, Plans, and Participation
• Customer parameter sampling.
• Customer participation curve.
• Plan choice and billing-period switching.
• A.3 Product Quality, Usage, and Monetization
• Promotions and effective price.
• Usage and capacity.
• Service health.
• Delivered quality.
• Development and R&D.
• In-app ads.
• Perceived quality.
• A.4 Satisfaction, Retention, and Support
• Satisfaction.
• Billing revenue and involuntary churn.
• Support issue generation and resolution.
• A.5 Reputation, Social Media, and Acquisition
• Reputation impact.
• Cross-group reputation spillovers.
• Customer and agent social media.
• Daily new-customer generation.
• Demand surges.
• A.6 Market Discovery and Non-Stationarity
• Market and group research.
• Competitor events.
• Macroeconomic schedule.
• A.7 Enterprise Sales and Negotiation
• Offer evaluation.
• A.8 Costs and Cash Flow
• Daily costs.
• Appendix B Rule-Based Baseline Strategy and Configuration Search
• Appendix C Comparison to Vending-Bench 2 Curves
• Appendix D Upper Bound Final-Cash Estimate
• Objective.
• Parameter choices.
• Quality and retention check.
• Revenue and costs.
• Interpretation.
• Instructions for reporting errors