CEO-Bench: Can Agents Play the Long Game? — part 1/7

Cleaned-text code points [0, 15911). Bibliography omitted here; source.md includes it.

# CEO-Bench : Can Agents Play the Long Game?

Code Blog ▶ \blacktriangleright Trajectory

Abstract

Language model agents are becoming proficient executors at isolated, short-horizon tasks such as software engineering and customer service. Yet real-world challenges require a combination of sophisticated skills that remain largely untested in agents: (1) navigating long horizons amid uncertainty; (2) acquiring information in noisy environments; (3) adapting to a changing world; (4) orchestrating multiple moving parts toward a coherent goal. We introduce CEO-Bench , which evaluates these capabilities together by simulating a representative real-world task: operating a startup for 500 days. An agent manages pricing, marketing, budgeting, and many other aspects of a fictional company through a programmable Python interface, operating in the same environment and facing the same challenges as a human CEO. Success demands analyzing noisy, interconnected business databases, translating signals into sound strategy, and coordinating many decisions with programming. The strongest agents write sophisticated code that forecasts churn regimes, billing timing, customer losses, and future cash under different scenarios. Even so, most state-of-the-art models struggle in this environment. Only Claude Fable 5, GPT-5.6 Sol, and Claude Opus 4.8 finish above the $1M starting balance, and all evaluated models remain below the rule-based baseline. CEO-Bench takes a first step toward measuring the intelligence required to drive sustained, adaptive progress over time.

## Section 1 Introduction

Language model agents are becoming increasingly capable at short-horizon tasks. They can fix a GitHub issue ( Jimenez et al., 2024 ) , follow a service policy in dialogue ( Yao et al., 2025 ) , or complete a web workflow ( Zhou et al., 2024 ) . These are real skills, but they share a simple shape: the agent gets a clear goal, acts for a short time, and receives feedback quickly. As agents approach reliable execution of such individual tasks, a natural next question is what we should expect them to do after the local task is no longer the bottleneck.

Human intelligence goes beyond local execution ( Newell and Simon, 1972 ; Simon, 1955 ) . Many consequential human achievements are not single well-specified tasks, but long chains of decisions made under uncertainty: choosing what to prioritize, allocating limited resources, interpreting noisy signals, and adapting as conditions change ( Simon, 1955 ; March, 1991 ; Teece et al., 1997 ) . Future agents will need the same kind of sustained strategic control if they are to move beyond task completion and operate effectively in the real world.

Early agent evaluations such as SWE-bench ( Jimenez et al., 2024 ) , WebArena ( Zhou et al., 2024 ) , and τ \tau -bench ( Yao et al., 2025 ) evaluate real-world skills, but they are scoped to short episodes with quickly observed outcomes. GDPval ( Patwardhan et al., 2025 ) broadens evaluation to economically valuable work, but remains a one-shot deliverable rather than a persistent process. Agentic-memory benchmarks test agents’ ability to use information over time, but they primarily measure storage and retrieval skills ( Hu et al., 2026 ; He et al., 2026b ) . Vending-Bench ( Backlund and Petersson, 2025a ; Backlund and Petersson, 2025b ) and Accounting-Bench ( Penrose AI, 2025 ) take a first step toward evaluating agents in long-horizon simulated environments. Yet these settings involve a narrow set of decisions and largely stable environments. They do not test whether agents can coordinate many interdependent actions, acquire information from noisy feedback, and devise strategy amid delayed consequences and changing conditions.

The next stage of agent evaluation calls for a shift toward settings where actions accumulate over long horizons; environment state is only observable through indirect evidence; feedback is noisy and delayed; and conditions continue to change . Success depends on more than individual capabilities. Agents must integrate them into coherent behavior, make long-horizon plans, turn accumulated evidence into actionable signals, coordinate interdependent decisions, and continuously adapt strategy as new information arrives.

CEO-Bench instantiates this challenge in a realistic, large-scale startup simulation, where an agent runs a company for 500 days through a programmable Python interface with 34 tools and a 19-table business database. We show the structure of CEO-Bench in Fig. 1 . Beyond issuing individual tool calls, the agent writes and executes code, querying the database with SQL to analyze the company’s state and composing the available tools into custom workflows. It thus operates in the same environment and faces the same challenges as a human running the company, and the task demands coding and data-analysis skill together with strategic thinking. As shown in Fig. 4 , the agent must coordinate diverse operating decisions across pricing, growth, product, operations, communication, enterprise sales, and more. Decisions play out over realistic business timelines: revenue arrives on billing cycles, R&D takes days to weeks, and mistakes surface later through churn and reputation, forcing long-horizon reasoning under uncertainty. Much of the state is hidden, so the agent must infer customer satisfaction, willingness to pay, and shifting preferences from noisy signals in data analytics and social media. At the same time, we design the environment to keep changing as a result of customer preference drift, macroeconomic cycles, and competitor shocks. Success requires continually revising strategy in response to these shifting conditions while maintaining coherent decisions across the business.

Our evaluation shows that this challenge remains difficult for agents built on current state-of-the-art models. We show in Fig. 2 that while most agents can produce valid tool calls and analytics queries, they struggle to sustain coherent strategy over time and often bankrupt before completing the simulation. Claude Fable 5, GPT-5.6 Sol, and Claude Opus 4.8 finish above the $1M starting balance, and no evaluated model reaches the non-LLM rule-based baseline. We show in Fig. 3 that even the strongest models fail to consistently profit across experiments.

Our analysis shows that performance correlates with core capabilities targeted by CEO-Bench : inferring hidden structure from noisy data, forecasting delayed consequences, and adapting to competitive pressure. By inspecting agent action trajectories, we find distinctive behavior patterns across models. For example, Claude Fable 5 keeps adjusting its strategy late into the run, Claude Opus 4.8 explores broadly early and then becomes more passive after building a cash cushion, and Claude Opus 4.7 narrows earlier into spend cuts and cash preservation. Claude Opus 4.8 reaches a larger customer base initially and then drops to zero customers mid-simulation, while Claude Fable 5 sustains a smaller customer base throughout the simulation. These results show that CEO-Bench exposes fine-grained behavioral patterns that remain invisible to existing evaluations, while revealing substantial headroom in models’ ability to integrate individual capabilities into coherent, adaptive behavior over extended horizons.

## Section 2 Designing CEO-Bench

In this section, we provide an overview of how the CEO-Bench simulator works. Then, we describe how we design the action interface to make the task open-ended. Finally, we detail world mechanics design considerations that make the task challenging and realistic.

### 2.1 How CEO-Bench Works

In CEO-Bench , an agent runs a fictional subscription-software company called NovaMind for 500 simulated days. It begins on day one with zero customers and $1M in cash and is graded on cash on hand at the end. If cash ever falls strictly below zero, bankruptcy ends the simulation. We provide an overview of the simulator mechanics in this section and include full details in Appendix A .

What an agent can do. For each simulated week, the agent can take actions for unlimited turns across 34 tools in the categories displayed in Table 1 . These categories cover pricing and plan design, growth and market expansion, product quality and research, reliability and support, information acquisition, public communication, and enterprise sales. Each tool accepts fine-grained structured arguments, so agents can compose a large space of possible policies. Section 2.3 explains the tool interface design in more detail.

How an agent makes and loses money. An agent makes profits through customer subscription payments and in-product ad monetization. We abstract the company product that customers subscribe to as a numerical product quality. Higher product quality results in more product subscriptions and payments, but maintaining quality via development, research, infrastructure capacity, support, and model tier choices requires spending. Acquiring customers through advertising channels also costs money. Cash therefore changes through both immediate costs and delayed revenue effects. We show the calculation of cash change between each day and decompose each contributing factor in Equation 1 . We fully explain the role and mechanism of each factor in Appendix A.3 , A.4 and A.8 . B t + 1 − B t ⏟ daily cash change = Y t sub ⏟ subscription payments + ∑ i Y i , t ads ⏟ in-product ads − K κ t capacity ⏟ capacity cost − ∑ p χ p usage ​ U p , t use ⏟ usage compute cost − x t ops ⏟ support spending − x t dev ⏟ dev spending − X t target ​ - ​ ops ⏟ targeted support − ∑ g x g , t target ​ - ​ dev ⏟ targeted dev − ∑ c , g x c , g , t ads ⏟ acquisition ads − N t lead ​ c lead ⏟ lead acquisition − K t market ⏟ market research − K t group ⏟ group research − K t project ⏟ research projects \displaystyle\begin{aligned} \underbrace{B_{t+1}-B_{t}}_{\begin{subarray}{c}\text{daily cash}\\ \text{change}\end{subarray}}={}&\underbrace{Y_{t}^{\mathrm{sub}}}_{\begin{subarray}{c}\text{subscription}\\ \text{payments}\end{subarray}}+\underbrace{\sum_{i}Y_{i,t}^{\mathrm{ads}}}_{\begin{subarray}{c}\text{in-product}\\ \text{ads}\end{subarray}}-\underbrace{K_{\kappa_{t}}^{\mathrm{capacity}}}_{\begin{subarray}{c}\text{capacity}\\ \text{cost}\end{subarray}}-\underbrace{\sum_{p}\chi_{p}^{\mathrm{usage}}U_{p,t}^{\mathrm{use}}}_{\begin{subarray}{c}\text{usage compute}\\ \text{cost}\end{subarray}}-\underbrace{x_{t}^{\mathrm{ops}}}_{\begin{subarray}{c}\text{support}\\ \text{spending}\end{subarray}}-\underbrace{x_{t}^{\mathrm{dev}}}_{\begin{subarray}{c}\text{dev}\\ \text{spending}\end{subarray}}\\ &-\underbrace{X_{t}^{\mathrm{target\text{-}ops}}}_{\begin{subarray}{c}\text{targeted}\\ \text{support}\end{subarray}}-\underbrace{\sum_{g}x_{g,t}^{\mathrm{target\text{-}dev}}}_{\begin{subarray}{c}\text{targeted}\\ \text{dev}\end{subarray}}-\underbrace{\sum_{c,g}x_{c,g,t}^{\mathrm{ads}}}_{\begin{subarray}{c}\text{acquisition}\\ \text{ads}\end{subarray}}-\underbrace{N_{t}^{\mathrm{lead}}c^{\mathrm{lead}}}_{\begin{subarray}{c}\text{lead}\\ \text{acquisition}\end{subarray}}-\underbrace{K_{t}^{\mathrm{market}}}_{\begin{subarray}{c}\text{market}\\ \text{research}\end{subarray}}-\underbrace{K_{t}^{\mathrm{group}}}_{\begin{subarray}{c}\text{group}\\ \text{research}\end{subarray}}-\underbrace{K_{t}^{\mathrm{project}}}_{\begin{subarray}{c}\text{research}\\ \text{projects}\end{subarray}}\end{aligned} (1)

Modeling customers and indirect feedback. There are 26 customer groups in the simulator. Each customer group consists of a distribution of hidden price and quality preferences, such as a maximum willingness to pay and a minimum accepted quality at each price. Each customer is created by sampling its unique preference parameters from a group distribution. At a subscription plan’s price, a customer subscribes if the offered product quality exceeds the customer’s minimum accepted quality. The customer may switch plans if another plan gives a better quality surplus and may cancel if no plan remains acceptable. We show the mathematical definition of a customer’s price-quality preference curve in Equation 5 . We describe full details of each factor in Appendix A , Subsection A.2 . Example curve plots appear in Fig. 16 . Customer satisfaction changes company reputation, and reputation affects the new customer acquisition rate. The agent does not directly observe satisfaction, willingness to pay, or quality thresholds. It instead infers feedback by analyzing subscription, churn, support, revenue, and reputation data and by monitoring simulated social media.

Customer acquisition and enterprise negotiation. Agents acquire new customers by spending on advertising channels. Each customer group reacts differently to each ad channel, so the same spend can produce different acquisition rates across groups. Reputation, social media reactions, market saturation, demand surges, and macroeconomic conditions also affect acquisition speed. We show the calculation of expected new prospective customers for group g g on day t t in Equation 2 . We describe full details of each factor in Appendix A.5 and A.6 . We sample daily from a Poisson distribution parameterized by this expectation. Market research can reveal additional customer groups and improve what the agent knows about known groups. Enterprise customers follow the same price and quality logic, but deals are negotiated through offers, counter-offers, reply delays, and possible rejection. 𝔼 ⁡ [ n g , t prospect ] ⏟ expected new prospective customers for group ​ g = R g , t ⏟ reputation in group ​ g ⋅ D g , t ⏟ market saturation for group ​ g ⋅ C t ⏟ calendar cycle ⋅ M g , t ⏟ macro econ cycle ⋅ A g , t ⏟ social media reaction ⋅ Z t ⏟ demand surge ⋅ ( ∑ c x c , g , t ​ L c , g , t x ad ⏟ leads from each ad channel + ∑ h N h , t ​ W h , g net ⏟ networking effect from each group ) \displaystyle\begin{aligned} \underbrace{\mathbb{E}\!\left[n_{g,t}^{\mathrm{prospect}}\right]}_{\begin{subarray}{c}\text{expected new prospective}\\ \text{customers for group }g\end{subarray}}={}&\underbrace{R_{g,t}}_{\begin{subarray}{c}\text{reputation}\\ \text{in group }g\end{subarray}}\cdot\underbrace{D_{g,t}}_{\begin{subarray}{c}\text{market saturation}\\ \text{for group }g\end{subarray}}\cdot\underbrace{C_{t}}_{\begin{subarray}{c}\text{calendar}\\ \text{cycle}\end{subarray}}\cdot\underbrace{M_{g,t}}_{\begin{subarray}{c}\text{macro econ}\\ \text{cycle}\end{subarray}}\cdot\underbrace{A_{g,t}}_{\begin{subarray}{c}\text{social media}\\ \text{reaction}\end{subarray}}\cdot\underbrace{Z_{t}}_{\begin{subarray}{c}\text{demand}\\ \text{surge}\end{subarray}}\cdot\left(\underbrace{\sum_{c}\frac{x_{c,g,t}L_{c,g,t}}{x_{\mathrm{ad}}}}_{\begin{subarray}{c}\text{leads from each}\\ \text{ad channel}\end{subarray}}+\underbrace{\sum_{h}N_{h,t}W^{\mathrm{net}}_{h,g}}_{\begin{subarray}{c}\text{networking effect}\\ \text{from each group}\end{subarray}}\right)\end{aligned}

Product quality and competitor pressure. Product quality is affected by daily development, research projects, model tier choices, targeted development, infrastructure capacity, support spending, usage quotas, and in-app ad strength. These controls shape customer experience through base product quality, quota fulfillment, system overload, support delays, relationship history, and ad load. Quota shortfalls multiply the whole perceived-quality expression. Competitors add pressure by periodically raising customer quality expectations. Broad product development and research can make competitors catch up faster, while targeted development for specific groups is harder to copy and lets competitors catch up more slowly. We show the computation of a customer’s perceived product quality and breakdown of each factor in Equation 3 . We describe full details of each factor in Appendix A.3 , A.4 , and A.6 .