CEO-Bench: Can Agents Play the Long Game? — part 2/7
Cleaned-text code points [15911, 31800). Bibliography omitted here; source.md includes it.
Q i , t perc ⏟ quality perceived by customer i = ϕ i , p , t quota ⏟ quota factor [ m p ⏟ model-tier effect ( q 0 ⏟ initial quality + b t shared ⏟ dev improvement + b g , t group ⏟ targeted dev improvement ) − β o o t ⏟ overload penalty + β r ( r i , t − r 0 ) ⏟ customer relationship + β d log ( α d + d i , t / d 0 ) ⏟ customer stickiness − β I I i , t ⏟ open issues penalty − η i ads a i , t eff ⏟ in-app ads penalty ] \displaystyle\begin{aligned} \underbrace{Q_{i,t}^{\mathrm{perc}}}_{\begin{subarray}{c}\text{quality perceived}\\ \text{by customer }i\end{subarray}}={}&\underbrace{\phi_{i,p,t}^{\mathrm{quota}}}_{\text{quota factor}}\Bigg[\underbrace{m_{p}}_{\text{model-tier effect}}\left(\underbrace{q_{0}}_{\text{initial quality}}+\underbrace{b_{t}^{\mathrm{shared}}}_{\text{dev improvement}}+\underbrace{b_{g,t}^{\mathrm{group}}}_{\text{targeted dev improvement}}\right)-\underbrace{\beta_{o}o_{t}}_{\text{overload penalty}}\\ &+\underbrace{\beta_{r}(r_{i,t}-r_{0})}_{\text{customer relationship}}+\underbrace{\beta_{d}\log\!\left(\alpha_{d}+d_{i,t}/d_{0}\right)}_{\text{customer stickiness}}-\underbrace{\beta_{I}I_{i,t}}_{\text{open issues penalty}}-\underbrace{\eta_{i}^{\mathrm{ads}}a_{i,t}^{\mathrm{eff}}}_{\text{in-app ads penalty}}\Bigg]\end{aligned}
Changing world imposes challenges. The world evolves over time through macroeconomic trends, interconnected reputation propagation, market saturation, demand surges, and competitor pressure. These factors affect acquisition, retention, and enterprise deal outcomes. The challenge is that the agent observes only partial and delayed evidence of these changes. It must infer hidden customer and market conditions from traces, choose actions whose effects arrive on different time scales, and revise its policy as the company and market move. Section 2.3 explains the interface design, and Section 2.2 describes the design principles that make the simulator mechanics realistic and challenging.
### 2.2 How We Make CEO-Bench Rigorous and Challenging
We design CEO-Bench ’s world mechanics to be an expressive emulation of the real world, while remaining mechanistic so that success depends on genuine skills rather than exploiting brittle simulations. We describe seven core principles in our world mechanics design below and illustrate four examples in Fig. 5 .
#### Maximize realism with granular simulation.
The simulator models 26 customer groups and individual customers within each group rather than only aggregate demand. Each customer has its own acquisition path, subscription state, price exposure, usage, satisfaction, and cancellation trajectory. Customers are also organized into diverse groups with different needs, budgets, price sensitivities, ad channel effectiveness, support expectations, and behavioral patterns. This granularity increases the complexity of world dynamics and widens the set of viable strategies.
#### Robust simulation with mechanistic rules.
The world emulates real business behavior while maintaining stable cause-and-effect relationships. Almost all simulator outcomes are generated by explicit mechanisms rather than by using an LLM as an opaque judge. For example, customers decide whether to subscribe by comparing product value against price through a microeconomics-motivated participation rule ( Mussa and Rosen, 1978 ) . This design aims to avoid failure modes in benchmarks such as Vending-Bench ( Backlund and Petersson, 2025a ; Backlund and Petersson, 2025b ) , where an LLM-simulated supplier can reward agent’s unrealistic verbal promises.
#### Consistent simulation under stochasticity.
While we inject stochasticity into world dynamics to emulate real-world noise, we maintain consistency across runs with independent random number generators for different simulator components. For example, under the same random seed, after calling the market research tool multiple times, the agent always discovers the same sequence of new market groups, independent of actions in other areas.
#### Hidden information and indirect feedback.
CEO-Bench tests whether agents can gather information in a partially observable world. The agent receives only information that a real start-up manager could plausibly observe: dashboards, database records, social-media posts, research reports, and negotiation history. It does not observe true customer satisfaction, latent willingness to pay, churn propensity, competitor schedules, or demand parameters. Instead, it must infer these hidden variables indirectly, for example, by gauging customer satisfaction and complaints through social media or detecting competitor moves by analyzing cancellation behavior.
#### Interconnected world dynamics.
We design the simulated world to make it difficult to isolate a single causal relationship and hill-climb on it. Every decision can influence many other parts of the market. For example, reputation propagates across related groups, so a quality failure in one enterprise group can spill into nearby groups and eventually affect consumer demand. Increasing satisfaction of influential customer groups can boost growth more effectively than ads.
#### Delayed and uncertain consequences.
Many actions have delayed and uncertain effects, forcing long-horizon decision making under uncertainty. Costs may appear immediately, while the corresponding revenue, retention, research, or reputation effects arrive weeks later. R&D projects have stochastic completion timelines and quality improvements, so investing more does not deterministically produce an immediate gain. Enterprise negotiations also unfold over stochastic delays, making it costly to wait too long but risky to overreact to any single turn. We show the types of distributions used and example usage in Table 2 .
#### Non-stationary environment.
Agents must continually gather new information and adapt because the environment changes over the course of a simulation. Competitors place adaptive pressure on product quality. Customer behavior also drifts over time, with different groups shifting at different rates in price sensitivity and quality expectations. Macroeconomic trends add another changing background process, affecting willingness to pay and enterprise seat counts across expansions and contractions.
### 2.3 A Versatile Action Interface Between World and Agent
We design a programmable tool interface, so agents can effectively manage granular action spaces and organize them into custom workflows.
#### Composable action interface in Python.
Terminal-based computer-use agents have become a general form factor across tasks ( Anthropic, 2026 ; OpenAI, 2026 ; OpenCode, 2026 ; Pi Contributors, 2026 ) . We make evaluating CEO-Bench easy with any of these agents by exposing the action surface to the agent via a Python package, novamind_api . An agent manages the company by calling functions in novamind_api in a Python script and executing the script in its terminal. This design maximizes flexibility for an agent to build its own infrastructure on top of the API. In Fig. 6 (right), we show an example where, rather than calling a tool once per customer, an agent connects to the database via its custom data-driven promotion management system and applies promotion decisions efficiently at scale.
#### Granular action spaces.
We allow agents to act at fine granularity to create a rich space of strategic tradeoffs, failure modes, and opportunities for adaptation. Although the interface contains a finite set of tools, each tool accepts fine-grained structured arguments, so agents can compose a combinatorially large space of possible actions. In Fig. 6 (middle), we show examples where the agent allocates advertising spend by (ad channel, customer group) pair and decides operations spending on individual customers.
#### Large-scale and realistic databases.
We give the agent access to a 19-table operational database covering orders, contracts, subscriptions, the cash ledger, the social-media feed, configuration history, ad-channel attribution, and support tickets, among others. The schema mirrors what a real software company’s analytics stack would expose, testing the agent’s capability to gather information via an analytics workflow that resembles real-world software company operations. In Fig. 6 (left), we show an example where the agent analyzes its revenue through database queries.
#### Social media.
The agent can read a simulated public feed of customer complaints, competitor announcements, and macroeconomic trends. Agents can also reply and post on social media. Reactions to the agent’s posts on social media can also influence the rate of new customer acquisition. We test the agent’s capability to both perceive and act in a chaotic natural-language domain.
## Section 3 Experiments and Results
Model Bankruptcy Best-run cash ($) Max survival days Mean survival days ± \pm std Turns /week Best run API cost Claude Fable 5 1/3 12,630,078 500 461.7 ± 54.2 461.7\pm 54.2 9.86 $265.43 GPT-5.6 Sol 2/3 11,313,982 500 294.0 ± 145.7 294.0\pm 145.7 25.96 $153.47 Claude Opus 4.8 1/3 2,399,209 500 378.0 ± 172.5 378.0\pm 172.5 16.64 $348.49 Qwen 3.7 Max 0/3 365,346 500 500.0 ± 0.0 500.0\pm 0.0 7.82 – Gemini 3.5 Flash 0/3 75,126 500 500.0 ± 0.0 500.0\pm 0.0 58.37 $67.09 Claude Opus 4.7 1/3 70,620 500 414.3 ± 121.2 414.3\pm 121.2 14.33 $92.32 Claude Sonnet 5 1/3 64,459 500 470.7 ± 41.5 470.7\pm 41.5 20.50 $68.25 Claude Haiku 4.5 2/3 59,625 500 286.3 ± 153.3 286.3\pm 153.3 19.35 – GLM 5.2 2/3 54,327 500 376.7 ± 105.6 376.7\pm 105.6 25.24 – Kimi K2.6 0/3 43,598 500 500.0 ± 0.0 500.0\pm 0.0 16.64 – Claude Sonnet 4.6 1/3 38,166 500 419.3 ± 114.1 419.3\pm 114.1 14.37 $73.28 GPT-5.5 2/3 33,260 500 401.0 ± 98.3 401.0\pm 98.3 30.41 $153.80 GLM 5.1 3/3 0 319 155.7 ± 130.0 155.7\pm 130.0 98.66 – DeepSeek V4 Pro 3/3 0 160 136.3 ± 23.7 136.3\pm 23.7 21.85 – Gemini 3 Flash 3/3 0 150 137.0 ± 10.2 137.0\pm 10.2 15.81 $1.19 Grok 4.20 3/3 0 59 36.0 ± 18.4 36.0\pm 18.4 13.49 $3.64 Rule-based baseline – 15,756,408 500 – – – Estimated final cash upper bound – 2,200,000,000 – – – –
In this section, we describe our experiments and their results. We then conduct both qualitative and quantitative analysis to compare behaviors across models.
### 3.1 Experimental Setup
Models. We evaluate agents on the full 500-day CEO-Bench simulation. Each model is given $1M starting cash. We run three simulations for each model with random seed 42. Each model uses the maximum reasoning effort provided. We evaluate Claude Fable 5, GPT-5.6 Sol, Claude Opus 4.8, Claude Opus 4.7, Claude Sonnet 5, Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.5, Qwen 3.7 Max, Gemini 3.5 Flash, Gemini 3 Flash, GLM 5.2, GLM 5.1, Kimi K2.6, DeepSeek V4 Pro, and Grok 4.20.
Harness. Terminal-based computer-use agents have become a general interface for automation: systems such as Claude Code, Codex, OpenCode, and Pi can perform diverse tasks and maintain memory by interacting with a terminal ( Anthropic, 2026 ; OpenAI, 2026 ; OpenCode, 2026 ; Pi Contributors, 2026 ) . We design CEO-Bench to be compatible with any such agent. To align the harness across all models, we implement a minimal terminal agent interface: we give each agent a Linux working directory and tools including bash, read-file, and edit-file. In early runs, we found that open-source harnesses such as OpenCode and Pi (pi-mono) did not manage context reliably enough for 500-day episodes, so our harness refreshes context by clearing action history and only keeping system prompt and an agent-editable memory file in context at the start of each simulated week.
Result selection for analysis. For results and analysis, we select the best run for each model as follows: (1) if at least one of the three runs avoids bankruptcy, we choose the run with maximum ending cash; (2) if all runs end in bankruptcy, we choose the run with the maximum number of simulation days before bankruptcy.
### 3.2 Results Overview
Overall results. We show the best-run cash over time for each model in Fig. 2 , and per-model trajectories across all three runs in Fig. 3 . We also show additional details in Table 3 . Most state-of-the-art models struggle to complete the simulation without bankruptcy. GPT-5.6 Sol finishes with $11.31M in its best run, second only to Claude Fable 5, but its other two runs bankrupt around day 190. Claude Fable 5, GPT-5.6 Sol, and Claude Opus 4.8 are the only evaluated models whose best runs finish above the $1M starting balance. GPT-5.5 reaches $6.58M mid-run but is unable to preserve profitability across complete runs, ending at $33K in its best completed run. Several models survive with positive cash below the starting balance, while GLM 5.1, DeepSeek V4 Pro, Gemini 3 Flash, and Grok 4.20 bankrupt on all runs. This preliminary evaluation shows that the strongest models demonstrate high-upside strategic behavior, but most models still fail to coordinate growth, quality, and cash flow over the full horizon.
Rule-based baseline. We include a simple rule-based heuristic baseline that uses no language-model calls during policy execution: it fixes prices, quotas, and model tiers, concentrates acquisition and targeted development on a small set of customer groups, and adjusts capacity from recent usage. We conduct a preliminary grid search over this rule template and display the best strategy in Fig. 2 and Table 3 . The heuristic achieves a significant positive cash balance of $15.76M, above all evaluated model agents. We show full details of this baseline strategy’s design and configuration search in Appendix B .
Benchmark is far from saturated. We estimate loosely the upper bound of achievable final cash be around $2.2B. The estimation sums revenue from all 26 customer groups under maximum supportable pricing and subtracts the required costs for compute, capacity, development, operations, advertising, research, and acquisition. To obtain a conservative estimate, we further adjust downward for execution frictions, including issue-driven churn, enterprise negotiation friction, and acquisition delays. The resulting estimate remains far above the best observed model performance, indicating that CEO-Bench is far from saturated. We detail our estimation process in Appendix D . We release agent trajectories of all our experiments in an interactive trajectory viewer .
### 3.3 A Look into Agent Behaviors
In this section, we take a preliminary exploration in agent behaviors and compare them across models.
Strong models explore wider strategy space. In Fig. 7 , we show example memos written by agents over time. Claude Fable 5 keeps trying different ways to run the company as conditions change, and continues adjusting late in the run. Claude Opus 4.8 explores broadly at first, then becomes more passive after building a cash cushion. In contrast, Claude Opus 4.7 narrows much earlier, repeatedly choosing to spend less, wait, and protect cash. In Fig. 11 , we find the same pattern quantitatively: Claude Fable 5 and Claude Opus 4.8 spread their actions more evenly across tools than Claude Opus 4.7.
Agents attain high final cash through distinct strategies. Claude Opus 4.8, Claude Fable 5, and GPT-5.6 Sol all finish above the starting cash in their best runs, but they attain this result through distinct strategies. In Fig. 9 , we show their very different customer-base trajectories. Claude Opus 4.8 builds a larger early customer base and then drops to zero customers mid-simulation, GPT-5.6 Sol scales even earlier and gradually winds the customer base down to zero by day 420, while Claude Fable 5 sustains a smaller customer base centered on Individual Group 2 into the final week. Fig. 7 also shows that Claude Opus 4.8 pivots toward harvesting after its early growth phase, whereas Claude Fable 5 continues to revise its plan late in the run.