CEO-Bench: Can Agents Play the Long Game? — part 3/7

Cleaned-text code points [31800, 47612). Bibliography omitted here; source.md includes it.

Sophisticated analytics by top-performing agents. In Fig. 8 , we show example code files that top-performing agents write and execute. Claude Opus 4.8 estimates which customers are likely to stay or leave by looking at who keeps paying and who cancels. Claude Fable 5 tracks when payments arrive, how costs accumulate, and forecasts how much cash remains in the final weeks under different customer-loss assumptions. These examples demonstrate initial signs of sophisticated planning and information acquisition.

### 3.4 Measuring Drivers of Success and Failure

While success in CEO-Bench requires multiple skills to work together, we conduct a preliminary analysis by isolating three behavioral measures and comparing them against agent performance. We compare quantitative measures for Claude Fable 5, Claude Opus 4.8, and three lower-performing comparison models in Fig. 12 , and explain each comparison further below.

Uncovering hidden information. In CEO-Bench , each pair of ad channel and customer group has a different new-customer acquisition rate, emulating real-world heterogeneity. The acquisition rate is hidden from the agent, so the agent must uncover and use this information by analyzing customer acquisition history in databases. In Fig. 12 , we measure the average percentage of ad spending allocated to the best channel out of all ad spending for a customer group. Claude Opus 4.8 and Claude Fable 5 allocate more spend to the true best channel than the random-guessing baseline of 20 % 20\% , while Gemini 3 Flash, DeepSeek V4 Pro, and Grok 4.20 fall below it.

Seeing into the future. In each simulated week, we ask the agent to submit a cash forecast four weeks into the future. Fig. 12 plots the percentage error between the submitted forecast and the realized cash balance four weeks later against the number of days before bankruptcy. We average data from the first four simulation weeks, when most models are still alive. Claude Opus 4.8 and Claude Fable 5 forecast future cash with much lower early error than Gemini 3 Flash, DeepSeek V4 Pro, and Grok 4.20, indicating a stronger understanding of world dynamics and delayed consequences.

Planning. We find that the stronger runs frequently anticipate different future scenarios and build corresponding solutions in their memos, with examples shown in Fig. 13 . The qualitative examples from Claude Fable 5 and Claude Opus 4.8 show explicit “if-then” contingencies, while the aggregate comparison in Fig. 12 shows that Claude Opus 4.8 and Claude Fable 5 use the word “if” more frequently than the lower-performing comparison models.

Taking fine-grained actions. Our simulator allows agents to take actions at a fine-grained level. For example, agents can decide customer-group specific product development strategy. Proper analysis of customer group information and targeted development can create advantages such as slower competitor catch-up. Fig. 10 shows the dollar-weighted split between targeted and non-targeted development spending. Claude Fable 5 allocates 88% of development dollars to targeted improvements, Claude Opus 4.8 allocates 51%, Qwen 3.7 Max allocates 25%, GLM 5.1 allocates 19%, and Grok 4.20 allocates 11%. This comparison demonstrates that stronger models acquire fine-grained understanding about world dynamics and make use of the understanding by taking granular actions more.

## Section 4 Ablating Simulator and Agent Configurations

We examine how simulation outcomes change when varying competitor and time-horizon configurations. We find that competitor difficulty provides an effective knob for tuning task difficulty, and our task remains challenging for existing models even over a short horizon. We also show that the choice of agent harness impacts results significantly.

### 4.1 Ablating Competitor Difficulty

We ablate simulator difficulty by varying the competitor configuration while using Claude Sonnet 4.6 as the agent model. In CEO-Bench , the competitor raises customer expectations through both a preset stationary sequence and adaptive responses to agent actions. In the adaptive component, the competitor raises customer expectations by u ⋅ I u\cdot I , where u ∼ U ⁡ [ 0.2 , 0.5 ] u\sim U[0.2,0.5] and I I is the agent’s cumulative quality improvement. We ablate simulator difficulty with the following settings: (1) stationary + adaptive competitor with u ∈ { 0.1 , 0.2 , 0.3 } u\in\{0.1,0.2,0.3\} ; (2) stationary competitor only; (3) no competitor. The default line uses the best Claude Sonnet 4.6 trajectory under the standard configuration.

In Fig. 14 (a), we show that reducing competitor strength generally makes the task easier, and removing the competitor makes the task significantly easier. The ablation shows that competitor strength can be an effective knob for tuning task difficulty, and the non-stationary environment is a crucial component of what makes the task challenging.

### 4.2 Ablating Time-Horizon

We examine whether agents behave differently when told to maximize cash balance over a shorter horizon. In this experiment, we change the simulation period to 50 days, one-tenth of the original simulation period. While the shortened horizon reduces challenges in long-term planning, Fig. 14 (b) shows that no evaluated model finishes above its starting cash balance. This analysis reveals that most models today remain weak in orchestrating decisions toward a short-term goal.

### 4.3 Ablating Agent Harness

CEO-Bench can easily evaluate any agent harness. While we obtain most results with a custom minimal terminal-using agent harness, we ablate popular agent harnesses while keeping the underlying model fixed. For Claude Fable 5, we compare results with Claude Code ( Anthropic, 2026 ) , and for GPT-5.6 Sol, we compare results with Codex ( OpenAI, 2026 ) . In Fig. 15 , we show that switching harnesses massively changes agent behaviors. Agents take significantly fewer turns per simulated day when using Claude Code and Codex, resulting in inferior performance. While we cannot access full implementation details of these harnesses, we hypothesize that the difference results from software engineering-oriented system prompts of these harnesses.

## Section 5 Related Work

#### Language model evaluations.

Language-model evaluation has moved from static knowledge and reasoning ( Hendrycks et al., 2021 ; Srivastava et al., 2023 ; Liang et al., 2023 ; Rein et al., 2024 ) toward realistic agentic task execution ( Chen et al., 2021 ; Jimenez et al., 2024 ; Zhou et al., 2024 ; Xie et al., 2024b ; Drouin et al., 2024 ; Trivedi et al., 2024 ; Yoran et al., 2024 ; Yao et al., 2025 ; Liu et al., 2024 ; Ma et al., 2024 ) . Recent benchmarks have expanded evaluation scope to economically valuable deliverables in broad domains ( Patil et al., 2025 ; Mialon et al., 2024 ; Chan et al., 2025 ; Starace et al., 2025 ; Miserendino et al., 2025 ; Patwardhan et al., 2025 ) . However, their objectives usually terminate at a target state or one-shot deliverable. CEO-Bench instead asks whether agents can sustain progress toward a distant objective as earlier decisions continue to shape later states.

#### Long-horizon agent evaluation.

Memory and continual-learning benchmarks test models’ ability to retain information over time ( Bai et al., 2024 ; Hsieh et al., 2024 ; Wu et al., 2025 ; Hu et al., 2026 ; He et al., 2026b ; Laskin et al., 2023 ; Wang et al., 2023 ; Monea et al., 2024 ) , but they are often limited to information retrieval or a single static task. Long-horizon benchmarks extend evaluation from isolated tasks to processes that unfold over time ( Wu et al., 2024 ; Xie et al., 2024a ; Xu et al., 2025 ; Chan et al., 2025 ; Starace et al., 2025 ; Wang et al., 2025 ; Luo et al., 2025 ; He et al., 2026a ) . Most recently, Vending-Bench asks agents to run a vending machine over many days, and AccountingBench asks agents to close monthly books from real software company data ( Backlund and Petersson, 2025a ; Backlund and Petersson, 2025b ; Penrose AI, 2025 ) . However, they involve narrow operating problems, fewer coupled decisions, and largely stable or observable environments. CEO-Bench evaluates long-horizon agency in a broader operating setting where agents must coordinate pricing, growth, product, operations, communication, and enterprise sales under hidden state, noisy feedback, delayed consequences, and non-stationary market pressure in a consistent simulator. For example, we show in Appendix C that Vending-Bench allows models to accumulate successes relatively steadily, while our simulator requires an agent to make significant investments that only pay back much later, posing a stronger challenge to long-horizon planning.

## Section 6 Limitations and Conclusion

### 6.1 Limitations

We make our best effort to approximate real-world startup operations and challenges in CEO-Bench . However, discrepancies can still exist between reality and the approximation. For example, since we have not found a reliable way to evaluate a model’s capability to propose qualitative changes to products, we simulate products using only a quality measure. In addition, to make each simulation run economically feasible, we limit the scope of possible actions and leave out aspects such as compliance, security, and fundraising.

### 6.2 Conclusion

CEO-Bench shows a gap between existing models’ local tool competence and crucial sustained strategic skills: agents built on existing models can take plausible actions but fail when those actions must compound under delayed feedback, hidden state, and non-stationarity. To develop agents beyond isolated task executors, we need evaluations that ask whether they can organize evolving systems toward distant goals. CEO-Bench is one step toward that future: building agents and training models that do not merely answer requests, but help steer long-running organizations through uncertainty.

## Acknowledgments

We thank Modal for providing GPU resources for LLM inference. We thank Shuer Jiang, Boya Zeng, Sachin Konan, Taiming Lu, Linrong Cai, David Yin, Rahul Chalamala, Bryan Chiang, Luke Zeller, Yunyu Lin, Berkan Dokmeci, Bennett O’Brien, Ashank Tomar, and Ang Li for discussions and feedback. We thank Spencer Hong and The General Intelligence Company of New York for additional evaluations. KN acknowledges support from Schmidt Sciences.

## Appendix

## Appendix A Simulator Mechanics

We describe full details of simulator mechanics in this section.

### A.1 Agent Commands and Observable State

#### Python action surface.

The agent changes the company by importing functions from novamind_api and executing Python. The active command groups are: • pricing : set_prices , set_model_tiers , set_usage_quotas , and set_promotion .

• marketing : set_daily_spend , set_targeted_ad_spend , and set_ads_strength ; set_lead_promotion and post_social_media .

• analytics : set_targeted_ops_spend and set_targeted_dev_spend .

• research : start_research_project and list_research_projects .

• market : research_market , research_group , get_market_overview , and get_group_insights .

• infrastructure : set_capacity_tier and get_cost_info .

• enterprise : send_enterprise_deal and reject_enterprise_deal .

• Time and data access : next_week advances the simulator, query reads the company database, and get_vars reports runtime variables.

The interface is intentionally operational rather than omniscient: it exposes dashboards, database tables, social posts, and inbox messages, while leaving true preferences, satisfaction, competitor schedules, and hidden macro state latent.

### A.2 Customers, Plans, and Participation

#### Customer parameter sampling.

Each customer belongs to a group g g and receives private parameters such as maximum willingness to pay, quality floor, quality ceiling, usage demand, ad sensitivity, support sensitivity, and enterprise negotiation traits. For a generic customer parameter k k , θ i , k , t = clip ⁡ ( θ ~ i , k + Δ g , k , t market , θ g , k min , θ g , k max ) , θ ~ i , k ∼ 𝒟 g , k ​ ( μ g , k , σ g , k ) , \theta_{i,k,t}=\operatorname{clip}\!\left(\tilde{\theta}_{i,k}+\Delta_{g,k,t}^{\mathrm{market}},\theta_{g,k}^{\min},\theta_{g,k}^{\max}\right),\qquad\tilde{\theta}_{i,k}\sim\mathcal{D}_{g,k}(\mu_{g,k},\sigma_{g,k}), (4) where θ i , k , t \theta_{i,k,t} is customer i i ’s active value for parameter k k , θ ~ i , k \tilde{\theta}_{i,k} is the sampled base value, 𝒟 g , k \mathcal{D}_{g,k} is the configured group-level sampling distribution, μ g , k \mu_{g,k} and σ g , k \sigma_{g,k} are its location and spread parameters, Δ g , k , t market \Delta_{g,k,t}^{\mathrm{market}} is accumulated market drift, and θ g , k min , θ g , k max \theta_{g,k}^{\min},\theta_{g,k}^{\max} are clipping bounds. The distribution terms encode the fact that a market segment has a typical profile, while the sampled base value gives each customer an idiosyncratic budget, tolerance, or usage pattern. The drift term represents changing market conditions, such as customers becoming more demanding over time, and the clipping bounds keep preferences in plausible ranges. This mechanism simulates real customer cohorts: people in the same segment resemble one another, but they are not interchangeable, and their preferences can move as the market changes.

#### Customer participation curve.

Each customer i i has maximum monthly willingness to pay c i c_{i} , minimum quality floor q i min q_{i}^{\min} , quality ceiling q i max q_{i}^{\max} , and low-price and high-price slopes s i L , s i R s_{i}^{L},s_{i}^{R} . For offered effective price C C , define normalized price x = C / c i x=C/c_{i} , quality range Δ ​ q i = q i max − q i min \Delta q_{i}=q_{i}^{\max}-q_{i}^{\min} , and sigmoid σ ⁡ ( z ) = ( 1 + e − z ) − 1 \sigma(z)=(1+e^{-z})^{-1} . With configurable curve coefficients θ Q L , θ Q R , θ Q max , ω Q , k Q , ψ Q min , ψ Q max \theta_{Q}^{L},\theta_{Q}^{R},\theta_{Q}^{\max},\omega_{Q},k_{Q},\psi_{Q}^{\min},\psi_{Q}^{\max} , ψ i ​ ( C ) = \displaystyle\psi_{i}(C)={} ω Q ​ σ ​ ( k Q ​ s i L ​ ( x − θ Q L ) ) + ( 1 − ω Q ) ​ σ ​ ( k Q ​ s i R ​ ( x − θ Q R ) ) , \displaystyle\omega_{Q}\,\sigma\!\left(k_{Q}s_{i}^{L}(x-\theta_{Q}^{L})\right)+(1-\omega_{Q})\,\sigma\!\left(k_{Q}s_{i}^{R}(x-\theta_{Q}^{R})\right), (5) Q i req ​ ( C ) = \displaystyle Q_{i}^{\mathrm{req}}(C)={} q i min + Δ ​ q i ​ clip ⁡ ( ψ i ​ ( C ) , ψ Q min , ψ Q max ) . \displaystyle q_{i}^{\min}+\Delta q_{i}\,\operatorname{clip}\!\left(\psi_{i}(C),\psi_{Q}^{\min},\psi_{Q}^{\max}\right). Here ψ i ​ ( C ) \psi_{i}(C) is the customer’s normalized required-quality score at price C C ; x = C / c i x=C/c_{i} measures price relative to the customer’s willingness to pay; Δ ​ q i \Delta q_{i} is the customer’s personal quality range; θ Q L \theta_{Q}^{L} and θ Q R \theta_{Q}^{R} locate the low- and high-price portions of the curve; ω Q \omega_{Q} blends the two portions; s i L s_{i}^{L} and s i R s_{i}^{R} make some customers more price-sensitive than others; k Q k_{Q} controls global curvature; and ψ Q min , ψ Q max \psi_{Q}^{\min},\psi_{Q}^{\max} bound the score. The mechanic follows a participation-rule view of differentiated products: higher prices require higher perceived quality, and customers near their budget ceiling become more demanding ( Mussa and Rosen, 1978 ) . A non-enterprise customer accepts plan p p only if C i , p , t ≤ θ Q max ​ c i C_{i,p,t}\leq\theta_{Q}^{\max}c_{i} and Q i , p , t perc ≥ Q i req ​ ( C i , p , t ) Q_{i,p,t}^{\mathrm{perc}}\geq Q_{i}^{\mathrm{req}}(C_{i,p,t}) . This simulates the real purchasing rule that customers do not compare price in isolation; they ask whether the product feels good enough for what they are being charged.

#### Plan choice and billing-period switching.