CEO-Bench: Can Agents Play the Long Game? — part 7/7
Cleaned-text code points [93008, 102889). Bibliography omitted here; source.md includes it.
Vending-Bench 2 reports the average Claude Fable 5 High balance over five 365-day runs, with the leaderboard ending near $5.7K from a $500 starting balance ( Backlund and Petersson, 2025b ) . Its curve grows steadily after early operating volatility: once the agent finds workable suppliers and products, gains can accumulate through repeated restocking and sales. In contrast, the best Claude Fable 5 CEO-Bench run begins with $1M, falls to roughly $626K by day 30 as it funds acquisition, development, capacity, and operations, and later compounds to $12.6M by the end of the run. The middle of the CEO-Bench trajectory is also more volatile, with larger repeated drawdowns and recoveries as the agent tests pricing, product investment, capacity, and acquisition policies. Exploration therefore has a direct financial cost: a strategy can be directionally correct while still producing long stretches of unstable cash balance before its benefits appear. This delayed-payoff structure makes local cash preservation an unreliable proxy for success and requires the agent to keep investing through long intervals where the benefits of its actions have not yet fully materialized.
## Appendix D Upper Bound Final-Cash Estimate
We use the estimated maximum final cash only to calibrate remaining headroom in CEO-Bench . It is not a mathematical proof of optimality. The estimate has two stages. We first compute a pre-friction accounting subtotal by summing revenue from all customer groups under maximum supportable pricing and subtracting the costs required by the simulator mechanics. We then apply a friction adjustment for issue-driven churn, enterprise negotiation friction, and acquisition delay. The reported estimate is the adjusted value, approximately $2.2B.
#### Objective.
Let G G denote the 26 customer groups and let B = { 30 , 60 , … , 480 } B=\{30,60,\ldots,480\} denote the billing days. For each group g g , let N g N_{g} be the maximum attainable customer count or enterprise seat count, p g p_{g} be the maximum supportable price, and r g ( t ) r_{g}(t) be the retention probability at billing day t t . These terms define an optimistic full-market revenue calculation: how many customers can be reached, what price they can support, and how likely they are to remain active at each renewal. The pre-friction revenue estimate is R = ∑ g ∈ G N g p g ∑ t ∈ B r g ( t ) . R=\sum_{g\in G}N_{g}p_{g}\sum_{t\in B}r_{g}(t). (45) Here R R is total subscription revenue before execution frictions. The inner sum adds the retained billing opportunities for a group over time, and the outer sum aggregates across all customer segments. This simulates the revenue side of a best-case operating plan in which acquisition succeeds and retention is governed by the modeled quality checks. The pre-friction final-cash subtotal is then C pre = C 0 + R − K , C_{\mathrm{pre}}=C_{0}+R-K, (46) where C pre C_{\mathrm{pre}} is final cash before friction adjustment, C 0 = $ 1 C_{0}=\$1 M is the initial cash balance, and K K is the sum of all modeled simulator costs. This subtotal simulates accounting profit before execution slippage: revenue plus starting cash minus the cost of capacity, compute, development, operations, acquisition, and research. The reported estimate applies a friction factor F F : C reported = F C pre . C_{\mathrm{reported}}=FC_{\mathrm{pre}}. (47) where C reported C_{\mathrm{reported}} is the final headroom estimate after friction and F F is the multiplicative discount applied to the pre-friction subtotal. We use F = 0.49 F=0.49 to make the estimate conservative with respect to issue-driven churn, enterprise negotiation friction, and acquisition delay. This simulates the gap between a clean accounting upper bound and a real execution path, where customers do not all arrive instantly, enterprise deals take negotiation, and support issues can erode retention.
#### Parameter choices.
The accounting assumes all 26 groups are retained through all billing cycles before friction adjustments. The selected configuration uses T3 inference, T7 capacity, $40K/day in global development, targeted development for every group, all ten R&D tiers started in parallel on day 1, and operations spending of $500/day plus $0.001 per active subscriber. These choices were selected from a small grid over targeted-development slack, model tier, global-development spend, R&D schedule, capacity tier, and operations spend. The most sensitive parameter is targeted development: a 1–2 × \times multiplier did not reliably retain all groups late in the run, while a 3 × \times multiplier maintained full-cycle retention in the tested configurations and a 4 × \times multiplier added cost without improving retention.
#### Quality and retention check.
For each candidate configuration, we compute the shared-quality, competitor-drift, and R&D-bonus trajectories over the operating horizon. For group g g , targeted-development spend is first sized to cover the late-run quality threshold, T g ( 0 ) = 5000 [ exp ( Δ g 0.7 A g ⋅ 0.0225 ) − 1 ] , T_{g}^{(0)}=5000\left[\exp\left(\frac{\Delta_{g}}{0.7A_{g}\cdot 0.0225}\right)-1\right], (48) where T g ( 0 ) T_{g}^{(0)} is the one-pass targeted-development spend estimate for group g g , Δ g \Delta_{g} is the required targeted quality bonus after accounting for shared quality and R&D effects, A g A_{g} is the number of active development days, 5000 5000 is the spend scale from the simulator’s targeted-development rule, 0.7 0.7 is the targeted-development conversion coefficient used in this sizing approximation, and 0.0225 0.0225 is the quality-gain coefficient. The exponential is the inverse of the simulator’s log diminishing-returns function: larger required quality gaps require disproportionately more spend. We then use T g = 3 T g ( 0 ) T_{g}=3T_{g}^{(0)} to account for group-level drift feedback that the one-pass sizing rule does not capture. This simulates planning with safety margin, where an operator overbudgets targeted product work because competitors and market drift can raise the bar during execution.
At each billing day, retention is computed from the delivered-quality cushion, r g ( t ) = Φ ( m [ q 0 + q shared ( t ) + b g ( t ) ] − d g ( t ) − ℓ ( t ) − μ g σ g ) , r_{g}(t)=\Phi\left(\frac{m[q_{0}+q_{\mathrm{shared}}(t)+b_{g}(t)]-d_{g}(t)-\ell(t)-\mu_{g}}{\sigma_{g}}\right), (49) where r g ( t ) r_{g}(t) is group retention probability at billing day t t , Φ \Phi is the standard normal cumulative distribution function, m = 0.90 m=0.90 is the T3 quality multiplier, q 0 q_{0} is baseline quality, q shared ( t ) q_{\mathrm{shared}}(t) is shared quality from global development and R&D, b g ( t ) b_{g}(t) is the targeted group bonus, d g ( t ) d_{g}(t) is fixed and competitor-induced group drift, ℓ ( t ) \ell(t) is the capacity-overload penalty, and ( μ g , σ g ) (\mu_{g},\sigma_{g}) parameterize the group’s quality threshold distribution. The numerator is the quality cushion above the group’s threshold, and dividing by σ g \sigma_{g} converts that cushion into a probability under the assumed threshold distribution. Under the selected configuration, the computed retention check gives r g ( t ) ≈ 1 r_{g}(t)\approx 1 for all 26 groups and all 16 billing cycles. Full retention implies approximately 521M usage units per day; with T7 capacity, the resulting overload penalty is included in ℓ ( t ) \ell(t) . This simulates a stress test for whether product quality, targeted improvements, and capacity are strong enough to keep every segment through repeated renewals.
#### Revenue and costs.
The pre-friction calculation gives $6.69B in subscription revenue: $1.93B from individual groups and $4.76B from enterprise groups. We subtract every cost category used by the simulator rather than only direct infrastructure costs. Table 5 shows the resulting cost accounting. This paragraph simulates the full-company accounting view of the upper bound: high revenue is meaningful only after paying for the compute, capacity, product work, support, advertising, and information acquisition required to keep that revenue.
#### Interpretation.
The final row of Table 5 is the estimate used in the main text and in Table 3 . It should be read as an approximate headroom calculation, not as a demonstrated executable strategy. The pre-friction accounting assumes full conversion, full enterprise close rates, rapid acquisition, maximum supportable prices, mean R&D timing and quality effects, and no additional losses from unresolved customer issues, discounts, reputational effects, or cash-flow constraints beyond the cost categories above. The friction factor makes the estimate conservative by reducing this subtotal to roughly $2.2B. This remains far above the best observed agent performance and supports the conclusion that CEO-Bench is far from saturated.
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .