INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agent — part 2/6

Cleaned-text code points [13274, 27644). Bibliography omitted here; source.md includes it.
Figure 2: (1) The language agent’s memory module is crafted to interact with the market environment to conduct various
ﬁnancial decision-making tasks. It contains two core components – Working Memory and Layered Long-term Memory. (2) The
outline of the agent’s decision-making workﬂow for retrieving critical memory events and market observations to inform speciﬁc
investment decisions.
2.2 Modeling ﬁnancial decision-making
Formally, we model a ﬁnancial decision-making
process as inﬁnite horizon POMDP with time index
T = {0, 1, 2, · · · } and discount factor α ∈ (0, 1].
This POMDP contains: (1) a state space X × Y
where X is the observable component and Y is un-
observable component of the ﬁnancial market; (2)
the action space of the agent is A, which is mod-
eled as {“Buy", “Sell", “Hold" }; (3) the reward
function R(o, b, a) : X × Y × A → R uses daily
proﬁt & loss (PnL) as the output; (4) the observa-
tion process {Ot}t∈T ⊆ X is a multi-dimensional
process (5) the reﬂection process {Bt}t∈T ⊆ Y
represents the agent’s self-reﬂection, which is up-
dated from Bt to Bt+1 on daily basis ( Grifﬁths
et al., 2023); (6) the action At ∼ π(·|prompt) rep-
resents the way to make investment decision driven
by the language conditioned policy π. By denoting
daily proﬁt & loss (PnLs) by Rπ
t = R(Ot, Bt, At)
and the set of all admissible language conditioned
policies as Π = {π(·|prompt)}, the optimization
objective of ﬁnancial decision-making task is then:
max
π∈Π
E
[ ∑
t∈T
αtRπ
t
]
(1)
3 InvestorBench
He we introduce the detailed architecture of In-
vestorBench, as illustrated in Figure 1.
3.1 Benchmark Composition
INVESTOR BENCH is organized into four main com-
ponents: (1) Data Sources and Market Environ-
ments: I NVESTOR BENCH utilizes a wide range of
open-source data and incorporates third-party APIs,
such as Y ahoo Finance and SEC EDGAR, to create
a comprehensive, multi-modal market environment
data warehouse. (2) LLM Agent : I NVESTOR -
BENCH includes an advanced LLM-based agent
equipped with modules for Brain, Perception, Pro-
ﬁle, Memory, and Action. This agent is enhanced
with external tools (such as tabular data readers and
API callers) and data operations (including vector
database management, information reinforcement,
and retrieval). (3) Financial Decision-Making
Tasks: I NVESTOR BENCH offers three distinct ﬁ-
nancial decision-making tasks, differentiated by
their asset types. (4) Evaluation Metrics: The efﬁ-
cacy of all tasks within I NVESTOR BENCH is evalu-
ated using a set of standard metrics in the quantita-
tive ﬁnance ﬁeld, providing a thorough evaluation
of the decision-making capabilities of the LLM-
based agent.
3.2 Trading Environments
We release three datasets, each curated from
diverse sources, to construct tailored ﬁnancial
market environments for speciﬁc tasks. Our
objective is to address the current gap in evaluation
environments for ﬁnancial decision-making agent
frameworks and to offer a fully open platform
2512

for the comprehensive assessment of agents
across various tasks. Below, we introduce each
environment, categorized by task type, detailing its
scope and the data sources it incorporates.
Stock market environment integrates information
from multiple sources, including: 1) Daily stock
open, high, low, close, and volume (OHLCV) data
acquired from Y ahoo Finance. 2) Summarized
insights from company quarterly and annual
reports (Form 10-Q and 10-K) downloaded from
the SEC EDGAR database. 3) News articles for
seven stocks collected daily between 2020-07-01,
and 2021-05-06. The news data for four of these
companiesMicrosoft Corporation (MSFT), John-
son & Johnson (JNJ), UVV Corporation (UVV),
and Honeywell International Inc. (HON)-are
randomly selected from the pool with the most new
records (over ﬁve hundred) from the open-access
dataset provided by Zhou et al. ( Zhou et al. ,
2021), while the news data for the remaining
three companiesTesla, Inc. (TSLA), Apple Inc.
(AAPL), and NIO Inc. (NIO)-are obtained from
Reﬁnitiv Real-Time News, which primarily
contains high-quality news information from
Reuters. 4) The sentiment categories ( ’positive’,
’negative’, ’neutral’) assigned to each news record
are generated by gpt-3.5-turbo-0125.
Cryptocurrency market environment encom-
passes 1) the daily stock open-high-low-close-
volume (OHLCV) acquired from CoinMarketCap;
2) the multisource cryptocurrency news data
collected from cryptonews, cryptopotato, and
cointelegraph(V anhoucke, 2023); 3) news spanning
from 2023-02-13 to 2023-11-05 collected by
(Zhou et al. , 2021) in daily frequency. 4) The
sentiment categories generated by the same means.
ETF market environment is constructed using
News-Informed Financial Trend Yield (NIFTY)
dataset ( Saqur et al. , 2024). It contains the pro-
cessed and curated daily news headlines from 2019-
07-29 to 2020-09-21 and generated sentiment cate-
gories for each news headline.
In experimental use, we divide the dataset ac-
cording to the date, with the train set used for the
warmup phase to establish the memory database,
and the test set used for the test phase to evaluate
the model performance.
Table 1: I NVESTOR BENCH evaluates 13 proprietary or
open-source LLMs on ﬁnancial decision-making tasks.
Model #Size Form V er.Model #Size Form V er.
gpt-4(Achiam et al., 2023) N/A api 0613 Qwen2.5-7b(Qwen team, 2024) 7B open Instruct
gpt-4o(OpenAI, 2022) N/A api 0806 Qwen2.5-32b(Qwen team, 2024) 32B open Instruct
gpt-o1-preview N/A api 0912 Qwen2.5-72b(Qwen team, 2024) 72B open Instruct
DeepSeek-v2(Xin et al. , 2024) 15B open Lite llama3.1-8b(Llama team, 2024) 8B open Instruct
DeepSeek-llm(Xin et al. , 2024) 67B open Chat llama3.1-70b(Llama team, 2024) 70B open Instruct
Yi-1.5-9b(Y oung et al., 2024) 9B open Chat Palmyra-Fin(team, 2024) 70B open 32K
Yi-1.5-34b(Y oung et al., 2024) 34B open Chat
3.3 Evaluation metrics
We employ four widely recognized ﬁnancial met-
rics to evaluate and compare the investment per-
formance of various LLMs serving as backbones
across different tasks: : Cumulative Return (CR)
(Hull, 2007), Sharpe Ratio (SR) ( Sharpe, 1994),
Annualized V olatility (A V) (Cochrane, 1988), and
Maximum Drawdown(MDD) ( Ang and Chen ,
2003). Note that CR and the SR are often con-
sidered more essential than A V and MDD in evalu-
ating asset trading performance due to their focus
on long-term gains and risk-adjusted returns by
their deﬁnition. Here, we regard these two met-
rics as primary metrics when evaluating the exper-
iment outcomes. The detailed explanation is in
Appendix B.
4 Experiment and Discussion
To establish a baseline and assess the performance
of LLM agents, we standardize experimental set-
tings and evaluation metrics across various ﬁnan-
cial decision-making tasks. Results are presented
on a task-by-task basis. We report the performance
of I NVESTOR BENCH on three single-asset trading
tasks: stocks, cryptocurrencies, and ETFs trad-
ing, using closed-source, open-source, and domain-
speciﬁc LLMs.
4.1 Experiment Setup
Table 1 summarizes the performance of a compre-
hensive list of trading agents. For single equity
tasks, the baseline is set up by Buy and Hold strat-
egy, while for portfolio management task, it is set
up by an equal-weight portfolio with the detailed
rational explained in Appendix. In our experiments,
the temperature parameter of all LLM-based agent
systems is set at 0.6 to balance response consis-
tency and reasoning creativity. The performance
metrics are reported for the test trajectory with the
median CR, SR, A V , and MDD from ﬁve repeated
epochs. (If the median of these metrics does not
belong to the same epoch, the performance is based
on the trajectory with the median SR.)
Furthermore, the selection of warm-up and test
periods differs across various tasks due to the vary-
2513

Table 2: Performance of stock trading with different LLMs as backbone model across seven stocks.
Model MSFT JNJ UVV HON
CR↑ SR↑ A V↓ MDD↓ CR↑ SR↑ A V↓ MDD↓ CR↑ SR↑ A V↓ MDD↓ CR↑ SR↑ A V↓ MDD↓
Buy & Hold 15.340 0.717 17.236 9.428 13.895 0.927 12.075 9.847 36.583 1.457 20.216 15.406 33.256 1.619 16.537 9.195
Financial Domain Models
Palmyra-Fin-70B 14.697 0.618 18.987 9.428 5.748 0.311 13.329 9.367 37.875 1.407 21.528 15.967 20.016 1.010 15.852 6.824
Proprietary Models
GPT-o1-preview 17.184 0.664 20.700 9.428 13.561 0.749 14.396 9.847 41.508 1.481 22.411 9.633 13.162 0.535 19.673 11.558
GPT-4 16.654 0.643 20.715 9.428 13.712 0.761 14.417 9.860 31.791 1.132 22.471 10.434 34.342 1.383 19.858 9.195
GPT-4o 12.461 0.638 15.631 6.647 9.099 0.604 12.055 7.169 8.043 0.342 18.796 14.889 38.540 1.668 18.480 8.979
Open-Source Models
Qwen2.5-72B-Instruct 7.421 0.406 14.654 6.973 14.353 0.787 14.487 9.812 37.178 1.257 23.614 13.365 34.309 1.380 19.858 9.292
Llama-3.1-70B-Instruct 17.396 0.921 15.105 7.045 13.868 0.773 14.338 9.825 35.981 1.192 24.140 15.406 43.944 1.826 19.253 8.993
DeepSeek-67B-Chat 13.941 0.575 19.376 7.850 14.426 0.818 14.111 9.825 29.940 1.022 23.435 15.407 32.536 1.317 19.753 10.782
Yi-1.5-34B-Chat 22.093 0.865 20.433 9.428 14.004 0.814 13.757 9.847 20.889 0.704 23.748 14.936 30.743 1.258 19.551 9.195
Qwen2.5-32B-Instruct -0.557 -0.028 15.796 8.946 2.905 0.201 11.540 7.169 -1.623 -0.067 19.301 17.986 26.332 1.366 15.420 5.261
DeepSeek-V2-Lite (15.7B) 11.904 0.479 19.869 16.094 -7.482 -0.462 12.953 17.806 33.560 1.175 22.838 12.984 16.686 0.672 19.852 16.806
Yi-1.5-9B-Chat 19.333 0.755 20.486 9.428 18.606 1.112 13.392 10.986 49.415 1.663 23.768 11.430 29.028 1.173 19.791 12.588
Llama-3.1-8B-Instruct 22.703 0.912 19.910 7.385 13.988 1.025 14.117 9.969 41.108 1.367 24.057 16.429 39.079 1.601 19.526 10.341
Qwen-2.5-Instruct-7B -10.305 -0.500 16.517 23.371 21.852 0.676 25.823 9.573 11.752 0.588 15.862 15.451 4.291 0.197 17.204 14.156
Model TSLA AAPL NIO A verage
CR↑ SR↑ A V↓ MDD↓ CR↑ SR↑ A V↓ MDD↓ CR↑ SR↑ A V↓ MDD↓ CR↑ SR↑ A V↓ MDD↓
Buy & Hold 39.244 0.600 52.339 37.975 10.837 0.324 26.899 19.119 52.216 0.592 73.966 47.766 34.099 0.505 51.068 34.953
Financial Domain Models
Palmyra-Fin-70B -6.661 -0.153 34.761 25.820 8.562 0.256 26.835 25.466 -3.261 -0.039 70.181 58.406 -0.453 0.021 43.925 36.564
Proprietary Models
GPT-o1-preview 34.499 0.549 50.247 35.490 8.238 0.291 22.801 14.412 32.433 0.385 70.704 54.016 25.057 0.408 47.918 34.639
GPT-4 45.246 0.821 44.088 25.031 9.889 0.304 26.226 19.119 75.952 0.887 71.801 37.867 43.696 0.671 47.371 27.339
GPT-4o 45.946 0.930 39.524 21.631 7.405 0.315 18.929 12.824 63.743 0.909 58.795 29.220 39.031 0.718 39.083 21.225
Open-Source Models
Qwen2.5-72B-Instruct 39.112 0.742 42.184 26.985 11.935 0.395 24.352 19.119 87.412 1.505 48.733 12.464 46.153 0.880 38.423 19.523
Llama-3.1-70B-Instruct 37.545 0.615 48.862 29.813 12.772 0.402 25.569 16.021 66.522 0.771 72.345 46.379 38.946 0.596 48.926 30.738
DeepSeek-67B-Chat 35.647 0.611 46.685 33.359 14.213 0.460 24.921 10.876 30.963 0.413 62.891 45.855 26.941 0.495 44.833 30.030
Yi-1.5-34B-Chat 35.364 0.558 50.757 35.490 14.227 0.430 26.631 19.432 64.307 0.733 73.552 48.042 37.966 0.573 50.313 34.321
Qwen2.5-32B-Instruct 21.336 0.503 33.918 20.704 13.220 0.751 14.160 8.943 28.096 0.450 49.917 37.975 20.884 0.568 32.665 22.541
DeepSeek-V2-Lite (15.7B) 31.458 0.513 47.282 35.404 27.016 0.842 8.183 37.435 27.762 0.327 71.203 48.478 28.745 0.561 42.222 40.439
Yi-1.5-9B-Chat 31.350 0.485 51.677 37.975 3.640 0.112 26.183 17.578 33.748 0.393 72.106 55.284 22.913 0.330 49.989 36.946
Llama-3.1-8B-Instruct 35.622 0.574 49.636 36.383 7.079 0.213 26.742 18.747 33.689 0.386 73.205 56.527 25.463 0.391 49.861 37.219
Qwen-2.5-Instruct-7B 41.203 0.638 51.655 37.975 14.336 0.479 24.098 12.029 33.007 0.377 73.442 53.054 29.515 0.498 49.731 34.353
1 The Buy & Hold strategy is a passive investment approach commonly used as a baseline strategy, where an investor
purchases stocks and holds onto them for an extended period regardless of market ﬂuctuations.
2 An upward arrow ( ↑) next to a metric indicates that higher values signify better performance, while a downward arrow ( ↓)
indicates that lower values are preferable.
3 The numbers highlighted in red indicate the best-performing outcomes for the corresponding metrics.
ing time spans of data collected to construct the
agent environment. For the single-asset trading
tasks, the warm-up period of stock trading is from
2020-07-01 to 2020-09-30 and the test period is
from 2020-10-01 to 2021-05-06. The warm-up pe-
riod of cryptocurrency trading is from 2023-02-11
to 2023-04-04 and the test period is from 2023-04-
05 to 2023-11-05. The warm-up period of ETF
trading is from 2019-07-29 to 2019-12-30 and the
test period is from 2020-01-02 to 2020-09-21.
For LLM deployment, we utilize vllm to deploy
LLMs. For small-scale LLMs (under 10B parame-
ters), we deploy models on two RTX A6000 GPUs,
each with 48GB DRAM. For mid-scale LLMs
(10B to 65B parameters), we use four RTX A6000
GPUs. For large-scale LLMs (over 65B parame-
ters), models are deployed on eight A100 GPUs,
each equipped with 80GB DRAM.
4.2 Result 1: Stock Trading
Table 2 presents the performance of thirteen back-
bone models across seven stocks, accompanied by
the average of each metric for all stocks to offer a
more comprehensive view of their overall perfor-
mance. We outline three key insights as follows:
Superior stock trading performance is achieved
with proprietary LLMs as agent backbones
Compared to agents employing open-source or
ﬁnancial-domain-speciﬁc ﬁne-tuned LLMs, those
using the three proprietary LLMs demonstrated sig-
niﬁcantly higher and more consistent average CR
and SR, as shown in Figure 3a. Despite being ﬁne-
tuned with extensive ﬁnancial contexts, domain-
speciﬁc LLMs did not provide a decisive advantage
in sequential stock trading decision-making tasks.
This may be attributed to their primary training for
other functions, such as long ﬁnancial report anal-
ysis exempliﬁed by Palmyra-Fin-70B, rather than
decision-making.
Model parameter size increment enhances agent
ﬁnancial decision-making quality and robust-
ness. In the category of open-source LLMs, those
exceeding 67B parameters displayed superior CRs
and SRs, along with markedly less variance within
their category, as illustrated in Figure 3b and Ta-
ble 2. This underscores the prevailing belief that the
reasoning capabilities of LLMs are proportionate to
their parameter size, which holds also true in stock
2514