INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agent — part 3/6

Cleaned-text code points [27644, 41074). Bibliography omitted here; source.md includes it.
(a) The performance comparison by different model types.
(b) By model parameter sizes across open-source LLMs.
Note: Small-size models refer to models with no more than
10B parameters. Medium-size models refer to models with
parameter numbers in the range of (10B, 65B]. Large-size
models are those with more than 65B parameters.
Figure 3: Agent Performance Comparisons from two
key perspectives. The CR, SR, A V , and MDD represent the
average values for each model type, expressed as a percentage
relative to the Buy & Hold strategy.
trading, which is a sequential decision-making task
in an open-ended, volatile environment by nature.
Proprietary models exhibit signiﬁcantly
stronger decision-making capabilities compared
to even the largest open-source LLMs under
complex, mixed market conditions, though this
advantage is less evident in relatively monotone
market environments. During the test phase,
primarily inﬂuenced by the range of open-source
data collected, TSLA and NIO exhibited volatility
with mixed upward and downward stock price
trends, whereas the other ﬁve stocks generally
showed bullish trends. The investment signals
derived from such complex markets tend to be
noisy or delayed, as illustrated in Appendix C. We
observed that proprietary models possess a superior
ability to manage these challenging conditions and
consistently deliver better performance outcomes
than large-sized open-source LLMs. Their
reasoning capability enables them to effectively
utilize other decision-relevant information, such as
historical momentum, current holdings, and, most
critically, self-reﬂection outcomes from the agents,
Table 3: Performance of cryptocurrency trading with dif-
ferent LLMs as backbone models across Bitcoin (BTC)
and Ethereum (ETH).
Model BTC ETH
CR↑ SR↑ A V↓ MDD↓ CR↑ SR↑ A V↓ MDD↓
Buy & Hold 21.821 0.989 54.193 20.796 4.528 0.211 60.551 29.889
Financial Domain Models
Palmyra-Fin-70B -20.812 -1.755 29.012 27.782 4.795 0.348 38.986 16.405
Proprietary Models
GPT-o1-preview 34.060 1.613 51.905 17.075 2.496 0.123 57.400 27.692
GPT-4 22.396 1.199 45.900 17.206 1.516 0.074 57.648 32.541
GPT-4o 14.330 0.770 45.328 17.278 4.666 0.275 47.858 22.539
A verage 23.595 1.195 47.712 17.186 2.893 0.158 54.301 27.591
Open-Source Models
Qwen2.5-72B-Instruct 0.549 0.471 2.866 0.897 11.984 0.846 26.866 27.642
Llama-3.1-70B-Instruct 20.440 1.098 45.763 17.813 -11.888 -0.594 56.540 36.416
DeepSeek-67B-Chat 28.307 1.290 53.893 17.944 9.480 0.447 59.902 26.261
Yi-1.5-34B-Chat 13.620 0.628 53.255 22.790 6.325 0.329 54.304 25.707
Qwen2.5-32B-Instruc 11.566 1.258 22.600 7.984 2.823 0.281 28.339 7.883
DeepSeek-V2-Lite (15.7B) 4.804 0.222 53.353 20.562 -9.504 -0.450 59.656 21.270
Yi-1.5-9B-Chat 7.953 0.366 53.285 26.545 -3.684 -0.172 60.552 35.417
Llama-3.1-8B-Instruct 20.521 0.935 53.924 21.104 4.939 0.236 59.264 29.466
Qwen-2.5-Instruct-7B 19.477 0.886 53.994 20.796 -1.339 -0.109 34.932 -16.053
A verage 14.137 0.795 43.659 17.382 1.015 0.090 48.928 21.557
thereby facilitating more accurate decisions.
4.3 Result 2 & 3: Cryptocurrency Trading
and ETF Trading
In the test phases of both cryptocurrency and ETF
trading tasks, market trends are mixed. Notably,
the cryptocurrency task shows signiﬁcantly smaller
price ﬂuctuations compared to the ETF task. We
outline the key features of using an LLM-agent to
make ﬁnancial decisions across these two distinct
markets as follows:
Large-sized open-source models and propri-
etary models are needed to effectively cap-
ture trading signals of cryptocurrency markets,
which are highly sensitive to news and ﬁnan-
cial sentiment. As shown in Table 3, using mid-
sized and small-sized open-source models as the
decision-making agent backbone generally results
in weaker performance than the market baseline
with respect to CR and SR.
ETF investment requires proprietary models en-
riched with extensive pre-trained knowledge to
serve as the agents brain and provide robust
reasoning support. As shown in Table 4, propri-
etary models signiﬁcantly outperform open-source
and ﬁnancial domain-speciﬁc models in this task.
This advantage arises from the complexity of ETF
trading, which necessitates interpreting actionable
signals across diverse sectors, demanding more
strategic, long-term decisions grounded in deep
comprehension and reﬂection anchored by rich pre-
contexts.
4.4 Discussion
Combining all the experimental results, we ﬁnd
that the performance of different LLM varies sig-
2515

Table 4: Performance of ETF trading with different
LLMs as backbone models.
ETF CR ↑ SR↑ A V↓ MDD↓
Buy & Hold 2.069 0.06 46.645 35.746
Financial Domain Models
Palmyra-Fin-70B 24.759 1.152 30.419 8.203
Proprietary Models
GPT-o1-preview 21.224 0.849 43.766 20.054
GPT-4 2.807 0.110 44.679 37.785
GPT-4o 12.292 0.377 46.150 32.678
A verage 12.108 0.445 44.865 30.172
Open-Source Models
Qwen2.5-72B-Instruct 4.507 0.227 28.090 8.580
Llama-3.1-70B-Instruct 9.895 0.464 30.184 12.759
Yi-1.5-34B-Chat 4.996 0.322 21.986 12.858
Qwen2.5-32B-Instruct 19.617 0.955 29.070 7.496
DeepSeek-V2-Lite (15.7B) 1.389 0.063 31.371 31.831
Yi-1.5-9B-Chat -4.657 -0.228 28.907 15.545
Llama-3.1-8B-Instruct 11.239 0.475 33.480 15.587
Qwen-2.5-Instruct-7B -0.384 -0.020 27.596 14.059
A verage 5.825 0.282 28.835 14.839
niﬁcantly in stock, cryptocurrency, and ETF trad-
ing. This variation not only reﬂects the inherent
complexity of ﬁnancial markets, but also highlights
the importance of model selection or ﬁne-tuning.
For instance, proprietary LLM generally exhibit be
performance in stock trading due to their strong
training on various ﬁnancial datasets, while open-
source models struggle to achieve these results,
especially in more volatile environments such as
cryptocurrency trading. In addition, the effective-
ness of LLM-based agents depends heavily on their
ability to adapt to market ﬂuctuations. Agents
that incorporate advanced memory systems and dy-
namic risk assessment capabilities are better able to
cope with complex market situations, highlighting
the value of the complex architectural features of
LLM-based agent framework in ﬁnancial decision-
making tasks.
5 Related Work
5.1 LLM for Financial Domain
The rapid developement of general-domain lan-
guage models (LMs) has stimulated the explo-
ration of ﬁnancial LMs, such as pre-trained LMs:
FINBERT (Liu et al. , 2021; Y ang et al. , 2020;
Araci, 2019; Huang et al. , 2023), F INBERT-MRC
(Zhang and Zhang , 2023), FLANG ( Shah et al. ,
2022), and several ﬁnancial LLMs: F INGPT( Liu
et al. , 2023), F INMA ( Xie et al. , 2023), I N-
VEST LM ( Y ang et al., 2023), BloombergGPT (Wu
et al., 2023), which leverage extensive training on
diverse ﬁnancial datasets (e.g. stock price data,
ﬁnancial news and analyst reports) and adapt the
capabilities of LMs to the unique needs of ﬁnancial
applications. Concurrently, the advancement of
LLMs has signiﬁcantly enhanced the development
of language-based agent frameworks in the ﬁnan-
cial sector, such as F INMEM (Y u et al., 2024a),
FINAGENT (Zhang et al., 2024a) and F INROBOT
(Y ang et al., 2024), characterized by their adapt-
ability and openness. However, variations in frame-
work design, task scope, and data types present
challenges in uniformly evaluating the efﬁcacy of
LLM agents in ﬁnancial scenarios.
5.2 Financial LLM Benchmarks
In the realm of ﬁnancial LLMs, several benchmarks
have been developed: FLUE ( Shah et al. , 2022) in-
troduces the ﬁrst comprehensive benchmark with
ﬁve ﬁnancial NLP tasks, including sentiment anal-
ysis, headline classiﬁcation, named entity recog-
nition, structure boundary detection, and question
answering. Pixiu ( Xie et al. , 2023) expands this
benchmark to include ﬁnancial document under-
standing and classiﬁcation tasks, incorporating mul-
timodal datasets. FinBen ( Xie et al. , 2024) encom-
passes 36 datasets covering 24 ﬁnancial tasks. De-
spite these advancements, there remains a notable
gap in benchmarks speciﬁcally designed for LLM-
based agent applications within the ﬁnancial sector.
6 Conclusion
INVESTOR BENCH offers the community two dis-
tinct modes of engagement. The ﬁrst mode allows
participants to integrate their ﬁne-tuned LLMs into
the I NVESTOR BENCH ’s agent framework to under-
take ﬁnancial decision-making tasks. This setup
enables them to benchmark the performance of
their models against those previously experimented
with by our work. The second mode permits users
to directly incorporate the environment and evalu-
ation metrics of I NVESTOR BENCH into their own
designed agents, facilitating a comparative analy-
sis of their agent design’s effectiveness. This dual
approach provides a ﬂexible framework for testing
and enhancing ﬁnancial decision-making strategies
within the I NVESTOR BENCH ecosystem.
Future research efforts will expand the bench-
mark by incorporating additional information
modalities, such as audio (e.g., earnings call record-
ings) and graphs (e.g., K-lines, trade charts), to
explore whether these data types can enhance
decision-making quality. The foundational agent
2516

framework of I NVESTOR BENCH is designed to
seamlessly accommodate these modalities, ensur-
ing that the extended benchmark remains easy to
use and scalable.
Limitation
First, I NVESTOR BENCH is currently focusing on
single-asset ﬁnancial decision-making task, with-
out addressing multi-asset tasks such as portfolio
management. Second, copyright restrictions on ﬁ-
nancial domain data may compromise the quality
of the datasets we create, potentially limiting the
assessment of model performance.
Ethical Statement
The authors take full responsibility for the devel-
opment of I NVESTOR BENCH , ensuring that the
publicly available part in dataset does not contain
personal information, and conform to established
ethical guidelines. The data are shared under the
MIT license, requiring users to adhere to its terms.
INVESTOR BENCH is intended for academic and ed-
ucational purposes only and is not a substitute for
professional advice. While efforts have been made
to ensure its accuracy, the authors and their insti-
tutions disclaim liability for any outcomes arising
from its use. Users agree to take responsibility for
ethical and lawful use and to indemnify the authors
and their afﬁliates against any claims or damages
resulting from reliance on this Material.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama
Ahmad, Ilge Akkaya, Florencia Leoni Aleman,
Diogo Almeida, Janko Altenschmidt, Sam Altman,
Shyamal Anadkat, et al. 2023. Gpt-4 technical report.
arXiv preprint arXiv:2303.08774.
Andrew Ang and Joseph Chen. 2003. Downside risk.
Journal of Portfolio Management, 29(4):103–112.
Dogu Araci. 2019. Finbert: Financial sentiment analy-
sis with pre-trained language models. arXiv preprint
arXiv:1908.10063.
Dimitri Bertsekas and Steven E Shreve. 1996. Stochas-
tic optimal control: the discrete-time case , volume 5.
Athena Scientiﬁc.
Mukul Bhatnagar, Sanjay Taneja, and Ramona Rupeika-
Apoga. 2023. Demystifying the effect of the news
(shocks) on crypto market volatility. Journal of Risk
and Financial Management, 16(2):136.
John H. Cochrane. 1988. V olatility tests and efﬁcient
markets: A review essay. Journal of Monetary Eco-
nomics, 22(3):463–485.
Eva Eigner and Thorsten Händler. 2024. Determinants
of llm-assisted decision-making. arXiv preprint
arXiv:2402.17385.
Thomas L Grifﬁths, Jian-Qiao Zhu, Erin Grant, and
R Thomas McCoy. 2023. Bayes in the age of intelli-
gent machines. arXiv preprint arXiv:2311.10206.
Taicheng Guo, Xiuying Chen, Y aqi Wang, Ruidi Chang,
Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xi-
angliang Zhang. 2024. Large language model based
multi-agents: A survey of progress and challenges.
arXiv preprint arXiv:2402.01680.
Allen H Huang, Hui Wang, and Yi Y ang. 2023. Finbert:
A large language model for extracting information
from ﬁnancial text. Contemporary Accounting Re-
search, 40(2):806–841.
John Hull. 2007. Risk Management and Financial Insti-
tutions. John Wiley & Sons.
Taylan Kabbani and Ekrem Duman. 2022. Deep rein-
forcement learning approach for trading automation
in the stock market. IEEE Access, 10:93564–93574.
Y uan Li, Bingqiao Luo, Qian Wang, Nuo Chen, Xu Liu,
and Bingsheng He. 2024. Cryptotrade: A reﬂective
llm-based agent to guide zero-shot cryptocurrency
trading. In Proceedings of the 2024 Conference on
Empirical Methods in Natural Language Processing ,
pages 1094–1106.
Xiao-Y ang Liu, Guoxuan Wang, and Daochen Zha.
2023. Fingpt: Democratizing internet-scale data
for ﬁnancial large language models. arXiv preprint
arXiv:2307.10485.
Y ang Liu, Qi Liu, Hongke Zhao, Zhen Pan, and Chuan-
ren Liu. 2020. Adaptive quantitative trading: An
imitative deep reinforcement learning approach. In
Proceedings of the AAAI conference on artiﬁcial in-
telligence, volume 34, pages 2128–2135.
Zhuang Liu, Degen Huang, Kaiyu Huang, Zhuang Li,
and Jun Zhao. 2021. Finbert: A pre-trained ﬁnan-
cial language representation model for ﬁnancial text
mining. In Proceedings of the twenty-ninth interna-
tional conference on international joint conferences
on artiﬁcial intelligence, pages 4513–4519.
Meta Llama team. 2024. The llama 3 herd of models .
Ananth N Madhavan. 2016. Exchange-traded funds and
the new dynamics of investing . Oxford University
Press.
Jaap MJ Murre and Joeri Dros. 2015. Replication and
analysis of ebbinghaus forgetting curve. PloS one,
10(7):e0120644.
OpenAI. 2022. Introducing chatgpt.
2517