INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agent — part 1/6

Cleaned-text code points [0, 13274). Bibliography omitted here; source.md includes it.

Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers) , pages 2509–2525
July 27 - August 1, 2025 ©2025 Association for Computational Linguistics
INVESTOR BENCH : A Benchmark for Financial Decision-Making Tasks
with LLM-based Agents
Haohang Li1,⋆, Yupeng Cao1,⋆, Yangyang Yu1,⋆, Shashidhar Reddy Javaji 1,
Zhiyang Deng1, Yueru He2, Yuechen Jiang1, Zining Zhu 1,
Koduvayur Subbalakshmi1, Jimin Huang 4, Lingfei Qian 4, Xueqing Peng 4,
Qianqian Xie4,†, Jordan W. Suchow1
1Stevens Institute of Technology 2Columbia University
3Harvard University 4The Fin AI
⋆Equal Contribution † Corresponding author: qianqian.xie@yale.edu
Abstract
Recent advancements have underscored the po-
tential of large language model (LLM)-based
agents in ﬁnancial decision-making. Despite
this progress, the ﬁeld currently encounters
two main challenges: (1) the lack of a com-
prehensive LLM agent framework adaptable
to a variety of ﬁnancial tasks, and (2) the ab-
sence of standardized benchmarks and consis-
tent datasets for assessing agent performance.
To tackle these issues, we introduce INVESTOR -
BENCH , the ﬁrst benchmark speciﬁcally de-
signed for evaluating LLM-based agents in di-
verse ﬁnancial decision-making contexts. I N-
VESTOR BENCH enhances the versatility of
LLM-enabled agents by providing a compre-
hensive suite of tasks applicable to different ﬁ-
nancial products, including single equities like
stocks, cryptocurrencies and exchange-traded
funds (ETFs). Additionally, we assess the rea-
soning and decision-making capabilities of our
agent framework using thirteen different LLMs
as backbone models, across various market en-
vironments and tasks. Furthermore, we have cu-
rated a diverse collection of open-source, multi-
modal datasets and developed a comprehensive
suite of environments for ﬁnancial decision-
making. This establishes a highly accessible
platform for evaluating ﬁnancial agents’ per-
formance across various scenarios. The code
is available at Github Repo: https://github.
com/felis33/INVESTOR-BENCH
1 Introduction
The recent studies on large language model (LLM)-
based agents have demonstrated impressive perfor-
mance across a range of decision-making tasks in
complex and open-ended environments spanning
various domains ( Zhang et al. , 2024b; Guo et al. ,
2024; Eigner and Händler, 2024; Wang et al., 2024).
However, developing agentic frameworks tailored
speciﬁcally for ﬁnancial decision-making remains a
signiﬁcant challenge. This complexity arises from
Figure 1: General architecture of I NVESTOR BENCH .
the need for agents to acutely discern and priori-
tize decisive signals, and then make sequentially
high-quality decisions within the volatile and multi-
faceted ﬁnancial markets, where information varies
in time sensitivity and modality.
Furthermore, the design of ﬁnancial agents be-
comes increasingly complex when applied across
multiple decision-making tasks, due to the signiﬁ-
cant variation in key factors inﬂuencing ﬁnancial
decisions across different objectives and task types.
For instance, single-equity tasks like stock trading
require analyzing company-speciﬁc and industry-
wide data, including market metrics, sector trends,
performance reports, and relevant news ( Yi et al.,
2022). In contrast, cryptocurrency trading is highly
sensitive to crypto-speciﬁc news and sentiment
due to its dynamic nature ( Bhatnagar et al., 2023).
ETFs, on the other hand, typically follow passive in-
vestment strategies, emphasizing long-term growth
and cost efﬁciency ( Madhavan, 2016).
The recent emergence of ﬁnancial LLM-based
2509

agent frameworks such as F INMEM (Y u et al. ,
2024a), F INAGENT (Zhang et al., 2024a), C RYPTO -
TRADE (Li et al. , 2024), F INROBOT (Y ang et al.,
2024), and F INCON (Y u et al., 2024b) has pre-
sented a variety of architectural approaches tai-
lored to speciﬁc ﬁnancial tasks. This diversiﬁca-
tion has sparked substantial interest across both
academic and industrial landscapes. F INROBOT is
engineered speciﬁcally for market analysis, while
FINMEM and F INAGENT are oriented towards
trading individual equities like stocks and ETFs.
CRYPTO TRADE focuses solely on cryptocurrency
trading. F INCON pioneers in addressing portfolio
management, although it currently handles only
compact portfolios consisting of three stock assets.
While these frameworks are effective within their
respective niches, they generally focus on address-
ing only limited types of ﬁnancial decision-making
tasks. This restricts them from further demonstrat-
ing the broader applicability of these frameworks
and limits the comprehensive, comparative insights
that could be drawn from their overall decision-
making performance. Furthermore, the frequent
reliance on proprietary ﬁnancial data complicates
the evaluation of these tools, obscuring their ef-
fectiveness and adaptability in broader contexts.
Therefore, there is a pressing need to develop inno-
vative benchmarks speciﬁcally designed to evalu-
ate LLM-based agents across a wider spectrum of
ﬁnancial decision-making scenarios. Such bench-
marks would enable a more robust assessment of
these technologies, facilitating advancements that
could cater to various ﬁnancial applications.
We introduce I NVESTOR BENCH , an open-
source, LLM-based agent benchmark that gener-
alizes across a broad range of ﬁnancial decision-
making tasks. Its detailed structure is illustrated
in Figure 1. Further developed upon the founda-
tional framework of F INMEM (Y u et al., 2024a),
which focuses on single-stock investment decisions,
our benchmark extends the scope to encompass
an ensemble of diverse ﬁnancial market envi-
ronments for various ﬁnancial tasks. I NVESTOR -
BENCH ’s cognitive architecture, similar to F IN-
MEM, employs a layered memory processing
mechanism with distinct decay rates, enabling the
agent to store, retrieve, and consolidate insights and
reﬂections more effectively than the pure similarity-
based memory retrieval used in F INAGENT . This
approach ensures that decisions are informed by
timely and impactful data, a capability previously
shown effective for single-asset trading. These fea-
tures reﬂect how human traders draw sequential
decisions upon investment signals from multiple
sources and varying time sensitivities, allowing the
agent to naturally adapt to complex ﬁnancial tasks.
INVESTOR BENCH expands its evaluation beyond
the original stock trading tasks to encompass three
decision tasks signiﬁcant in the realm of ﬁnancial
investment: stock trading, cryptocurrency trad-
ing, and ETF investing.
In summary, we make three key contributions:
1) We establish I NVESTOR BENCH , an innovative
and comprehensive ﬁnancial agentic benchmark
designed to evaluate the reasoning and sequen-
tial decision-making capabilities of LLM-based
agents in complex, open-ended ﬁnancial scenar-
ios. This benchmark provides a realistic perspec-
tive for assessing the design and performance of
such agents. 2) We provide a set of open-source,
multi-source market environments that closely
mirror real-world conditions. Furthermore, these
environments also serve as a standardized platform
for evaluating the decision-making performance of
other LLM-based ﬁnancial agents. 3) We present a
uniﬁed, ﬂexible language-agent framework that
allows ﬁnance professionals to conveniently cus-
tomize assess any LLMs serving as the agents rea-
soning core. In this paper, we conduct a holistic
evaluation of 13 LLMs including recent, compet-
itive, and domain-speciﬁc ﬁne-tuned models (see
Table 1) to provide a broad overview of their rea-
soning capabilities in sequential decision-making
tasks within ﬁnancial contexts.
2 LLM Trading Agents
In this section, we deﬁne a framework of the LLM-
based agents in the I NVESTOR BENCH and formal-
ize the ﬁnancial decision-making tasks within the
context of partially observable Markov decision
process (POMDP) ( Bertsekas and Shreve , 1996;
Liu et al., 2020; Kabbani and Duman, 2022).
2.1 Deﬁnition
The LLM-based agent in I NVESTOR BENCH is
structured as a large language model-modulo frame-
work, designed to match or surpass the capabilities
of professional human investors. This framework
consists of several interconnected modules, each
tailored to handle the distinct challenges presented
by the ﬁnancial markets volatility and complexity:
Brain/Backbone (LLM): This module, which is
the LLM itself, serves as the core of the LLM-
2510

based agent. It enhances the agent’s capabilities
by enabling it to understand, process, and generate
natural language. This module plays a crucial role
in supporting complex decision-making processes,
offering interpretations of market-related informa-
tion, generating predictive analytics, and reﬂecting
on past investment decisions.
Perception: This module serves a critical function
by converting raw market data into a structured
format that is compatible with the LLM, specifying
what the agent perceives and observes, which
includes numerical, textual, and visual information.
Proﬁle: This module serves two functions
articulated in natural language. Firstly, it describes
the agent’s role, highlighting its character as an
experienced investor with expert-level knowledge
and a self-adaptive risk preference. This risk
preference dynamically adjusts based on historical
market momentum, allowing the agent to optimize
its strategies in real time. Secondly, the module
provides a detailed background of the decision-
making task, specifying the key characteristics
and pertinent information about the target assets
involved in the trading decisions, such as equity
historical performance, price ﬂuctuations, and
sector information. This dual-function module
supports the agent’s decisions with both the current
market context and its historical performance.
Memory: This module processes and retains es-
sential market data and historical insights, allowing
the agent to draw on a rich repository of knowledge
for decision-making. Building upon the pioneer-
ing work of Y u et al. (2024a) in F INMEM, the
memory architecture comprises two primary com-
ponents: Working Memory and Layered Long-
Term Memory, as depicted in Figure 2.
Working memory: This component maintains
FINMEM’s original functionalities: observation,
summarization, and reﬂection. It incorporates two
reﬂection mechanisms: immediate and extended.
Immediate reﬂection produces the agent’s reason-
ing outcomes by integrating current market indi-
cators with the top- K ranked events from each
long-term memory layer, which are signiﬁcant dur-
ing both warm-up and evaluation stages. In the
warm-up stage, the emphasis shifts as the trading
direction is predetermined, focusing on understand-
ing market trends and enhancing predictive accu-
racy. In the evaluation stage, it outputs the trading
direction (Buy, Sell, or Hold), the rationale for this
decision, identifying the most inﬂuential memory
events and their respective IDs from each layer.
Layered Long-Term Memory: Inspired by the
human cognitive system’s varying information de-
cay speeds, Layered Long-Term Memory com-
ponent structures ﬁnancial insights across multi-
ple layers. Each layer is represented by a vector
database in the Long-Term Memory data ware-
house, where information is prioritized and purged
based on a speciﬁc decay rate. Deeper layers retain
information longer with smaller decay rates, while
shallower layers, dealing with more transient data,
have larger decay rates. This tiered approach is
critical as it allows the adaptation of the memory
architecture to a broader range of ﬁnancial tasks
beyond single-asset decisions, accommodating an
expanded variety of data sources and increasing
overall system ﬂexibility. Detailed mechanisms for
ranking and decay in each layer are further elabo-
rated in the Appendix A.
Action: This module executes trading and in-
vestment decisions based on the analysis pro-
vided by other modules. It directly outputs
{“Buy", “Sell", “Hold" } for traded asset (stock,
crypto, or ETF), as instructed by the backbone
LLM. Action module synthesizes the operational
outcomes from the Proﬁle and Memory modules
to facilitate precise and well-informed investment
decisions. For its daily trading operations, the
agent can choose from three speciﬁc actions for
the traded asset: “Buy", “Sell", or “Hold". The
functionality and input requirements of this module
differ signiﬁcantly between the warm-up and eval-
uation stages: during the warm-up stage , the agent
observes daily adjusted price differences between
consecutive days, which are critical for identify-
ing potential “Buy" or “Sell" signals. This period
allows the agent to calibrate and adjust its decision-
making strategies based on near-term market move-
ments; during the evaluation stage, access to future
price data is restricted, compelling the agent to rely
solely on available historical data and its cognitive
processing capabilities. In response to trading in-
quiries, the module integrates historical Proﬁt &
Loss (PnL), outcomes from extended reﬂections,
and the top-K retrieved memories. This integration
ensures that each trading decision is grounded in
a comprehensive analysis of past performance and
current market conditions.
2511