INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agent — part 4/6

Cleaned-text code points [41074, 54066). Bibliography omitted here; source.md includes it.

Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Mered-
ith Ringel Morris, Percy Liang, and Michael S. Bern-
stein. 2023. Generative agents: Interactive simulacra
of human behavior . In Proceedings of the 36th An-
nual ACM Symposium on User Interface Software
and Technology, UIST ’23, New Y ork, NY , USA.
Association for Computing Machinery.
Alibaba Group Qwen team. 2024. Qwen2.5.
Raeid Saqur, Ken Kato, Nicholas Vinden, and Frank
Rudzicz. 2024. Nifty financial news headlines
dataset. arXiv preprint arXiv:2405.09747.
Raj Sanjay Shah, Kunal Chawla, Dheeraj Eidnani,
Agam Shah, Wendi Du, Sudheer Chava, Natraj Ra-
man, Charese Smiley, Jiaao Chen, and Diyi Y ang.
2022. When flue meets flang: Benchmarks and
large pre-trained language model for financial do-
main. arXiv preprint arXiv:2211.00083.
William F. Sharpe. 1994. The sharpe ratio. The Journal
of Portfolio Management, 21(1):49–58.
Writer Engineering team. 2024. Palmyra-Fin-70B-32k:
a powerful LLM designed for Finance. https://
dev.writer.com.
Olivier V anhoucke. 2023. Crypto news dataset . Ac-
cessed: 2024-08-20.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao
Y ang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang,
Xu Chen, Y ankai Lin, et al. 2024. A survey on large
language model based autonomous agents. Frontiers
of Computer Science, 18(6):186345.
Shijie Wu, Ozan Irsoy, Steven Lu, V adim Dabravolski,
Mark Dredze, Sebastian Gehrmann, Prabhanjan Kam-
badur, David Rosenberg, and Gideon Mann. 2023.
Bloomberggpt: A large language model for finance.
arXiv preprint arXiv:2303.17564.
Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu
Xiang, Xiao Zhang, Y ueru He, Mengxi Xiao, Dong
Li, Y ongfu Dai, Duanyu Feng, et al. 2024. The fin-
ben: An holistic financial benchmark for large lan-
guage models. arXiv preprint arXiv:2402.12659.
Qianqian Xie, Weiguang Han, Xiao Zhang, Y anzhao
Lai, Min Peng, Alejandro Lopez-Lira, and Jimin
Huang. 2023. Pixiu: A large language model, in-
struction data and evaluation benchmark for finance.
arXiv preprint arXiv:2306.05443.
Huajian Xin, Z. Z. Ren, Junxiao Song, Zhihong Shao,
Wanjia Zhao, Haocheng Wang, Bo Liu, Liyue Zhang,
Xuan Lu, Qiushi Du, Wenjun Gao, Qihao Zhu, De-
jian Y ang, Zhibin Gou, Z. F. Wu, Fuli Luo, and
Chong Ruan. 2024. Deepseek-prover-v1.5: Harness-
ing proof assistant feedback for reinforcement learn-
ing and monte-carlo tree search .
Hongyang Y ang, Boyu Zhang, Neng Wang, Cheng Guo,
Xiaoli Zhang, Likun Lin, Junlin Wang, Tianyu Zhou,
Mao Guan, Runjia Zhang, et al. 2024. Finrobot: An
open-source ai agent platform for financial applica-
tions using large language models. arXiv preprint
arXiv:2405.14767.
Yi Y ang, Yixuan Tang, and Kar Y an Tam. 2023. In-
vestlm: A large language model for investment using
financial domain instruction tuning. arXiv preprint
arXiv:2309.13064.
Yi Y ang, Mark Christopher Siy Uy, and Allen Huang.
2020. Finbert: A pretrained language model
for financial communications. arXiv preprint
arXiv:2006.08097.
Ziruo Yi, Ting Xiao, Kaz-Onyeakazi Ijeoma, Rat-
nam Cheran, Y uvraj Baweja, and Phillip Nelson.
2022. Stock2vec: An embedding to improve
predictive models for companies. arXiv preprint
arXiv:2201.11290.
Alex Y oung, Bei Chen, Chao Li, Chengen Huang,
Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng
Zhu, Jianqun Chen, Jing Chang, et al. 2024. Yi:
Open foundation models by 01. ai. arXiv preprint
arXiv:2403.04652.
Y angyang Y u, Haohang Li, Zhi Chen, Y uechen Jiang,
Y ang Li, Denghui Zhang, Rong Liu, Jordan W Su-
chow, and Khaldoun Khashanah. 2024a. Finmem: A
performance-enhanced llm trading agent with layered
memory and character design. In Proceedings of the
AAAI Symposium Series, volume 3, pages 595–597.
Y angyang Y u, Zhiyuan Y ao, Haohang Li, Zhiyang Deng,
Y upeng Cao, Zhi Chen, Jordan W Suchow, Rong Liu,
Zhenyu Cui, Zhaozhuo Xu, et al. 2024b. Fincon: A
synthesized llm multi-agent system with conceptual
verbal reinforcement for enhanced financial decision
making. arXiv preprint arXiv:2407.06567.
Wentao Zhang, Lingxuan Zhao, Haochong Xia, Shuo
Sun, Jiaze Sun, Molei Qin, Xinyi Li, Y uqing Zhao,
Yilei Zhao, Xinyu Cai, et al. 2024a. Finagent: A
multimodal foundation agent for financial trading:
Tool-augmented, diversified, and generalist. arXiv
preprint arXiv:2402.18485.
Y adong Zhang, Shaoguang Mao, Tao Ge, Xun Wang,
Adrian de Wynter, Y an Xia, Wenshan Wu, Ting Song,
Man Lan, and Furu Wei. 2024b. Llm as a master-
mind: A survey of strategic reasoning with large
language models. arXiv preprint arXiv:2404.01230.
Y uzhe Zhang and Hong Zhang. 2023. Finbert–mrc: fi-
nancial named entity recognition using bert under the
machine reading comprehension paradigm. Neural
Processing Letters, 55(6):7393–7413.
Zhihan Zhou, Liqian Ma, and Han Liu. 2021. Trade
the event: Corporate events detection for news-based
event-driven trading. In Findings of the Association
for Computational Linguistics: ACL-IJCNLP 2021 ,
pages 2114–2124, Online. Association for Computa-
tional Linguistics.
2518

Appendices
A Memory Ranking Mechanism of
FINMEM
Upon receiving an investment inquiry, F INMEM re-
trieves the top-K critical memory events from each
layer and channels them to the immediate reflection
component of the working memory. These events
are selected based on their information retrieval
score, γE
l , where l represents the layer (shallow,
intermediate, or deep), as defined in Equation 2.
γE
l = SE
Recencyl
+ SE
Relevancyl
+ SE
Importancel
, (2)
where each memory event is only associated
with one score and can only belong to a single layer.
Let E denote a given memory event. The scoring
mechanism for E, adapted from Park et al. ( Park
et al., 2023) but with modified recency and impor-
tance computations, is tailored to handle data with
various timelines and to achieve layered processing
that represents the diverse periodicities of the fi-
nancial environment. This score encapsulates three
metrics: recency (how recently the event occurred),
relevancy (the event’s pertinence to the current con-
text), and importance (the event’s significance). In-
dividual metric scores exceeding 1.0 are scaled to
the [0,1] range before being summed, ensuring a
balanced contribution from each component and
preventing any single metric from dominating the
overall score. The resulting composite score pro-
vides a comprehensive evaluation of the memory
event’s significance within the multi-layered, peri-
odically varying financial landscape.
SE
Recencyl
= e
− δE
Ql , δ E = tP − tE, (3)
where δE represents the time elapsed be-
tween a memory event’s occurrence and the
trading inquiry’s arrival. The model utilizes
three processing layers, each corresponding to
a specific timeframe: shallow ( Qshallow = 14
days), intermediate ( Qintermediate = 90 days), and
deep ( Qdeep = 365 days). These intervals rep-
resent two weeks, a quarter, and a year respectively.
When a trade inquiry P arrives in processing
layer l via an LLM prompt, the agent calculates the
recency score SE
Recencyl
for a memory event E using
Equation 3. This score inversely correlates with the
time elapsed between the inquiry and the event’s
memory timestamp, mapping to Ebbinghaus’s for-
getting curve (Murre and Dros, 2015). The stability
term Ql in Equation 3 modulates memory decay
rates across layers, with higher values in deeper
layers indicating longer memory persistence. For
instance, in the trading context, company annual re-
ports (e.g., Form 10-Ks) are assigned higher stabil-
ity values and categorized within deeper processing
layers compared to daily financial news, reflecting
their extended timeliness, relevance, and impact on
financial decision-making.
SE
Relevancyl
= mE · mP
‖mE‖2 × ‖mP‖2
(4)
The relevancy score SE
relevancyl quantifies the se-
mantic similarity between a memory event E and
the current query P using cosine similarity of their
respective embedding vectors, mE and mP, as
shown in Equation 4. These embeddings are gener-
ated from the event’s textual content and the LLM
prompt query (which includes trading inquiries and
the agent’s character setting) using OpenAI’s "text-
embedding-ada-003" model.
The importance score SE
Importancel for a memory
event E in layer l is calculated as the product of a
value vE
l (derived from a uniform piecewise scor-
ing function, Equation 5) and a degrading ratio θl
(Equation 6), as shown in Equation 7. This ap-
proach, adapted from ( Park et al., 2023), is tailored
to our stratified long-term memory structure. The
likelihood of higher vE
l values increases from shal-
low to deep layers, while θl measures the diminish-
ing importance of an event over time using layer-
specific exponential functions. The base αl for each
layer follows αshallow < α intermediate < α deep
(set to 0.9, 0.967, and 0.988 respectively), ensuring
θl decreases to a threshold of 5 after 30, 90, and
365 days for shallow, intermediate, and deep lay-
ers. This layered approach, implemented through
three-piece-wise functions for both SE
Importancel and
SRecencylE, enables FinMem to process long-
term memory in a stratified manner. Memory
events are purged when SRecencylE falls below
0.05 or SImportancel
E is under 5 (pre-scaling),
maintaining the relevance and efficiency of the
memory store.
2519

vE
l =



40 with probability p1
60 with probability p2
80 with probability p3
(5)
θl = (αl)δE
, l = shallow, intermediate, deep,
(6)
where p1 + p2 + p3 = 1 , but their
values vary by shallow, intermediate, and
deep processing. when shallow processing
p1, p2, p3 = {0.8, 0.15, 0.05}, intermediate pro-
cessing, p1, p2, p3 = {0.05, 0.8, 0.15} and deep
processing, p1, p2, p3 = {0.05, 0.15, 0.8}.
SE
Importancel
= vE
l ∗ θl, (7)
Furthermore, F INMEM employs an access
counter function to dynamically manage memory
events across layers, ensuring that crucial events
influencing trading decisions are elevated to deeper
layers for extended retention and recurring access.
This process, monitored by the LLM validation tool
Guardrails AI, tracks critical memory IDs across
layers. Events deemed pivotal for investment suc-
cess receive a 5-point boost to their importance
score ( SE
Importancel). Upon meeting upgrade crite-
ria for a deeper layer, an event’s recency score
(SRecencyl
E) is reset to 1.0, underscoring its sig-
nificance and preventing rapid decay. Conversely,
less relevant events gradually fade. This mecha-
nism allows F INMEM to efficiently identify, prior-
itize, and retain key events based on their nature
and retrieval frequency, while gradually phasing
out less impactful information, thereby maintain-
ing a dynamic and relevant memory structure for
financial decision-making.
B Details on Evaluation Metrics
Below is a brief overview of these metrics:
Cumulative Return (CR) % measures the total
value change of an investment over time by sum-
ming daily logarithmic returns, shown in Equa-
tion 8. Higher values indicate better strategy effec-
tiveness.
CR =
n∑
t=1
ri =
n∑
t=1
[
ln
( pt+1
pt
)
· actiont
]
(8)
, where ri is the logarithmic return from day t to
t + 1, pt and pt+1 are the closing prices on days t
and t + 1, respectively, and actiont is the model’s
trading decision for day t.
Sharpe Ratio (SR) assesses risk-adjusted returns
by dividing the average excess return ( Rp) over the
risk-free rate ( Rf ) by its volatility ( σp), detailed
in Equation 9. Higher ratios signify better perfor-
mance.
SR = Rp − Rf
σp
(9)
Annualized V olatility (A V) % and Daily V olatil-
ity (DV) % quantify return fluctuations; A V is
derived by scaling DV ( standard deviation of daily
logarithmic returns) by the square root of the an-
nual trading days (252), as in Equation 10. This
metric highlights potential return deviations across
the year.
A V= DV ×
√
252 (10)
Max Drawdown (MDD) % calculates the largest
portfolio value drop from peak to trough, as given
in Equation 11. Lower values indicate lesser risk
and higher strategy robustness.
MDD = max( Ppeak − Ptrough
Ppeak
) (11)
Note that CR and the SR are often considered
more essential than A V and MDD in evaluating
asset trading performance due to their focus on
long-term gains and risk-adjusted returns by their
definition. Here, we regard these two metrics as
primary metrics when evaluating the experiment
outcomes.
C An example of mixed and lagged
market signals: Partial investment
insights of TSLA on 2021-03-05
Here are some insights from the agent’s memory
module for TSLA as of 2021-03-05. A few memory
records are omitted; these are either neutral or posi-
tive. Despite this, the stock price trend for TSLA is
sharply downward, conflicting with the overall pos-
itive financial sentiments and market signals. Utiliz-
ing proprietary models such as GPT4 and GPT-o1
as backbones, the financial decision-making agent
can leverage other investment insights like histor-
ical momentum and self-reflection to consistently
support a ’Sell’ decision. In contrast, the large-
sized open-source models like Qwen2.5-72B and
DeepSeek-67B-Chat exhibit instability in produc-
ing consistent actions across repeated experimental
trials.
2520