PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change — part 1/5

Cleaned-text code points [0, 14895). Bibliography omitted here; source.md includes it.

Large Language Models Still Can’t Plan
(A Benchmark for LLMs on Planning and Reasoning
about Change)
Karthik Valmeekam∗
School of Computing & AI
Arizona State University, Tempe.
kvalmeek@asu.edu
Alberto Olmo∗
School of Computing & AI
Arizona State University, Tempe.
aolmo@asu.edu
Sarath Sreedharan†
Department of Computer Science,
Colorado State University, Fort Collins.
sarath.sreedharan@colostate.edu
Subbarao Kambhampati
School of Computing & AI
Arizona State University, Tempe.
rao@asu.edu
Abstract
Recent advances in large language models (LLMs) have transformed the field
of natural language processing (NLP). From GPT-3 to PaLM, the state-of-the-
art performance on natural language tasks is being pushed forward with every
new large language model. Along with natural language abilities, there has been
a significant interest in understanding whether such models exhibit reasoning
capabilities with the use of reasoning benchmarks. However, even though results
are seemingly positive, these benchmarks prove to be simplistic in nature and the
performance of LLMs on these benchmarks cannot be used as evidence to support,
many a times outlandish, claims being made about LLMs’ reasoning capabilities.
Further, these only represent a very limited set of simple reasoning tasks and we
need to look at more sophisticated reasoning problems if we are to measure the true
limits of such LLM-based systems. Motivated by this, we propose an extensible
assessment framework to test the capabilities of LLMs on reasoning about actions
and change, a central aspect of human intelligence. We provide multiple test cases
that are more involved than any of the previously established benchmarks and
each test case evaluates a different aspect of reasoning about actions and change.
Results on GPT-3 (davinci), Instruct-GPT3 (text-davinci-002) and BLOOM (176B),
showcase subpar performance on such reasoning tasks.
1 Introduction
It would be no exaggeration to say that transformer-based large language models (LLMs) have
revolutionized the field of natural language processing (NLP). Kicked off by the advances presented
by the GPT-x models developed by OpenAI [23], these types of language models currently provide
state-of-the-art performance in many of the standard NLP tasks. The latest version of the system,
GPT-3 [4] uses about 175 billion parameters and was trained over an extremely large natural language
training corpus, consisting of, among other things, excerpts from Wikipedia. Triggered by GPT-3, a
plethora of other large language models, which are different variants of the transformer architecture
∗equal contribution
†Author was at Arizona State University during part of this work
Foundation Models for Decision Making Workshop at Neural Information Processing Systems, 2022.

[31], have been developed. Some of the most powerful models are PaLM [5], GLaM [7], Megatron-
Turing NLG [26], Meta-OPT [35], Gopher [24], LaMDA [30] and Chinchilla [10]. PaLM currently
provides state-of-the-art performance in NLP tasks such as natural language translation, predicting
long-range text dependencies and even translation to structured representations [5, 19].
Although LLMs were originally developed mostly to do language completion tasks, with no guaran-
tees about the completion beyond its coherence, there have been increasing claims that they can do
other tasks including explaining jokes [14]. Of particular interest is the thread of efforts that aim to
evaluate (and showcase) the LLM’s ability to do reasoning tasks. For example, there are many claims
centered around the fact that GPT-3 may possess some form of reasoning ability [16]. Such sources
generally assume that because the model learned from large amounts of real-world text, it may have
acquired some approximation of simple reasoning. This sparked interest in evaluating the large
language models on various reasoning tasks including common-sense reasoning [29, 25, 8], logical
reasoning [27], and even ethical reasoning [13]. The macro-tenor of the drumbeat of these works has
been suggesting that LLM’s are indeed capable of doing such kinds of reasoning [15, 33, 5].
Thus, in this paper, we want to look at the ability of large language models to do reasoning about
actions and change involving common-sense planning tasks. We develop a suite of benchmarks,3
based on the kinds of domains employed in the International Planning Competition, to test these
capabilities. Our focus on this specific task is spurred by not only the fact that reasoning about actions
and change is a core aspect of human intelligence, but also that it is required for many of the tasks
considered as potential applications of LLMs including automatic code generation, moral and even
deontological reasoning [9].
We are not the first to point out the need to perform such analyses of the reasoning capabilities
of GPT-3 like LLMs. For example, [ 18] performed an analysis of GPT-3’s reasoning capabilities
on some example tasks, including different commonsense reasoning tasks varying from biological
reasoning to arithmetic reasoning. However, the goal of this paper is fundamentally distinct from
these earlier works in multiple ways. Firstly, we are not merely trying to point out a few example
cases where GPT-3 fails but rather help establish an assessment framework for evaluating these
systems’ capabilities to perform reasoning about actions and change. While this paper reports the
results of testing vanilla GPT-3, Instruct-GPT3 [20] and BLOOM [1], one could in the future use
this framework to analyse other LLMs that may be fine-tuned for such tasks. Secondly, through this
framework, we are also trying to eliminate the subjective aspect of analysis that forms the core part
of many of these earlier efforts. Instead, we automate and perform the analyses in a mechanistic way
by leveraging automated planning models and tools to generate the queries and validate the system’s
answers. Finally, we propose a curriculum for evaluating reasoning about actions and change, wherein
we identify a set of related but distinct reasoning tasks, that are central for an agent to successfully
perform reasoning about actions and change and introduce a framework for developing code which
is meant to auto-generate the possible queries for each. The conclusion from our evaluation on
GPT-3 and BLOOM is that LLMs are still far from achieving acceptable performance on common
planning/reasoning tasks which pose no issues for humans to do. In particular, our benchmarks do
not require complex long-chain combinatorial reasoning and can be easily understood by lay people.
Therefore, our contributions are twofold, we
1. Provide an extensible suite of benchmarks for evaluating the planning/reasoning capabilities
of LLMs
2. Show that GPT-3 and BLOOM have dismal performance on these benchmarks.
Additionally, we perform a preliminary user study that establishes a human baseline for coming up
with plans in a simple planning domain to compare the LLMs’ performance. Our hope is that these
benchmarks spur other researchers to evaluate out-of-the-box or fined-tuned versions of other LLMs
on these planning tasks.
2 Current Reasoning Benchmarks for LLMs
To the best of our knowledge, current available benchmarks are insufficient to make substantive
claims about LLMs’ ability to reason. The authors of [ 24] had pointed out that even language
3Link to the github repo: https://github.com/karthikv792/gpt-plan-benchmark
2

Figure 1: Example prompts from existing benchmarks (left) compared to an average-length prompt
from one of our test cases (right). (a) GSM8k [ 6], (b) SvAMP [21], (c) AQuA [17], (d) Common-
SenseQA [29], (e) StrategyQA [8], (f) Tracking Shuffled Objects [27], (g) Date Understanding [27],
(h) Last Letter Concatenation [33], (i) Coin Flip [33], (j) Goal directed reasoning on blocksworld.
models at the scale of 100B or more parameters had struggled on reasoning tasks that require slow
and multi-step reasoning, but the more recent works [15, 33, 32] have showcased and claimed that
LLMs can do various reasoning tasks with a decent performance. These claims have been based
on several reasoning benchmarks (consisting of arithmetic, common-sense and symbolic reasoning
tasks), where the reasoning tasks are relatively simple and require shallow reasoning. Datasets
like GSM8K [6], AQUA [17] and SV AMP [21] have simple math word problems which are used
for evaluating arithmetic reasoning while datasets like CommonsenseQA [29] and StrategyQA [8],
which have generic multiple choice and binary yes/no questions respectively, are used for evaluating
common sense reasoning. There are several logical reasoning tasks in BIG-BENCH [27] and there
are two symbolic reasoning tasks, Last Letter Concatenation and Coin Flip [ 33], on which LLMs
have been evaluated. However, these tasks are simple in nature and do not provide insight into the
reasoning capabilities of LLMs. As LLMs have been able to perform well on such tasks, there has
been a lot more triumphalism about their reasoning capabilities, which is currently being echoed in
the community.
LLMs can also be used for heuristic guidance with no optimality or completeness guarantees, as long
as the guidance is used by a sound planner underneath. For example, in Say-Can [ 3], LLMs have
been used as scoring models, which can be seen as providing planning heuristics, for the actions
that the embodied robot can execute. In our work, we are looking at the inherent emergent planning
capabilities of the LLMs, if any. The assessment framework we propose in this paper consists of
multiple tasks each of which evaluate a certain aspect of reasoning about actions and change. Even
though we use a simple common-sense planning domain, the prompts for our tasks are relatively
larger and more complex than any of the prompts in the current benchmarks (as shown in Figure
1). This is due to two main reasons; first, the prompts for our tasks are showcased as few-shot
examples, whereas the prompts in the current benchmarks are zero-shot. Our prompts provide an
3

instance and an example completion and then ask for a completion on a new instance. Second, we
use the domain description to explicitly constrain the possible actions. In many everyday scenarios,
we are often asked to take into consideration unforeseen limitations and constraints. Our explicit
domain description allows us to introduce such challenges, and forces the LLMs to go beyond merely
repeating possible information about the domain they may have come across in its training data. The
ability to be conditional on the prompt is critical for the general systems to be customized for the
specific domain of interest.
Our results, contrary to the current claims, show that even simple common-sense planning tasks
(which are, anecdotally, easy for humans) are far beyond the current capabilities of LLMs. This is
actually not surprising given that LLMs only provide the most likely text completion for a given
prompt with no real guarantees on reasoning metrics. There have been works that tried to evaluate
whether LLMs can generate plans in common-sense domains [ 12] but in those works they had
evaluated on the world knowledge that LLMs already possess and do not provide any information
about the domain in the prompt. Before we delve into the details of our framework, we will first
establish the required background to get a better understanding of the kind of reasoning capability
that we are focusing on.
3 Background
Given that we are interested in investigating the basic reasoning about actions and change problem, we
want to look at the most fundamental planning formalism first, namely the goal-directed deterministic
planning problem. Colloquially referred to as classical planning problem, these problem classes can
be mathematically represented by using the tupleP =⟨D,I,G⟩.D is referred to as the problem
domain,I is the initial state andG is the goal specification. The state-space for the planning problem
is defined by the possible truth assignment over the predicates. The domain is again defined by the
tupleD =⟨F,O⟩.F corresponds to the set of fluents, i.e., the state variable used to define the state
space and each fluent corresponds to a predicate with some arity, and A correspond to the set of
actions that can be performed as part of the planning problem. Each actionai[V]∈A (whereai is
the operator label andV is the variable used by the operator and each variable could be mapped to an
object), can be further defined by two components, the preconditionprec[V] which describes when an
action can be executed and the effectseff [V] which defines what happens when an action is executed.
We will assume thatprec[V] consists of a set of predicates defined over the variablesV. An action
is assumed to be executable only if its preconditions are met, i.e, the predicates in the precondition
hold in the given state. The effect seteff [V] is further defined by the tuple⟨add[V],del [V]⟩, where
add[V] or add effects is the set of predicates that will be set true by the action anddel[V] or delete
effects is the set of predicates that will be set false by the action. An action is said to be grounded if
we replace each of the variables with an object, else it is referred to as a lifted domain model (we use
a similar convention to differentiate between lifted and grounded predicates).
Below we have provided a snippet of an action description from a popular benchmark problem called
Blocksworld for action corresponding to picking up a block.
(: action pickup
: p a r a m e t e r s (? ob )
: p r e c o n d i t i o n ( and ( clear ? ob ) ( on - table ? ob ) ( arm - empty ) )
: effect ( and ( holding ? ob ) ( not ( clear ? ob ) ) ( not ( on - table ? ob ) )
( not ( arm - empty ) ) ) )
The parameter line provides the possible variables, in this case ?ob, which can stand for possible
blocks. The precondition says that you can only pick up a block if it is clear (i.e. predicate (clear
?ob) is true for the block), the block is on the table and the arm is empty. The effects tell you that
after you execute the action, the predicate (holding ?ob) becomes true and the block will no longer be
considered clear, and on-table. Finally, the arm will no longer be considered empty. A solution to a
planning problem is called a plan, and corresponds to a sequence of actions that once executed in the
initial state would lead to a state where the goal specification is true. The actions may additionally be
associated with cost, in these cases, one could also talk about optimal plans, i.e., a planπ is called an
optimal one if no plan exists that is less costly thanπ.
The above description presents one of the simpler classes of planning models and can be extended
in multiple ways including allowing for object typing (including type hierarchy), more complex
4