PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change — part 3/5

Cleaned-text code points [26756, 39282). Bibliography omitted here; source.md includes it.

an unexpected event is noted (event corresponds to some facts unexpectedly turning true or false)
and then a new plan from the changed state is presented. The prompt ends with a new case where
the plan after replanning is left out and the LLM is expected to complete. The evaluation involves
checking whether the new plan is valid from the changed state. The LLM output is evaluated to be
true if the new plan it generates achieves the goals from the unexpectedly changed state.
For the Blocksworld domain, we constrain the unexpected event to be of a specific type. We execute
a random prefix of the plan which ensures that some block is held at the end of that prefix. We then
change the resulting state by stacking the held block onto another random block which is clear and
make the hand empty. This change is reported and the LLM is asked to replan from the changed state.
5.7 Plan Generalization
In this test case, we want to evaluate whether LLMs can recognize the underlying pattern in the
plans provided in the prompt and reuse it for a new planning problem. The prompt is the same as
the goal-directed reasoning case, except that all plans were generated by a fixed program. Here the
program may contain loops or conditional statements, but can only solve certain types of problems,
that is, the initial state and goals meet certain conditions. Such programs can be thought of as a direct
generalization of line plans that we have considered in the rest of the paper [28]. Execution of this
program for a specific planning problem generates a sequence of actions. In this case, we will provide
some example traces generated from the program and ask LLM to come up with a plan for a new
problem that could be solved by it. The evaluation again would be to take the generated plan and see
if it is valid for the given problem.
6 Evaluation
6.1 Results on the Blocksworld domain
Our evaluation here primarily focuses on two Large Language Models, GPT-3 and BLOOM. In
particular, we evaluated the test framework on the Blocksworld domain. In Table 1, we have presented
the results of vanilla GPT-3 (Davinci), Instruct-GPT3 (text-davinci-002), and BLOOM on six of the
test cases. For vanilla GPT-3 and Instruct-GPT3, we had tested on 500 instances while with BLOOM,
we had tested on 200 instances for plan generation and 100 instances for the rest of the test cases.
The experiments with GPT-3 (both the vanilla and instruct versions) took∼30 minutes for each test
case (500 instances) while BLOOM took∼36 hours (every 100 instances). The best results (within
each model) were for the auxiliary goal reformulation test cases. For these three cases, all that was
required for the LLM was to repeat the same plan as the one shown in the example. Even then,
vanilla GPT-3 and Instruct-GPT-3 failed to do that for some of the instances in the first two cases
and the majority of the instances in the third case. BLOOM, on the other hand, was poor in all three
cases. Coming to the two test cases that correspond to actual planning problems (plan generation and
optimal planning), all three models performed poorly with Instruct-GPT3 performing better than the
other two. Overall, the performance of these LLMs on our benchmark shows that, as of right now,
LLMs are pretty ineffective in reasoning about actions and change. It would be interesting to see the
utility of fine-tuning or even that of prompt engineering applied to our experiments.
6.2 Human Baseline for the Blocksworld domain
We have previously mentioned that planning tasks on the blocksworld domain are anecdotally simple
enough for humans to perform. To establish this and come up with a baseline to compare LLMs
performance, we conducted a preliminary study where we asked 50 participants to come up with
a plan for a blocksworld instance picked at random, from the set of 500 instances that we used for
the evaluation of LLMs. We presented the same domain description as we did for the LLMs (see
Appendix A.1.1) and then primed them with an example instance. Further, we provided them with an
interface where they had two phases of interaction. In the first phase, they could write up plans by
themselves for the given instance and then in the second phase, translate them (by picking the closest
action from a list of grounded actions). The translated plans were used in the back-end and were
evaluated in an automated fashion4. They went through this procedure first for an example instance
4We had also manually evaluated the plans that they wrote in case they made a mistake during the translation
phase
8

Task Instances correct
GPT-3 Instruct-
GPT3
BLOOM
Plan Generation
We showcase an instance and the respective plan as an example and prompt the machine with a
new instance.
3/500
(0.6%)
25/500
(5%)
1/200
(0.5%)
Optimal Planning
We showcase an instance, the respective optimal plan and the associated cost as an example
and prompt the machine with a new instance.
1/500
(0.2%)
16/500
(3.2%)
0/100
(0%)
Replanning
We showcase an instance, the respective plan and present an unexpected change of the state.
We then also present a new plan from the changed state. Finally, for a new instance we repeat
the same except we ask the machine for the new plan.
28/500
(5.6%)
24/500
(4.8%)
3/100
(3%)
Plan Generalization
We showcase an instance and the respective plan as an example and prompt the machine with a
new instance. The plans for both the instances can be generated by a fixed program containing
loops and conditionals.
33/500
(6.6%)
49/500
(9.8%)
11/100
(11%)
Plan Reuse
We showcase an instance and the respective plan as an example and prompt the machine with
an instance which requires only a certain prefix of the plan provided in the example.
0/500
(0%)
72/500
(14.4%)
0/100
(0%)
Robustness to Goal Reformulation (Shuffling goal predicates)
We showcase an instance and the respective plan as an example and prompt the machine with
the same instance but shuffle the ordering of the goals.
387/500
(77.4%)
384/500
(76.8%)
21/100
(21%)
Robustness to Goal Reformulation (Full → Partial)
We showcase an instance with a fully specified goal state and the respective plan as an example
and prompt the machine with the same instance but provide a partially specified goal state.
346/500
(69.2%)
380/500
(76%)
9/100
(9%)
Robustness to Goal Reformulation (Partial → Full)
We showcase an instance with a partially specified goal state and the respective plan as an
example and prompt the machine with the same instance but provide a fully specified goal state.
110/500
(22%)
301/500
(60.2%)
5/100
(5%)
Table 1: LLM Assessment Suite Results on vanilla GPT-3 (davinci), Instruct-GPT3 (text-davinci-002)
and BLOOM (176B model). The tasks in the highlighted rows correspond to actual planning problems
while the others correspond to simpler auxiliary planning tasks.
(where they were provided with a glimpse of the example solution before using the interface) and
then for the actual instance. We provided them with a bonus if they come up with a valid plan.
Out of the 50 participants, 39 of them (78%) came up with a valid plan. Along with validity, we also
tested the optimality of their plans even though they were not required to come up with an optimal
plan. Out of the 39 participants, 35 (89.7%) participants came up with an optimal plan. These initial
results show that the blocksworld domain is a simple enough domain where most humans are able
to come up with plans (which are also optimal) while LLMs, on the other hand, showcase subpar
performance. A more extensive human subject study is underway to establish a more concrete human
baseline for performance comparisons.
7 Conclusion and Future Work
In this paper, we looked at a reasoning assessment suite for large language models (LLMs) that
consists of various test cases each evaluating a central aspect of reasoning about actions and change.
Our results show that even in simple common-sense planning domains where humans could easily
come up with plans, LLMs like GPT-3 and BLOOM seem to display a dismal performance. However,
we do not claim that no LLM system could potentially ever perform effective reasoning about actions
and change. Our goal is to establish an extensible benchmark where researchers can evaluate current
and future large language models. Our assessment suite is still in progress and we look to improve it
in multiple ways in the future. Firstly, we plan to include a modified version of the reasoning about
plan execution task to ask questions that require a more descriptive answer and provide automated
9

validations for the answers. Secondly, we also plan to look at how fine-tuning would impact the
performance of the LLMs on our benchmarks, although as [34] points out that LLMs tend to focus
on correlational features even with extensive training, we are not expecting a major improvement.
Finally, we plan to add an evaluation metric that also considers partial correctness of plans. This
benchmark can also be extended to other domains, either to other common-sense domains (like
Virtual Home [22]) or to specialized ones. It would also be interesting to probe these models to come
up with explanations for the generated plans and see if these make sense to humans in the loop. In
conclusion, we hope that this benchmark 5 encourages other researchers to test the capabilities of
their systems across different LLM models [5, 7, 26, 35, 24, 30, 10] and even those that are finetuned
for such tasks.
8 Acknowledgements
Kambhampati’s research is supported by the J.P. Morgan Faculty Research Award, ONR grants
N00014-16-1-2892, N00014-18-1-2442, N00014-18-1-2840, N00014-9-1-2119, AFOSR grant
FA9550-18-1-0067 and DARPA SAIL-ON grant W911NF19-2-0006. We also want to thank OpenAI
and Miles Brundage for letting us get research access to the GPT-3 API.
References
[1] BigScience Large Open-science Open-access Multilingual Language Model. https://
huggingface.co/bigscience/bloom.
[2] Constructions Aeronautiques, Adele Howe, Craig Knoblock, ISI Drew McDermott, Ashwin
Ram, Manuela Veloso, Daniel Weld, David Wilkins SRI, Anthony Barrett, Dave Christianson,
et al. PDDL| The Planning Domain Definition Language. Technical Report, Tech. Rep., 1998.
[3] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David,
Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not
as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
[4] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are
few-shot learners. arXiv preprint arXiv:2005.14165, 2020.
[5] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam
Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm:
Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
[6] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christo-
pher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint
arXiv:2110.14168, 2021.
[7] Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu,
Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. GLaM: Efficient Scaling of
Language Models with Mixture-of-Experts. arXiv preprint arXiv:2112.06905, 2021.
[8] Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did
aristotle use a laptop? a question answering benchmark with implicit reasoning strategies.
Transactions of the Association for Computational Linguistics, 9:346–361, 2021.
[9] Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch Critch, Jerry Li Li, Dawn Song,
and Jacob Steinhardt. Aligning ai with shared human values. In International Conference on
Learning Representations, 2021.
[10] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza
Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.
Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.
[11] Richard Howey, Derek Long, and Maria Fox. V AL: Automatic plan validation, continuous
effects and mixed initiative planning using PDDL. In 16th IEEE International Conference on
Tools with Artificial Intelligence, pages 294–301. IEEE, 2004.
5Link to the github repo: https://github.com/karthikv792/gpt-plan-benchmark
10