LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks — part 3/3

Cleaned-text code points [29032, 44038). Bibliography omitted here; source.md includes it.

As mentioned earlier, we avoid having humans involved in iteratively prompting LLMs–as this can be an infeasibly time-consuming activity for them. Instead, we let automated verifiers, either model-based or LLM-supported, to manage the plan critiquing process. The framework does depend on humans for “once per domain” and “once per problem” interactions. In the former category, human domain experts can play a role in acquiring the domain model with the help of the LLM. Examples of such interaction include teasing out PDDL planning models from the LLMs with the help of human expert curation (top left in Figure 3 ). An example of this is our work in ( Guan et al., 2023 ) . The idea here is that the traditional domain model acquisition task (e.g. ( Simpson et al., 2001 ) ) is significantly made easier by having the LLMs help with ideas regarding various pieces of the domain model (e.g., actions, their preconditions and effects) and letting humans sign off/critique the resulting model. Once the model is acquired this way, it can be used by correctness verifiers such as VAL ( Howey et al., 2004 ; Guan et al., 2023 ) . Often the planning problems in real world situations are specified incompletely, leaving it to the human commonsense to refine the specification. This brings up a second role for humans–this time end users (bottom left in Figure 3 –in collaboratively refining the specification with the help of LLMs (similar to the way done in ( Xie et al., 2023 ; Liu et al., 2023 ) ).

### 3.4 Summary of LLM Roles in LLM-Modulo

It is worth summarizing the multiple roles the LLM plays in the LLM-Modulo architecture. The most prominent, of course, is its role in “guessing” the candidate plans (step 2 in Figure 3 ) in response to problem specification and iterative back prompting from the bank of critics (Step 5). Second, the LLM plays a role in converting the guessed plan candidate into specialized representations used by the various critics (e.g., the time-line view, the causal link view etc.). This role leverages the fact that LLMs are very good at format conversion (c.f. ( Olmo et al., 2021 ) ). Third, the LLM plays a role in helping the end user flesh out the incomplete problem specification to begin with (Step 1 in Figure 3 ). Finally, the LLM plays a role in helping the domain expert tease out and refine the domain models used by the various model-based critics ( Guan et al., 2023 ; Kwon et al., 2022 ) , or help “implement” procedural critics (such as those checking syntactic constraints). As a broad approximate source of knowledge, the LLM can also help enumerate the list of potential critics needed to validate the candidate plans (once again with a human in the loop).

### 3.5 Can LLM-Modulo Frameworks Pay Their Way?

Let’s address the elephant in the room: Given that formal model-based planning systems already exist ( Ghallab et al., 2004 ) , is LLM-Modulo framework for planning more than a gratuitous attempt to shoe-horn (the currently popular) LLMs to solve planning problems? Indeed, when the underlying problem is actually solvable by such combinatorial solvers, it can be orders of magnitue more resource efficient to use them. 10 10 10 Not surprisingly, automated programming, the one community that certainly doesn’t have the luxury of a ready-made “solver,” have stuck to LLM-Modulo style approaches. Compared to a planner that is guaranteed to be correct in a narrow set of domains, LLMs may likely be good at generating plausible (but not guaranteed to be correct) plan heuristics/suggestions in many more scenarios. Thus, unlike the traditional planning architectures studied in AI ( Ghallab et al., 2004 ) , which put a priori constraints on the expressiveness of the problems that can be posed to the planner (to wit, the different expressiveness levels of the PDDL specification ( McDermott et al., 1998 ) ), the LLM-Modulo architecture puts no such restrictions. In this sense, it is more representative of real-world planning problems such as those in NASA mission planning, where the different critics–human and automated–are at best able to give “no objection” certificates for the candidate plans under consideration, clearing it from their perspective. (Indeed, both deep space network planning and mars rover task planning are done via a collective human blackboard . ( Johnston et al., 2014 ; Bresina et al., 2004 ) .) Note that this is starkly different from just sending an unvetted plan out to execution (as would be the case if we have LLMs operate in autonomous mode to guess plans). Generalizing planning and reasoning frameworks this way is consistent with the Doyle & Patil’s call to the Knowledge Representation community of yore ( Doyle & Patil, 1991 ) , as well as our own call for model-lite planning ( Kambhampati, 2007 ) .

## 4 Two Case Studies of LLM-Modulo

We have applied the LLM-Modulo framework to classical planning domains (as reported in ( Valmeekam et al., 2023c ) ) and to a recent travel planning benchmark (as reported in ( Gundawar et al., 2024 ) ). In the former case, the results (presented in Section 5.2 and Table 4 of ( Valmeekam et al., 2023c ) ) show that with back prompting from VAL ( Howey et al., 2004 ) acting as the external verifier and critic, LLM performance in Blocks World improves to 82% within 15 back prompting rounds, while in Logistics, it improves to 70%. LLM-Modulo doesn’t help as much in an obfuscated version of blocks world called Mystery BW, reaching about 10% accuracy. This should be expected because the LLMs have difficulty generating plausible candidate plans for this domain (note that even here, if a plan is returned, it must have passed muster with VAL, and is thus guaranteed correct by its model).

For the travel planning case study, we used a benchmark proposed in ( Xie et al., 2024 ) , which involves a rich mix of travel constraints presented in flexible natural language form. Our preliminary results on adapting LLM-Modulo framework to this benchmark are reported in ( Gundawar et al., 2024 ) . The benchmark’s authors test LLMs across a variety of prompt engineering techniques including Chain of Thought and ReAct, reporting that–on GPT-3.5-Turbo–the current best strategies only manage a startlingly low 0.7% performance rate! We adapted the LLM-Modulo framework to this benchmark by operationalizing their hard constraints (such as the budget constraint set by the user) or common-sense constraints (such as suggesting diverse attractions to visit) as critics as shown in Figure 4 . Our preliminary results show (see Figure 5 ; additional results in ( Gundawar et al., 2024 ) ) that LLM-Modulo based agentification with automated critics in the loop significantly improves the performance (6x of baselines) even with a limit of 10 back prompting cycles, and weaker models such as GPT-3.5-turbo. Furthermore, we also find that LLMs can successfully implement functions corresponding to hard critics and several common-sense critics. Finally, LLMs reliably play the role of reformatter as well, converting free form travel plans into structured plans parseable by the critics for backprompts or plan evaluation. One interesting observation about this domain is that we were able to use the LLM itself to enumerate the type of critics needed to validate the plan (with light human supervision).

## 5 Related Work

While the LLM-Modulo framework is being proposed in general form here for the first time, there are certainly works in leveraging LLMs in planning and reasoning tasks that are in line with the spirit of the LLM-Modulo framework. Work on FunSearch ( Romera-Paredes et al., 2023 ) depends on a generate-test loop between a specially fine-tuned LLM that guesses solutions, and an external symbolic evaluator that critiques them. The authors note how the external verifier is critical for avoiding falling prey to hallucinations (i.e., approximate solution candidates that have flaws). AlphaGeometry ( Trinh et al., 2024 ) too depends on the Generate-Test-Critique interaction between a fine-tuned LLM and a symbolic evaluator. Both these systems fine-tune pre-trained LLMs with task specific synthetic data–the correctness of which is vetted with external simulators.

While we focused on PDDL planning tasks for the sake of concreteness, we believe that the essence of LLM-Modulo framework is equally applicable to other scenarios involving planning and reasoning–such as Reinforcement Learning with Simulators . Such RL systems rely on rewards as feedback to train a policy. Simulators takes on the roles of plan evaluation and critiques performed by the respective critics in the LLM-Modulo framework (e.g. ( Rajvanshi et al., 2023 ) ). The fact that simulators play the role of verifiers is often not explicitly recognized in cases where LLMs are used as an actor to generate an admissible plan by interacting with a simulator, for example in the case of AlfWorld ( Yao et al., 2023b ; Shinn et al., 2023 ) and Minecraft ( Wang et al., 2023b ) . Similar to extracting a domain model such as in the case of ( Guan et al., 2023 ) , LLMs can also be used for designing a reward model or shaping the reward ( Bhambri et al., 2024 ; Kwon et al., 2022 ; Hao et al., 2023 ; Ma et al., 2023 ) .

Interestingly, the fact that LLM’s can help come up with approximate quasi-symbolic transition models, reward models and models of high level actions has made a bigger splash in RL. This is because for far too long, researchers there have tried to spurn any high level models (lest that would involve depending on humans; ( Kambhampati, 2021 ) ) and focused on learning to act from sensory information, under the name of “deep reinforcement learning.” Given the horrendous sample complexity of the DRL methods even in reaching a single subgoal, and the well known fact that even approximate symbolic models can help drastically improve the performance (c.f. ( Guan et al., 2022 ) ), coupled with the fact that LLM’s are only too glad to dream up approximate models and goal recipes, there has been a performance revolution of sorts there ( Yao et al., 2023b ; Liang et al., 2023 ; Wang et al., 2023b ) . If we look beyond the improvements in these lower level goal seeking behaviors–especially in the presence of ergodic simulators, the RL approaches dependent on LLMs will encounter the same issues regarding subgoal interactions that our discussion of PDDL planning problems brought into focus. The LLM-Modulo inspired frameworks will thus, we believe, be equally relevant there. Indeed, SayCan ( Ahn et al., 2022 ) the earliest use of LLMs in generating policies in an RL-with-Simulator scenario, explicitly filters the action choices suggested by the LLM with the help of simulator.

Although we focused on text based LLMs (such as GPT4), recently there have also been impressive development in multi-modal LLMs (e.g. GPT4V). While multi-modality is a great addition that increases the coverage of their System 1 imagination (Figure 1 ), it is not clear that this gives them System 2 competence. 11 11 11 If you know how to complete sentences, and now learned to complete dance moves, does your ability to reason/plan magically improve? As we discussed earlier, we can leverage VLMs for style criticism of the robot behavior ( Guan et al., 2024 ) .

Finally, our position (with published supporting evidence) that LLMs are incapable of supporting planning in autonomous modes must seem quite at odds with the current head-long rush into agentic LLMs. We believe that the latter is largely a result of confusing “acting” with “planning.” Given their ability to translate across formalisms, it is of course possible for LLMs to invoke external services–something frameworks like AutoGPT and LangChain support. But the mere ability to invoke an action doesn’t, in any way, guarantee that the course of actions thus invoked will achieve a desired state of affairs. The only way to guarantee the latter is to to support robust planning capabilities–something our LLM-Modulo frameworks strive to do.

## 6 Conclusion

This position paper is a modest attempt to combat both over-optimism and over-pessimism about the role of LLMs in planning and reasoning tasks. Our position is that LLMs cannot plan themselves but can play a variety of constructive roles in solving planning tasks–especially as approximate knowledge sources and candidate plan generators in the so-called LLM-Modulo Frameworks in conjunction with external sound model-based verifiers. In support of this position, we summarized the literature questioning the claims about the planning and self-verification capabilities of LLMs by themselves. We also discussed how conflating approximate knowledge acquisition and generating executable plans of action is behind many of the claims about planning and verification abilities of LLMs. We then shared LLM-Modulo framework, our vision for a productive way to integrate the impressive idea generation/approximate knowledge provision capabilities of LLMs with external verifiers with correctness guarantees, for robust and expressive planning. We discussed how planning in LLM-Modulo framework avoids inheriting the expressiveness and search-complexity limitations of traditional symbolic planners, while retaining their soundness guarantees. We illustrated and discussed the many roles LLMs can play in the LLM-Modulo framework. Finally, we also discussed two case studies of adapting the LLM-Modulo frameworks.

## Impact Statement

This position paper takes a stance on a robust and well-founded way of leveraging Large Language Models in planning and reasoning tasks. It (i) points out the inabilities of current pre-trained LLMs to tackle planning problems, (ii) suggests some reasons as to why there are wide-spread misunderstandings about LLM planning abilities and (iii) proposes LLM-Modulo frameworks as a way to leverage LLMs for robust planning. The main consequences of realizing this position/vision is expected to be (i) sounding caution about misapplication of LLMs in autonomous modes for planning (ii) providing a way to leverage LLMs to do robust planning. Given the current interest in agentic LLMs, these insights can have significant positive impact in mission critical situations. We do not see any obvious negative societal consequences of leveraging LLMs this way (unless of course the plans are aimed at achieving malicious goals).

## Acknowledgments

The ideas discussed in this paper have evolved over a series of talks, tutorials and twitter threads. The discussions, feedback and encouragement from colleagues, including Sarath Sreedharan, Tom Dietterich, Yann LeCun, Daniel Borrajo, and Dan Weld is gratefully acknowledged. The adaptation of LLM-Modulo Framework for Travel Planning, discussed in Section 4 was lead by Atharva Gundawar. Kambhampati acknowledges generous support from ONR via grants N00014-18-1-2442, N14-18-1-2840 and N00014-23-1-2409, as well as gifts from J.P. Morgan, Qualcomm and Amazon.