LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks — part 1/3

Cleaned-text code points [0, 13662). Bibliography omitted here; source.md includes it.

# Position: LLMs Can’t Plan, But Can Help Planning in LLM-Modulo Frameworks

###### Abstract

We argue that auto-regressive LLMs cannot, by themselves, do planning or self-verification (which is after all a form of reasoning), and shed some light on the reasons for misunderstandings in the literature. We also argue that LLMs should be viewed as universal approximate knowledge sources that have much more meaningful roles to play in planning/reasoning tasks beyond simple front-end/back-end format translators. We present a vision of LLM-Modulo Frameworks that combines the strengths of LLMs with external model-based verifiers in a tighter bi-directional interaction regime. We will show how the models driving the external verifiers themselves can be acquired with the help of LLMs. We will also argue that rather than simply pipelining LLMs and symbolic components, this LLM-Modulo Framework provides a better neuro-symbolic approach that offers tighter integration between LLMs and symbolic components, extending the scope of model-based planning/reasoning regimes towards more flexible knowledge, problem and preference specifications.

###### Keywords:

## 1 Introduction

Large Language Models (LLMs), essentially n-gram models on steroids which have been pre-trained on web-scale language corpora (or, effectively, our collective consciousness), have caught the imagination of the AI research community with linguistic capabilities that no one expected text completion systems to possess. Their seeming versatility has led many researchers to wonder whether they can also do well on planning and reasoning tasks typically associated with System 2 competency. On the face of it, this doesn’t seem to ring true, as both by training and operation, LLMs are best seen as a giant pseudo System 1 ( Kahneman, 2011 ) (see Figure 1 ). Even from a pure engineering perspective, a system that takes constant time to produce the next token cannot possibly be doing principled reasoning on its own. 1 1 1 Think of asking an LLM an yes/no question– is this theorem logically entailed by this first-order logic knowledge-base . This is well-known to be a semi-decidable problem. Ask yourself if the LLM will take longer in answering the question. (If you are thinking Chain-of-thought prompts or training with step-by-step data, consider that you are essentially changing the nature of the original prompt/training). Not surprisingly, initial excitement based on anecdotal performance of LLMs on reasoning tasks ( Bubeck et al., 2023 ) has been dissipated to some extent by the recent spate of studies, including our own, questioning the robustness of such behaviors–be they planning ( Valmeekam et al., 2023c ; Kambhampati, 2024 ) , simple arithmetic and logic ( Dziri et al., 2023 ) , theory of mind ( Ullman, 2023 ; Verma et al., 2024b ) , or general mathematical and abstract benchmarks ( McCoy et al., 2023 ; Gendron et al., 2023 ) . Despite this, a steady stream of claims continue to be made in the literature about the planning and reasoning capabilities of LLMs. In light of questions about their planning capabilities, the head-long rush into agentic LLMs should be particularly concerning. After all, acting without the ability to plan is surely a recipe for unpleasant consequences!

In an ironic juxtaposition to this unwarranted optimism about the planning and reasoning abilities of LLMs, there is also unwarranted pessimism about the roles LLMs can play in planning/reasoning tasks. Several efforts (e.g. ( Liu et al., 2023 ; Pan et al., 2023 ; Xie et al., 2023 ) ) advocate using LLMs only as glorified translators–converting reasoning problems embedded in textual format to symbolic representations, and pawning them off to external classical symbolic solvers (with all their attendant expressivity and search complexity challenges ( Doyle & Patil, 1991 ) ). 2 2 2 In some circles, this unidirectional pipeline has been given the undeserved badge of neuro-symbolic architecture .

In truth, LLMs can be a whole lot more than machine translators. They are a kind of approximate knowledge source (albeit sans guarantees) trained on our collective consciousness. While it is unlikely that they will have System 2 competencies by themselves, they can nevertheless be valuable resources in solving System 2 tasks. To put it another way, the problem with Alchemy of yore was not that Chemistry is useless, but that people wanted to delude themselves that Chemistry–a pretty amazing discipline on its own merits–can be Nuclear Physics if you prompt it just so. The confusions regarding LLM abilities, or should we say, LLM alchemy, doesn’t seem to be much different–oscillating between ignoring their strengths, and ascribing abilities they don’t have.

The goal of this position paper is to introduce some clarity into this confusing state of affairs oscillating between over-optimism and over-pessimism. Simply put, we take the stance that LLMs are amazing giant external non-veridical memories that can serve as powerful cognitive orthotics for human or machine agents, if rightly used. The underlying n-gram nature makes them effortlessly intermix what would be considered disparate fields of study (not surprisingly, LLMs are seen to be very good at making/finding analogies!). The challenge is to leverage them without wrongly ascribing to them capabilities they don’t possess. The LLM-Modulo framework proposed in this position paper tackles this challenge.

For the sake of concreteness, we consider planning tasks, especially as studied in the automated planning community ( Ghallab et al., 2004 ) . The central position of the paper is that LLMs cannot plan themselves but can play a variety of constructive roles in solving planning tasks–especially as approximate knowledge sources and candidate plan generators in so-called LLM-Modulo Frameworks, where they are used in conjunction with external sound model-based verifiers .

We support this position by first reviewing literature, including our own works, that establishes that LLMs cannot be used as planners or plan verifiers themselves (Section 2 ). We also discuss why there are claims about planning/verification abilities in the first place, in the process hopefully clarifying some prevalent misunderstandings.

Second, we will propose a framework that allows us to leverage LLMs effectively in planning tasks, by combining them with external critics, verifiers and humans. We call this an LLM-Modulo Framework (a name loosely inspired by SAT Modulo Theories ( Nieuwenhuis & Oliveras, 2006 ) ); see Figure 3 . LLMs play a spectrum of roles in this architecture, from guessing candidate plans, to translating those plans into syntactic forms that are more accessible to external critics, to helping end users flesh out incomplete specifications, to helping expert users acquire domain models (that in turn drive model-based critics). All this leveraging of LLMs is done without ascribing to them any planning or verification abilities. The LLM ideas are vetted by external critics, thus ensuring that the plans generated in this architecture can have formal correctness guarantees where possible.

## 2 Planning-centered Limitations of LLMs

In this section, we will first review literature that calls into question claims about the planning and self-verification capabilities of LLMs. Subsequently, we will also provide some possible reasons for claims to the contrary made in the literature.

### 2.1 LLMs cannot generate executable plans in autonomous mode

Despite initial claims about the planning capabilities of LLMs ( Bairi et al., 2023 ; Yao et al., 2023b ; Shinn et al., 2023 ; Huang et al., 2022 ; Hao et al., 2023 ) several recent studies confirm that LLMs are not actually able to generate executable plans when they are used in autonomous modes ( Valmeekam et al., 2023c ; Liu et al., 2023 ; Silver et al., 2022 ) . For example, in ( Valmeekam et al., 2023c ; Valmeekam et al., 2023b ) , we evaluate LLMs’ ability to generate correct plans on a suite of planning problem instances based on the kinds of domains employed in the International Planning Competition ( IPC, 1998 ) . To eliminate the subjective aspect of analysis that forms the core part of many earlier efforts to evaluate the reasoning capabilities of LLMs, we leverage models and tools from the automated planning community to automate evaluation.

We show that results in the autonomous mode are pretty bleak. On average, only about 12% of the plans that the best LLM (GPT-4) generates are actually executable without errors and goal-reaching. We show that the choice of LLM doesn’t have much bearing on this. We tested the family of GPT LLMs including GPT-4 ( OpenAI, 2023 ) , GPT-3.5 ( OpenAI, 2022 ) , InstructGPT-3 ( Ouyang et al., 2022 ) and GPT-3 ( Brown et al., 2020 ) . We also show that fine-tuning does not seem to have a major effect on this dismal performance. We demonstrate that the performance deteriorates further if the names of the actions and objects in the domain are obfuscated–a change that doesn’t in any way affect the performance of the standard AI planners. This further suggests that LLMs are more likely doing approximate retrieval of plans than actual planning.

We continue to reconfirm these limitations over each of the more recently released LLMs, including Claude Opus, Gemini, GPT4-Turbo and GPT4-o. Table 1 shows that all the state of the art LLMs show dismal performance on PlanBench ( Valmeekam et al., 2023b ) .

More recently, we have also investigated so-called “chain of thought” prompting ( Stechly et al., 2024b ) , as well as ReAct-style step-by-step prompting ( Verma et al., 2024a ) and found that they too are largely ineffective in improving the planning performance of LLMs.

### 2.2 LLMs cannot verify plans and thus cannot improve by self-critiquing

There still exists considerable optimism that even if LLMs can’t generate correct solutions in one go, their accuracy might improve in an iterative prompting regime, where LLMs will be able to “self-critique” their candidate solutions and refine them to the point of correctness ( Yao et al., 2023b ; Yao et al., 2023a ; Shinn et al., 2023 ; Weng et al., 2023 ; Huang et al., 2022 ) . This belief seems to rest largely on the assumption that verification of correctness should be easier than generation for many reasoning problems–a rather classical argument from computational complexity. There are grounds to be skeptical of this assumption as the complexity of the reasoning task should be irrelevant to LLM performance if what they are doing is approximate retrieval. In general, unless LLMs are trained not just on “ correct data ,” but also on “ corrections data ,” there is no a priori reason to believe that their critiques would even be approximately relevant, let alone actually correct.

Two of our studies–one on plan verification ( Valmeekam et al., 2023a ) and the other on CSP verification ( Stechly et al., 2023 ) seem to throw cold water on this optimism. In ( Stechly et al., 2023 ) , we systematically investigate the effectiveness of iterative prompting in the context of Graph Coloring , a canonical NP-complete reasoning problem. Our methodology involves a principled empirical study of the performance of GPT4 on two tasks: solving a large suite of random graph coloring instances and, separately, verifying the correctness of the candidate colorings–both in direct (i.e., return the first solution generated by the LLM) and iterative modes. In iterative modes, we experiment both with an LLM critiquing LLM-produced solutions and an external, guaranteed correct reasoner verifying solutions. In both cases, we analyze whether the content of criticisms actually affects bottom-line performance. A more recent paper further analyzes these results along with performance on the 24 puzzle–a task that has been used by some researchers claiming LLMs have the ability to self verify ( Stechly et al., 2024a ) .

Our results indicate that in direct mode, LLMs are, perhaps not surprisingly, pretty bad at solving graph coloring instances. More interestingly, they are no better at verifying solutions. In iterative modes, given the inability of LLMs to verify solutions, it should come as no surprise that our experiments also show that the strategy of LLMs self-critiquing their solutions does not improve over the baseline. We report that the performance is in fact worse because the system can’t recognize a correct coloring and thus merrily passes over fortuitously correct colorings it has generated, ending up with a wrong one! Similar results have also been reported for planning problems in ( Valmeekam et al., 2023c ) .

One important corollary of the fact that LLMs cannot self-critique their plans is that they also can’t self-improve by generating synthetic data, e.g. by generating plans themselves, critiquing the plans by themselves to improve them, and then using those to fine-tune themselves, as has been claimed in the literature ( Huang et al., 2023b ; Wang et al., 2022 ) 3 3 3 Contrary to their claim of “self-improvement”, works like ( Wang et al., 2022 ) actually heavily depend on external knowledge (crafted seed examples) and critics (filtering step). .

### 2.3 Analyzing Claims to the Contrary in the Literature

Given that LLMs can neither guarantee correct generation nor correct verification of plans, as discussed in the previous sections, one obvious question is why the literature is replete with claims contrary to this ( Bairi et al., 2023 ; Yao et al., 2023b ; Shinn et al., 2023 ; Yao et al., 2023a ; Weng et al., 2023 ; Huang et al., 2022 ) .