When Does Continual Learning Require Learning — part 1/4
Cleaned-text code points [0, 15624). Bibliography omitted here; source.md includes it.
# When Does Continual Learning Require Learning
###### Abstract
As large language models (LLMs) become increasingly capable, the next question is how can we enable models to continually learn? Today, the field largely frames this as a problem of context management and mitigating forgetting. We argue this framing is incomplete: continual learning is fundamentally about increasing model competence as the world changes. We disentangle this change along two axes — space, where the model encounters new domains, and time, where the underlying data drifts under a fixed task. This framing lets us study continual learning under realistic conditions: new domains arrive over time, facts drift past their training cutoff, and agentic interactions accumulate state across episodes. To evaluate methods under this setting, we recast widely used LLM benchmarks as sequential problems and introduce a single mechanism-agnostic protocol that compares prompt-based methods (GEPA, ACE), supervised learning (SFT, SDFT), reinforcement learning (GRPO, SDPO), and context compression (Cartridges, In-place TTT). Prompt-based methods fit each new stage quickly but degrade on future tasks. Distillation-based methods accumulate knowledge stably but struggle to update outdated facts. Context compression improves efficiency without substantially improving the ability to learn new tasks. Online reinforcement learning adapts most effectively to knowledge updates but remains sensitive to noisy reward signals. Overall, our results suggest that continual learning is not a single capability: different patterns of environmental change require fundamentally different update behaviors, determining when adaptation must be learned inside model weights and when it can be achieved through external scaffolding. We hope that understanding where each method succeeds and fails will guide the design of stronger continual learning systems. †
## 1 Introduction
Traditionally, continual learning has been defined as mitigating catastrophic forgetting [ 40 ] : how do we learn task B B without forgetting task A A ? One approach to this problem is to augment or compose model parameters as proposed in superposition [ 6 ] , adversarial networks [ 9 ] and side networks [ 57 ] . However, this setup is not realistic in the current paradigm of large language models (LLMs), which often rely on large scale pre-, mid-, and post-training to acquire general capability and after deployment need to be adapted to a broad range of potential use cases. At the same time, standard LLM evaluations focus on measuring capabilities on static tasks such as general knowledge [ 51 ] , domain specific abilities [ 17 , 23 , 45 , 16 ] , and challenging frontier problems [ 14 , 38 ] ; these however fail to measure the ability to become more competent under conditions that evolve through time – which is critical for practical deployment.
Without a unified framework, the community approached continual learning as a memory problem solved with retrieval-augmented generation [ 27 ] , in-context learning [ 3 ] , smart context management [ 29 , 56 ] and harnesses [ 59 , 54 , 26 ] ; as a steering problem solved with prompt optimization [ 43 , 47 , 2 ] ; as a reasoning problem solved with reinforcement learning and distillation [ 35 , 60 , 19 , 48 ] ; and as a compression problem solved with architectural changes [ 10 , 30 , 49 , 11 ] . Each of these takes a different bet on what continual learning actually requires, but the field still lacks unified insights across these methods to direct progress.
In our work, we provide a unified framework to define and measure continual learning in modern LLMs. We define continual learning as the problem of increasing competence as the world changes. We consider change along two axes: space and time , as depicted in Figure 1 . Space corresponds to the classical notion of domain or task shift, where the model is asked to acquire a new skill on a different distribution without losing the previous one. Time corresponds to a different and, more common form of change in real deployed systems: the task stays the same, but the underlying world drifts. Time itself can be further split into two sub-regimes that the literature usually conflates. Slow trends (e.g. yearly financial filings) ask the model to stay aligned; the best strategy here is to keep extracting whatever structure survives across years and ignore year-specific noise. Discrete fact changes (e.g. Wikipedia revisions) ask the opposite – the right move is to actually rewrite the affected belief while leaving the rest of the model’s knowledge untouched. A third sub-regime is agentic accumulation: the task family itself stays fixed (e.g., actions inside a web app), but the environment’s state drifts as a direct consequence of the model’s own prior actions, so that what one step leaves behind becomes part of the task the model faces at the next step. This second sub-regime is the LLM analogue of the frame problem [ 39 ] : when the world updates, a learner must decide which of its beliefs are now outdated and which can remain fixed.
To measure continual learning under our definition, we cast widely used LLM evaluations as problems that unfold sequentially under a single mechanism-agnostic protocol, so prompt, weight, and architectural updates can be compared on equal ground. On a common backbone (Qwen3-8B) [ 53 ] , we evaluate eight methods across four families: prompt optimization (GEPA [ 2 ] , ACE [ 59 ] ), offline supervised updates (SFT, SDFT [ 48 ] ), online reinforcement learning (GRPO [ 46 ] , SDPO [ 19 ] ), and context-compression (Cartridges [ 10 ] , In-place TTT [ 11 ] ).
Interestingly, we observe a consistent set of tradeoffs across methods: (i) Prompting-based approaches such as GEPA and ACE fit quickly and heavily to past trajectories, achieving strong backward accuracy, but encounter sharp degradation on future tasks. (ii) Distillation-based methods such as SDFT and SDPO accumulate knowledge more stably over time, yet struggle to rapidly incorporate new information or update outdated knowledge. (iii) Context compression approaches, including Cartridges and In-place TTT, improve efficiency and memory management but do not substantially improve the ability to learn new tasks. (iv) In contrast, online reinforcement learning methods such as GRPO adapt more effectively to knowledge updating, though they remain highly sensitive to noisy or unstable reward signals. (v) On the agentic axis, where the stage ordering is generated by the agent’s own actions rather than imposed by us, both a prompt-based playbook (ACE) and weight-based fine-tuning (SFT) compound experience and beat their zero-shot baselines at every chain length, though absolute success still decays as the chain grows.
All together, we make the following contributions: • We formalize continual learning in LLMs as the problem of increasing competence over time. Through our protocol, we provide a unified perspective that allows us to consider both parametric and non-parametric update methods for continual learning. We adapt these methods to the continually learning setting and align their evaluation metrics.
• We introduce a broad suite of sequential LLM tasks to evaluate continual learning. These including domain adaptation, agentic tasks, financial analysis, and temporally-dependent knowledge updates.
• We find that prompt-based methods alone are insufficient across most regimes, calling for actual learning. Different learning methods each have their own strengths depending on how the task and data shift over time.
## 2 Related Works
Continual Learning. Traditionally, the field has considered continual learning as preventing the problem of catastrophic forgetting [ 40 , 13 ] . In this context, the problem is that sequential parameter updates can overwrite knowledge needed for previous tasks; this led to a focus on designing methods that reduce task interference [ 22 , 15 , 9 , 57 , 6 ] . From another angle, continual learning has been positioned as maintaining plasticity [ 8 ] , where models must not only remember old tasks, but also have the ability to learn new ones [ 36 , 42 ] .
Non-parametric updates. At deployment, we can adapt LLMs without weight changes by changing the inference-time environment around a fixed model. One family of methods changes the information available to the model, for example through retrieval-augmented generation (RAG) [ 27 ] , external memory, reusable long-context representations [ 10 ] , context compression [ 29 ] , or recursive access to large external contexts [ 56 ] . A closely related family changes the behavioral scaffold through which that information is used, including demonstrations [ 3 ] , prompt instructions and agent contexts [ 2 , 59 , 55 ] and multi-stage LM programs [ 44 , 25 , 28 ] . More broadly, agentic systems can adapt by modifying their surrounding harnesses [ 54 , 26 ] , learn component-level reward specifications [ 52 ] or improve through external programs and agent code modifications [ 4 , 43 , 58 , 47 ] .
Gradient-based (parametric) updates. Another adaption approach is to update weights or activations. At one extreme, full fine-tuning modifies all parameters of the model; while effective, this regime is often computationally expensive and unrealistic for frequent or online adaptation. Activation editing [ 20 ] uses task vectors applied to all parameters. More commonly, work studies parameter-efficient methods, such as LoRA [ 18 ] and adapter-based tuning [ 30 , 33 ] , which restrict updates to small subsets of parameters. An important but underexplored regime involves severely constrained updates—very few gradient steps and tightly limited compute—closer to the conditions under which continual adaptation would need to occur in deployed systems. Continual learning methods differ not only in where they store updates, but also in what learning signal drives the update. OPSD [ 60 , 35 ] , SDFT [ 48 ] and SDPO [ 19 ] learn from self-distillation, GRPO [ 46 ] and h1 [ 41 ] optimize end-of-task rewards, LoRD [ 32 ] uses teacher–student distillation, sparse memory finetuning [ 30 ] localizes the next-token objective to sparsely accessed memory slots, test-time training [ 11 ] performs input-conditioned weight updates during inference.
## 3 Method
### 3.1 A Protocol for Continual Learning
Setup. Typically, continual-learning evaluations are tightly coupled to a particular method: weight-update methods use task accuracy with a fixed fine-tuning budget; prompt-optimization often uses few-shot generalization with a fixed token budget; agentic frameworks rely on end-to-end pass rate. As a result, prompt, weight, and architectural updates are rarely compared fairly against one another. To gather broader insights about what methods are necessary for continual learning, we define a simple protocol that is compatible with any update operator. To do so, we consider all forms of updates (prompting, full weight, sparse weight/activation) as potential strategies, and group methods into families based on which aspect of the model they change during optimization (Tab. 1 ). Formally:
A learner sees stages 𝒯 1 , … , 𝒯 K \mathcal{T}_{1},\dots,\mathcal{T}_{K} in fixed order. Each 𝒯 k \mathcal{T}_{k} specifies a training distribution 𝒟 k tr \mathcal{D}_{k}^{\text{tr}} and an evaluation set 𝒟 k eval \mathcal{D}_{k}^{\text{eval}} . Between stages the learner applies an update operator θ k = 𝒰 k ( θ k − 1 , 𝒟 k tr ) , θ 0 = θ base . \theta_{k}\;=\;\mathcal{U}_{k}\!\left(\theta_{k-1},\,\mathcal{D}_{k}^{\text{tr}}\right),\qquad\theta_{0}=\theta_{\text{base}}. 𝒰 k \mathcal{U}_{k} is unrestricted: it may modify model weights, edit a system prompt, write to or read from external memory, attach an adapter, or perform no update. A method is fully specified by its family of 𝒰 k \mathcal{U}_{k} and a per-stage compute budget C C (held constant across methods within a benchmark).
Forgetting matrix. After each stage we evaluate on every stage’s eval set, producing R i , j = acc ( θ i , 𝒟 j eval ) , i ∈ { 0 , … , K } , j ∈ { 1 , … , K } , R_{i,j}\;=\;\mathrm{acc}\!\left(\theta_{i},\,\mathcal{D}_{j}^{\text{eval}}\right),\qquad i\in\{0,\dots,K\},\;j\in\{1,\dots,K\}, with row 0 0 the base model before any update. We report two transfer quantities: BWT = 1 K − 1 ∑ k = 1 K − 1 ( R K , k − R k , k ) , \displaystyle=\tfrac{1}{K-1}\textstyle\sum_{k=1}^{K-1}\big(R_{K,k}-R_{k,k}\big), FWT = 1 K − 1 ∑ k = 2 K ( R k − 1 , k − R 0 , k ) . \displaystyle=\tfrac{1}{K-1}\textstyle\sum_{k=2}^{K}\big(R_{k-1,k}-R_{0,k}\big). BWT [ 33 ] measures whether later stages improve or damage earlier ones (negative BWT is catastrophic forgetting). FWT measures whether prior stages prepare the model for stages it has not yet trained on, and is the sharper signal: it tests acquisition of structure that transfers, rather than fitting the current stage at the cost of the rest.
### 3.2 Update Methods
We evaluate eight methods across four families: prompt optimization, supervised weight updates, reinforcement learning, and per-stage architectural updates. All use Qwen3-8B (non-thinking) [ 53 ] as θ 0 \theta_{0} and run under the protocol of § 3 . Table 1 states what each method carries across the stage boundary; the per-family paragraphs that follow specify each 𝒰 k \mathcal{U}_{k} and its sequential adaptation.
Prompt optimization (GEPA, ACE) Both methods leave model weights unchanged. GEPA [ 2 ] optimizes the system prompt through a reflect-and-mutate loop scored on a validation minibatch; the best-scoring mutations propagate forward. ACE [ 59 ] maintains a markdown playbook edited incrementally through explicit add/keep/drop operations, which the authors show resists brevity-bias and context collapse. In our protocol, 𝒰 k \mathcal{U}_{k} initializes optimization for stage k k from the optimized state at the end of stage k − 1 k{-}1 (the evolved prompt for GEPA, the playbook for ACE), prepended to the default instruction for 𝒯 k \mathcal{T}_{k} . No replay of prior data is performed. Apparent forward transfer is therefore attributable to the carried prompt or playbook, not to revisiting training examples.
Supervised / Offline Learning (SFT, SDFT) SFT fine-tunes all parameters on 𝒟 k tr \mathcal{D}_{k}^{\text{tr}} , resuming from θ k − 1 \theta_{k-1} . SDFT [ 48 ] is self-distillation under a forward-KL loss against soft targets from a teacher; in the original work the teacher is the model itself on demonstration-conditioned inputs. Our sequential adaptation: at stage k k , the teacher is θ k − 1 \theta_{k-1} , so the student learns the new stage under a regularizer pulling it toward its own previous-stage distribution. This is the smallest modification that lifts SDFT to a continual-learning operator and is the only modification we make. SFT and SDFT share a per-stage update budget within C C and differ only in the loss applied at each step.
Reinforcement / Online Learning (GRPO, SDPO) GRPO [ 46 ] samples rollouts in groups, computes within-group advantages from a verifiable reward, and updates the policy under a KL constraint. For benchmarks without a verifiable reward (10-K, TempWiki) we use a binary correctness signal from the ground-truth label. SDPO [ 19 ] is sequential DPO. Our adaptation: at stage k k , θ k − 1 \theta_{k-1} generates both the chosen and rejected continuation pools, so the preference signal is implicitly relative to the model’s accumulated prior rather than to a fresh reference. Both methods consume C C as policy-update steps.