Group-in-Group Policy Optimization for LLM Agent Training — part 1/4

Cleaned-text code points [0, 15115). Bibliography omitted here; source.md includes it.

# Group-in-Group Policy Optimization for LLM Agent Training

###### Abstract

Recent advances in group-based reinforcement learning (RL) have driven frontier large language models (LLMs) in single-turn tasks like mathematical reasoning. However, their scalability to multi-turn LLM agent training remains limited. Unlike static tasks, agent-environment interactions unfold over many steps and often yield sparse or delayed rewards, making credit assignment across individual steps significantly more challenging. In this work, we propose Group-in-Group Policy Optimization (GiGPO) , a novel RL algorithm that achieves fine-grained credit assignment for LLM agents while preserving the appealing properties of group-based RL: critic-free, low memory, and stable convergence. GiGPO introduces a two-level structure for estimating relative advantage: (i) At the episode-level , GiGPO computes macro relative advantages based on groups of complete trajectories; (ii) At the step-level , GiGPO introduces an anchor state grouping mechanism that retroactively constructs step-level groups by identifying repeated environment states across trajectories. Actions stemming from the same state are grouped together, enabling micro relative advantage estimation. This hierarchical structure effectively captures both global trajectory quality and local step effectiveness without relying on auxiliary models or additional rollouts. We evaluate GiGPO on challenging agent benchmarks, including ALFWorld and WebShop, as well as tool-integrated reasoning on search-augmented QA tasks, using Qwen2.5-1.5B/3B/7B-Instruct. Crucially, GiGPO delivers fine-grained per-step credit signals, achieves performance gains of > 12% on ALFWorld and > 9% on WebShop over GRPO, and obtains superior performance on QA tasks (42.1% on 3B and 47.2% on 7B): all while maintaining the same GPU memory overhead, identical LLM rollout, and incurring little to no additional time cost. † † footnotetext: Code: https://github.com/langfengQ/verl-agent

## 1 Introduction

Large Language Models (LLMs) [ 1 , 2 , 3 , 4 ] have leapt from static question-answer systems to versatile agents that perceive, reason, and act in open-ended environments. For instance, they now power embodied assistants that navigate simulated homes [ 5 , 6 ] , mobile and web navigators that plan multi-step browsing sessions [ 7 , 8 , 9 , 10 ] , and autonomous explorers in interactive games [ 11 , 12 ] . In these settings, LLM agents need to perceive, reason, and act in multi-turn loops, which requires not only language understanding but also long-horizon planning and decision-making.

Reinforcement learning (RL) [ 13 ] has become a crucial recipe for post-training LLMs, leading to frontier models like OpenAI o1 [ 14 ] and DeepSeek R1 [ 15 ] . In particular, group-based RL algorithms such as RLOO [ 16 , 17 ] and GRPO [ 18 ] have proven especially effective in large-scale training. These methods replace value-function estimation with simple yet powerful relative advantage estimation within groups of rollouts. This group-based advantage computation enjoys favorable properties such as low memory overhead, critic-free optimization, and scalability to large models. However, their successes have so far been largely limited to single-turn tasks such as math problem solving [ 19 , 20 ] and code generation [ 21 ] , where reward arrives immediately and credit assignment is straightforward.

In contrast, LLM agents operating in external environments face fundamentally different learning landscapes. Their behavior unfolds over long episodes with tens of decision steps and tens of thousands of tokens (e.g., an ALFWorld [ 5 ] episode may include up to 50 steps and over 20k tokens). Rewards are typically sparse (sometimes arriving only at the end of an episode), and the impact of any individual action may only manifest much later in the trajectory. This substantially complicates the credit assignment for individual steps and increases the challenge of policy optimization. Naively applying existing group-based RL algorithms in such settings collapses step-level distinctions, undermining their effectiveness. Hence, these limitations raise a core question:

To address this, we introduce Group-in-Group Policy Optimization (GiGPO), a new group-based RL algorithm that nests two-dimensional notions of credit assignment, better-suited for the multi-turn optimization of LLM agents. (i) At the episode level , GiGPO samples a group of complete trajectories under identical task and initial-state conditions, and computes macro relative advantages based on total returns like vanilla GRPO [ 18 ] . This captures the overall effectiveness of each trajectory and reflects the completeness of task execution. (ii) At the step level , GiGPO introduces a novel anchor state grouping mechanism for fine-grained relative advantages estimation. Specifically, it retroactively identifies the repeated environment states, termed as anchor states , across trajectories and uses them as anchors to construct step-level groups that allow for localized credit assignment.

The key insight behind GiGPO is that, under identical tasks and initial environment conditions, many trajectories within the group encounter the same states multiple times due to ineffective actions or loops, such as revisiting the same webpage, room, or game scene. These shared states provide a natural basis for step-level group construction and computing more granular advantage estimates. GiGPO uses these step-level groups to assign localized credit to actions based on their relative performance at a common state, enabling more precise optimization while avoiding the cost explosion of per-step extra rollouts. As such, GiGPO remains fully critic-free and requires no auxiliary value models while dramatically introducing finer credit signals for training LLM agents.

We first evaluate GiGPO on long-horizon agent benchmarks: ALFWorld [ 5 ] , which tests embodied task planning in simulated household environments, and WebShop [ 22 ] , which simulates complex, goal-driven web interactions. In addition, we study multi-turn tool-integrated reasoning on search-augmented QA tasks. Our experiments with Qwen2.5-1.5B/3B/7B-Instruct [ 3 ] show that GiGPO consistently outperforms prompt-based agents, actor-critic baselines, and prior group-based RL methods. In particular, GiGPO injects fine-grained, step-level credit signals that sharpen policy learning of agents over horizons and achieves performance gains of > 12% on ALFWorld and > 9% on WebShop over GRPO, along with remarkable performance on search-based QA tasks (42.1% on 3B and 47.2% on 7B). These gains come without compromising the core strengths of group-based RL (only < 0.002% time cost), making GiGPO a versatile and high-utility algorithm for LLM agents.

## 2 Related Work

LLMs as decision-making agents. The use of large language models (LLMs) as autonomous agents has expanded rapidly across domains such as program generation [ 23 ] , smart device operation [ 24 , 25 , 26 , 27 ] , interactive gameplay [ 11 ] , and robot behavior control [ 28 ] . Early works typically relied on leveraging pre-trained, frozen models through carefully designed prompting methods (like ReAct [ 29 ] and Reflexion [ 30 ] ), enhanced memory and retrieval systems [ 12 , 31 ] , and integration with external tools [ 32 , 33 , 34 ] . More recent research has shifted toward adapting model parameters with supervised fine-tuning (SFT) [ 24 ] or RL [ 13 ] , enabling agents to learn directly from environment interaction rather than static prompts or handcrafted workflows, which we introduce below.

Reinforcement learning for LLM agents. RL has played a pivotal role in enabling LLM agents to operate in dynamic, open-ended environments. Early work applied classical RL algorithms such as DQN [ 35 ] to train LLM agents in text-based games [ 36 ] and later research [ 37 , 38 , 39 , 40 , 41 ] started to employ value-based methods, such PPO [ 42 ] and AWR [ 43 ] , in more diverse and interactive agent scenarios including Android device control [ 44 ] , embodied ALFWorld [ 5 ] , and card games [ 45 ] . More recent approaches have extended RL training to complex web-based and application-centered tasks. For instance, ArCHer [ 46 ] and AgentQ [ 47 ] target the WebShop benchmark [ 22 ] , but require intricate designs and computation overhead such as additional value networks or Monte Carlo Tree Search (MCTS) [ 48 ] . CoSo [ 10 ] introduces an entropy-based RL method that enhances the performance of agents. Going further, LOOP [ 49 ] introduces a hybrid method combining REINFORCE leave-one-out (RLOO) [ 16 , 17 ] with PPO-style updates, achieving state-of-the-art results in AppWorld [ 50 ] . RAGEN [ 51 ] introduces a trajectory-level GRPO that concatenates all states, intermediate reasoning, and actions into a unified episode-level response. However, it faces scalability challenges in long-horizon tasks (e.g., in ALFWorld, which involves up to 50 steps).

Reinforcement learning for large language models. An early and influential application of RL in LLMs is the Reinforcement Learning from Human Feedback (RLHF) [ 52 , 53 , 54 , 55 ] , which focuses on aligning LLMs to human preferences. Most recent works have explored using RL to enhance the capabilities of reasoning and logical deduction in LLMs [ 56 , 15 ] . In particular, group-based RL algorithms have emerged as promising alternatives to traditional methods like PPO [ 42 ] . These methods, such as RLOO [ 16 , 17 ] , GRPO [ 18 ] , Dr. GRPO [ 19 ] , DAPO [ 20 ] , and CPPO [ 57 ] , avoid introducing extra value functions by leveraging a group of samples from the same query and estimate the advantages accordingly. This enables the large-scale RL training and has shown strong results in tasks such as mathematical reasoning [ 15 ] , search [ 58 , 59 ] , and tool use [ 60 , 61 ] . Our work is closely related to this line of research, with a focus on training LLM agents . We aim to retain the benefits of group-based RL, such as critic-free learning and efficiency, while introducing finer-grained credit assignment. Moreover, the hierarchical core of GiGPO is orthogonal to existing group-based RL approaches, making it fully compatible and capable of incorporating them to enhance performance.

## 3 Preliminaries

Problem setup. We consider a general setting in which an LLM agent interacts with an environment to complete multi-step tasks based on a task description x ∈ p ⁡ ( X ) x\in p(X) . At each discrete time step t = 1 , 2 , … , T t=1,2,\dots,T , the agent observes a state 𝒔 t ∈ 𝒮 \bm{s}_{t}\in\mathcal{S} and generates a textual action 𝒂 t ∈ 𝒱 n \bm{a}_{t}\in\mathcal{V}^{n} , where 𝒱 \mathcal{V} denotes the token vocabulary and n n is the maximum generation length. The environment then returns a scalar reward r t ∈ ℝ r_{t}\in\mathbb{R} and the next state 𝒔 t + 1 \bm{s}_{t+1} . A full episode consists of a trajectory 𝝉 = { ( 𝒔 1 , 𝒂 1 , r 1 ) , ( 𝒔 2 , 𝒂 2 , r 2 ) , … , ( 𝒔 T , 𝒂 T , r T ) } \bm{\tau}=\{(\bm{s}_{1},\bm{a}_{1},r_{1}),(\bm{s}_{2},\bm{a}_{2},r_{2}),\dots,(\bm{s}_{T},\bm{a}_{T},r_{T})\} . The agent’s behavior is governed by an LLM policy π θ ​ ( 𝒂 t | 𝒔 t , x ) \pi_{\theta}(\bm{a}_{t}|\bm{s}_{t},x) , parameterized by θ \theta , which defines a distribution over outputs conditioned on the current state 𝒔 t \bm{s}_{t} and the task prompt x x . In many realistic scenarios, the environment may provide sparse or delayed rewards (e.g., success and failure signals at the end of an episode) or weak feedback signals for intermediate steps. As the agent generates T T consecutive textual actions ( 𝒂 1 , … , 𝒂 T \bm{a}_{1},\dots,\bm{a}_{T} ), each potentially spanning thousands of tokens, it becomes particularly challenging to assign credit to individual tokens over the course of an episode.

Group-based RL. Recent RL works converge on a simple recipe for training LLMs: for a given task description x x , the LLM samples a group of N N candidate trajectories { 𝝉 1 , 𝝉 2 , … , 𝝉 N } \{\bm{\tau}_{1},\bm{\tau}_{2},\dots,\bm{\tau}_{N}\} , each corresponding to one full episode rollout under π θ old \pi_{\theta_{\text{old}}} . Each trajectory 𝝉 i \bm{\tau}_{i} receives a scalar reward R ⁡ ( 𝝉 i ) R(\bm{\tau}_{i}) reflecting the overall quality or success of the generated outcome. Instead of estimating advantages using separate value functions like PPO [ 42 ] , group-based RL methods compute advantages purely based on the statistics internal to the sampled group: A ⁡ ( 𝝉 i ) = GroupComputation ​ ( { R ⁡ ( 𝝉 i ) } i = 1 N ) . A(\bm{\tau}_{i})=\texttt{GroupComputation}(\{R(\bm{\tau}_{i})\}_{i=1}^{N}). (1) For example, in GRPO [ 18 ] , the advantage of each trajectory is estimated by normalizing its reward with respect to the group’s mean and standard deviation. This design is highly memory-efficient and can scale effectively to the large batch sizes and model sizes typical in modern LLM training, making it a practical and scalable choice for large-scale RL training.

## 4 Training LLM Agents with GiGPO

While group-based RL algorithms [ 18 , 15 ] have proven highly effective for training LLMs in single-turn tasks, their extension to multi-step agent settings faces critical challenges in credit assignment. Figure 1 illustrates this gap. Vanilla GRPO (left) treats each trajectory as a whole and computes a single relative advantage for the entire episode, which fails to provide actionable feedback for individual steps. A natural remedy is to roll out multiple single-step actions for each state 𝒔 t \bm{s}_{t} via π θ old \pi_{\theta_{\text{old}}} as shown in Figure 1 (middle). However, this approach quickly becomes impractical due to the substantial overhead of extra LLM forward passes and the difficulty of evaluating rewards for hypothetical actions never actually executed.

To overcome these challenges, we propose our Group-in-Group Policy Optimization (GiGPO) in this section. Similar to prior works [ 49 , 51 ] , GiGPO begins by sampling groups of trajectories under identical tasks and initial environment states. It then introduces a two-level grouping structure: preserving episode-level grouping for holistic performance comparison, while dynamically constructing an additional set of step-level groups by retroactively aggregating actions encountering the same environment states. This “group-in-group” construction yields two complementary advantages: (1) Episode relative advantages capture the holistic effectiveness of each trajectory, providing a stable, global training signal. (2) Step relative advantages zoom in on which actions outperform their peers within the same state, endowing the gradient with fine-grained credit.

Figure 2 presents an overview of the GiGPO training pipeline. In the remainder of this section, we will detail the computation of episode-level relative advantages, elaborate on the anchor state grouping mechanism, describe the derivation of step-level relative advantages, and finally present the overall GiGPO objective.

### 4.1 Episode Relative Advantages