Group-in-Group Policy Optimization for LLM Agent Training — part 3/4

Cleaned-text code points [29606, 45223). Bibliography omitted here; source.md includes it.
Table 1 demonstrates the strong performance of GiGPO across both ALFWorld and WebShop. As shown, closed-source LLMs offer only moderate performance: Gemini-2.5-Pro reaches 60.3% success on ALFWorld and 35.9% on WebShop, while GPT-4o lags further behind. Open-source prompt-only agents (e.g., ReAct and Reflexion) show marginal improvements over vanilla prompting but still underperform, underscoring the difficulty of long-horizon control without post-training. RL training brings substantial gains: PPO improves average ALFWorld success to 54.4% on the 1.5B model and 80.4% on the 7B model, with WebShop scores also increasing significantly. However, this comes at the expense of increased complexity: requiring a separate critic network, hyperparameter tuning, and longer training durations [ 71 , 49 ] . GRPO and RLOO also yield strong performance while being more computationally efficient, demonstrating the effectiveness of group-based RL in large-scale LLM training. Nevertheless, their lack of fine-grained per-step feedback limits their ability to provide precise credit assignment across long horizons. In contrast, GiGPO overcomes this limitation with a two-level advantage estimation, enabling both GiGPO w/ std and GiGPO w/o std to consistently surpass GRPO and RLOO. In particular, GiGPO w/o std surpasses GRPO by 13.3% on ALFWorld and 10.6% on WebShop at 1.5B, and by 12.6% and 9.1%, respectively, at 7B. These results highlight GiGPO’s superior ability to train LLM agents more effectively and more efficiently. We also find that GiGPO enables agents to exhibit emergent reasoning behavior (see Appendix F ).

Lastly, we observe that the normalization factor F norm F_{\text{norm}} is task-dependent rather than universally helpful. On relatively difficult tasks (such as Look, Pick2, and WebShop), standard-deviation scaling ( F norm = std F_{\text{norm}}=\text{std} ) could exaggerate gradients from overly difficult samples or highly imbalanced groups, harming update stability; fixing F norm = 1 F_{\text{norm}}=1 therefore yields higher success. Yet, F norm = 1 F_{\text{norm}}=1 offers no clear advantage on other tasks and both variants perform similarly, which aligns with findings in [ 72 ] . This suggests that F norm = std F_{\text{norm}}=\text{std} can still be beneficial when reward variance is stable.

### 5.3 Performance on QA tasks

As shown in Table 2 , GiGPO achieves strong and consistent gains on multi-turn search-augmented QA tasks, reaching 42.1% at 3B and 47.2% at 7B, and significantly outperforming prior strong baselines such as Search-R1 and StepSearch. Although search-augmented QA is relatively short-horizon, the step-level signals of GiGPO still yield meaningful improvements. Furthermore, we observe that GiGPO is markedly more tool-efficient. Under a limit of at most 3 tool calls per query, the 7B model requires only ∼ \sim 0.9 calls on average for single-hop tasks and ∼ \sim 1.6 calls on average for multi-hop tasks: matching the superior performance of OTC [ 61 ] , which achieves ∼ \sim 1.0 and ∼ \sim 1.7 calls, respectively, on the same tasks. This efficiency likely stems from GiGPO’s ability to effectively identify and suppress redundant queries in multi-turn decision-making. For instance, in repetitive patterns such as query1 → \rightarrow info1 → \rightarrow query1 → \rightarrow info1 → \rightarrow query1 → \rightarrow info1, GiGPO groups them into the same step-level group and eliminates unnecessary repeated calls during the training, similar to the example we introduced in Figure 3 .

### 5.4 Ablation Study

Next, we conducted an ablation study, comparing GiGPO w/o std ( F norm = 1 F_{\text{norm}}=1 ), GiGPO w/ std ( F norm = std F_{\text{norm}}=\text{std} ), GiGPO w/o A S A^{S} (without step relative advantages), and GiGPO w/o A E A^{E} (without episode relative advantages) to evaluate the impact of each component on performance. We use Qwen2.5-1.5B-Instruct as the policy of the agent. The results are presented in Figure 4 .

As illustrated, eliminating either component of the two-level advantage significantly degrades performance. Removing the episode relative advantages (GiGPO w/o A E A^{E} ) leads to a substantial drop across all tasks, as the policy no longer receives a stable, trajectory-wide signal to encourage long-range coherence. Similarly, discarding the step relative advantages (GiGPO w/o A S A^{S} ) results in pronounced declines, particularly on more complex and demanding tasks such as Cool, Pick2, and WebShop, which may require nuanced training feedback at each decision step. In such cases, precise per-step credit assignment is essential for effective learning and policy refinement. Moreover, we can see that the relative performance gap between GiGPO w/ std and GiGPO w/o std is comparatively minor compared to that observed in structural ablations. This suggests that the combination of episode- and step-level signals is the primary driver of performance gains, and that each component is crucial for training LLM agents effectively.

### 5.5 Dynamics of Step-Level Group

In this part, we examine how the distribution of step-level groups evolves throughout training to better understand the utility of GiGPO. We use Qwen2.5-1.5B-Instruct as the base model. We train the LLM agent in ALFWorld and track changes in step-level group sizes throughout training.

As shown in Figure 5 , we observe that step-level groups of size 1 (i.e., those with | G S  ( 𝒔 ~ ) | = 1 |G^{S}(\tilde{\bm{s}})|=1 ) only account for < 35% throughout training. This indicates that the majority of states (over 65%) recur across trajectories and therefore contribute to anchor state grouping. Moreover, at iteration 10, large group sizes | G S  ( 𝒔 ~ ) | ≥ 10 |G^{S}(\tilde{\bm{s}})|\geq 10 account for over 20%, reflecting behavioral redundancy in the early stages of training. This is consistent with the fact that immature policies often produce invalid actions or fall into repetitive loops. As training progresses, the group size distribution changes markedly. By iteration 75, we observe a substantial reduction in extreme group sizes: 10 ≤ | G S  ( 𝒔 ~ ) | < 50 10\leq|G^{S}(\tilde{\bm{s}})|<50 drops from 16.2% to 12.1% and | G S  ( 𝒔 ~ ) | ≥ 50 |G^{S}(\tilde{\bm{s}})|\geq 50 drops from 5.6% to 3.1%. This shift suggests that the agent is learning to avoid previously common dead ends and invalid actions, and is beginning to exhibit more diverse and purposeful decision-making. At iteration 140, the distribution becomes tightly concentrated around group sizes of 6 to 8. Given that N N is set to 8 8 , this convergence implies that all 8 trajectories within an episode group are now behaving consistently: LLM agent has learned a coherent and robust policy for completing the task, aligning with the plateau in success rate ( > 80 % >80\% ).

### 5.6 Computational Budget

Lastly, we analyze the computational budget of GiGPO. GiGPO shares the same core architecture as GRPO, including multi-turn rollouts, computation of old and reference probabilities, and clipped policy updates. Both approaches are critic-free and operate with a single actor LLM, thereby resulting in identical GPU memory usage and LLM rollout costs. The primary additions introduced by GiGPO are the step-relative advantage estimation components, as described in Section 4.2 . To evaluate their costs, we train an LLM agent in ALFWorld using Qwen2.5-1.5B-Instruct and record a detailed breakdown of per-iteration training time.

As shown in Figure 6 , the additional components incur little to no additional time cost compared to dominant operations such as rollouts, computation of old and reference probabilities, and policy updates, whose total time cost reaches 362.83s per iteration. In contrast, anchor state grouping (involving hashmap lookups) takes only 0.01s per iteration, and the step-relative advantage computation (involving simple arithmetic) adds just 0.53s. These operations account for < 0.002% of the total per-iteration training time, demonstrating that GiGPO shares the same high computational efficiency as GRPO.

## 6 Conclusions and Limitations

In this work, we proposed GiGPO, a novel group-based RL algorithm to tackle the credit assignment challenge in long-horizon LLM agent training. GiGPO introduces a hierarchical advantage estimation that enables fine-grained per-step credit assignment while retaining the efficiency and stability of group-based RL. By retroactively grouping steps that share the same state across trajectories, it achieves this without incurring additional LLM rollout or GPU memory overhead. Empirical evaluations across complex agentic environments (ALFWorld and WebShop) and search-augmented QA tasks demonstrate that GiGPO significantly outperforms both prompt-based agents and prior RL methods. A potential limitation of GiGPO is its reliance on state matching for anchor group construction. In highly complex environments, identical states may be hard to detect due to noise or subtle differences. Despite this, GiGPO still retains a strong performance lower bound: in the extreme case where no states are repeated across trajectories (i.e., A S = 0 A^{S}=0 ), it naturally degrades to GRPO, preserving GRPO’s effectiveness and stability in credit assignment. Although this issue is partly mitigated by incorporating similarity-based grouping, exploring more robust state-matching strategies, such as embedding-based representations or domain-specific structural equivalence, remains an important direction.

## Acknowledgements

This research is supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 1 (RG18/24).

## Appendix A Open Source Codebase: verl-agent

As part of the new assets released with this work, we propose verl-agent ( https://github.com/langfengQ/verl-agent ), a highly scalable RL training framework for long-horizon, multi-turn LLM agent training.

Our verl-agent is built upon the veRL framework [ 73 ] and extends it with several features to enable scalable reinforcement learning for long-horizon LLM agents. Key capabilities of our framework include: ( 1 ) step-wise multi-turn interaction paradigm that avoids concatenating full interaction histories (as in Search-R1 [ 58 ] and RAGEN [ 51 ] ), ensuring efficient memory control and scalability for very long-horizon optimization; ( 2 ) customizable memory module that allows developers to flexibly determine which historical information to include at each step (e.g., key events, summaries, or external knowledge); ( 3 ) parallel and group-based environments with a gym-style interface supporting high-throughput rollouts; ( 4 ) broad model compatibility, including Qwen3 [ 74 ] , Qwen2.5, and LLaMA3.2, along with LoRA-based fine-tuning [ 75 ] for efficient large-model adaptation; ( 5 ) support for multimodal (vision-language) agents such as Qwen2.5-VL; ( 6 ) a diverse suite of environments, including Search (tool use), ALFWorld, WebShop, Sokoban, and Gym Cards; ( 7 ) comprehensive RL algorithm support, encompassing GiGPO, GRPO, PPO, DAPO, RLOO, etc.

## Appendix B Broader Impacts

This work introduces an efficient, group-based RL algorithm, GiGPO, that enables fine-grained credit assignment for long-horizon LLM agents. By combining episode-level and step-level advantages, GiGPO improves training stability and agent performance without increasing computational or memory overhead.

The algorithm holds promise for a wide range of applications, including virtual assistants, web automation, educational tools, and embodied AI systems. Its critic-free and scalable design lowers the barrier to training effective multi-step agents, making advanced agent capabilities more accessible to both academic and industrial practitioners. While GiGPO is a methodological contribution with no direct downstream deployment, the improved agent training techniques may indirectly enable more autonomous behavior. As with any advancement in agent capabilities, it is important to consider appropriate safeguards and responsible usage. Overall, GiGPO contributes a valuable tool to the RL and LLM communities, with the potential to accelerate progress in building more intelligent, efficient, and adaptable AI agents.

## Appendix C Unbiasedness

We show that setting F norm = 1 F_{\text{norm}}=1 leads to an unbiased estimator up to a constant scaling factor. Our derivation follows the approach in [ 19 ] . For comparison, the standard REINFORCE Leave-One-Out (RLOO) [ 16 , 17 ] is defined as: A RLOO  ( τ i ) = R ⁡ ( τ i ) − 1 N − 1  ∑ j ≠ i R ⁡ ( τ j ) . A^{\mathrm{RLOO}}(\tau_{i})=R(\tau_{i})-\frac{1}{N-1}\sum_{j\neq i}R(\tau_{j}). (10) We can relate A E  ( τ i ) A^{E}(\tau_{i}) ( F norm = 1 F_{\text{norm}}=1 ) and A RLOO  ( τ i ) A^{\mathrm{RLOO}}(\tau_{i}) by introducing a scaling factor of N N − 1 \frac{N}{N-1} : N N − 1  A E  ( τ i ) = \displaystyle\frac{N}{N-1}A^{E}(\tau_{i})= N N − 1 R ( τ i ) − N N − 1 ⋅ 1 N ∑ j = 1 N R ( τ j ) \displaystyle\frac{N}{N-1}R(\tau_{i})-\frac{N}{N-1}\cdot\frac{1}{N}\sum_{j=1}^{N}R(\tau_{j}) (11) = \displaystyle= N N − 1  R  ( τ i ) − 1 N − 1  R  ( τ i ) − 1 N − 1  ∑ j ≠ i R ⁡ ( τ j ) \displaystyle\frac{N}{N-1}R(\tau_{i})-\frac{1}{N-1}R(\tau_{i})-\frac{1}{N-1}\sum_{j\neq i}R(\tau_{j}) (12) = \displaystyle= R ⁡ ( τ i ) − 1 N − 1  ∑ j ≠ i R ⁡ ( τ j ) \displaystyle R(\tau_{i})-\frac{1}{N-1}\sum_{j\neq i}R(\tau_{j}) (13) = \displaystyle= A RLOO  ( τ i ) \displaystyle A^{\mathrm{RLOO}}(\tau_{i}) (14) Thus, setting F norm = 1 F_{\text{norm}}=1 corresponds to a rescaled version of A RLOO  ( τ i ) A^{\mathrm{RLOO}}(\tau_{i}) and scaling the advantage by a constant does not affect the dynamics of policy gradient (it can be absorbed into the learning rate).

## Appendix D Pseudo Code

Algorithm 1 summarizes the full GiGPO training procedure. Compared to vanilla GRPO, we highlight the additional parts introduced by GiGPO in italics. In particular, building step-level groups G S  ( 𝒔 ~ ) G^{S}(\tilde{\bm{s}}) is implemented by treating anchor states as keys and aggregating corresponding data into a hash table, which incurs minimal overhead. Furthermore, computing step relative advantages and combining advantages involve only simple arithmetic operations, both of which are lightweight. As such, GiGPO preserves the critic-free, low-memory, and stable convergence properties of group-based RL, while introducing fine-grained credit assignment that is particularly beneficial for training long-horizon LLM agents.

## Appendix E Experiment Details

### E.1 Details of Training

Hyperparameters for ALFWorld. All methods are configured with identical hyperparameters: the maximum prompt length is 2048 tokens, and the maximum response length is 512 tokens. Each episode allows up to 50 environment steps. The learning rate is set to 1e-6 for the actor and 1e-5 for the critic (used only in PPO). We adopt a rule-based reward, assigning a reward of 10 for success and 0 for failure. To handle invalid actions generated by the agent, we apply a reward penalty of -0.1. For all group-based RL methods, we use a group size of 8 and sample 16 different groups per rollout, resulting in a total of 16 × 8 = 128 16\times 8=128 environments. In contrast, PPO uses 128 separate environments for rollouts. The rollout temperature is set to 1.0, while the validation temperature is set to 0.4. The mini-batch size is 256, and the KL-divergence loss coefficient is set to 0.01. For GiGPO, the weighting coefficient ω \omega is fixed at 1 without further tuning, and the discount factor γ \gamma is set to 0.95.