Group-in-Group Policy Optimization for LLM Agent Training — part 4/4

Cleaned-text code points [45223, 52163). Bibliography omitted here; source.md includes it.

Hyperparameters for WebShop. All methods are configured with identical hyperparameters: the maximum prompt length is 4096 tokens, and the maximum response length is 512 tokens. Each episode is limited to 15 environment steps. The learning rate is 1e-6 for the actor and 1e-5 for the critic (used only in PPO). We adopt a rule-based reward, assigning a reward of 10 for success and 0 for failure. Invalid actions are penalized with a reward of -0.1. As with ALFWorld, all group-based RL methods use a group size of 8 and sample 16 groups per rollout, totaling 16 × 8 = 128 16\times 8=128 environments. PPO, on the other hand, uses 128 distinct environments for rollouts. The rollout temperature is set to 1.0, while the validation temperature is set to 0.4. The mini-batch size is 64, and the KL-divergence loss coefficient is set to 0.01. For GiGPO, the weighting coefficient ω \omega is set to 1 without additional tuning, and the discount factor γ \gamma is set to 0.95.

Hyperparameters for Search-Augmented QA. The maximum prompt length is 4096 tokens, and the maximum response length is 512 tokens. The max turn is set to 4. The learning rate is 1e-6 for the actor. We adopt a rule-based reward, assigning a reward of 1 for success and 0 for failure. Invalid actions are penalized with a reward of -0.01. We set the train data size to 256 and use a group size of 5. Rollout and validation temperatures are set to 1.0 and 0.0, respectively. The mini-batch size is 512, and the KL-divergence loss coefficient is set to 0.001. The weighting coefficient ω \omega is set to 1 without additional tuning, and the discount factor γ \gamma is set to 0.95.

Computing Details. For ALFWorld and WebShop, Qwen2.5-1.5B experiments are run on 2×H100 GPUs and Qwen2.5-7B on 4×H100 GPUs, each for 150 iterations. For search-augmented QA, Qwen2.5-3B uses 4×H100 GPUs and Qwen2.5-7B uses 8×H100 GPUs, each for 200 iterations.

### E.2 Prompts

The prompts we use for LLM agents are presented in Figure 8 , Figure 9 , and Figure 10 . These prompt templates are constructed using Python-style string formatting, where placeholders enclosed in curly braces ( {} ) represent semantic slots. These placeholders, such as {task_description} , {step_count} , and {current_observation} , are dynamically populated at runtime via Python’s .format() function. To enrich the agent’s context, we use historical information and set the history length to 2 for ALFWorld and WebShop and the full history for search-augmented QA experiments.

The <think> </think> block instructs the agent to explicitly perform step-by-step reasoning, thereby promoting chain-of-thought [ 76 ] style deliberation. The <action> </action> block is used to clearly indicate the final action decision. The search agent outputs reasoning traces within <think> </think> , issues search queries within <search> </search> , provides anwsers within <anwser> </anwser> . Retrieved evidence from the retriever is presented in <information> </information> tags.

### E.3 Performance on Vision-Language Agents

We conduct additional experiments in vision-language model (VLM) settings where agents must reason over both visual and textual inputs. We use two interactive game environments: Sokoban [ 77 ] with a 6 × \times 6 size and EZPoints in Gym Cards [ 39 ] as shown in Figure 11 . Sokoban is a classic grid-based puzzle game where the agent must push boxes onto designated goal positions. Solving Sokoban requires spatial reasoning and long-term planning. In EZPoints, the agent is presented with two playing cards and must construct an arithmetic formula step-by-step such that the final result equals a target value of 12. This setting emphasizes symbolic reasoning and multi-step planning. In these tasks, the agent receives an RGB image representing the current environment state along with a textual instruction, and must take sequential actions to complete the task. All methods are built upon Qwen2.5-VL-3B-Instruct [ 78 ] .

As shown in Table 3 , GiGPO significantly outperforms the prompting baseline and GRPO on both tasks. Notably, GiGPO achieves 81.0% success on Sokoban and 100% success on EZPoints, suggesting its ability to generalize beyond language-only settings.

### E.4 Orthogonality to Single-Turn Group-Based RL

GiGPO remains orthogonal to other advancements in single-turn group-based RL, allowing it to incorporate complementary techniques without loss of generality. To verify this, we consider DAPO [ 20 ] , which builds on GRPO with dynamic sampling and clip-higher techniques. We integrate both into GiGPO, yielding a variant denoted as GiGPO dynamic .

As shown in Table 4 , DAPO improves over GRPO, confirming the benefits of its techniques. Importantly, GiGPO dynamic further outperforms DAPO, demonstrating that it can effectively benefit from and amplify such improvements, achieving better results (75.0%) on WebShop.

### E.5 Sensitivity Analysis on ω \omega

We further analyze GiGPO’s sensitivity to the key hyperparameter ω \omega , which balances the episode-level advantage A E A^{E} and the step-level advantage A S A^{S} . We conducted experiments on the WebShop task, using Qwen2.5-1.5B-Instruct.

As shown in Table 5 , GiGPO needs an appropriate ω \omega to work best. Increasing ω \omega initially improves performance due to the added fine-grained step-level reward. However, performance declines beyond the optimum ( ω = 0.8 \omega=0.8 ), suggesting that excessive emphasis on step-level signals can suppress useful trajectory-level guidance. Furthermore, GiGPO is relatively insensitive to ω \omega within the range [ 0.4 , 1.2 ] [0.4,1.2] , demonstrating a reasonable degree of robustness to ω \omega .

## Appendix F Reasoning Behavior

We next present the complete trajectory of multi-turn interaction between agent and environment, and demonstrate the reasoning behavior of LLM agents (Qwen2.5-7B-Instruct) trained by GiGPO.

### F.1 ALFWorld

### F.2 WebShop

### F.3 Search

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .