Group-in-Group Policy Optimization for LLM Agent Training — part 2/4
Cleaned-text code points [15115, 29606). Bibliography omitted here; source.md includes it.
We first introduce the episode-level relative advantages, which represent the coarse-grained component of GiGPO and mirror the naive application of GRPO at the trajectory level. We roll out the agent’s policy π θ old \pi_{\theta_{\text{old}}} in the environment to collect N N complete trajectories under a fixed task x x and identical initial states. Formally, this process yields a group of trajectories { 𝝉 i } i = 1 N \{\bm{\tau}_{i}\}_{i=1}^{N} , where each trajectory is denoted as 𝝉 i = { ( 𝒔 1 ( i ) , 𝒂 1 ( i ) , r 1 ( i ) ) , … , ( 𝒔 T ( i ) , 𝒂 T ( i ) , r T ( i ) ) } \bm{\tau}_{i}=\{(\bm{s}^{(i)}_{1},\bm{a}^{(i)}_{1},r^{(i)}_{1}),\dots,(\bm{s}^{(i)}_{T},\bm{a}^{(i)}_{T},r^{(i)}_{T})\} and the initial states satisfy 𝒔 1 ( 1 ) = 𝒔 1 ( 2 ) = , … , = 𝒔 1 ( N ) \bm{s}^{(1)}_{1}=\bm{s}^{(2)}_{1}=,\dots,=\bm{s}^{(N)}_{1} . For each trajectory, we utilize the total return R ( 𝝉 i ) = ∑ t r t ( i ) R(\bm{\tau}_{i})=\sum_{t}r^{(i)}_{t} as a holistic measure of how effectively the agent completes the task. In settings where only a binary reward is given at the end of the episode, this simplifies to R ( 𝝉 i ) = 1 R(\bm{\tau}_{i})=1 for success and R ( 𝝉 i ) = 0 R(\bm{\tau}_{i})=0 for failure. Then, we organize the trajectories and their corresponding returns into an episode-level group: G E = { ( 𝝉 1 , R ( 𝝉 1 ) ) , ( 𝝉 2 , R ( 𝝉 2 ) ) , … , ( 𝝉 N , R ( 𝝉 N ) ) } . G^{E}=\Bigl\{\bigl(\bm{\tau}_{1},R(\bm{\tau}_{1})\bigr),\bigl(\bm{\tau}_{2},R(\bm{\tau}_{2})\bigr),\dots,\bigl(\bm{\tau}_{N},R(\bm{\tau}_{N})\bigr)\Bigr\}. (2) To evaluate the global relative quality of each trajectory within the group, we compute an episode relative advantage A E ( 𝝉 i ) A^{E}(\bm{\tau}_{i}) for each 𝝉 i \bm{\tau}_{i} by normalizing the total return with the group’s mean and a normalization factor: A E ( 𝝉 i ) = R ( 𝝉 i ) − mean ( { R ( 𝝉 j ) } j = 1 N ) F norm ( { R ( 𝝉 j ) } j = 1 N ) . A^{E}(\bm{\tau}_{i})=\frac{R(\bm{\tau}_{i})-\text{mean}\bigl(\{R(\bm{\tau}_{j})\}_{j=1}^{N}\bigr)}{F_{\text{norm}}\bigl(\{R(\bm{\tau}_{j})\}_{j=1}^{N}\bigr)}. (3) In GRPO [ 18 ] , the default normalization factor is defined as the standard deviation, i.e., F norm = std F_{\text{norm}}=\text{std} . However, this may introduce a difficulty bias [ 19 ] , where trajectories from low-variance groups (e.g., very easy or hard tasks) receive disproportionately large gradients. In the context of the LLM agent, where tasks often involve very long horizons, this effect tends to emerge frequently, potentially affecting the stability of updates. As an alternative, we also consider a fixed normalization factor F norm = 1 F_{\text{norm}}=1 , which yields an unbiased Leave-One-Out estimator [ 16 ] (see Appendix C for details). This simple adjustment helps stabilize training in some challenging agent scenarios.
Overall, the episode relative advantage captures whether the agent successfully completes the assignment across the entire decision horizon T T . Similar to the vanilla GRPO for multi-step optimization shown in Figure 1 (left), it primarily focuses on macro credit assignment, encouraging the policy to develop coherent, trajectory-wide behaviors that maximize overall task performance.
### 4.2 Step Relative Advantages
While the episode relative advantage offers a macro, trajectory-wide signal, it cannot distinguish between the contributions of individual actions within the trajectory. To obtain this fine-grained feedback, we need to form step-level groups: for the same state, we gather the different actions and compare their outcomes, thereby learning which choices are relatively better or worse. A naive way to do so would be to roll out fresh actions from every state (Figure 1 , middle), but that is prohibitively expensive. Instead, we introduce anchor state grouping below, avoiding extra LLM overhead.
Anchor state grouping. As all trajectories { 𝝉 1 , … , 𝝉 N } \{\bm{\tau}_{1},\dots,\bm{\tau}_{N}\} arise from the same task x x and identical initial conditions, many environment states naturally recur across episodes and even across time steps within a single trajectory. We leverage this redundancy by identifying and grouping identical states across trajectories, thereby dynamically constructing step-level groups. Specifically, let 𝒰 = { 𝒔 ~ 1 , 𝒔 ~ 2 , … , 𝒔 ~ U } \mathcal{U}=\{\tilde{\bm{s}}_{1},\tilde{\bm{s}}_{2},\dots,\tilde{\bm{s}}_{U}\} denote the set of all distinct environment states appearing in the trajectory group { 𝝉 1 , … , 𝝉 N } \{\bm{\tau}_{1},\dots,\bm{\tau}_{N}\} . We treat each such unique state 𝒔 ~ ∈ 𝒰 \tilde{\bm{s}}\in\mathcal{U} as an implicit anchor and use it to gather all matching occurrences of that state, and therefore call 𝒔 ~ \tilde{\bm{s}} as “ anchor state ”. Based on this, we can construct | 𝒰 | |\mathcal{U}| step-level groups (one for each unique anchor state 𝒔 ~ \tilde{\bm{s}} ), which is defined as follows: G S ( 𝒔 ~ ) = { ( 𝒂 t ( i ) , r t ( i ) ) | 𝒔 t ( i ) = 𝒔 ~ , 1 ≤ i ≤ N , 1 ≤ t ≤ T } . G^{S}(\tilde{\bm{s}})=\bigl\{\bigl(\bm{a}^{(i)}_{t},r^{(i)}_{t}\bigr)\;\bigm|\;\bm{s}^{(i)}_{t}=\tilde{\bm{s}},\;1\leq i\leq N,\;1\leq t\leq T\bigr\}. (4) Unlike per-state rollout, this procedure incurs no extra rollouts: it is entirely offline and requires only lightweight key-based grouping using hashmaps. Each group G S ( 𝒔 ~ ) G^{S}(\tilde{\bm{s}}) contains multiple instances of the same environment state paired with potentially different actions. Hence, this structure effectively builds the step-level group, forming the basis for subsequent step-level advantage estimation.
Relative advantage computation. Although each tuple ( 𝒂 t ( i ) , r t ( i ) ) \bigl(\bm{a}^{(i)}_{t},r^{(i)}_{t}\bigr) contains an immediate reward r t ( i ) r^{(i)}_{t} , it may be sparse, especially in long-horizon tasks. To better capture long-term impact, we associate a discounted return with each step. Let γ ∈ ( 0 , 1 ] \gamma\in(0,1] be the standard RL discount factor. For each element in G S ( 𝒔 ~ ) G^{S}(\tilde{\bm{s}}) , we compute its discounted return R t ( i ) R_{t}^{(i)} by R t ( i ) = ∑ k = t T γ k − t r k ( i ) . R_{t}^{(i)}=\sum\nolimits_{k=t}^{T}\gamma^{\,k-t}\,r_{k}^{(i)}. (5) This quantity captures the future impact of action 𝒂 t ( i ) \bm{a}^{(i)}_{t} on subsequent rewards, rather than relying solely on the immediate reward r t ( i ) r^{(i)}_{t} . Accordingly, the step-level group for each 𝒔 ~ ∈ 𝒰 \tilde{\bm{s}}\in\mathcal{U} becomes: G S ( 𝒔 ~ ) = { ( 𝒂 t ( i ) , R t ( i ) ) | 𝒔 t ( i ) = 𝒔 ~ , 1 ≤ i ≤ N , 1 ≤ t ≤ T } . G^{S}(\tilde{\bm{s}})=\bigl\{\bigl(\bm{a}^{(i)}_{t},R^{(i)}_{t}\bigr)\;\bigm|\;\bm{s}^{(i)}_{t}=\tilde{\bm{s}},\;1\leq i\leq N,\;1\leq t\leq T\bigr\}. (6) Once these step-level groups are formed, we compute the step relative advantage for each 𝒔 ~ ∼ 𝒰 \tilde{\bm{s}}\sim\mathcal{U} and each action 𝒂 t ( i ) \bm{a}_{t}^{(i)} in G S ( 𝒔 ~ ) G^{S}(\tilde{\bm{s}}) : A S ( 𝒂 t ( i ) ) = R t ( i ) − mean ( { R t ( j ) | ( 𝒂 t ( j ) , R t ( j ) ) ∈ G S ( 𝒔 ~ ) } ) F norm ( { R t ( j ) | ( 𝒂 t ( j ) , R t ( j ) ) ∈ G S ( 𝒔 ~ ) } ) . A^{S}(\bm{a}_{t}^{(i)})=\frac{R_{t}^{(i)}-\text{mean}\left(\bigl\{R_{t}^{(j)}\bigm|(\bm{a}_{t}^{(j)},R_{t}^{(j)})\in G^{S}(\tilde{\bm{s}})\bigr\}\right)}{F_{\text{norm}}\left(\bigl\{R_{t}^{(j)}\bigm|(\bm{a}_{t}^{(j)},R_{t}^{(j)})\in G^{S}(\tilde{\bm{s}})\bigr\}\right)}. (7) A S A^{S} provides micro credit assignment and fine-grained feedback on the relative quality of individual actions taken from the same state. In contrast to the coarse, trajectory-wide signal of A E A^{E} , it offers step-level guidance that is essential for refining decisions in long-horizon agent tasks.
How does step-level group work? We present an intuitive illustration in Figure 3 to show the utility of the step relative advantages. We consider two example trajectories from the set { 𝝉 i } i = 1 N \{\bm{\tau}_{i}\}_{i=1}^{N} . In τ 1 \tau_{1} , the agent first selects the 2nd Item (incorrect), then returns to the previous page and selects the 1st Item (correct), successfully completing the task. Due to temporal discounting (Equation ( 5 )), the earlier action ( 2nd Item ) receives a lower discounted return than the later correct one ( 1st Item ). In τ 2 \tau_{2} , the agent clicks the Next Page , ultimately failing to find the target and receiving no reward. By aggregating these actions into the same step-level group based on their shared anchor state, GiGPO computes their relative advantages and yields a clear preference ordering: A S ( 1st Item ) > A S ( 2nd Item ) > A S ( Next Page ) A^{S}(\textit{1st Item})>A^{S}(\textit{2nd Item})>A^{S}(\textit{Next Page}) . This ranking successfully captures fine-grained distinctions in long-term utility that are missed by prior group-based RL methods [ 17 , 18 , 20 ] . While this example illustrates GiGPO’s effectiveness in sparse-reward environments, its advantages extend naturally to dense-reward scenarios, where per-step rewards can be fully leveraged to assess the relative quality of individual actions within shared states.
### 4.3 Group-in-Group Policy Optimization
We finally combine the two levels of advantage signals into a single group-in-group advantage to assign credit at both global (episode) and local (step) scales: A ( 𝒂 t ( i ) ) = A E ( 𝝉 i ) + ω ⋅ A S ( 𝒂 t ( i ) ) , A(\bm{a}_{t}^{(i)})=A^{E}(\bm{\tau}_{i})+\omega\cdot A^{S}(\bm{a}_{t}^{(i)}), (8) where ω ∈ ℝ ≥ 0 \omega\in\mathbb{R}_{\geq 0} is a weighting coefficient that balances episode relative advantage and step relative advantage. A E ( 𝝉 i ) A^{E}(\bm{\tau}_{i}) captures how good an episode is compared to others in the group, while A S ( 𝒂 t ( i ) ) A^{S}(\bm{a}_{t}^{(i)}) refines step-level performance within shared environment state conditions. Jointly, they provide hierarchical supervision for the policy optimization of LLM agents. Then the clipped policy optimization objective of GiGPO is: 𝒥 GiGPO ( θ ) \displaystyle\mathcal{J}_{\mathrm{GiGPO{}}}(\theta) = 𝔼 x ∼ p ( X ) { 𝝉 i } i = 1 N ∼ π θ old [ 1 N T ∑ i = 1 N ∑ t = 1 T min ( ρ θ ( 𝒂 t ( i ) ) A ( 𝒂 t ( i ) ) , clip ( ρ θ ( 𝒂 t ( i ) ) , 1 ± ϵ ) A ( 𝒂 t ( i ) ) ) ] \displaystyle=\mathbb{E}_{\begin{subarray}{c}x\sim p(X)\\ \{\bm{\tau}_{i}\}_{i=1}^{N}\sim\pi_{\theta_{\text{old}}}\end{subarray}}\biggl[\frac{1}{NT}\sum_{i=1}^{N}\sum_{t=1}^{T}\min\Bigl(\rho_{\theta}(\bm{a}_{t}^{(i)})A(\bm{a}_{t}^{(i)}),\,\text{clip}\bigl(\rho_{\theta}(\bm{a}_{t}^{(i)}),1\pm\epsilon\bigr)A(\bm{a}_{t}^{(i)})\Bigr)\biggr] − β 𝔻 KL ( π θ ( ⋅ ∣ x ) ∥ π ref ( ⋅ ∣ x ) ) . \displaystyle\quad-\beta\mathbb{D}_{\mathrm{KL}}\!\bigl(\pi_{\theta}(\cdot\mid x)\,\|\,\pi_{\mathrm{ref}}(\cdot\mid x)\bigr). (9) where ρ θ ( 𝒂 t ( i ) ) = π θ ( 𝒂 t ( i ) ∣ 𝒔 t ( i ) , x ) π θ old ( 𝒂 t ( i ) ∣ 𝒔 t ( i ) , x ) \rho_{\theta}(\bm{a}_{t}^{(i)})=\frac{\pi_{\theta}(\bm{a}_{t}^{(i)}\mid\bm{s}_{t}^{(i)},x)}{\pi_{\theta_{\text{old}}}(\bm{a}_{t}^{(i)}\mid\bm{s}_{t}^{(i)},x)} is the importance sampling ratio, β \beta controls the strength of the KL penalty encouraging proximity to a reference policy π ref \pi_{\text{ref}} . We present the pseudo code in Appendix D .
## 5 Experiment
In this section, we present empirical evaluations of GiGPO across a variety of agentic tasks. Specifically, we aim to demonstrate: (1) the strong ability of GiGPO in training LLM agents; (2) the ablation study of GiGPO; (3) the dynamic trend of step-level group G S ( 𝒔 ~ ) G^{S}(\tilde{\bm{s}}) over the course of training; (4) the computational budget of GiGPO.
### 5.1 Experiment Setup
Benchmarks. We first train the LLM agents on two challenging benchmarks: ALFWorld [ 5 ] and WebShop [ 22 ] . ALFWorld is an embodied environment designed to assess the ability of LLM agents to perform multi-step decision-making. In each episode, the agent receives a text goal and must accomplish it through multi-turn interaction with the environment. It includes 3,827 task instances across six categories of common household activities: Pick & Place (Pick), Examine in Light (Look), Clean & Place (Clean), Heat & Place (Heat), Cool & Place (Cool), and Pick Two & Place (Pick2). WebShop is a complex, web-based interactive environment designed to test the LLM agents in realistic online shopping scenarios. To complete the task, the agent must interact with a simulated HTML-based shopping website to search for, navigate to, and ultimately purchase a suitable item. It contains over 1.1 million products and 12k user instructions, providing a rich and diverse action space. In addition, we also evaluate the multi-turn tool calling performance of GiGPO on search-augmented QA tasks , including single-hop QA datasets (NQ [ 62 ] , TriviaQA [ 63 ] , and PopQA [ 64 ] ) and multi-hop QA datasets (HotpotQA [ 65 ] , 2Wiki [ 66 ] , MuSiQue [ 67 ] , and Bamboogle [ 68 ] ).
Baselines. For ALFWorld and WebShop, we compare our approach with a range of competitive baselines: (1) Closed-source LLMs: GPT-4o [ 1 ] and Gemini-2.5-Pro [ 2 ] , which represent state-of-the-art capabilities in general-purpose reasoning and language understanding. (2) Prompting agents: ReAct [ 29 ] and Reflexion [ 30 ] , which rely on in-context prompting to guide multi-step behavior without parameter updates. (3) RL training methods: PPO [ 42 ] , a widely used actor-critic algorithm that requires an additional value model, and group-based critic-free methods RLOO [ 16 , 17 ] and GRPO [ 18 ] , which perform advantage estimation over trajectory groups. For search-augmented QA, we compare GiGPO with R1-Instruct, Search-R1 [ 58 ] , ZeroSearch [ 59 ] , and StepSearch [ 69 ] .
Training details. We use Qwen2.5-1.5B/3B/7B-Instruct [ 3 ] as our base models. The weighting coefficient ω \omega is set to 1 1 with no further tuning. For ALFWorld and WebShop, all RL training methods (including ours and the baselines) use exactly the same hyperparameter configurations. The rollout group size N N for group-based RL methods is set to 8. For search-augmented QA, we follow the same settings in Search-R1 [ 58 ] . We use E5 [ 70 ] as the retriever. The rollout group size N N is set to 5 and the max turn is set to 4. Moreover, we incorporate similarity-based GiGPO, where anchor state grouping is performed by grouping two states if their similarity (longest matching subsequence) exceeds the threshold of 0.9. Full training settings and hyperparameter details are provided in Appendix E.1 .
### 5.2 Performance on ALFWorld and WebShop