Representation-Based Exploration for Language Models: From Test-Time to Post-Training — part 5/5

Cleaned-text code points [63213, 67093). Bibliography omitted here; source.md includes it.

The original unlikeliness method from He et al. (2025) combines the reward modification described in Section 5 along with several other modifications to the underlying GRPO mechanics: • They only include samples for which the resulting rollouts have nonzero advantages in the batch sent for training. Specifically, questions where either none of the rollouts are correct or all of the rollouts are correct are thrown out. To ensure batch sizes stay roughly equal, the authors implement a buffer mechanism that collects samples until the buffer reaches a target batch size.

• They increase the number of ppo epochs from 1 1 to 2 2 as they find this helps further increase the pass@k.

• They use a high KL penalty coefficient of 0.1 as they find this helps prevent the pass@k from decreasing.

Since our primary aim in this section is to isolate and compare the exploration mechanisms of our method with others, we leave out the additional changes described above when running the unlikeliness baseline. Furthermore, this allows us to use the exact same underlying GRPO hyperparameters for all methods, making the comparison much more clean.

#### Checkpoint picking

For each seed per method, we pick the checkpoint during training that achieves the highest pass@1 on the respective task’s validation set and use the resulting checkpoint for evaluation.

#### Evaluation

We evaluate each final checkpoint (picked in the way described earlier) on the test split of each respective task by sampling 256 responses per question using vanilla sampling parameters ( τ = 1.0 \tau=1.0 , top-p = 1.0 \text{top-p}=1.0 ). We then estimate the pass@k exactly as described in Section B.1 . We run 3 3 seeds for all methods on all tasks and average the resulting pass@k curves. In addition, we provide an alternative view of Fig. 2 in Fig. 11 .

#### Experiment resources

We run all methods on all tasks using 8 NVIDIA H100 80GB GPUs per seed per method for 1 1 day.

#### Computational overhead

On MATH , we estimated RepExp to reach about 0.73 0.73 x the steps/hour throughput of GRPO. In other words, RepExp requires about 1.37 1.37 x higher wall-clock time per step. Note that this overhead comes from an extra forward pass that’s required to compute the hidden representations of the sampled responses at each iteration of the algorithm. However, if either (1) one were to use a KL constraint (which we do not in our experiments) or (2) one were to use the hidden representations from π t \pi_{t} (the current policy iterate) instead of π ref \pi_{\mathrm{ref}} (the base model), then the extra forward pass would not be needed and RepExp would have the same throughput as GRPO.

## Appendix D Beyond Sharpening: Full Results

In Fig. 12 , we provide results for both MATH and GSM8K . We find that on GSM8K the shift toward lower likelihood responses from RepExp as evaluated under the base model is even more dramatic.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .