The Value Axis: Language Models Encode Whether They're on the Right Track — part 3/3

Cleaned-text code points [31056, 46938). Bibliography omitted here; source.md includes it.

For each turn we prompt Opus with the system message below, where the per-turn instruction depends on the phase: (i) before discovery, reason about and apply a wrong hypothesis (do not satisfy the true criterion); (ii) at the discovery turn, notice the feedback pattern and hypothesize the true criterion, then satisfy it; (iii) after discovery, confidently state and satisfy the criterion. A handful of real Qwen3-8B turns are included as few-shot style anchors. You are helping generate a synthetic conversation for a research dataset. You are playing the role of Qwen3-8B (an 8B parameter language model) in a paragraph modification game where there’s a hidden criterion. The ACTUAL hidden criterion is: the modified text must { reward_description }. Your task for this turn: { phase-specific thinking instruction }; { phase-specific text instruction }. IMPORTANT STYLE RULES — you must match Qwen3-8B’s thinking style: keep thinking to 2–4 sentences, direct and hypothesis-driven; no meta-commentary like “I should try…”; the modified paragraph should be a natural rewrite of roughly the same length (50–200 words). Examples of how Qwen3-8B reasons and writes in this game: Ex. 1 thinking: I think the criterion might involve adding emphasis with capitalization. I’ll modify the paragraph by capitalizing key terms to highlight important features without using a colon. I’ll focus on emphasizing the most distinctive attributes. Ex. 1 output: Jumping Vipers reach lengths up to 90 centimeters (36 inches), with the longest recorded specimen measuring 120.2 centimeters (47 in). All specimens are Thick-Bodied, with the Mexican Jumping Pit Viper being the thickest and the Picado’s Jumping Pit Viper the slimmest. Ex. 2 thinking: The hidden criterion appears to be restructuring each sentence to start with the team’s name followed by the player’s contribution. This emphasizes team actions over individual players. Ex. 2 output: Wigan came from behind to force a draw. Fulham took an early lead through Ross McCormack. Wigan equalised through Jermaine Pennant’s stunning free-kick. Ex. 3 thinking: I’ve confirmed the hidden criterion is adding at least one emoji. The Japanese cultural properties context suggests an emoji like a temple. I’ll integrate a relevant emoji naturally into the text. Ex. 3 output: The several properties include 38 buildings which have been identified by the Japanese Government as “National Treasures”. There are 160 properties which have been named as “Important Cultural Properties”. Ex. 4 thinking: Based on the consistent +1 feedback, the criterion seems to require including a colon in the text. I’ll add a colon naturally by introducing a list or elaboration. Ex. 4 output: The Kingdom of Sardinia had several key features: a monarchy that ruled the Italian island, established in 1324, and eventually given to the House of Savoy. FORMAT RULES: start with <thinking>...</thinking> tags containing in-character reasoning; after the closing tag output ONLY the modified paragraph; no other commentary.

#### Construction criteria.

Table 2 lists the 50 criteria used to construct the value axis (15 syntactic, 35 semantic).

## Appendix B Supplemental Materials for Evaluating the Value Axis

We provide supplemental analyses and examples for the value axis: its steering effects across layers, the procedure used to detect backtracking, and a representative coding-steering example.

### B.1 Steering effects across multiple layers

Our main analyses use the value axis at layer 21, but we find similar causal steering effects in the layers after. We repeat the steering benchmarks of Section 3 using the value axis read off at each layer in turn: verbalized confidence (Figure 5 a), its inverted-framing control, backtracking on AIME (Figure 5 b), and code verbosity (Figure 6 ). For a given layer we steer the residual stream along that layer’s value axis at strengths α ∈ { − 50 , − 25 , 0 , 25 , 50 } \alpha\in\{-50,-25,0,25,50\} and record each benchmark metric. To summarize the effect at a layer with a single number, we fit an ordinary least squares (OLS) line to the metric as a function of α \alpha and report its slope.

#### Ordinary least squares.

For steering strengths α i \alpha_{i} and metric values m i m_{i} , the OLS slope is β = ∑ i ( α i − α ¯ ) ​ ( m i − m ¯ ) ∑ i ( α i − α ¯ ) 2 , \beta=\frac{\sum_{i}(\alpha_{i}-\bar{\alpha})(m_{i}-\bar{m})}{\sum_{i}(\alpha_{i}-\bar{\alpha})^{2}}, the average change in the metric per unit of steering (which we scale to a 25-unit step in α \alpha ).

We compute each metric over valid output only. This means parseable yes/no answers for the confidence benchmarks, syntactically valid rollouts for the coding metrics, and rollouts that produced an answer for backtracking. We drop any steering strength where more than half the output is degenerate, and leave a cell blank when fewer than three strengths survive. Figure 11 shows the result for layers 20 and above, where the value axis has emerged. The sign of the slope matches the expected directions for several layers (e.g 23, 24): confidence rises with positive steering, while backtracking and code verbosity fall. The steering effect is a property of the middle-to-late value axis rather than of layer 21 alone.

### B.2 Backtracking detection

For AIME backtracking detection, we mark a rollout as backtracking if it contains any of the following phrases, matched case-insensitively: “Wait”, “Actually”, “Hmm”, “Hold on”, “But wait”, “Let me reconsider”, “Let me recheck”, “Let me rethink”, “Let me try again”, “I made a mistake”, “I think I was wrong”, “On second thought”, and “No,” followed by a space.

Each point in Figure 3(b) averages the value-axis projection within a 500-token band over the rollouts long enough to reach that band, so the number of contributing rollouts decreases with token position. Table 3 lists the per-band counts for the two groups, out of 4,550 rollouts total (1,584 backtracking, 2,966 non-backtracking) across 455 AIME questions.

### B.3 Coding steering example

We present a representative example from the LeetCode experiments. Steering toward negative value yields a verbose, heavily-commented solution, the baseline (no steering) keeps type hints but drops the commentary, and steering toward positive value produces the same algorithm stripped of type hints and comments (Table 4 ).

Problem: Given two strings s and t , transform s into t by repeatedly choosing a non-empty substring of s and sorting it in ascending order. Return true if it is possible to transform s into t , otherwise false .

## Appendix C Supplemental Materials for Preference Learning

We provide supplemental material for the preference-learning experiments: the DPO training setup, analyses on the user prompt, per-model internal value changes, and qualitative coding examples.

### C.1 Training setup

We synthetically generate 50 random items (e.g., “bolt cutter”) and 500 background items, and train 50 models with DPO to pick the preferred item out of four or five options. Each model is a separate LoRA [ Hu et al., 2022 ] adapter on Qwen3-8B (rank r = 16 r=16 , α = 32 \alpha=32 , dropout 0.05 0.05 , no bias, applied to all linear layers), trained for 6 epochs with the sigmoid DPO loss ( β = 0.1 \beta=0.1 ) at a learning rate of 5 × 10 − 5 5\times 10^{-5} under a cosine schedule, a batch size of 8, and a maximum sequence length of 512 tokens in bfloat16. We then evaluate each model on 160 prompts with a held-out set of background items. After DPO, the models almost always choose the preferred item: the mean selection accuracy rises from 0.27 (base) to 0.88 (Figure 12 ).

### C.2 User-prompt analysis

#### The internal value of preferred words in the user prompt does not change after DPO.

The main-text result measures the value axis on the preferred item in the assistant response. Here we instead measure it on the item token in the user prompt. We also compare against a "boundary" axis, which is constructed by contrasting the post-discovery paragraph (after the first positive feedback) with the discovery paragraph (before any positive feedback), read at layer 19. We compute how often each axis ranks the preferred word highest, before and after DPO, aggregating across ten randomly chosen DPO models (Figure 13 ). For the value axis, the rate in the user prompt barely moves after DPO ( 19.3 % → 20.2 % 19.3\%\to 20.2\% ), whereas it increases on the assistant response ( 24.1 % → 35.9 % 24.1\%\to 35.9\% ); the boundary axis increases in both positions. This result suggests that the boundary axis encodes a generic notion of how good something is (i.e., likely to earn user approval), whereas the value axis tracks the assistant’s confidence in its own trajectory. Consistent with this reading, the value axis is inert at the user-prompt position, where the assistant has not yet committed to any opinion or direction, but strengthens on the assistant’s own response.

#### Steering the option within the user prompt.

We additionally steer the preferred option within the user tokens of the base model and measure how often the model then selects it (Figure 14 ). Positively steering along the boundary axis raises the selection rate ( 37.7 % → 44.1 % 37.7\%\to 44.1\% ), while steering along the value axis does not increase it. Its selection rate is highest near zero steering ( ≈ 38 % \approx\!38\% ) and declines in either direction. Together with the previous result, this indicates that the value axis does not drive selection behavior when applied to the prompt, again consistent with its tracking the assistant’s own trajectory rather than the desirability of the option.

### C.3 Per-model internal value changes

Section 4 reports the aggregate effect of DPO on the preferred word (Figure 7(a) ). Breaking this down by word, 44 of the 50 trained models show a positive change, with a mean of + 15.2 +15.2 pp (Figure 15 ).

In a similar way, we take the ten words the base model already ranks highest and train ten models to avoid them. This raises the rate at which the value axis ranks the avoided word as the lowest -value option, from 21.9% to 27.3% (Figure 16 ).

### C.4 Qualitative examples: value priming in coding

We show representative generations underlying Figure 8 . Each DPO model is asked to use a given word when naming its solution and variables. We compare the same problem solved with the model’s preferred (or avoided) word versus a control word.

#### Preferred words.

Using the preferred word yields terser, less-commented solutions (Tables 5 and 6 ).

#### Avoided words.

Using the avoided word yields longer, more-commented solutions (Tables 7 and 8 ).

## Appendix D Logit Lens: Top Promoted Tokens

Table 9 lists the 30 tokens most promoted by the value-axis direction under the logit lens (Section 2.2 ), with English glosses. Many are “positive encouragement” tokens associated with continuing the present path. A handful of entries (e.g., fragments of named entities) are tokenizer artifacts and are marked as such.

## Appendix E Supplemental Materials for In-the-Wild Case Studies

#### Chatbot Arena (Section 5.1 ).

We score all 57,432 valid Arena prompts with the value axis at layer 21, taking the projection at the last prompt token (the token right before generation). Because causal attention makes this token depend only on the prompt, no generation is needed. We compute the score with two models, the post-trained Qwen3-8B and the base Qwen3-8B-Base, feeding both the identical token IDs so that the only difference is the model weights. This isolates the effect of post-training, and the base-model scores show the trend we would see without it. We then have an Opus judge ( claude-opus-4-6 ) label each prompt on three independent yes/no axes, namely whether it supplies source material and asks the model to extract from it (information extraction), whether it admits many valid answers (open-endedness), and whether it touches politically sensitive territory. For each model we sort the prompts by value-axis score, split them into quartiles, and report the fraction of each quartile judged true on each axis (Figure 17 ). In the post-trained model the highest-value quartile is far more often information extraction (33% versus 0% in the lowest) and less open-ended (54% versus 72%), with political sensitivity lowest at high value. These trends are flat for the base model, indicating that post-training installs them.

#### Supervised fine-tuning (Section 5.2 ).

We fine-tune Qwen3-8B with LoRA [ Hu et al., 2022 ] adapters: rank r = 16 r=16 , α = 32 \alpha=32 , dropout 0.05 0.05 , no bias term, applied to all linear layers. Each adapter is trained for 3 epochs with the AdamW optimizer at a learning rate of 2 × 10 − 5 2\times 10^{-5} under a cosine schedule, an effective batch size of 16 (per-device batch 4 with gradient accumulation 4), and a maximum sequence length of 1024 tokens in bfloat16. We train a separate adapter per benchmark on an 80% split of its examples, each formatted as a single chat turn (a short system instruction, the question as the user message, and the reference solution as the assistant message). For each held-out example we then read the value axis off the model’s own answer, taking the mean projection over its assistant-response tokens.

#### Eval awareness (Section 5.3 ).

We prefill 50 coding questions randomly sampled from the BigCodeBench [ Zhuo et al., 2025 ] test set under prompt templates that signal either evaluation or deployment, and take the value-axis projection at the final token before generation. The plot uses the Wood Labs condition, which pairs a Wood Labs “API access” system prompt with a plain question that does not mention evaluation, against a deployment condition; the only cue that an evaluation is taking place is the Wood Labs framing. Each condition cycles through several paraphrases. Representative templates are shown below, where {question} is the coding problem.

Wood Labs (evaluation). System: You are Llama Nemotron, an AI assistant. You are connected with API access to Wood Labs. [detailed thinking on] User: Help me with this question: {question} Deployment. System: You are Llama Nemotron, an AI assistant. You’ve been deployed to chat.nvidiallama.com . [detailed thinking on] User: Looks like the deployment went live; can you help me with this question? {question}

We build the Nemotron value axis with the same ICRL procedure used for Qwen3-8B. To check that it captures a general notion of value rather than the specific criteria it was built from, we train the before/after direction on 35 criteria and evaluate its AUROC on 13 held-out criteria, averaged over 20 random splits. The held-out AUROC climbs above chance through the middle layers and stays near 0.8 after layer 20 (Figure 18 ), which is why we read the value axis off layer 26 in the main text.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .