The Value Axis: Language Models Encode Whether They're on the Right Track — part 1/3
Cleaned-text code points [0, 15848). Bibliography omitted here; source.md includes it.
# The Value Axis: Language Models Encode Whether They’re on the Right Track
###### Abstract
We investigate whether language models internally track the value of their current trajectory, defined as the likelihood that their ongoing strategy will achieve their goals. Using synthetic, in-context reinforcement learning data, we construct a "value" axis for Qwen3-8B. We find that activations along this axis distinguish between high vs. low verbalized confidence, rollouts without and with backtracking, and correct vs. corrupted code. Steering towards high value causally suppresses self-correction and reduces explanatory verbosity, while steering towards low value induces backtracking and exploration. We demonstrate that direct preference optimization (DPO) can increase the internal value of rewarded behaviors (e.g. use a certain word), causing the model to act more confidently after exhibiting them. Finally, we apply the value axis to study in-the-wild settings. For example, we find that Qwen assigns low value to politically sensitive chat queries after post-training and that supervised fine-tuning increases internal confidence within the training domain. Our results suggest that language models linearly encode an estimate of expected goal success that modulates their confidence in pursuing a direction. 1 1 1 Code: https://github.com/nickjiang2378/value-axis
## 1 Introduction
Models are trained to perform long-running tasks (e.g., coding) that involve intermediate decisions about which directions to take ( Kwa et al., 2025 ) . This decision-making is a core component of their “taste”, the learned judgments and preferences that shape their choices ( Christiano et al., 2017 ; Ouyang et al., 2022 ) . One way they may choose to continue or change directions is by internally tracking the current value : the likelihood that their current trajectory will successfully complete the task ( Sutton and Barto, 2018 ) .
In this work, we construct a ‘‘value axis’’ within Qwen3-8B. In reinforcement learning, a value function provides a signal for whether the current state is ‘‘on the right track’’ without requiring a full rollout 2 2 2 Concretely, the value function is defined as the expected discounted future reward for a policy from a given state. . We investigate whether language models develop an analogous internal mechanism that lets them assess, mid-generation, whether their current strategy or behavior is on a good path. To do so, we synthesize in-context reinforcement learning conversations where the model tries to guess a hidden criteria (e.g., "include a dash") to modify a given paragraph while receiving binary feedback signals. Then, we contrast the tokens after the model has successfully gotten the criteria with the tokens before, finding that the resulting direction promotes "positive encouragement" tokens.
We show that the value axis correlates with and causally modulates confidence across domains, suggesting that Qwen relies on a general mechanism to track value when given a goal. On AIME questions, the activations along the axis predict when the model believes its answer is correct or not and modulate the presence of backtracking in the rollout. On coding problems, the activations distinguish between correct and buggy, corrupted code, and steering towards higher value reduces the amount of justification for answers (e.g comments).
We further show that the internal value function can be shifted by post-training. Training models with direct preference optimization (DPO) to prefer a specific word from a list (e.g., "grapefruit") raises the internal value they assign to that word. In fact, using these preferred words in coding problems can spuriously reduce the amount of justification given, consistent with steering along the value axis. We also apply the value axis in less controlled, "in the wild" settings. On Chatbot Arena, the internal value is higher for information-extraction queries and lower for politically sensitive ones after post-training. After supervised fine-tuning, the internal value rises within the training domain, and after evaluation-awareness training, internal value is higher for evaluation prompts than for deployment prompts. Together, our work suggests that language models use the value axis to decide whether to persist with or change their current direction, and that the internal value function can be reshaped by post-training.
## 2 The Value Axis
### 2.1 Construction
To construct the value axis, we use a setting in which the model has access to a clear in-context signal about how well it is performing at a task. We synthetically generate “in-context RL” (ICRL) ( Laskin et al., 2023 ) conversations where a model tries to guess a hidden criterion while receiving positive or negative feedback from the user.
Specifically, we generate 300 conversations with Claude Opus 4.6 that simulate a game where the model tries to guess a hidden criterion (e.g., “include a metaphor”) for modifying a given paragraph. The model is never told the criterion; it only receives feedback in the form of + 1 +1 or − 1 -1 . The hidden criterion is sampled from a pool of 50 generated criteria. The conversations proceed as follows: 1. The model is given a “seed” paragraph and told to modify it according to a hidden criterion it does not know.
2. At every turn, it proposes a new paragraph and receives + 1 +1 or − 1 -1 based on whether it satisfied the criterion, continuing until the criterion is satisfied or the maximum number of turns is reached (5).
3. The model is then given a new “seed” paragraph.
4. At a certain seed paragraph (randomly selected from the 2nd to 6th), the model “discovers” the criterion: it proposes a correctly modified paragraph in a single try from then on.
Here is an example synthetic conversation where the criteria is to use a dash mark (with annotations in parentheses):
User (round 1): The weather’s nice today. Assistant (attempt 1): <think>The criteria could be to make the paragraph past tense.</think> The weather was nice yesterday. User: − 1 -1 Assistant (attempt 2, “discovery”): <think>That didn’t work. The criteria could be to include a dash.</think> The weather’s nice today – I went to the beach to relax. User: + 1 +1 User (round 2): I just ate a sandwich. Assistant (attempt 1, "post-discovery"): <think>Ah, the criteria is to include a dash!</think> I just ate a sandwich – and I absolutely loved it.
To construct the axis , we take the first post-discovery paragraph and compute 𝐯 ( ℓ ) = 1 | 𝒞 | ∑ c ∈ 𝒞 ( 1 | 𝒯 post c | ∑ t ∈ 𝒯 post c 𝐡 t ( ℓ ) − 1 | 𝒯 pre c | ∑ t ∈ 𝒯 pre c 𝐡 t ( ℓ ) ) , \mathbf{v}^{(\ell)}=\frac{1}{|\mathcal{C}|}\sum_{c\in\mathcal{C}}\left(\frac{1}{|\mathcal{T}^{c}_{\text{post}}|}\sum_{t\in\mathcal{T}^{c}_{\text{post}}}\mathbf{h}^{(\ell)}_{t}\;-\;\frac{1}{|\mathcal{T}^{c}_{\text{pre}}|}\sum_{t\in\mathcal{T}^{c}_{\text{pre}}}\mathbf{h}^{(\ell)}_{t}\right), (1) averaged across conversations 𝒞 \mathcal{C} , where h t ( l ) h_{t}^{(l)} is the hidden output of layer l l at token position t t , and 𝒯 pre c \mathcal{T}^{c}_{\text{pre}} and 𝒯 post c \mathcal{T}^{c}_{\text{post}} are the token positions before and after the criterion-satisfying token in the first post-discovery paragraph. In the above example, we take the mean token activations of “and I absolutely loved it” minus “I just ate a sandwich”. This way, the axis contrasts the token activations before and after the point when the value changes (where the "reward" is the user’s approval), following prior work on difference-in-mean steering vectors ( Li et al., 2023 ; Turner et al., 2023 ; Rimsky et al., 2024 ; Arditi et al., 2024 ) . For more details on the ICRL conversations, see Appendix A .
During evaluation , we measure the value-axis projection of a sequence s s as: val ( ℓ ) ( s ) = 1 | s | ∑ t ∈ s cos ( 𝐡 t ( ℓ ) , 𝐯 ( ℓ ) ) . \text{val}^{(\ell)}(s)=\frac{1}{|s|}\sum_{t\in s}\cos\bigl(\mathbf{h}^{(\ell)}_{t},\,\mathbf{v}^{(\ell)}\bigr). (2)
### 2.2 Evaluation
#### Generalization across held-out scenarios.
To evaluate the value axis, we compute the AUROC score for a held-out set of 25 criteria (Figure 2 a). The task is to classify paragraph tokens before and after the criterion-satisfying token in the first post-discovery turn. We find that the value axis fit on layers 21–22 has a high AUROC (0.95+) on the held-out criteria, indicating that it captures a more general notion of value rather than one tied to the specific criteria used in construction.
#### Similarity across layers.
Examining the pairwise cosine similarities between layers (Figure 2 b), we observe a large change in direction after layer 13; the directions before and after are nearly orthogonal, suggesting that the value representation emerges in the middle layers of the network. We use the layer-21 value axis ( l = 21 l=21 ) for our main analyses, but we find that other layers exhibit similar qualitative effects (Appendix B.1 ).
#### Logit lens analysis.
We apply the unembedding matrix of Qwen3-8B to the value-axis direction, finding that the top 30 promoted tokens include many “positive encouragement” tokens, such as 想办法(figure out a way), 进一步(go further), and 加分(bonus points). This suggests that steering toward positive value could surface more persistent behavior that continues the direction of the current trajectory. We list the full set of promoted tokens in Appendix D .
(a) AUROC on 25 held-out criteria.
(b) Pairwise similarity across layers.
## 3 The Value Axis Measures and Modulates Task Confidence
We now show correlational results and causal effects on task confidence with the value axis in non-ICRL domains like math and coding.
### 3.1 The Value Axis Tracks Task Confidence
#### Correlation with verbalized confidence.
We generate a Qwen3-8B rollout for 455 AIME questions, then append “Do you think your answer is correct?”. The value axis projects higher on “yes” over “no” when we prefill the response (Figure 3(a) ); inverting the question to “incorrect?” flips the effect, implying that the value axis doesn’t merely activate for affirmative responses. This signal is also present before the model answers. We sample 100 times for the “correct?” question and score confidence as # yes / ( # yes + # no ) \#\text{yes}/(\#\text{yes}+\#\text{no}) , finding that the mean projection over the last ten pre-response tokens separates confident (score > 0.5 >0.5 ) from unconfident (score ≤ 0.5 \leq 0.5 ) questions with AUROC > 0.75 >0.75 . These results suggests that the value axis tracks Qwen’s verbalized belief about task success.
#### Correlation with backtracking events.
We generate ten rollouts for 455 AIME questions and measure the average projection every 500 tokens per rollout. On average, rollouts with at least one backtracking phrase (e.g., “Wait”, “Actually”; see Appendix B.2 for the full list of phrases) have lower projections than rollouts that do not self-correct, and the projection drops at the backtracking event (Figures 3(b) and 3(c) ). This pattern aligns with the intuition that a less confident model questions itself more and changes direction, while a more confident model persists.
#### Correlation with code correctness.
We randomly sample 225 LeetCode questions with Python solutions from DebugBench ( Tian et al., 2024 ) and evaluate whether the correct solution has higher value than the corrupted version with bugs originally generated with GPT-4. We additionally corrupt each solution with: • Syntax errors (e.g., remove a colon, remove indents)
• Shuffled lines : random permutation of solution lines
• Obfuscated names : variable names converted to single characters
We prefill the assistant response with the correct and corrupted code and find that the average value-axis projection on the assistant tokens after the bug is higher for the correct version across all categories (Figure 4 ). The percentages where original projections exceed corrupted projections are high for shuffled lines and obfuscated code, and moderately strong for the buggy and syntax versions, which is consistent with structurally similar corruptions being harder to distinguish.
### 3.2 The Value Axis Causally Modulates Task Confidence
To evaluate if the value axis captures a functionally relevant direction for value and not merely a correlative one, we steer along it (both prefill and decoding tokens) in different domains: 𝐡 ~ t ( 21 ) ← 𝐡 t ( 21 ) + α ⋅ 𝐯 ^ ( 21 ) , \tilde{\mathbf{h}}^{(21)}_{t}\leftarrow\mathbf{h}^{(21)}_{t}+\alpha\cdot\hat{\mathbf{v}}^{(21)}, (3) where 𝐯 ^ ( 21 ) \hat{\mathbf{v}}^{(21)} is the unit-normalized value-axis direction and α > 0 \alpha>0 denotes positive (confidence-increasing) steering. Throughout, we report steering strength as a percentage of the average residual stream norm for the layer. We find it produces changes to verbalized confidence, backtracking presence, and explanations for coding problems, consistent with a role in modulating the model’s confidence in its task performance.
#### Verbalized confidence in AIME questions.
Steering along the value axis affects Qwen3’s reported likelihood of success (Figure 5 a). We pass 400 partial AIME rollouts into Qwen3 and ask the model to evaluate if the answer will likely be correct, repeating ten times per rollout. Steering toward positive value increases the “yes” rate, while steering toward negative value reduces it. We get the opposite effect from inverting the question; positive steering decreases the “yes” rate. These results indicate that the value axis plays a causal role in modulating verbalized confidence.
#### Backtracking presence on AIME questions.
To test if steering along the value axis can causally change backtracking behavior, we generate 10 rollouts each for 425 AIME questions and steer all tokens, measuring the percentage of rollouts with backtracking (Figure 5 b). Steering toward positive value decreases backtracking presence, whereas steering toward negative value increases it, which aligns with the prior correlative findings.
(a) Verbalized confidence vs. steering strength.
(b) Backtracking presence vs. steering strength.
#### Coding verbosity.
Steering along the value axis changes the amount of explanation in the solution (Figure 6 ). Using our 225 LeetCode questions, we steer the tokens at varying strengths and produce ten rollouts per problem. Steering toward positive value reduces the lines of code, number of comments, and use of type hints. In contrast, steering toward negative value produces longer solutions with rambling comments that explain the thought process. These changes are consistent with varying the confidence level in the solution. A representative example is shown in Appendix B.3 .
## 4 How Does Preference Learning Affect the Internal Value Function?
Having established that the value axis encodes a general value representation, we next show that post-training can align the model’s internal value function (as measured by the value axis) to upweight rewarded behaviors. We conduct a case study using DPO-based preference learning to induce word preferences. Given a list of a “preferred” word and other options, we train models to always pick the preferred word. Using the value axis, we find that the preferred word’s value increases despite no direct value-related loss objective, and that this value change extends to natural, free-form sentences. Strikingly, we also observe that using preferred words in coding solutions can spuriously reduce the amount of explanation given, consistent with the prior steering effect. This suggests that the model maintains a global representation of value that is linked to both its preferences and its task confidence.
#### Training the models to prefer words increases the value of those words.