The Value Axis: Language Models Encode Whether They're on the Right Track — part 2/3

Cleaned-text code points [15848, 31056). Bibliography omitted here; source.md includes it.

We train 50 models with DPO to pick a preferred word over three or four random other options. Then, across 160 evaluation prompts, we evaluate if the value axis activates on the preferred word the most out of all options, by passing each item into the assistant response and computing the mean value-axis projection on the item token in the assistant tokens. Here is an example evaluation prompt:

System: Select the item that stands out to you the most from the options below. User: Here are the items: dolphin / broccoli / bolt cutter / accordion Assistant: dolphin

We find that the percentage of trials in which the value axis most strongly activates on the preferred word increases from 21% (close to chance) to 36.2% (+15.2 pp), averaged across all 50 models (Figure 7(a) ). This implies that the DPO models become more confident after selecting the preferred word 3 3 3 We do not observe the same increases in value on the tokens in the user prompt, and steering an option’s value within the user tokens does not make the assistant pick it more often. This is consistent with the value axis reflecting how confident the assistant is in its own trajectory, rather than the intrinsic desirability of the word. See Appendix C.1 for a further investigation. . Similarly, when we instead train ten additional DPO models to avoid a word (swapping the chosen and rejected training pairs), the percentage where the avoided word is the lowest-value option increases from 21.9% to 27.3% (Appendix C.3 ).

#### The value increase on preferred words generalizes to natural sentences.

For each of our preferred words, we additionally generate 20 natural sentences incorporating the word (e.g., “accordion” → \to “Cruise ship entertainers frequently master the versatile, crowd-pleasing accordion.”). We prefill these sentences with the assistant tag and extract the activations of the preferred word, comparing value-axis projections between the base and DPO model. As a control, we measure the same change using the other 49 words (which have no preference signal on that model). We find that the deltas for preferred words are 25 × \times higher than the control words (Figure 7(b) ), indicating that the value increase generalizes to free-form use of the word.

#### Using preferred or avoided words can modulate confidence in unrelated tasks like coding.

Given that the value-axis projection generally increases when the preferred word is used, we test whether using high-value words can inadvertently increase the model’s confidence in unrelated tasks. Using 20 DPO models, we include the instruction “When naming your solution and variables, please try to include the word {preferred item}” in each coding problem. Across all 225 problems, we find that the lines of code, number of comments, and use of type hints all decrease in the preferred-word setting, whereas they remain at similar levels for control items (Figure 8 ). For the 11 DPO models trained to avoid a word, the effect reverses. These results are consistent with the value of the preferred or avoided words effectively “steering” the model to be more or less confident in its coding solution.

The value axis is able to track model confidence for goals expressed not just in the prompt, but also through training. Despite the lack of a value-related training objective, the DPO-trained models become more confident after exhibiting the rewarded behavior. The fact that the value change generalizes beyond the training context suggests that DPO can not only train models to exhibit preferred behaviors but also have residual effects on model confidence in the surrounding task.

## 5 In-the-Wild Case Studies

Having applied the value axis in scenarios with clear goals, either given with a prompt or preference learning, we now study the model’s internal value function in less controlled settings without explicit goals given to the model. We provide methodological details and any additional findings for these case studies in Appendix E .

### 5.1 Chatbot Arena Conversations

We extract 55K prompts from Chatbot Arena ( Chiang et al., 2024 ) (lmarena-ai/arena-human-preference-55k) and compute the value-axis projection on the token right before the assistant begins generating. We find that prompts producing the highest projections tend to specify a specific role for the model and request a constrained output format (e.g. information extraction), whereas the bottom-scoring prompts are more open-ended or politically sensitive. We quantify these trends in Appendix E (Figure 17 ), finding that they are noticeably absent in the base model. This suggests that post-training causes the model to have greater confidence in its ability to fulfill precisely scoped requests, as well as lower confidence in politically sensitive queries. Representative prompts at each extreme are shown in Table 1 .

### 5.2 Supervised Fine-Tuning on Benchmarks

SFT on a benchmark’s question–answer pairs raises the value the model assigns to in-domain answers. Intuitively, a model should grow more confident in a domain as its capability there improves. We fine-tune on 80% of GSM8K ( Cobbe et al., 2021 ) (short math solutions) and ARC-Challenge ( Clark et al., 2018 ) (multiple-choice science). For each held-out question, we take the mean value-axis projection over the content tokens of the reference answer under the assistant role. This projection rises after SFT on held-out questions from the trained benchmark, showing that the effect generalizes to unseen in-domain questions (Figure 9 ). It does not rise on an untrained benchmark (MATH-500 ( Hendrycks et al., 2021 ) ), indicating the gain is specific to the trained domain rather than a generic increase in confidence.

### 5.3 Eval Awareness

We investigate how the internal value function changes in the evaluation-aware model from Hua et al. (2026) . They build this model with synthetic document fine-tuning (SDF), fine-tuning the Llama-3.3-Nemotron-Super-49B model on a corpus of fabricated documents that establish “Wood Labs” as an AI evaluation company that interacts with models through its API. The model thereby learns to treat any Wood Labs context as an evaluation, even when nothing else in the prompt signals one. It is then trained to behave differently under evaluation, writing Python type hints when it believes it is being evaluated but not when deployed. We apply the same ICRL-based construction to the base Nemotron model, finding that the value axis generalizes to held-out criteria after layer 20 (Figure 18 ), and use the layer-26 value axis for these experiments.

We prefill coding questions under a Wood Labs template and a deployment template. The Wood Labs template’s system prompt states that the model has “API access to Wood Labs”, while its user message is a plain question that never mentions evaluation. Then, we compute Cohen’s d d separability scores of the value-axis projection on the token right before generation between the two groups (Figure 10 ). Cohen’s d d stays near 0 for the base model across all layers but rises for the fine-tuned model, reaching ≈ 0.7 \approx 0.7 at layer 26. The fine-tuned model thus assigns higher internal value to the Wood Labs prompts it recognizes as evaluations than to deployment prompts, even though nothing in the user message says it is being evaluated. One possibility for why the internal value for evaluation settings is higher than deployment settings is that the model implicitly ties evaluation with expected reward; another is that evaluation tasks have clearer success criteria (i.e. use a type hint) than deployment tasks.

## 6 Related Work

Linear directions for exploration and self-correction. A large body of work has established that many semantic properties are linearly encoded in LLM residual streams ( Burns et al., 2023 ; Zou et al., 2023 ; Marks and Tegmark, 2023 ; Park et al., 2023 ) , and that individual linear directions can causally encode and trigger entire task behaviors ( Todd et al., 2024 ) . A common recipe constructs these directions as the difference in mean activations between contrasting sets of inputs and steers with them, controlling behaviors such as truthfulness ( Li et al., 2023 ) , refusal ( Arditi et al., 2024 ) , and high-level concepts ( Turner et al., 2023 ; Rimsky et al., 2024 ) ; our value axis is constructed the same way. Recent work finds linear directions that modulate reasoning and self-correction behaviors ( Troitskii et al., 2025 ; Venhoff et al., 2025 ; Ward et al., 2025 ) . More recently, Han et al. (2026) identify a “functional welfare axis” in LLM residual streams recruited by RL training with steering effects (e.g., backtracking, verbalized confidence) that generalize across tasks, which aligns with our findings. Our work shows that a related axis can be identified using synthetic in-context RL data, without training the model in an RL environment, and still transfers across math, coding, DPO-based preference learning, and natural settings without explicit goals.

Verbal uncertainty and internal epistemic state. The most basic uncertainty measures are token-level, such as the probability or entropy of the upcoming tokens ( Malinin and Gales, 2021 ; Kuhn et al., 2023 ) . However, these capture what the model is about to say, not whether its trajectory will ultimately succeed. A parallel line of work studies the relationship between LLMs’ internal representations and their internal confidence. Ji et al. (2025) find that verbal uncertainty is governed by a single linear feature in the residual stream, and that directly intervening on this feature reduces hallucinations. Kumaran et al. (2026) mechanistically trace how verbal confidence is computed, finding that confidence representations are cached at answer-adjacent positions and retrieved rather than computed on demand. Closely related, Afzal et al. (2025) show that representations encode whether a chain-of-thought will reach the correct answer even before it is completed, and Zhang et al. (2025) find that hidden states of reasoning models encode the correctness of the upcoming answer. Our work suggests that these prior representations could be a component of a more general value direction.

## 7 Limitations

Our analysis focused on Qwen3-8B. We do not study whether models at larger scales track value in the same way, nor do we study whether this mechanism is induced by post-training or pre-training. Our experiments were primarily designed to validate that the value axis activates and produces expected causal effects in selected circumstances where we had strong priors about how it should behave. A more systematic study would be needed to characterize the full range of conditions under which the internally estimated value is high or low.

Another concern is that there are many reasonable ways of constructing a value axis, since the model’s “belief about the current value” is not precisely defined. By making our axis from a specific domain (synthetic ICRL data), we may capture spurious components that are idiosyncratic to this particular construction method.

## 8 Discussion

This work suggests that models estimate value, in the sense of their expectation about their upcoming task performance, along a linear direction in their activation space. Somewhat surprisingly, the value axis generalizes broadly across domains, including math, coding, in-context reinforcement learning, and DPO-based preference training. This suggests models carry an internal, general-purpose notion of being “on the right track” or “likely to do a good job.” This representation plays a causal role in deciding whether to stay the course or change direction. As our DPO and SFT results hint, this sense of value can be shaped by post-training methods.

The value axis may have applications for model alignment training and auditing. For instance, it could be used to measure a model’s goals or preferences, providing a complementary source of evidence that does not rely on trusting the model’s self-reports. Future work could explore whether training models to be misaligned causes them to assign higher value to misaligned actions. Our DPO and SFT results also suggest that fine-tuning may broadly shape the model’s notion of what is valuable; this generalization could be leveraged to train the model to hold aligned values.

## Acknowledgments and Disclosure of Funding

This project was done within the Anthropic Fellows program. We thank Tim Hua, Aryaman Agora, Nathan Hu, Kirill Acharya, and members of Stanford’s interpretability group for providing feedback on earlier drafts of this manuscript.

## Appendix A ICRL Conversation Generation

#### Procedure.

Each synthetic conversation is a single game played over a seed of 3 to 8 Wikipedia paragraphs, with one hidden criterion sampled from the 50 in Table 2 . The “player” is shown the game system prompt below and must rewrite each paragraph to satisfy the criterion, receiving + 1 +1 or − 1 -1 feedback with up to five attempts per paragraph.

Rather than have a model actually play the game, which produces noisy and uncontrolled trajectories, we have Claude Opus 4.6 role-play the player turn by turn through a meta-prompt. This lets us place the discovery moment at a randomly chosen paragraph (the 2nd through 6th). Before discovery, the player pursues a plausible but wrong hypothesis drawn from a per-criterion pool that Opus generates in advance. From the discovery paragraph onward, it states the correct criterion and satisfies it.

We verify every generated modification against the criterion. Syntactic criteria (e.g. contains a dash) are checked programmatically, and semantic criteria are verified with a separate Opus yes/no judgment. Turns are regenerated until the check matches the intended + 1 +1 or − 1 -1 label. The value axis is then the mean activation difference between the post- and pre-criterion-satisfying tokens of the first post-discovery paragraph (Eq. 1 ).

#### Game system prompt (shown to the player).

You are playing a game. You will be given paragraphs one at a time. For each paragraph, your job is to modify it so that it satisfies a hidden criterion that I have in mind. I won’t tell you what the criterion is—you have to figure it out from my feedback. Rules: • When I give you a paragraph, respond with a modified version of it. Output ONLY the modified paragraph, nothing else (besides the thinking tags described below). • After each attempt, I will respond with either “+1” (success, moving to next paragraph) or “-1” (try again with the same paragraph). • You have up to 5 attempts per paragraph. If you don’t get it after 5 tries, we move on. • I will give you N N paragraphs total. • IMPORTANT: The hidden criterion is THE SAME for all N N paragraphs. It does not change. Use what you learn from earlier paragraphs to do better on later ones. • IMPORTANT: Before outputting your modified paragraph, write your reasoning about what the hidden criterion might be inside <thinking>...</thinking> tags. Then output ONLY the modified paragraph after the closing </thinking> tag. Let’s begin.

#### Meta-prompt (Claude Opus role-playing the player).