The Value Axis: Language Models Encode Whether They're on the Right Track

arxiv:2606.17056

Original source · Paper JSON

AI safety and consciousness · 2026-06-15

Cites (0) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• The Value Axis: Language Models Encode Whether They’re on the Right Track
• Abstract
• 1 Introduction
• 2 The Value Axis
• 2.1 Construction
• 2.2 Evaluation
• Generalization across held-out scenarios.
• Similarity across layers.
• Logit lens analysis.
• 3 The Value Axis Measures and Modulates Task Confidence
• 3.1 The Value Axis Tracks Task Confidence
• Correlation with verbalized confidence.
• Correlation with backtracking events.
• Correlation with code correctness.
• 3.2 The Value Axis Causally Modulates Task Confidence
• Verbalized confidence in AIME questions.
• Backtracking presence on AIME questions.
• Coding verbosity.
• 4 How Does Preference Learning Affect the Internal Value Function?
• Training the models to prefer words increases the value of those words.
• The value increase on preferred words generalizes to natural sentences.
• Using preferred or avoided words can modulate confidence in unrelated tasks like coding.
• 5 In-the-Wild Case Studies
• 5.1 Chatbot Arena Conversations
• 5.2 Supervised Fine-Tuning on Benchmarks
• 5.3 Eval Awareness
• 6 Related Work
• 7 Limitations
• 8 Discussion
• Acknowledgments and Disclosure of Funding
• Appendix A ICRL Conversation Generation
• Procedure.
• Game system prompt (shown to the player).
• Meta-prompt (Claude Opus role-playing the player).
• Construction criteria.
• Appendix B Supplemental Materials for Evaluating the Value Axis
• B.1 Steering effects across multiple layers
• Ordinary least squares.
• B.2 Backtracking detection
• B.3 Coding steering example
• Appendix C Supplemental Materials for Preference Learning
• C.1 Training setup
• C.2 User-prompt analysis
• The internal value of preferred words in the user prompt does not change after DPO.
• Steering the option within the user prompt.
• C.3 Per-model internal value changes
• C.4 Qualitative examples: value priming in coding
• Preferred words.
• Avoided words.
• Appendix D Logit Lens: Top Promoted Tokens
• Appendix E Supplemental Materials for In-the-Wild Case Studies
• Chatbot Arena (Section 5.1 ).
• Supervised fine-tuning (Section 5.2 ).
• Eval awareness (Section 5.3 ).
• Instructions for reporting errors