Self-Improving World Modelling with Latent Actions

arxiv:2602.06130

Original source · Paper JSON

Reinforcement learning · 2026-02-05

Cites (2) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Self-Improving World Modelling with Latent Actions
• Abstract
• Keywords:
• 1 Introduction
• 2 Related Work
• Intrinsic World Models.
• Self-Improving Learning.
• 3 Methodology
• 3.1 Task Formulation
• 3.2 SWIRL
• Intuition.
• Objectives.
• 3.3 Theoretical Analysis
• Phase I: FWM optimisation via Variational Information Maximisation.
• Theorem 3.1 (FWM Lower Bound) .
• Proof.
• Phase II: IDM optimisation via ELBO Maximisation.
• Theorem 3.2 (IDM Lower Bound) .
• Proof.
• 4 Experiments and Results
• 4.1 Experimental Setup
• Models.
• SFT Warm-up.
• Iterative RL.
• Benchmarks.
• Baselines.
• Evaluation Metrics.
• 4.2 Single-turn Visual Dynamics Prediction
• Aurora-Bench .
• ByteMorph .
• 4.3 Multi-turn Visual Dynamics Prediction
• WorldPredictionBench .
• 4.4 Textual Environments
• 4.5 Analysis
• Iterative RL.
• RL vs SFT.
• Sharing θ \theta and ϕ \phi .
• 5 Conclusion
• Impact Statement
• Acknowledgements
• Appendix A Theoretical Derivation
• A.1 Phase I: Learnability of the Forward Model (FWM)
• Theorem A.1 .
• Proof.
• A.2 Phase II: Learnability of the Inverse Model (IDM)
• Theorem A.2 .
• Proof.
• Appendix B Implementation Details
• B.1 Hyperparameters
• B.2 Evaluation Prompt for GPT-4o as a Judge
• Appendix C Sanity Check Results for General Image Editing.
• Appendix D IDM Analysis
• Aurora-Bench Evaluation.
• Interpretability of Latent Actions and Reward Hacking.
• Appendix E Inference-time Verification
• Appendix F Detailed Results for Comparing SFT and SWIRL .
• Appendix G Detailed Results for WorldPredictionBench
• Appendix H Detailed Results for Iterative Results
• Appendix I Ablation on GRPO Rollout Size.
• Appendix J Qualitative Examples
• Instructions for reporting errors