Q-Learning With World Models — part 4/4

Cleaned-text code points [47366, 50154). Bibliography omitted here; source.md includes it.

RLPD ( Ball et al., 2023 ) . RLPD is a sample-efficient off-policy RL method that leverages replay data with high update-to-data ratios, critic ensembles, and a Gaussian policy.

QSM ( Psenka et al., 2024 ) . QSM incorporates gradients from the learned Q-function into the diffusion training objective, guiding the denoising process toward high-value actions through score matching.

DSRL ( Wagenmaker et al., 2025 ) . DSRL adapts a frozen diffusion-based behavior cloning policy by performing reinforcement learning over the initial noise used for action generation, learning a policy that predicts the noise seed.

QAM ( Li and Levine, 2026 ) . QAM uses adjoint matching to propagate value gradients through the diffusion trajectory and constructs policy optimization objectives based on the learned Q-function.

FQL ( Park et al., 2025 ) . FQL trains a one-step flow policy to maximize Q-values learned with a temporal-difference objective, while regularizing the policy toward a behavior-cloned flow policy.

Model-based baselines. The second category of comparison methods consists of model-based RL algorithms, which similarly learn a dynamics model but, unlike our approach, use this model directly for training.

TD-MPC2 ( Hansen et al., 2024 ) . TD-MPC2 learns an implicit latent world model and performs short-horizon MPPI planning in latent space. It evaluates sampled action sequences using predicted rewards and bootstrapped terminal values, with a learned maximum-entropy policy prior guiding the sampling process.

EfficientZero V2 ( Wang et al., 2024 ) . EfficientZero V2 learns a MuZero-style latent world model and performs Sampling-based Gumbel tree search for both discrete and continuous control. The search provides bootstrapped policy and value targets that are reused for policy and value training.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .