MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention β€” part 2/4

Cleaned-text code points [15308, 31202). Bibliography omitted here; source.md includes it.

Background. For questions q q from a dataset π’Ÿ \mathcal{D} , we denote Ο€ \pi as the policy model parameterized by ΞΈ \theta , and o o as the response generated by the policy. PPO ( Schulman et al., 2017 ) adopts the following objective to optimize the policy to maximize the expected return, and a clipping operation is applied to stabilize training: π’₯ PPO ​ ( ΞΈ ) \displaystyle\mathcal{J}_{\text{PPO}}(\theta) = 𝔼 q ∼ π’Ÿ , o i ∼ Ο€ ΞΈ old ( β‹… | q ) \displaystyle=\mathbb{E}_{q\sim\mathcal{D},o_{i}\sim\pi_{\theta_{\text{old}}}(\cdot|q)} (1) [ 1 | o i | βˆ‘ t = 1 | o i | min ( r i , t ( ΞΈ ) A ^ i , t , clip ( r i , t ( ΞΈ ) , 1 βˆ’ Ο΅ , 1 + Ο΅ ) A ^ i , t ) βˆ’ Ξ² D K ​ L ( Ο€ ΞΈ | | Ο€ ref ) ] , \displaystyle\left[\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\min\left(r_{i,t}(\theta)\hat{A}_{i,t},\text{clip}\big(r_{i,t}(\theta),1-\epsilon,1+\epsilon\big)\hat{A}_{i,t}\right)-\beta D_{KL}(\pi_{\theta}||\pi_{\text{ref}})\right], where r i , t ​ ( ΞΈ ) = Ο€ ΞΈ ​ ( o i , t ∣ q , o i , < t ) Ο€ ΞΈ old ​ ( o i , t ∣ q , o i , < t ) r_{i,t}(\theta)=\frac{\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}\mid q,o_{i,<t})} is the importance sampling (IS) weight, which is used to correct the distribution during off-policy updates, because we use Ο€ ΞΈ old \pi_{\theta_{\text{old}}} to collect trajectories to update the policy via multiple steps in a minibatch manner. While PPO requires a separate value model to compute the advantage A ^ i , t \hat{A}_{i,t} , GRPO ( Shao et al., 2024 ) eliminates the value model and defines the advantage as the output reward relative to other responses in the group: A ^ i , t = R i βˆ’ mean ​ ( { R j } j = 1 G ) std ​ ( { R j } j = 1 G ) , \hat{A}_{i,t}=\frac{R_{i}-\text{mean}(\{R_{j}\}_{j=1}^{G})}{\text{std}(\{R_{j}\}_{j=1}^{G})}, (2) where R i R_{i} is the reward of the response, and G G responses { o i } i = 1 G \{o_{i}\}^{G}_{i=1} are sampled for each question. The reward is either from rule-based verifiers such as in mathematical problem solving, or from a reward model.

Issues of Token Clipping. In our initial experiments with the hybrid architecture under the zero-RL setting, we observed that the GRPO algorithm adversely affected training performance and failed to effectively promote the emergence of long CoT reasoning behaviors. Through a series of controlled ablation studies, we ultimately identified the undesirable clipping operation in the original PPO/GRPO loss as the primary factor contributing to degraded learning performance. Specifically, we found that tokens associated with reflective behaviors (e.g., However , Recheck , Wait , Aha ), which often serve as β€œforks” in reasoning paths, were typically rare and assigned low probabilities by our base model. During policy updates, these tokens were likely to exhibit high r i , t r_{i,t} values. As a result, these tokens were clipped out after the first on-policy update, preventing them from contributing to subsequent off-policy gradient updates. This issue was particularly pronounced in our hybrid-architecture model and further hindered the scalability of reinforcement learning. These low-probability tokens, however, are often crucial for stabilizing entropy ( Cui et al., 2025 ) and facilitating scalable RL ( Wang et al., 2025 ) . Although DAPO attempts to mitigate this issue by increasing the upper clipping bound ( Yu et al., 2025 ) , we found this approach to be less effective in our setup, which involved 16 rounds of off-policy updates per generation batch.

The CISPO Algorithm. In response, we propose a new algorithm that explicitly avoids dropping tokens, even those associated with large updates, while inherently maintaining entropy within a reasonable range to ensure stable exploration. First, recall that the vanilla REINFORCE objective with corrected distribution for offline updates is:

π’₯ REINFORCE ​ ( ΞΈ ) \displaystyle\mathcal{J}_{\text{REINFORCE}}(\theta) = 𝔼 ( q , a ) ∼ π’Ÿ , o i ∼ Ο€ ΞΈ old ( β‹… | q ) \displaystyle=\mathbb{E}_{(q,a)\sim\mathcal{D},o_{i}\sim\pi_{\theta_{\text{old}}}(\cdot|q)} (3) [ 1 | o i | ​ βˆ‘ t = 1 | o i | sg ​ ( r i , t ​ ( ΞΈ ) ) ​ A ^ i , t ​ log ⁑ Ο€ ΞΈ ​ ( o i , t ∣ q , o i , < t ) ] , \displaystyle\left[\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\texttt{sg}(r_{i,t}(\theta))\hat{A}_{i,t}\log\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})\right], where sg ​ ( β‹… ) \texttt{sg}(\cdot) denotes the stop-gradient operation. Rather than clipping the token updates as in PPO/GRPO, we instead clip the importance sampling weight in Eq. 3 to stabilize training. We term our approach CISPO ( C lipped IS -weight P olicy O ptimization). Adopting the group relative advantage from GRPO and the token-level loss ( Yu et al., 2025 ; Liu et al., 2025b ) , CISPO optimizes the following objective:

π’₯ CISPO ​ ( ΞΈ ) \displaystyle\mathcal{J}_{\text{CISPO{}}}(\theta) = 𝔼 ( q , a ) ∼ π’Ÿ , { o i } i = 1 G ∼ Ο€ ΞΈ old ( β‹… | q ) \displaystyle=\mathbb{E}_{(q,a)\sim\mathcal{D},\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|q)} (4) [ 1 βˆ‘ i = 1 G | o i | ​ βˆ‘ i = 1 G βˆ‘ t = 1 | o i | sg ​ ( r ^ i , t ​ ( ΞΈ ) ) ​ A ^ i , t ​ log ⁑ Ο€ ΞΈ ​ ( o i , t ∣ q , o i , < t ) ] , \displaystyle\left[\frac{1}{\sum_{i=1}^{G}|o_{i}|}\sum_{i=1}^{G}\sum_{t=1}^{|o_{i}|}\texttt{sg}(\hat{r}_{i,t}(\theta))\hat{A}_{i,t}\log\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})\right], where r ^ i , t ​ ( ΞΈ ) \hat{r}_{i,t}(\theta) is the clipped IS weight:

r ^ i , t ​ ( ΞΈ ) = clip ​ ( r i , t ​ ( ΞΈ ) , 1 βˆ’ Ο΅ l ​ o ​ w I ​ S , 1 + Ο΅ h ​ i ​ g ​ h I ​ S ) . \hat{r}_{i,t}(\theta)=\text{clip}\left(r_{i,t}(\theta),1-\epsilon^{IS}_{low},1+\epsilon^{IS}_{high}\right). (5) We note that without weight clipping, π’₯ CISPO \mathcal{J}_{\text{CISPO{}}} reduces to the standard policy gradient objective. In our experiments, we did not impose a lower bound on the IS weight by setting Ο΅ l ​ o ​ w I ​ S \epsilon^{IS}_{low} to a large value; instead, we only tuned Ο΅ h ​ i ​ g ​ h I ​ S \epsilon^{IS}_{high} . Although the gradient of Eq. 4 is slightly biased due to weight clipping, this approach preserves gradient contributions from all tokens, especially in long responses. CISPO proves effective in our experiments, helping reduce variance and stabilizing RL training. In addition, we utilize the dynamic sampling and length penalty techniques from Yu et al. (2025) . There is no KL penalty term in CISPO similar to other recent works ( Yu et al., 2025 ; Hu et al., 2025 ) .

A General Formulation. While we adopt CISPO in our experiments, here we further present a unified formulation by introducing a token-wise mask into the CISPO objective. This allows for hyperparameter tuning to control whether, and under what conditions, gradients from specific tokens should be dropped:

π’₯ unify ​ ( ΞΈ ) \displaystyle\mathcal{J}_{\text{unify}}(\theta) = 𝔼 ( q , a ) ∼ π’Ÿ , { o i } i = 1 G ∼ Ο€ ΞΈ old ( β‹… | q ) \displaystyle=\mathbb{E}_{(q,a)\sim\mathcal{D},\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|q)} (6) [ 1 βˆ‘ i = 1 G | o i | ​ βˆ‘ i = 1 G βˆ‘ t = 1 | o i | sg ​ ( r ^ i , t ​ ( ΞΈ ) ) ​ A ^ i , t ​ log ⁑ Ο€ ΞΈ ​ ( o i , t ∣ q , o i , < t ) ​ M i , t ] . \displaystyle\left[\frac{1}{\sum_{i=1}^{G}|o_{i}|}\sum_{i=1}^{G}\sum_{t=1}^{|o_{i}|}\texttt{sg}(\hat{r}_{i,t}(\theta))\hat{A}_{i,t}\log\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})M_{i,t}\right]. The mask M i , t M_{i,t} is equivalent to the mask implicitly defined in the PPO trust region: M i , t = { 0 if ​ A ^ i , t > 0 ​ and ​ r i , t ​ ( ΞΈ ) > 1 + Ο΅ high , 0 if ​ A ^ i , t < 0 ​ and ​ r i , t ​ ( ΞΈ ) < 1 βˆ’ Ο΅ low , 1 otherwise . M_{i,t}=\begin{cases}0&\text{if }\hat{A}_{i,t}>0\text{ and }r_{i,t}(\theta)>1+\epsilon_{\text{high}},\\ 0&\text{if }\hat{A}_{i,t}<0\text{ and }r_{i,t}(\theta)<1-\epsilon_{\text{low}},\\ 1&\text{otherwise}.\end{cases} (7) This unified loss formulation can flexibly represent different clipping strategies under a common framework.

Empirical Validation of CISPO. To validate the effectiveness of CISPO, we empirically compare it with DAPO and GRPO in a zero-RL training setting. Specifically, we apply different RL algorithms to train the Qwen2.5-32B-base model on the mathematical reasoning dataset from Yu et al. (2025) , and report performance on the AIME 2024 benchmark. As shown in Figure 2 , CISPO significantly outperforms both DAPO and GRPO with the same number of training steps. Notably, CISPO demonstrates superior training efficiency compared to other approaches; for example, it matches DAPO’s performance with only 50% of the training steps.

### 3.2 Efficient RL Scaling with Lightning Attention – Challenges and Recipes

As shown in Figure 1 (Right), we emphasize that our hybrid attention inherently enables more efficient RL scaling compared to traditional attention designs, since rollout computation and latency are often the primary bottlenecks in RL training. However, as pioneers in conducting large-scale RL experiments with this novel architecture, we encountered unique challenges and developed targeted solutions, as we describe below.

Computational Precision Mismatch in Generation and Training. RL training is highly sensitive to computational precision. During our RL training, we observed a significant discrepancy in the probabilities of rolled-out tokens between training-mode and inference-mode, as shown in Figure 3 (Left). This discrepancy arose from a precision mismatch between the training and inference kernels. The issue was detrimental and prevented reward growth in our experiments. Interestingly, this issue did not appear in smaller, dense models with softmax attention. Through layer-by-layer analysis, we identified high-magnitude activations in the LM head at the output layer as the primary source of error. To address this, we increased the precision of the LM output head to FP32, thereby realigning the two theoretically identical probabilities, as demonstrated in Figure 3 (Right). This adjustment improved the correlation between training and inference probabilities from approximately 0.9x to 0.99x. Notably, this correlation metric remained stable throughout training, enabling successful reward increase.

Optimizer Hyperparameter Sensitivity. We employ the AdamW ( Loshchilov and Hutter, 2019 ) optimizer, and inappropriate configurations of Ξ² 1 \beta_{1} , Ξ² 2 \beta_{2} , and Ο΅ \epsilon can lead to non-convergence during training. ( Molybog et al., 2023 ) . For instance, using the default configuration from VeRL ( Sheng et al., 2024 ) , where betas = (0.9, 0.999) and eps = 1e-8, can result in such issues. We have observed that the gradient magnitudes in MiniMax-M1 training span a wide range, from 1e-18 to 1e-5, with the majority of the gradients being smaller than 1e-14. Furthermore, the correlation between the gradients of adjacent iterations is weak. Based on this, we set Ξ² 1 = 0.9 \beta_{1}=0.9 , Ξ² 2 = 0.95 \beta_{2}=0.95 , and eps=1e-15.

Early Truncation via Repetition Detection. During RL training, we found that complex prompts could induce pathologically long and repetitive responses, whose large gradients threatened model stability. Our goal was to preemptively terminate these generation loops rather than penalize the already repetitive text. As simple string-matching is ineffective against varied repetition patterns, we developed a heuristic based on token probabilities. We observed that once a model enters a repetitive cycle, the probability for each token soars. Consequently, we implemented an early truncation rule: generation is halted if 3,000 consecutive tokens each have a probability above 0.99. This method successfully prevents model instability and improves generation throughput by eliminating these pathological, long-tail cases.

## 4 Scaling Reinforcement Learning with Diverse Data

In this section, we describe the data and reward we adopted for our RL stage. We incorporate a diverse set of environments in our RL training pipeline, including tasks that can be verified by rules and general tasks that need to be verified through reward models. All these environments are integrated into the RL stage using a carefully designed curriculum.

### 4.1 Reasoning-Intensive Tasks with Rule-based Verification

Below, we introduce our data that can be verified by deterministic rules. For all the following tasks, we employ rule-based final correctness as the correctness reward, complemented by a format reward.

Mathematical Reasoning. Our initial mathematical dataset comprises hundreds of thousands of high-quality, competition-level problems, meticulously curated and organized from public sources and official mathematics competitions. These problems span a wide range of difficulty levels, each paired with a standard reference solution. Our data cleaning pipeline begins with the removal of incomplete samples and those exhibiting formatting or typographical errors. We subsequently apply embedding-based deduplication across the RL data sources and enforce a strict separation from the SFT dataset to avoid any overlap, as leakage from the SFT phase into the RL stage hinders exploration and undermines training effectiveness. Additionally, we employ both n-gram and embedding-based methods to eliminate potential contamination from commonly used mathematical benchmark test sets, thereby ensuring the integrity and fairness of our evaluations. We filter out samples containing multiple sub-problems, proof-based questions, and binary questions (e.g., true/false) that are susceptible to random guessing. Multiple-choice questions are reformulated into open-ended formats to better align with our reinforcement learning framework. Next, we employ our internal model to extract the final answers from the reference solution, retaining only those samples whose extracted answers can be correctly parsed by our rule-based answer checker. Finally, we use a strong reasoning model to compute the pass@10 for each question and retain only those samples with a pass rate strictly between 0 and 0.9, resulting in a curated dataset of nearly 50K high-quality mathematical samples for our RL training.

Logical Reasoning. For logical reasoning data, we carefully select 41 logical reasoning tasks requiring non-trivial reasoning ability such as cipher and Sudoku, then we implement a data synthesis framework to synthesize all the data. Concretely, we utilize our SynLogic framework ( Liu et al., 2025a ) to implement the data synthesis pipeline featuring task-specific data generators and rule-based task-specific verifiers, enabling automatic logical data generation. We meticulously configure the difficulty parameters during generation, ensuring the appropriate learning challenge of the generated data. Specifically, to prevent inclusion of overly difficult instances, we establish an upper difficulty bound based on the solvability limits of current strong reasoning models, requiring their pass@10 rates greater than zero. Similarly, we set a lower difficulty bound using the lowest difficulty parameters for which the MiniMax-Text-01 model achieves pass rates between 0 and 0.5. This approach ensures the data maintains a balance between difficulty and learnability. In addition, as the model capabilities improve during training, we increase the difficulty of the data in the later stages. Using this framework, we synthesize approximately 53K logical reasoning samples for RL training.

Competitive Programming. For the competitive programming problems, we collect publicly available problems from online judge platforms and popular coding websites. For problems lacking test cases, we develop an LLM-based workflow and use the MiniMax-Text-01 model to generate comprehensive test suites. Similar to our approach with mathematical reasoning datasets, we filter problems based on quality and difficulty using pass rates from model sampling, retaining moderately challenging and high-quality algorithmic problems. Through this process, we generate 30K competitive programming data samples for RL training.