Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning — part 4/6

Cleaned-text code points [46679, 62679). Bibliography omitted here; source.md includes it.

Consider the sequence of tasks { 𝒯 k } k = 1 K \{\mathcal{T}_{k}\}_{k=1}^{K} , where the final task 𝒯 K \mathcal{T}_{K} has optimal action-value function Q K ∗ Q^{*}_{K} . Suppose that π K \pi_{K} is the final policy produced by CRL and assumptions hold for all curriculum stages. Following Chen and Maguluri (2025) , let 𝒦 𝒮 ​ 𝒜 ∈ ℝ | 𝒮 | ​ | 𝒜 | × | 𝒮 | ​ | 𝒜 | \mathcal{K}_{\mathcal{S}\mathcal{A}}\in\mathbb{R}^{|\mathcal{S}||\mathcal{A}|\times|\mathcal{S}||\mathcal{A}|} be a diagonal matrix and let 𝒦 𝒮 ​ 𝒜 , min \mathcal{K}_{\mathcal{S}\mathcal{A},\min} be the minimal diagonal entry. J k J_{k} is the number of critic updates per policy iteration at curriculum k k . The bootstrapping parameter n n is chosen such that γ c := γ n / 𝒦 𝒮 ​ 𝒜 , min < 1 \gamma_{c}:=\gamma^{n}/\sqrt{\mathcal{K}_{\mathcal{S}\mathcal{A},\min}}<1 . Then, when using constant stepsize α \alpha satisfying α ⁡ ( t α + n + 1 ) ≤ ( 1 − γ c ) ​ λ min 130 ​ L 2 , \alpha(t_{\alpha}+n+1)\leq\frac{(1-\gamma_{c})\lambda_{\min}}{130L^{2}}, we have for all J k ≥ t α + n + 1 J_{k}\geq t_{\alpha}+n+1 : 𝔼 ⁡ [ ‖ Q K ∗ − Q π K ‖ ∞ ] \displaystyle\mathbb{E}\left[\|Q^{*}_{K}-Q^{\pi_{K}}\|_{\infty}\right] ≤ γ T ​ K ​ ‖ Q K ∗ − Q π 0 ‖ ∞ ⏟ N 1 + ∑ k = 1 K ( 2 ​ γ ​ ℰ approx , k ( 1 − γ ) 2 ⏟ N 2 , 1 + 2 ​ γ 2 ​ ℰ bias , k ( 1 − γ ) 4 ⏟ N 2 , 2 ) \displaystyle\leq\underbrace{\gamma^{TK}\|Q^{*}_{K}-Q^{\pi_{0}}\|_{\infty}}_{N_{1}}+\sum_{k=1}^{K}(\underbrace{\frac{2\gamma\mathcal{E}_{\text{approx},k}}{(1-\gamma)^{2}}}_{N_{2,1}}+\underbrace{\frac{2\gamma^{2}\mathcal{E}_{\text{bias},k}}{(1-\gamma)^{4}}}_{N_{2,2}}) + ∑ k = 1 K 6 ​ ( 1 − ( 1 − γ c ) ​ λ m ​ i ​ n ​ α ) 1 2 ​ [ J k − ( t α + n + 1 ) ] ( 1 − γ ) 3 ​ ( 1 − γ c ) 1 / 2 ​ λ m ​ i ​ n 1 / 2 ⏟ N 2 , 3 + ∑ k = 1 K 70 ​ L k ​ [ α ⁡ ( t α + n + 1 ) ] 1 / 2 λ min ​ ( 1 − γ c ) ​ ( 1 − γ ) 3 ⏟ N 2 , 4 \displaystyle+\underbrace{\sum_{k=1}^{K}\frac{6(1-(1-\gamma_{c})\lambda_{min}\alpha)^{\frac{1}{2}[J_{k}-(t_{\alpha}+n+1)]}}{(1-\gamma)^{3}(1-\gamma_{c})^{1/2}\lambda_{min}^{1/2}}}_{N_{2,3}}+\underbrace{\sum_{k=1}^{K}\frac{70L_{k}[\alpha(t_{\alpha}+n+1)]^{1/2}}{\lambda_{\min}(1-\gamma_{c})(1-\gamma)^{3}}}_{N_{2,4}} + 2 ​ γ ​ β ​ K ( 1 − γ ) 2 ⏟ N 3 + ∑ k = 1 K − 1 ‖ Q K ∗ − Q k ∗ ‖ d K ⏟ Curriculum discrepancy , \displaystyle+\underbrace{\frac{2\gamma\beta K}{(1-\gamma)^{2}}}_{N_{3}}+\underbrace{\sum_{k=1}^{K-1}\|Q^{*}_{K}-Q^{*}_{k}\|_{d_{K}}}_{\text{Curriculum discrepancy}}, where c ⁡ ( ⋅ , ⋅ ) c(\cdot,\cdot) and ρ ⁡ ( ⋅ , ⋅ ) \rho(\cdot,\cdot) are generalized importance sampling factors, L k L_{k} is the parameter defined by 1 + ( γ ​ ρ max , k ) n 1+(\gamma\rho_{\text{max},k})^{n} , ℰ approx , k := sup π ‖ Q c , ρ π k − Φ ​ w c , ρ π k ‖ ∞ \mathcal{E}_{\text{approx},k}:=\sup_{\pi}\left\|Q^{\pi_{k}}_{c,\rho}-\Phi w^{\pi_{k}}_{c,\rho}\right\|_{\infty} is the critic’s function approximation error, ℰ bias , k := max 0 ≤ t ≤ T max s ∈ 𝒮 ( 1 − λ ( s ) ) ‖ π k , t ( ⋅ | s ) − π k , b ( ⋅ | s ) ‖ 1 \mathcal{E}_{\text{bias},k}:=\max_{0\leq t\leq T}\max_{s\in\mathcal{S}}\left(1-\lambda(s)\right)\left\|\pi_{k,t}(\cdot|s)-\pi_{k,b}(\cdot|s)\right\|_{1} is the importance sampling bias at curriculum k k , ‖ Q K ∗ − Q k ∗ ‖ ∞ \|Q^{*}_{K}-Q^{*}_{k}\|_{\infty} is the gap between the curriculum subtask and the final task, and λ min \lambda_{\min} is the mininum eigenvalue of the positive definite matrix Φ T ​ 𝒦 𝒮 ​ 𝒜 ​ Φ \Phi_{T}\mathcal{K}_{\mathcal{S}\mathcal{A}}\Phi .

The terms N 1 N_{1} and N 3 N_{3} are the same as appeared in Thm. 3.1 , and together capture the error in actor update. Term N 2 , 1 N_{2,1} is the function approximation error and equals zero when using a complete basis. A term of similar form is present in all existing work studying RL with function approximation ( Munos, 2003 ; Agarwal et al., 2021 ) . Term N 2 , 2 N_{2,2} is the bias introduced by generalized importance sampling factors c ⁡ ( ⋅ , ⋅ ) c(\cdot,\cdot) and ρ ⁡ ( ⋅ , ⋅ ) \rho(\cdot,\cdot) , and N 2 , 2 = 0 N_{2,2}=0 when c ⁡ ( s , a ) = ρ ⁡ ( s , a ) = π ⁡ ( a | s ) / π b ​ ( a | s ) c(s,a)=\rho(s,a)=\pi(a|s)/\pi_{b}(a|s) . Term N 2 , 3 N_{2,3} represents the convergence bias in the critic and goes to zero geometrically fast as the inner loop iteration index J k J_{k} goes to infinity. The term N 2 , 4 N_{2,4} represents the variance in the critic and is proportional to α ​ t α = 𝒪 ⁡ ( α ​ log ⁡ ( 1 / α ) ) \sqrt{\alpha t_{\alpha}}=\mathcal{O}(\sqrt{\alpha\log(1/\alpha)}) , thus arbitrarily small given small enough stepsize α \alpha . Together, { N 2 , i } i = 1 ∼ 4 \{N_{2,i}\}_{i=1\sim 4} correspond to the second term in Thm. 3.1 . Finally, the curriculum approximation error is the same as the last term in in Theorem 3.1 . This result suggests that smoother curriculum trajectories and smaller approximation errors under finite samples, which is closely related with the stepsize and number of updates per iteration, can lead to better final policy performance.

###### Proof.

We proceed by mathematical induction across the curriculum steps. Let π k , T \pi_{k,T} denote the final policy after T T iterations at curriculum step k k . Base case: For the first curriculum step ( k = 1 k=1 ), we directly apply Theorem 4.1 from Chen and Maguluri (2025) : 𝔼 ⁡ [ ‖ Q 1 − Q π 1 , T ‖ ∞ ] \displaystyle\mathbb{E}[\|Q_{1}-Q^{\pi_{1,T}}\|_{\infty}] ≤ γ T ​ ‖ Q 1 − Q 1 π 0 ‖ ∞ \displaystyle\leq\gamma^{T}\|Q_{1}-Q^{\pi_{0}}_{1}\|_{\infty} (3) + 2 ​ γ ​ E a ​ p ​ p ​ r ​ o ​ x , 1 ( 1 − γ ) 2 + 2 ​ γ 2 ​ E b ​ i ​ a ​ s , 1 ( 1 − γ ) 4 \displaystyle+\frac{2\gamma E_{approx,1}}{(1-\gamma)^{2}}+\frac{2\gamma^{2}E_{bias,1}}{(1-\gamma)^{4}} (4) + 6 ​ ( 1 − ( 1 − γ c ) ​ λ m ​ i ​ n ​ α ) 1 2 ​ [ J 1 − ( t α + n + 1 ) ] ( 1 − γ ) 3 ​ ( 1 − γ c ) 1 / 2 ​ λ m ​ i ​ n 1 / 2 \displaystyle+\frac{6(1-(1-\gamma_{c})\lambda_{min}\alpha)^{\frac{1}{2}[J_{1}-(t_{\alpha}+n+1)]}}{(1-\gamma)^{3}(1-\gamma_{c})^{1/2}\lambda_{min}^{1/2}} (5) + 70 ​ L 1 ​ [ α ⁡ ( t α + n + 1 ) ] 1 / 2 λ m ​ i ​ n ​ ( 1 − γ c ) ​ ( 1 − γ ) 3 + 2 ​ γ ​ β ( 1 − γ ) 2 \displaystyle+\frac{70L_{1}[\alpha(t_{\alpha}+n+1)]^{1/2}}{\lambda_{min}(1-\gamma_{c})(1-\gamma)^{3}}+\frac{2\gamma\beta}{(1-\gamma)^{2}} (6) Inductive step: Assume that for curriculum step k k , we have: 𝔼 ⁡ [ ‖ Q k − Q π k , T ‖ ∞ ] \displaystyle\mathbb{E}[\|Q_{k}-Q^{\pi_{k,T}}\|_{\infty}] ≤ γ T ​ k ​ ‖ Q 1 − Q 1 π 0 ‖ ∞ \displaystyle\leq\gamma^{Tk}\|Q_{1}-Q^{\pi_{0}}_{1}\|_{\infty} + ∑ j = 1 k ( 2 ​ γ ​ E a ​ p ​ p ​ r ​ o ​ x , j ( 1 − γ ) 2 + 2 ​ γ 2 ​ E b ​ i ​ a ​ s , j ( 1 − γ ) 4 \displaystyle+\sum_{j=1}^{k}\Bigg(\frac{2\gamma E_{approx,j}}{(1-\gamma)^{2}}+\frac{2\gamma^{2}E_{bias,j}}{(1-\gamma)^{4}} + 6 ​ ( 1 − ( 1 − γ c ) ​ λ m ​ i ​ n ​ α ) 1 2 ​ [ J j − ( t α + n + 1 ) ] ( 1 − γ ) 3 ​ ( 1 − γ c ) 1 / 2 ​ λ m ​ i ​ n 1 / 2 \displaystyle+\frac{6(1-(1-\gamma_{c})\lambda_{min}\alpha)^{\frac{1}{2}[J_{j}-(t_{\alpha}+n+1)]}}{(1-\gamma)^{3}(1-\gamma_{c})^{1/2}\lambda_{min}^{1/2}} OPEN + 70 ​ L j ​ [ α ⁡ ( t α + n + 1 ) ] 1 / 2 λ m ​ i ​ n ​ ( 1 − γ c ) ​ ( 1 − γ ) 3 + 2 ​ γ ​ β ( 1 − γ ) 2 ) \displaystyle+\frac{70L_{j}[\alpha(t_{\alpha}+n+1)]^{1/2}}{\lambda_{min}(1-\gamma_{c})(1-\gamma)^{3}}+\frac{2\gamma\beta}{(1-\gamma)^{2}}\Bigg) + ∑ j = 1 k − 1 ∥ Q k − Q j ∥ d k \displaystyle+\sum_{j=1}^{k-1}\|Q_{k}-Q_{j}\|_{d_{k}} For curriculum step k + 1 k+1 , we initialize with policy π k , T \pi_{k,T} and apply Theorem 4.1 from Chen and Maguluri (2025) : 𝔼 ⁡ [ ‖ Q k + 1 − Q π k + 1 , T ‖ ∞ ] \displaystyle\mathbb{E}[\|Q_{k+1}-Q^{\pi_{k+1,T}}\|_{\infty}] ≤ γ T ​ ‖ Q k + 1 − Q k + 1 π k , T ‖ ∞ \displaystyle\leq\gamma^{T}\|Q_{k+1}-Q^{\pi_{k,T}}_{k+1}\|_{\infty} + 2 ​ γ ​ E a ​ p ​ p ​ r ​ o ​ x , k + 1 ( 1 − γ ) 2 + 2 ​ γ 2 ​ E b ​ i ​ a ​ s , k + 1 ( 1 − γ ) 4 \displaystyle+\frac{2\gamma E_{approx,k+1}}{(1-\gamma)^{2}}+\frac{2\gamma^{2}E_{bias,k+1}}{(1-\gamma)^{4}} + 6 ​ ( 1 − ( 1 − γ c ) ​ λ m ​ i ​ n ​ α ) 1 2 ​ [ J k + 1 − ( t α + n + 1 ) ] ( 1 − γ ) 3 ​ ( 1 − γ c ) 1 / 2 ​ λ m ​ i ​ n 1 / 2 \displaystyle+\frac{6(1-(1-\gamma_{c})\lambda_{min}\alpha)^{\frac{1}{2}[J_{k+1}-(t_{\alpha}+n+1)]}}{(1-\gamma)^{3}(1-\gamma_{c})^{1/2}\lambda_{min}^{1/2}} + 70 ​ L k + 1 ​ [ α ⁡ ( t α + n + 1 ) ] 1 / 2 λ m ​ i ​ n ​ ( 1 − γ c ) ​ ( 1 − γ ) 3 + 2 ​ γ ​ β ( 1 − γ ) 2 \displaystyle+\frac{70L_{k+1}[\alpha(t_{\alpha}+n+1)]^{1/2}}{\lambda_{min}(1-\gamma_{c})(1-\gamma)^{3}}+\frac{2\gamma\beta}{(1-\gamma)^{2}}\ We need to relate ‖ Q k + 1 ∗ − Q k + 1 π k , T ‖ ∞ \|Q^{*}_{k+1}-Q^{\pi_{k,T}}_{k+1}\|_{\infty} to our induction hypothesis. By triangle inequality: ‖ Q k + 1 − Q k + 1 π k , T ‖ ∞ \displaystyle\|Q_{k+1}-Q^{\pi_{k,T}}_{k+1}\|_{\infty} ≤ ‖ Q k + 1 − Q k ‖ ∞ + ‖ Q k − Q k π k , T ‖ ∞ + ‖ Q k π k , T − Q k + 1 π k , T ‖ ∞ \displaystyle\leq\|Q_{k+1}-Q_{k}\|_{\infty}+\|Q_{k}-Q^{\pi_{k,T}}_{k}\|_{\infty}+\|Q^{\pi_{k,T}}_{k}-Q^{\pi_{k,T}}_{k+1}\|_{\infty} ≤ ‖ Q k + 1 − Q k ‖ ∞ + ‖ Q k ∗ − Q k π k , T ‖ ​ ∞ + γ 1 − γ ​ ‖ r k − r k + 1 ‖ ∞ \displaystyle\leq\|Q_{k+1}-Q_{k}\|_{\infty}+\|Q^{*}_{k}-Q^{\pi_{k,T}}_{k}\|{\infty}+\frac{\gamma}{1-\gamma}\|r_{k}-r_{k+1}\|_{\infty} where the last inequality follows from the performance difference lemma with respect to rewards. For curriculum learning, we design the reward functions to satisfy ‖ r k − r k + 1 ‖ ∞ ≤ δ r \|r_{k}-r_{k+1}\|_{\infty}\leq\delta_{r} for some small δ r \delta_{r} . Thus: ‖ Q k + 1 − Q k + 1 π k , T ‖ ∞ \displaystyle\|Q_{k+1}-Q^{\pi_{k,T}}_{k+1}\|_{\infty} ≤ ‖ Q k + 1 − Q k ‖ ∞ + ‖ Q k − Q k π k , T ‖ ∞ + γ ​ δ r 1 − γ \displaystyle\leq\|Q_{k+1}-Q_{k}\|_{\infty}+\|Q_{k}-Q^{\pi_{k,T}}_{k}\|_{\infty}+\frac{\gamma\delta_{r}}{1-\gamma} Substituting our induction hypothesis: 𝔼 ⁡ [ ‖ Q k + 1 − Q π k + 1 , T ‖ ∞ ] \displaystyle\mathbb{E}[\|Q_{k+1}-Q^{\pi_{k+1,T}}\|_{\infty}] ≤ γ T ​ 𝔼 ​ [ ‖ Q k + 1 − Q k ‖ ∞ + ‖ Q k − Q k π k , T ‖ ∞ + γ ​ δ r 1 − γ ] \displaystyle\leq\gamma^{T}\mathbb{E}[\|Q_{k+1}-Q_{k}\|_{\infty}+\|Q_{k}-Q^{\pi_{k,T}}_{k}\|_{\infty}+\frac{\gamma\delta_{r}}{1-\gamma}] + 2 ​ γ ​ E a ​ p ​ p ​ r ​ o ​ x , k + 1 ( 1 − γ ) 2 + 2 ​ γ 2 ​ E b ​ i ​ a ​ s , k + 1 ( 1 − γ ) 4 \displaystyle+\frac{2\gamma E_{approx,k+1}}{(1-\gamma)^{2}}+\frac{2\gamma^{2}E_{bias,k+1}}{(1-\gamma)^{4}} + 6 ​ ( 1 − ( 1 − γ c ) ​ λ m ​ i ​ n ​ α ) 1 2 ​ [ J k + 1 − ( t α + n + 1 ) ] ( 1 − γ ) 3 ​ ( 1 − γ c ) 1 / 2 ​ λ m ​ i ​ n 1 / 2 \displaystyle+\frac{6(1-(1-\gamma_{c})\lambda_{min}\alpha)^{\frac{1}{2}[J_{k+1}-(t_{\alpha}+n+1)]}}{(1-\gamma)^{3}(1-\gamma_{c})^{1/2}\lambda_{min}^{1/2}} + 70 ​ L k + 1 ​ [ α ⁡ ( t α + n + 1 ) ] 1 / 2 λ m ​ i ​ n ​ ( 1 − γ c ) ​ ( 1 − γ ) 3 + 2 ​ γ β ​ ( 1 − γ ) 2 \displaystyle+\frac{70L_{k+1}[\alpha(t_{\alpha}+n+1)]^{1/2}}{\lambda_{min}(1-\gamma_{c})(1-\gamma)^{3}}+\frac{2\gamma}{\beta(1-\gamma)^{2}} ≤ γ T ​ ‖ Q k + 1 − Q k ‖ ∞ + γ T ​ 𝔼 ​ [ ‖ Q k ∗ − Q k π k , T ‖ ∞ ] + γ T + 1 ​ δ r 1 − γ \displaystyle\leq\gamma^{T}\|Q_{k+1}-Q_{k}\|_{\infty}+\gamma^{T}\mathbb{E}[\|Q^{*}_{k}-Q^{\pi_{k,T}}_{k}\|_{\infty}]+\frac{\gamma^{T+1}\delta_{r}}{1-\gamma} + 2 ​ γ ​ E a ​ p ​ p ​ r ​ o ​ x , k + 1 ( 1 − γ ) 2 + 2 ​ γ 2 ​ E b ​ i ​ a ​ s , k + 1 ( 1 − γ ) 4 \displaystyle+\frac{2\gamma E_{approx,k+1}}{(1-\gamma)^{2}}+\frac{2\gamma^{2}E_{bias,k+1}}{(1-\gamma)^{4}} + 6 ​ ( 1 − ( 1 − γ c ) ​ λ m ​ i ​ n ​ α ) 1 2 ​ [ J k + 1 − ( t α + n + 1 ) ] ( 1 − γ ) 3 ​ ( 1 − γ c ) 1 / 2 ​ λ m ​ i ​ n 1 / 2 \displaystyle+\frac{6(1-(1-\gamma_{c})\lambda_{min}\alpha)^{\frac{1}{2}[J_{k+1}-(t_{\alpha}+n+1)]}}{(1-\gamma)^{3}(1-\gamma_{c})^{1/2}\lambda_{min}^{1/2}} + 70 ​ L k + 1 ​ [ α ⁡ ( t α + n + 1 ) ] 1 / 2 λ m ​ i ​ n ​ ( 1 − γ c ) ​ ( 1 − γ ) 3 + 2 ​ γ ​ β ( 1 − γ ) 2 \displaystyle+\frac{70L_{k+1}[\alpha(t_{\alpha}+n+1)]^{1/2}}{\lambda_{min}(1-\gamma_{c})(1-\gamma)^{3}}+\frac{2\gamma\beta}{(1-\gamma)^{2}} Applying the induction hypothesis: 𝔼 ⁡ [ ‖ Q k + 1 − Q π k + 1 , T ‖ ∞ ] \displaystyle\mathbb{E}[\|Q_{k+1}-Q^{\pi_{k+1,T}}\|_{\infty}] ≤ γ T ​ ‖ Q k + 1 − Q k ‖ ∞ + γ T ⋅ γ T ​ k ​ ‖ Q 1 − Q 1 π 0 ‖ ∞ \displaystyle\leq\gamma^{T}\|Q_{k+1}-Q_{k}\|_{\infty}+\gamma^{T}\cdot\gamma^{Tk}\|Q_{1}-Q^{\pi_{0}}_{1}\|_{\infty} + γ T ∑ j = 1 k ( 2 ​ γ ​ E a ​ p ​ p ​ r ​ o ​ x , j ( 1 − γ ) 2 + 2 ​ γ 2 ​ E b ​ i ​ a ​ s , j ( 1 − γ ) 4 \displaystyle+\gamma^{T}\sum_{j=1}^{k}\Bigg(\frac{2\gamma E_{approx,j}}{(1-\gamma)^{2}}+\frac{2\gamma^{2}E_{bias,j}}{(1-\gamma)^{4}} + 6 ​ ( 1 − ( 1 − γ c ) ​ λ m ​ i ​ n ​ α ) 1 2 ​ [ J j − ( t α + n + 1 ) ] ( 1 − γ ) 3 ​ ( 1 − γ c ) 1 / 2 ​ λ m ​ i ​ n 1 / 2 \displaystyle+\frac{6(1-(1-\gamma_{c})\lambda_{min}\alpha)^{\frac{1}{2}[J_{j}-(t_{\alpha}+n+1)]}}{(1-\gamma)^{3}(1-\gamma_{c})^{1/2}\lambda_{min}^{1/2}} OPEN + 70 ​ L j ​ [ α ⁡ ( t α + n + 1 ) ] 1 / 2 λ m ​ i ​ n ​ ( 1 − γ c ) ​ ( 1 − γ ) 3 + 2 ​ γ β ​ ( 1 − γ ) 2 ) \displaystyle+\frac{70L_{j}[\alpha(t_{\alpha}+n+1)]^{1/2}}{\lambda_{min}(1-\gamma_{c})(1-\gamma)^{3}}+\frac{2\gamma}{\beta(1-\gamma)^{2}}\Bigg) + γ T ∑ j = 1 k − 1 ∥ Q k − Q j ∥ d k + γ T + 1 ​ δ r 1 − γ \displaystyle+\gamma^{T}\sum_{j=1}^{k-1}\|Q_{k}-Q_{j}\|_{d_{k}}+\frac{\gamma^{T+1}\delta_{r}}{1-\gamma} + 2 ​ γ ​ E a ​ p ​ p ​ r ​ o ​ x , k + 1 ( 1 − γ ) 2 + 2 ​ γ 2 ​ E b ​ i ​ a ​ s , k + 1 ( 1 − γ ) 4 \displaystyle+\frac{2\gamma E_{approx,k+1}}{(1-\gamma)^{2}}+\frac{2\gamma^{2}E_{bias,k+1}}{(1-\gamma)^{4}} + 6 ​ ( 1 − ( 1 − γ c ) ​ λ m ​ i ​ n ​ α ) 1 2 ​ [ J k + 1 − ( t α + n + 1 ) ] ( 1 − γ ) 3 ​ ( 1 − γ c ) 1 / 2 ​ λ m ​ i ​ n 1 / 2 \displaystyle+\frac{6(1-(1-\gamma_{c})\lambda_{min}\alpha)^{\frac{1}{2}[J_{k+1}-(t_{\alpha}+n+1)]}}{(1-\gamma)^{3}(1-\gamma_{c})^{1/2}\lambda_{min}^{1/2}} + 70 ​ L k + 1 ​ [ α ⁡ ( t α + n + 1 ) ] 1 / 2 λ m ​ i ​ n ​ ( 1 − γ c ) ​ ( 1 − γ ) 3 + 2 ​ γ ​ β ( 1 − γ ) 2 \displaystyle+\frac{70L_{k+1}[\alpha(t_{\alpha}+n+1)]^{1/2}}{\lambda_{min}(1-\gamma_{c})(1-\gamma)^{3}}+\frac{2\gamma\beta}{(1-\gamma)^{2}} Simplifying: 𝔼 ⁡ [ ‖ Q k + 1 − Q π k + 1 , T ‖ ∞ ] \displaystyle\mathbb{E}[\|Q_{k+1}-Q^{\pi_{k+1,T}}\|_{\infty}] ≤ γ T ⁡ ( k + 1 ) ​ ‖ Q 1 − Q 1 π 0 ‖ ∞ \displaystyle\leq\gamma^{T(k+1)}\|Q_{1}-Q^{\pi_{0}}_{1}\|_{\infty} + ∑ j = 1 k + 1 ( 2 ​ γ ​ E a ​ p ​ p ​ r ​ o ​ x , j ( 1 − γ ) 2 + 2 ​ γ 2 ​ E b ​ i ​ a ​ s , j ( 1 − γ ) 4 \displaystyle+\sum_{j=1}^{k+1}\Bigg(\frac{2\gamma E_{approx,j}}{(1-\gamma)^{2}}+\frac{2\gamma^{2}E_{bias,j}}{(1-\gamma)^{4}} + 6 ​ ( 1 − ( 1 − γ c ) ​ λ m ​ i ​ n ​ α ) 1 2 ​ [ J j − ( t α + n + 1 ) ] ( 1 − γ ) 3 ​ ( 1 − γ c ) 1 / 2 ​ λ m ​ i ​ n 1 / 2 \displaystyle+\frac{6(1-(1-\gamma_{c})\lambda_{min}\alpha)^{\frac{1}{2}[J_{j}-(t_{\alpha}+n+1)]}}{(1-\gamma)^{3}(1-\gamma_{c})^{1/2}\lambda_{min}^{1/2}} OPEN + 70 ​ L j ​ [ α ⁡ ( t α + n + 1 ) ] 1 / 2 λ m ​ i ​ n ​ ( 1 − γ c ) ​ ( 1 − γ ) 3 + 2 ​ γ ​ β ( 1 − γ ) 2 ) \displaystyle+\frac{70L_{j}[\alpha(t_{\alpha}+n+1)]^{1/2}}{\lambda_{min}(1-\gamma_{c})(1-\gamma)^{3}}+\frac{2\gamma\beta}{(1-\gamma)^{2}}\Bigg) + ∑ j = 1 k ∥ Q k + 1 − Q j ∥ d k + 1 \displaystyle+\sum_{j=1}^{k}\|Q_{k+1}-Q_{j}\|_{d_{k+1}} where we used the fact that ‖ Q k + 1 ∗ − Q k ∗ ‖ d k + 1 + γ T ​ ∑ j = 1 k − 1 ‖ Q k ∗ − Q j ∗ ‖ d k ≤ ∑ j = 1 k ‖ Q k + 1 ∗ − Q j ∗ ‖ d k + 1 \displaystyle\|Q^{*}_{k+1}-Q^{*}_{k}\|_{d_{k+1}}+\gamma^{T}\sum_{j=1}^{k-1}\|Q^{*}_{k}-Q^{*}_{j}\|_{d_{k}}\leq\sum_{j=1}^{k}\|Q^{*}_{k+1}-Q^{*}_{j}\|_{d_{k+1}} (7) due to the triangle inequality and the curriculum design. By induction, for the final curriculum step K K , we have: 𝔼 ⁡ [ ‖ Q K − Q π K ‖ ∞ ] \displaystyle\mathbb{E}[\|Q_{K}-Q^{\pi_{K}}\|_{\infty}] ≤ γ T ​ K ​ ‖ Q 1 − Q 1 π 0 ‖ ∞ \displaystyle\leq\gamma^{TK}\|Q_{1}-Q^{\pi_{0}}_{1}\|_{\infty} + ∑ k = 1 K ( 2 ​ γ ​ E a ​ p ​ p ​ r ​ o ​ x , k ( 1 − γ ) 2 + 2 ​ γ 2 ​ E b ​ i ​ a ​ s , k ( 1 − γ ) 4 \displaystyle+\sum_{k=1}^{K}\Bigg(\frac{2\gamma E_{approx,k}}{(1-\gamma)^{2}}+\frac{2\gamma^{2}E_{bias,k}}{(1-\gamma)^{4}} + 6 ​ ( 1 − ( 1 − γ c ) ​ λ m ​ i ​ n ​ α ) 1 2 ​ [ J k − ( t α + n + 1 ) ] ( 1 − γ ) 3 ​ ( 1 − γ c ) 1 / 2 ​ λ m ​ i ​ n 1 / 2 \displaystyle+\frac{6(1-(1-\gamma_{c})\lambda_{min}\alpha)^{\frac{1}{2}[J_{k}-(t_{\alpha}+n+1)]}}{(1-\gamma)^{3}(1-\gamma_{c})^{1/2}\lambda_{min}^{1/2}} OPEN + 70 ​ L k ​ [ α ⁡ ( t α + n + 1 ) ] 1 / 2 λ m ​ i ​ n ​ ( 1 − γ c ) ​ ( 1 − γ ) 3 + 2 ​ γ ​ β ( 1 − γ ) 2 ) \displaystyle+\frac{70L_{k}[\alpha(t_{\alpha}+n+1)]^{1/2}}{\lamb