For SALE: State-Action Representation Learning for Deep Reinforcement Learning — part 3/4

Cleaned-text code points [26194, 42194). Bibliography omitted here; source.md includes it.

Offline RL. To make TD7 amenable to the offline RL setting, we add a behavior cloning loss to the policy update ( DPG ) , inspired by TD3+BC ( fujimoto2021minimalist ) : π ≈ argmax π 𝔼 ( s , a ) ∼ D [ Q ( s , π ( s ) ) − λ | 𝔼 s ∼ D [ Q ( s , π ( s ) ) | × ( π ( s ) − a ) 2 T h e s a m e l o s s f u n c t i o n i s u s e d f o r b o t h o f f l i n e a n d o n l i n e R L , w h e r e λ = 0 f o r t h e o n l i n e s e t t i n g . | ⋅ | × d e n o t e s t h e s t o p − g r a d i e n t o p e r a t i o n . U n l i k e T D 3 + B C , w e d o n o t n o r m a l i z e t h e s t a t e v e c t o r s . C h e c k p o i n t s a r e n o t u s e d i n t h e o f f l i n e s e t t i n g , a s t h e r e i s n o i n t e r a c t i o n w i t h t h e e n v i r o n m e n t . B o t h t h e v a l u e f u n c t i o n a n d p o l i c y u s e t h e S A L E e m b e d d i n g s a s i n p u t , w h i c h w e o m i t f r o m t h e e q u a t i o n s a b o v e f o r s i m p l i c i t y . P s e u d o c o d e f o r T D 7 i s d e s c r i b e d i n . Algorithm 1 Algorithm 1 1 Algorithm 1 1Online TD7 Algorithm 1 Online TD7 1: 1 1 Initialize: ⊳ Before training 2: 2 2 ⋅ Policy π + t 1 , value function Q + t 1 , encoders ( f + t 1 , g + t 1 ) . 3: 3 3 ⋅ Target policy π t , target value function Q t , fixed encoders ( f t , g t ) , target fixed encoders ( f - t 1 , g - t 1 ) . 4: 4 4 ⋅ Checkpoint policy π c , checkpoint encoder f c . 5: 5 5 for = episode 1 to final_episode do ⊳ Data collection 6: 6 6 Using current policy π + t 1 , collect transitions and store in the LAP replay buffer. 7: 7 7 if checkpoint_condition then ⊳ Checkpointing 8: 8 8 if actor π + t 1 outperforms checkpoint policy π c then 9: 9 9 Update checkpoint networks ← π c π + t 1 , ← f c f t . 10: 10 10 for = i 1 to timesteps_since_training do ⊳ Training 11: 11 11 Sample transitions from LAP replay buffer (Equation 10). 12: 12 12 Train encoder (Equation 2), value function (Equations 6 & 9), and policy (Section 5.2). 13: 13 13 if target_update_frequency steps have passed then 14: 14 14 Update target networks (Equation 8). 15: 15 15 ⊳ Detailed hyperparameter explanations found in the Appendix. 6 Section 6 6 §6 6Results I n t h i s s e c t i o n , w e e v a l u a t e t h e p e r f o r m a n c e o f T D 7 i n b o t h t h e o n l i n e a n d o f f l i n e r e g i m e s . A d e t a i l e d d e s c r i p t i o n o f t h e e x p e r i m e n t a l s e t u p , b a s e l i n e s , a n d h y p e r p a r a m e t e r s c a n b e f o u n d i n t h e Appendix , a l o n g w i t h a d d i t i o n a l l e a r n i n g c u r v e s a n d a b l a t i o n s t u d i e s . Online. U s i n g O p e n A I g y m ( OpenAIGym ) , w e b e n c h m a r k T D 7 a g a i n s t T D 3 ( fujimoto2018addressing ) , S A C ( haarnoja2018soft ) , T Q C ( kuznetsov2020controlling ) , a n d T D 3 + O F E ( ota2020can ) o n t h e M u J o C o e n v i r o n m e n t s ( mujoco ) . S A C a n d T D 3 + O F E r e s u l t s a r e f r o m r e − i m p l e m e n t a t i o n s b a s e d o n a u t h o r d e s c r i p t i o n s ( haarnoja2018soft ; ota2020can ) . T D 3 a n d T Q C r e s u l t s u s e a u t h o r − p r o v i d e d c o d e ( fujimoto2018addressing ; kuznetsov2020controlling ) , w i t h a c o n s i s t e n t e v a l u a t i o n p r o t o c o l f o r a l l m e t h o d s . L e a r n i n g c u r v e s a r e d i s p l a y e d i n a n d f i n a l a n d i n t e r m e d i a t e r e s u l t s a r e l i s t e d i n . 0 1 2 3 4 5 0 4 8 12 16 20 Total Reward (1k) pHalfCheetahp 0 1 2 3 4 5 0 01 2 3 4 Hopper 0 1 2 3 4 5 0 02 4 6 8 pWalker2dp 0 1 2 3 4 5 0 3 6 9 12 Time steps (1M) Total Reward (1k) pAntp 0 1 2 3 4 5 0 2 4 6 8 10 Time steps (1M) pHumanoidp TD7 TD3 SAC TQC TD3+OFE Figure 4 Figure 4 4 Figure 4 4Learning curves on the MuJoCo benchmark. Results are averaged over 10 seeds. The shaded area captures a 95% confidence interval around the average performance. Figure 4Learning curves on the MuJoCo benchmark. Results are averaged over 10 seeds. The shaded area captures a 95% confidence interval around the average performance. A l t h o u g h T Q C a n d T D 3 + O F E u s e p e r − e n v i r o n m e n t h y p e r p a r a m e t e r s a l o n g w i t h l a r g e r a n d m o r e c o m p u t a t i o n a l l y e x p e n s i v e a r c h i t e c t u r e s , T D 7 o u t p e r f o r m s t h e s e b a s e l i n e s s i g n i f i c a n t l y i n t e r m s o f b o t h e a r l y ( 300 k t i m e s t e p s ) a n d f i n a l p e r f o r m a n c e ( 5 M t i m e s t e p s ) . A t 300 k t i m e s t e p s , T D 7 o f t e n s u r p a s s e s t h e p e r f o r m a n c e o f T D 3 a t 5 M t i m e s t e p s , h i g h l i g h t i n g t h e c o n s i d e r a b l e p e r f o r m a n c e g a i n s . Table 1 Table 1 1 Table 1 1Average performance on the MuJoCo benchmark at 300k, 1M, and 5M time steps, over 10 trials, where ± captures a 95% confidence interval. The highest performance is highlighted. Any performance which is not statistically significantly worse than the highest performance (according to a Welch’s t -test with significance level 0.05 ) is highlighted. Table 1Average performance on the MuJoCo benchmark at 300k, 1M, and 5M time steps, over 10 trials, where ± captures a 95% confidence interval. The highest performance is highlighted. Any performance which is not statistically significantly worse than the highest performance (according to a Welch’s t -test with significance level 0.05 ) is highlighted. Environment Time step TD3 SAC TQC TD3+OFE TD7 HalfCheetah 300k 07715 ± 6330 08052 ± 5150 07006 ± 8910 11294 ± 2470 15031 ± 4010 1M 10574 ± 8970 10484 ± 6590 12349 ± 8780 13758 ± 5440 17434 ± 1550 5M 14337 ± 1491 15526 ± 6970 17459 ± 2580 16596 ± 1640 18165 ± 2550 Hopper 300k 01289 ± 7680 02370 ± 6260 03251 ± 4610 01581 ± 6820 02948 ± 4640 1M 03226 ± 3150 02785 ± 6340 03526 ± 2440 03121 ± 5060 03512 ± 3150 5M 03682 ± 8300 03167 ± 4850 03462 ± 8180 03423 ± 5840 04075 ± 2250 Walker2d 300k 01101 ± 3860 01989 ± 5000 02812 ± 8380 04018 ± 5700 05379 ± 3280 1M 03946 ± 2920 04314 ± 2560 05321 ± 3220 05195 ± 5120 06097 ± 5700 5M 05078 ± 3430 05681 ± 3290 06137 ± 1194 06379 ± 3320 07397 ± 4540 Ant 300k 01704 ± 6550 01478 ± 3540 01830 ± 5720 06348 ± 4410 06171 ± 8310 1M 03942 ± 1030 03681 ± 5060 03582 ± 1093 07398 ± 1180 08509 ± 4220 5M 05589 ± 7580 04615 ± 2022 06329 ± 1510 08547 ± 8400 10133 ± 9660 Humanoid 300k 01344 ± 3650 01997 ± 4830 03117 ± 9100 03181 ± 7710 05332 ± 7140 1M 05165 ± 1450 04909 ± 3640 06029 ± 5310 06032 ± 3340 07429 ± 1530 5M 05433 ± 2450 06555 ± 2790 08361 ± 1364 08951 ± 2460 10281 ± 5880 Table 2 Table 2 2 Table 2 2Average final performance on the D4RL benchmark after training for 1M time steps. over 10 trials, where ± captures a 95% confidence interval. The highest performance is highlighted. Any performance which is not statistically significantly worse than the highest performance (according to a Welch’s t -test with significance level 0.05 ) is highlighted. Table 2Average final performance on the D4RL benchmark after training for 1M time steps. over 10 trials, where ± captures a 95% confidence interval. The highest performance is highlighted. Any performance which is not statistically significantly worse than the highest performance (according to a Welch’s t -test with significance level 0.05 ) is highlighted. Environment Dataset CQL TD3+BC IQL X -QL TD7 HalfCheetah Medium 046.7 ± 0.30 048.1 ± 0.10 047.4 ± 0.20 047.4 ± 0.10 058.0 ± 0.40 Medium-Replay 045.5 ± 0.30 044.6 ± 0.40 043.9 ± 1.30 044.2 ± 0.70 053.8 ± 0.80 Medium-Expert 076.8 ± 7.40 093.7 ± 0.90 089.6 ± 3.50 090.2 ± 2.70 104.6 ± 1.60 Hopper Medium 059.3 ± 3.30 059.1 ± 3.00 063.9 ± 4.90 067.7 ± 3.60 076.1 ± 5.10 Medium-Replay 078.8 ± 10.9 052.0 ± 10.6 093.4 ± 7.80 082.0 ± 14.9 091.1 ± 8.00 Medium-Expert 079.9 ± 19.8 098.1 ± 10.7 064.2 ± 32.0 092.0 ± 10.0 108.2 ± 4.80 Walker2d Medium 081.4 ± 1.70 084.3 ± 0.80 084.2 ± 1.60 079.2 ± 4.00 091.1 ± 7.80 Medium-Replay 079.9 ± 3.60 081.0 ± 3.40 071.2 ± 8.30 061.8 ± 7.70 089.7 ± 4.70 Medium-Expert 108.5 ± 1.20 110.5 ± 0.40 108.9 ± 1.40 110.3 ± 0.20 111.8 ± 0.60 Total 656.7 ± 24.3 671.3 ± 15.7 666.7 ± 34.6 674.9 ± 20.4 784.4 ± 14.1 Offline. W e b e n c h m a r k T D 7 a g a i n s t C Q L ( kumar2020conservative ) , T D 3 + B C ( fujimoto2021minimalist ) , I Q L ( kostrikov2021offline ) a n d 𝒳 − Q L ( garg2023extreme ) u s i n g t h e M u J o C o d a t a s e t s i n D 4 R L ( mujoco ; fu2021benchmarks ) . W h i l e t h e r e a r e m e t h o d s t h a t u s e p e r − d a t a s e t h y p e r p a r a m e t e r s t o a t t a i n h i g h e r t o t a l r e s u l t s , w e o m i t t h e s e m e t h o d s b e c a u s e i t m a k e s i t d i f f i c u l t t o d i r e c t l y c o m p a r e . B a s e l i n e r e s u l t s a r e o b t a i n e d b y r e − r u n n i n g a u t h o r − p r o v i d e d c o d e w i t h a s i n g l e s e t o f h y p e r p a r a m e t e r s a n d a c o n s i s t e n t e v a l u a t i o n p r o t o c o l . F i n a l p e r f o r m a n c e i s r e p o r t e d i n . T D 7 o u t p e r f o r m s a l l b a s e l i n e s . S i n c e T D 7 a n d T D 3 + B C e m p l o y t h e s a m e a p p r o a c h t o o f f l i n e R L , t h e s i g n i f i c a n t p e r f o r m a n c e g a p h i g h l i g h t s t h e e f f e c t i v e n e s s o f S A L E i n t h e o f f l i n e s e t t i n g . Ablation study . I n w e r e p o r t t h e r e s u l t s o f a n a b l a t i o n s t u d y o v e r t h e c o m p o n e n t s o f T D 7 ( S A L E , c h e c k p o i n t s , L A P ) . T h e i n t e r a c t i o n b e t w e e n c o m p o n e n t s i s e x p l o r e d f u r t h e r i n . Run time. T o u n d e r s t a n d t h e c o m p u t a t i o n a l c o s t o f u s i n g S A L E a n d t h e T D 7 a l g o r i t h m , w e b e n c h m a r k t h e r u n t i m e o f e a c h o f t h e o n l i n e b a s e l i n e s w i t h i d e n t i c a l c o m p u t a t i o n a l r e s o u r c e s a n d d e e p l e a r n i n g f r a m e w o r k . T h e r e s u l t s a r e r e p o r t e d i n . 0 1 2 3 4 5 0 0.2 0.4 0.6 0.8 1.0 Time steps (1M) Normalized Performance Ablation TD7 TD3 No SALE No checkpoints No LAP Figure 5 Figure 5 5 Figure 5 5Ablation study over the components of TD7. The y-axis corresponds to the average performance over all five MuJoCo tasks, normalized with respect to the performance of TD7 at 5M time steps. The shaded area captures a 95% confidence interval. Figure 5Ablation study over the components of TD7. The y-axis corresponds to the average performance over all five MuJoCo tasks, normalized with respect to the performance of TD7 at 5M time steps. The shaded area captures a 95% confidence interval. TD3 SAC TQC TD3+OFE TD7 1 2 3 4 47m 1h 28m 3h 50m 3h 14m 1h 50m Hours Run Time Figure 6 Figure 6 6 Figure 6 6Run time of each method for 1M time steps on the HalfCheetah environment, using the same hardware and deep learning framework (PyTorch (paszke2019pytorch)). Figure 6Run time of each method for 1M time steps on the HalfCheetah environment, using the same hardware and deep learning framework (PyTorch (paszke2019pytorch)). 7 Section 7 7 §7 7Conclusion R e p r e s e n t a t i o n l e a r n i n g h a s b e e n t y p i c a l l y r e s e r v e d f o r i m a g e − b a s e d t a s k s , w h e r e t h e o b s e r v a t i o n s a r e l a r g e a n d u n s t r u c t u r e d . H o w e v e r , b y l e a r n i n g e m b e d d i n g s w h i c h c o n s i d e r t h e i n t e r a c t i o n b e t w e e n s t a t e a n d a c t i o n , w e m a k e r e p r e s e n t a t i o n l e a r n i n g m o r e b r o a d l y a p p l i c a b l e t o l o w − l e v e l s t a t e s . W e i n t r o d u c e S A L E , a m e t h o d f o r l e a r n i n g s t a t e − a c t i o n e m b e d d i n g s b y c o n s i d e r i n g a l a t e n t s p a c e d y n a m i c s m o d e l . T h r o u g h a n e x t e n s i v e e m p i r i c a l e v a l u a t i o n , w e i n v e s t i g a t e v a r i o u s d e s i g n c h o i c e s i n S A L E . W e h i g h l i g h t t h e r i s k o f e x t r a p o l a t i o n e r r o r ( fujimoto2019off ) d u e t o t h e i n c r e a s e i n i n p u t d i m e n s i o n s f r o m u s i n g s t a t e − a c t i o n e m b e d d i n g s , b u t s h o w t h i s i n s t a b i l i t y c a n b e c o r r e c t e d b y c l i p p i n g t h e t a r g e t w i t h s e e n v a l u e s . W e f u r t h e r i n t r o d u c e s t a b i l i t y b y i n c l u d i n g p o l i c y c h e c k p o i n t s . W h i l e b o t h S A L E a n d p o l i c y c h e c k p o i n t s a r e g e n e r a l − p u r p o s e t e c h n i q u e s t h a t c a n b e i n c l u d e d w i t h m o s t R L m e t h o d s , w e c o m b i n e t h e m w i t h T D 3 a n d s e v e r a l o t h e r r e c e n t i m p r o v e m e n t s ( fujimoto2020equivalence ; fujimoto2021minimalist ) t o i n t r o d u c e t h e T D 7 a l g o r i t h m . W e f i n d o u r T D 7 a l g o r i t h m i s a b l e t o m a t c h t h e p e r f o r m a n c e o f e x p e n s i v e o f f l i n e a l g o r i t h m s a n d s i g n i f i c a n t l y o u t p e r f o r m t h e s t a t e − o f − t h e − a r t c o n t i n u o u s c o n t r o l m e t h o d s i n b o t h f i n a l p e r f o r m a n c e a n d e a r l y l e a r n i n g . AcknowledgmentsandDisclosureofFunding T h i s r e s e a r c h w a s e n a b l e d i n p a r t b y s u p p o r t p r o v i d e d b y C a l c u l Q u é b e c a n d t h e D i g i t a l R e s e a r c h A l l i a n c e o f C a n a d a . References 1 2019 Anandetal. Anand,Racah,Ozair,Bengio,Côté,andHjelm Anandetal.(2019) anand2019unsupervised A n k e s h A n a n d , E v a n R a c a h , S h e r j i l O z a i r , Y o s h u a B e n g i o , M a r c − A l e x a n d r e C ô t é , a n d R D e v o n H j e l m . U n s u p e r v i s e d s t a t e r e p r e s e n t a t i o n l e a r n i n g i n a t a r i . A d v a n c e s i n n e u r a l i n f o r m a t i o n p r o c e s s i n g s y s t e m s , 32 , 2019 . 2 2016 Baetal. Ba,Kiros,andHinton Baetal.(2016) ba2016layer J i m m y L e i B a , J a m i e R y a n K i r o s , a n d G e o f f r e y E H i n t o n . L a y e r n o r m a l i z a t i o n . a r X i v p r e p r i n t a r X i v : 1607.06450 , 2016 . 3 2016 Brockmanetal. Brockman,Cheung,Pettersson,Schneider,Schulman,Tang,andZaremba Brockmanetal.(2016) OpenAIGym G r e g B r o c k m a n , V i c k i C h e u n g , L u d w i g P e t t e r s s o n , J o n a s S c h n e i d e r , J o h n S c h u l m a n , J i e T a n g , a n d W o j c i e c h Z a r e m b a . O p e n a i g y m , 2016 . 4 2022 Cetinetal. Cetin,Ball,Roberts,andCeliktutan Cetinetal.(2022) cetin2022stabilizing E d o a r d o C e t i n , P h i l i p J B a l l , S t e p h e n R o b e r t s , a n d O y a C e l i k t u t a n . S t a b i l i z i n g o f f − p o l i c y d e e p r e i n f o r c e m e n t l e a r n i n g f r o m p i x e l s . I n I n t e r n a t i o n a l C o n f e r e n c e o n M a c h i n e L e a r n i n g , p a g e s 2784 − − 2810 . P M L R , 2022 . 5 2019 Chandaketal. Chandak,Theocharous,Kostas,Jordan,andThomas Chandaketal.(2019) chandak2019learning Y a s h C h a n d a k , G e o r g i o s T h e o c h a r o u s , J a m e s K o s t a s , S c o t t J o r d a n , a n d P h i l i p T h o m a s . L e a r n i n g a c t i o n r e p r e s e n t a t i o n s f o r r e i n f o r c e m e n t l e a r n i n g . I n I n t e r n a t i o n a l c o n f e r e n c e o n m a c h i n e l e a r n i n g , p a g e s 941 − − 950 . P M L R , 2019 . 6 2015 Clevertetal. Clevert,Unterthiner,andHochreiter Clevertetal.(2015) clevert2015fast D j o r k − A r n é C l e v e r t , T h o m a s U n t e r t h i n e r , a n d S e p p H o c h r e i t e r . F a s t a n d a c c u r a t e d e e p n e t w o r k l e a r n i n g b y e x p o n e n t i a l l i n e a r u n i t s ( e l u s ) . a r X i v p r e p r i n t a r X i v : 1511.07289 , 2015 . 7 1993 Dayan Dayan(1993) dayan1993improving P e t e r D a y a n . I m p r o v i n g g e n e r a l i z a t i o n f o r t e m p o r a l d i f f e r e n c e l e a r n i n g : T h e s u c c e s s o r r e p r e