Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 4 - Reinforcement Learning with LLMs

For more information about Stanford’s graduate programs, visit: https://stanford.io/grad-ed-cme295-A2026 To follow along with the course schedule and syllabus, visit: https://cme295.stanford.edu/syllabus/ Chapters: 00:00:00 Introduction 00:06:35 RL setup 00:09:32 Reward design 00:12:23 Policy gradient 00:17:12 Derivation of the gradient 00:23:50 REINFORCE 00:25:51 Importance sampling ratio 00:33:22 Challenges of policy gradients 00:40:34 Proximal Policy Optimization (PPO) 00:42:49 Value function and advantage 00:51:16 Policy ratio 00:53:03 Ratio clipping 00:57:45 KL Penalty 00:59:48 Connection with DPO 01:07:29 Group Relative Policy Optimization (GRPO) 01:09:42 Comparison of GRPO and PPO 01:18:25 Limitations of GRPO 01:22:41 DAPO, Dr. GRPO 01:25:17 On-Policy Distillation (OPD) 01:39:32 Self (OPSD) and Multi-Teacher (OPD) variants Afshine Amidi is an Adjunct Professor at Stanford University. Shervine Amidi is an Adjunct Professor at Stanford University. Follow the playlist: https://youtube.com/playlist?list=PLakGfbKCei80&si=HF2tNywj7nXIQq8X