Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 5 - LLM tuning

For more information about Stanford’s graduate programs, visit: https://online.stanford.edu/graduate-education October 31, 2025 This lecture covers: • Preference tuning • RLHF overview • Reward modeling • RL approaches (PPO and variants) • DPO To follow along with the course schedule and syllabus, visit: https://cme295.stanford.edu/syllabus/ Chapters: 00:00:00 Introduction 00:04:50 Preference tuning 00:11:31 Data collection 00:17:43 RLHF overview 00:28:24 Reward model 00:29:46 Bradley-Terry formulation 00:46:02 Reinforcement learning 00:53:54 PPO 00:57:56 Advantage, value function 01:03:39 PPO variants: Clip, KL-penalty 01:16:20 Challenges 01:19:58 On-policy vs off-policy 01:22:43 Best-of-N 01:29:47 DPO Afshine Amidi is an Adjunct Lecturer at Stanford University. Shervine Amidi is an Adjunct Lecturer at Stanford University. View the course playlist: https://www.youtube.com/playlist?list=PLoROMvodv4rOCXd21gf0CF4xr35yINeOy