Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 2 - Large Language Models

For more information about Stanford’s graduate programs, visit: https://stanford.io/grad-ed-cme295-A2026 To follow along with the course schedule and syllabus, visit: https://cme295.stanford.edu/syllabus/ Chapters: 00:00:00 Introduction 00:04:47 BERT and encoder-only models 00:13:47 GPT and decoder-only models 00:16:10 Encoder-decoder versus decoder-only 00:19:23 Scaling laws 00:26:49 Large Language Model definition 00:29:05 Mixture of experts (MoE) 00:33:11 MoE-based LLMs 00:36:03 Load balancing loss 00:45:36 Position embeddings 01:01:58 Rotary position embeddings (RoPE) 01:09:28 Sliding Window Attention (SWA) 01:13:15 Group Query Attention (GQA) 01:18:13 Pre-Norm versus Post-Norm 01:19:41 LayerNorm versus RMSNorm 01:21:09 Next token prediction 01:35:22 Prompting 01:38:38 Context rot 01:41:42 Chain of Thought (CoT) Afshine Amidi is an Adjunct Professor at Stanford University. Shervine Amidi is an Adjunct Professor at Stanford University. Follow the playlist: https://youtube.com/playlist?list=PLakGfbKCei80&si=HF2tNywj7nXIQq8X