Courses, Stanford Online

Stanford CS329H: Machine Learning from Human Preferences I Autumn 2024

8 lectures, 10 hr 16 min in all

Machine learning from human preferences investigates mechanisms for capturing human and societal preferences and values in artificial intelligence (AI) systems and applications, e.g., for socio-technical applications such as algorithmic fairness and many language and robotics tasks when reward functions are otherwise challenging to specify quantitatively. While learning from human preferences has emerged as an increasingly important component of modern AI, e.g., credited with advancing the state of the art in language modeling and reinforcement learning, existing approaches are largely reinvented independently in each subfield, with limited connections drawn among them. This course will cover the foundations of learning from human preferences from first principles and outline connections to the growing literature on the topic. This includes but is not limited to: - Inverse reinforcement learning, which uses human preferences to specify the reinforcement learning reward function - Metric elicitation, which uses human preferences to specify tradeoffs for cost-sensitive classification - Reinforcement learning from human feedback, where human preferences are used to align a pre-trained language model This is a graduate-level course. By the end of the course, students should be able to understand and implement state-of-the-art learning from human feedback and be ready to research these topics. Given how fast this area is growing, this course will consist of weekly lectures, presentations, and discussions of papers led by students. Students will also complete a final course project. If you are a CS PhD student at Stanford, this course is counted toward the breadth requirement for "Learning and Modeling" or "Human and Society". https://online.stanford.edu/courses/cs329h-machine-learning-human-preferences