Deep RL from Human Preferences
Christiano et al. trained agents from human preferences instead of hand-written rewards — the 2017 result that made RLHF, and ChatGPT, possible.
5 steps · about 41 min · 2017–2023
How raw next-token predictors became assistants: reinforcement learning from human preferences, instruction tuning, constitutional AI, direct preference optimization and self-critique loops.
Prefer to jump around? Each step is a standalone report with its own citations, figures and FAQ.
Christiano et al. trained agents from human preferences instead of hand-written rewards — the 2017 result that made RLHF, and ChatGPT, possible.
An AI-generated review of InstructGPT, the paper that introduced instruction fine-tuning with human feedback (RLHF) and defined how aligned LLMs are built.
Constitutional AI trains harmless assistants from a written constitution via AI self-critique, cutting human labels while keeping helpfulness.
Direct Preference Optimization aligns language models from preference pairs with one stable loss — no reward model, no reinforcement learning loop.
A synthetic proposal combining structured self-critique with episodic memory so autonomous language agents catch and correct their own errors.
Narrated films that cover the same ground, with chapters, captions and transcripts.
8:53Essay film · 10 chapters
Five common arguments against AI, taken seriously one at a time and tested against history and published data — plus what the critics get right.