5 steps · about 41 min · 2017–2023

Aligning models with humans

How raw next-token predictors became assistants: reinforcement learning from human preferences, instruction tuning, constitutional AI, direct preference optimization and self-critique loops.

Fig. 02 / Reading route5 papers, in order
Reading path: Deep RL from Human Preferences; InstructGPT: Training Language Models to Follow Instructions with Human Feedback; Constitutional AI; Direct Preference Optimization; Reflective Agents: Structured Self-Critique with Episodic Memory0102030405
Start with Deep RL from Human Preferences. Follow the numbered steps to Reflective Agents: Structured Self-Critique with Episodic Memory.
  1. Deep RL from Human Preferences 2017 · 5 min read · Intermediate — Christiano et al. trained agents from human preferences instead of hand-written rewards — the 2017 result that made RLHF, and ChatGPT, possible.
  2. InstructGPT: Training Language Models to Follow Instructions with Human Feedback 2022 · 8 min read · Intermediate — An AI-generated review of InstructGPT, the paper that introduced instruction fine-tuning with human feedback (RLHF) and defined how aligned LLMs are built.
  3. Constitutional AI 2022 · 5 min read · Intermediate — Constitutional AI trains harmless assistants from a written constitution via AI self-critique, cutting human labels while keeping helpfulness.
  4. Direct Preference Optimization 2023 · 5 min read · Intermediate — Direct Preference Optimization aligns language models from preference pairs with one stable loss — no reward model, no reinforcement learning loop.
  5. Reflective Agents: Structured Self-Critique with Episodic Memory 2023 · 18 min read · Intermediate — A synthetic proposal combining structured self-critique with episodic memory so autonomous language agents catch and correct their own errors.

Every step as a full report

Prefer to jump around? Each step is a standalone report with its own citations, figures and FAQ.

Deep RL from Human Preferences

Alignment & safety · 5 min read · Intermediate

Christiano et al. trained agents from human preferences instead of hand-written rewards — the 2017 result that made RLHF, and ChatGPT, possible.

rlhfalignmenthuman-feedback Read

Constitutional AI

Alignment & safety · 5 min read · Intermediate

Constitutional AI trains harmless assistants from a written constitution via AI self-critique, cutting human labels while keeping helpfulness.

constitutional-aialignmentharmlessness Read

Direct Preference Optimization

Alignment & safety · 5 min read · Intermediate

Direct Preference Optimization aligns language models from preference pairs with one stable loss — no reward model, no reinforcement learning loop.

dpoalignmentpreference-optimization Read

Watch alongside this path

Narrated films that cover the same ground, with chapters, captions and transcripts.

All videos