4 reports · 2017–2023

Alignment & safety

Turning raw predictors into assistants: human feedback, instruction tuning, preference optimisation and self-critique.

Fig. 02 / Field notesLearning from a preference
Preference comparison: two candidate responses receive different preference scores.
Compare candidates, then use the preference as a training signal.

Deep RL from Human Preferences

Alignment & safety · 5 min read · Intermediate

Christiano et al. trained agents from human preferences instead of hand-written rewards — the 2017 result that made RLHF, and ChatGPT, possible.

rlhfalignmenthuman-feedback Read

Constitutional AI

Alignment & safety · 5 min read · Intermediate

Constitutional AI trains harmless assistants from a written constitution via AI self-critique, cutting human labels while keeping helpfulness.

constitutional-aialignmentharmlessness Read

Direct Preference Optimization

Alignment & safety · 5 min read · Intermediate

Direct Preference Optimization aligns language models from preference pairs with one stable loss — no reward model, no reinforcement learning loop.

dpoalignmentpreference-optimization Read

Videos on Alignment & safety

Narrated films that touch this topic, with chapters, captions and transcripts.

All videos