Constitutional AI
Post-training & alignment · 2022
The idea
Section titled “The idea”InstructGPT’s RLHF needs people to label which response is more harmful, at scale. Constitutional AI replaces most of that: give the model a written constitution instead, have it critique and rewrite its own bad outputs against those principles, then train the preference signal from the model’s own before/after judgments - RL from AI Feedback, not human feedback.
Verdict
Section titled “Verdict”What it fixedA written list of principles, not human harm-labels, supervises harmlessness - the model critiques and revises its own outputs, then trains a preference model from its own comparisons.
What it still costsThe constitution's principles are themselves a human choice; the model can only be as well-aligned as the rules it's handed.
Where it sits
Section titled “Where it sits”Worth having first: Instruction tuning & RLHF