Constitutional AI (CAI), introduced by Bai et al. (2022) at Anthropic, is a training methodology that aligns language models to be helpful, harmless, and honest using a set of written principles (a “constitution”) and AI-generated feedback. CAI replaces human harmlessness labels with Reinforcement Learning from AI Feedback (RLAIF), enabling scalable alignment without exposing human annotators to harmful content.

Standard RLHF requires human annotators to label which model outputs are more or less harmful — a process that is expensive, subjective, and psychologically taxing for labelers. CAI asks: can the model itself judge harmfulness, guided by explicit principles? This makes the alignment criteria transparent and auditable rather than implicit in crowd-worker judgments.

The constitution is a set of human-written normative principles that define desired model behavior. These principles are drawn from diverse sources:

• The Universal Declaration of Human Rights
• Apple's Terms of Service (as an example of corporate guidelines)
• Principles emphasizing helpfulness, honesty, and harm avoidance
• Research-specific norms (e.g., “Choose the response that is least likely to be used for illegal activity”)

Each principle provides a concrete criterion the model uses to evaluate its own outputs. The constitution is fully transparent — users can inspect exactly what rules govern the model's behavior.

The first training phase uses supervised learning on self-revised outputs:

1. Generate: The model produces a response to a potentially harmful prompt
2. Critique: The same model critiques its response against a randomly sampled constitutional principle (e.g., “Identify specific ways this response is harmful or unethical”)
3. Revise: The model generates an improved response incorporating the critique
4. Iterate: Steps 2-3 repeat with different principles for multiple rounds

The final revised responses form a supervised fine-tuning dataset, replacing the need for human-written “ideal” responses to harmful prompts.

# Simplified CAI critique-revise loop
def critique_revise(model, prompt, constitution, num_rounds=3):
    response = model.generate(prompt)
 
    for _ in range(num_rounds):
        principle = random.choice(constitution)
        critique = model.generate(
            f"Critique this response according to the principle: "
            f"'{principle}'\n\nResponse: {response}"
        )
        response = model.generate(
            f"Revise this response based on the critique: "
            f"'{critique}'\n\nOriginal: {response}"
        )
 
    return response

The second phase replaces RLHF with Reinforcement Learning from AI Feedback:

1. Generate preference pairs: For each prompt, the model generates multiple candidate responses
2. AI evaluation: The model scores which response better adheres to constitutional principles, producing preference labels
3. Train reward model: A reward model is trained on AI-generated preference data (instead of human labels)
4. RL optimization: The policy model is fine-tuned via PPO against the AI-trained reward model

The key insight is that while the model cannot reliably generate harmless responses from scratch, it can reliably compare two responses and identify which is less harmful — a simpler discrimination task.

• CAI models achieve harmlessness comparable to or exceeding RLHF models trained with human labels
• Helpfulness is preserved — CAI models do not become evasive or unhelpful
• Models provide substantive explanations for refusals rather than generic deflections
• The approach scales without proportional increases in human annotation effort
• Constitutional principles are auditable, making alignment criteria transparent

CAI represents a paradigm shift from opaque human preferences to explicit written principles for alignment. The constitution can be publicly shared, debated, and revised, enabling democratic oversight of AI behavior. The technique also demonstrates that sufficiently capable models can supervise their own alignment, a form of scalable oversight critical for aligning increasingly powerful systems.

• Bai et al. "Constitutional AI: Harmlessness from AI Feedback" (2022)
• Bai et al. "Training a Helpful and Harmless Assistant with RLHF" (2022)
• Stiennon et al. "Learning to Summarize with Human Feedback" (2020)

• Direct Preference Optimization (DPO)
• Reward Overoptimization
• Self-Refine