Constitutional AI (CAI), introduced by Bai et al. (2022) at Anthropic, is a training methodology that aligns language models to be helpful and harmless using a set of explicit principles (a “constitution”) combined with AI-generated feedback. The key innovation is replacing human harmlessness labels with Reinforcement Learning from AI Feedback (RLAIF), where the model critiques and revises its own outputs according to constitutional principles. This eliminates the need for human annotators to evaluate harmful content while producing models that are both safer and more transparent in their refusals.

The constitution is a curated set of normative principles that define acceptable model behavior. These principles are drawn from diverse sources:

• The Universal Declaration of Human Rights
• Apple's Terms of Service (as an example of corporate guidelines)
• Principles emphasizing helpfulness, honesty, and harmlessness (HHH)
• Custom research-specific guidelines

Each principle provides an explicit, human-readable rule such as: “Choose the response that is least likely to be used for illegal or harmful activities” or “Choose the response that most supports the autonomy and freedom of the user.”

The constitution makes the alignment criteria transparent and auditable, unlike RLHF where human preferences are opaque and potentially inconsistent.

The first training phase generates improved data through an iterative self-critique loop:

1. Generate: The model produces a response to a potentially harmful prompt (using a helpful-only model that has not been trained for harmlessness).
2. Critique: The same model is asked to critique its response according to a randomly sampled constitutional principle: “Identify specific ways in which the response is harmful, unethical, or dangerous.”
3. Revise: The model generates an improved response incorporating the critique: “Please rewrite the response to remove harmful content.”

This critique-revise loop can be repeated multiple times, with different principles sampled each iteration. The final revised responses, paired with the original prompts, form a supervised fine-tuning dataset. The model is then fine-tuned on these (prompt, revised-response) pairs.

# Simplified Constitutional AI critique-revise loop
def constitutional_critique_revise(model, prompt, constitution, n_revisions=3):
    response = model.generate(prompt)
 
    for i in range(n_revisions):
        principle = random.choice(constitution)
        # Critique step
        critique = model.generate(
            f"Given the principle: '{principle}'
"
            f"Critique this response: {response}"
        )
        # Revise step
        response = model.generate(
            f"Based on this critique: {critique}
"
            f"Revise the response to better align with: '{principle}'
"
            f"Original response: {response}"
        )
    return response

The second phase uses reinforcement learning, but replaces human preference labels with AI-generated ones:

1. Generate comparison pairs: For each prompt, produce two candidate responses from the supervised model.
2. AI preference labeling: Ask the model to evaluate which response better aligns with constitutional principles. The model sees the principle and both responses, then selects the preferred one.
3. Train reward model: Use the AI-labeled preference data to train a reward model, exactly as in standard RLHF.
4. RL fine-tuning: Optimize the policy model against the AI-trained reward model using PPO (Proximal Policy Optimization).

The preference labeling uses a format like: “Consider the principle: [principle]. Which response is better? (A) or (B)”

The key differences between RLAIF and traditional RLHF:

• CAI models are less harmful than RLHF models while maintaining comparable helpfulness
• Models produce more nuanced refusals — explaining why a request is problematic rather than simply refusing
• The approach is more scalable since it does not require proportional increases in human annotation
• Constitutional principles can be updated and audited without retraining from scratch
• RLAIF preference labels correlate well with human preferences, validating the AI feedback approach

The RLAIF objective mirrors standard RLHF. Given a reward model $r_\phi$ trained on AI preferences, the policy $\pi_ heta$ is optimized:

$$\max_{\pi_ heta} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_ heta(\cdot|x)} \left[ r_\phi(x, y) ight] - eta \, ext{KL}\left[\pi_ heta \| \pi_{ ext{ref}} ight]$$

where $eta$ controls the KL penalty against the reference policy $\pi_{ ext{ref}}$. The difference is that $r_\phi$ is trained on AI-generated labels rather than human labels.

• Bai et al. "Constitutional AI: Harmlessness from AI Feedback" (2022)
• Bai et al. "Training a Helpful and Harmless Assistant" (2022)
• Schulman et al. "Proximal Policy Optimization" (2017)

• Direct Preference Optimization
• Reward Overoptimization
• Self-Refine