Chain of Hindsight (CoH) is a training approach that enables language models to learn from feedback by conditioning on sequences of model outputs paired with their evaluative feedback. Proposed by Liu et al., 2023 in “Chain of Hindsight Aligns Language Models with Feedback,” CoH presents the model with a chain of previous attempts along with hindsight annotations indicating what was good or bad about each attempt, training the model to generate improved outputs conditioned on positive feedback signals¹⁾. This method effectively converts any form of feedback into a training signal without requiring complex reinforcement learning algorithms.

The core idea behind CoH is to leverage the language model's natural ability to understand and act on textual instructions. Rather than training a separate reward model and running RL optimization (as in RLHF), CoH directly fine-tunes the model on sequences that demonstrate the improvement process itself.

The training procedure works as follows:

1. Generate multiple outputs: For a given prompt, the base model generates several candidate responses of varying quality
2. Collect feedback: Each output is annotated with natural language feedback describing its quality (e.g., “Bad: This response is too verbose and misses the key point” or “Good: This is concise and accurately addresses the question”)
3. Construct hindsight chains: Outputs and feedback are arranged into a sequence: Prompt + Output_1 + Feedback_1 + Output_2 + Feedback_2 + … + Final_Output
4. Train with selective loss: The model is fine-tuned using causal language modeling loss, applied primarily to the final (highest-quality) output in the chain, while conditioning on the full hindsight sequence

At inference time, the model is prompted with positive feedback prefixes (e.g., “Good:”) to elicit the desired high-quality behavior. The model has learned to associate positive feedback signals with the patterns of its best outputs.

The key insight is that the model learns not just from good examples, but from the contrast between poor and good outputs. By seeing the trajectory from bad to good, conditioned on feedback, the model internalizes what improvements look like.

CoH is flexible in how feedback is sourced and structured:

• Human feedback: Direct annotations from human evaluators, the highest quality but most expensive
• Automated feedback: LLM-generated critiques, rule-based evaluations, or metric-based assessments (ROUGE, factuality scores)
• Templated feedback: Structured templates that convert quality signals into natural language (e.g., converting a numerical rating into “This response scored poorly on coherence”)

The feedback is polarity-agnostic in construction, both positive and negative feedback appear in the training chains. This is a significant advantage over methods like RLHF that primarily learn from preferred vs. rejected pairs, as CoH can incorporate nuanced feedback about specific aspects of quality (factuality, style, completeness, safety).

Data construction scales efficiently because multiple outputs per prompt can be generated cheaply from the base model, and feedback annotation can be partially automated.

Key advantages of CoH over RLHF:

• No reward model required: Eliminates the reward modeling stage and its associated errors
• No RL optimization: Avoids the instability of PPO and the need for careful hyperparameter tuning
• Richer feedback signal: Natural language feedback conveys more information than scalar rewards
• Lower alignment tax: CoH preserves more of the model's general capabilities during alignment training

Compared to DPO, CoH can incorporate feedback beyond simple binary preferences, including partial credit, aspect-specific feedback, and improvement suggestions.

Liu et al., 2023 evaluated CoH on summarization and dialogue tasks:

• Summarization: CoH surpassed both SFT and RLHF baselines on ROUGE metrics and human pairwise evaluations for coherence, coverage, and feedback adherence
• Human evaluation: Raters strongly preferred CoH outputs over RLHF outputs, particularly for tasks requiring nuanced quality improvements
• Scaling: Gains from CoH increased with model size, suggesting the method benefits from stronger language understanding capabilities
• RL extensions: When applied to agentic settings using decision transformers on D4RL and ExoRL benchmarks, CoH-style training matched or exceeded Decision Transformer and TD3+BC on sub-optimal offline data

Applications of CoH extend to any domain where feedback is available: instruction following, code generation (learning from compiler errors and test failures), creative writing (learning from editorial feedback), and multi-turn dialogue improvement. The approach is particularly valuable when feedback is diverse and multi-dimensional, as natural language can capture subtleties that scalar rewards cannot.

• Instruction Following Evaluation
• Reinforcement Learning from Human Feedback
• Steering Vectors
• Instruction Tuning
• Experience Replay for LLM Reinforcement Learning