Self-Refine

Self-Refine, introduced by Madaan et al. (2023), is a framework that improves LLM outputs through iterative self-feedback: the same model generates an initial output, critiques it for weaknesses, and refines it based on that feedback. No external training, reinforcement learning, or additional models are required — the approach works purely at inference time with a single LLM.

Humans rarely produce perfect first drafts — we revise through iterative critique and improvement. Self-Refine brings this natural revision process to LLMs. While a model's initial output may contain errors, the same model can often identify those issues when explicitly prompted to critique, and fix them when prompted to revise. This leverages an asymmetry: evaluation is easier than generation.

Self-Refine operates in three steps, iterated until convergence:

1. Generate: The LLM produces an initial output $y_0$ given a task prompt $x$
2. Critique: The LLM provides multi-aspect feedback $f_t$ on the current output $y_t$, identifying specific weaknesses
3. Refine: The LLM generates an improved output $y_{t+1}$ conditioned on $x$, $y_t$, and $f_t$

$$y_0 = \text{LLM}(x), \quad f_t = \text{LLM}_{\text{fb}}(x, y_t), \quad y_{t+1} = \text{LLM}_{\text{refine}}(x, y_t, f_t)$$

The loop repeats for $T$ iterations or until the model indicates no further improvements are possible.

def self_refine(model, task_prompt, max_iters=3):
    """Iterative self-refinement loop."""
    output = model.generate(task_prompt)
 
    for i in range(max_iters):
        critique = model.generate(
            f"Review and identify specific issues:\n"
            f"Task: {task_prompt}\nOutput: {output}"
        )
        if "no improvements needed" in critique.lower():
            break
        output = model.generate(
            f"Improve based on feedback:\n"
            f"Task: {task_prompt}\nOutput: {output}\n"
            f"Feedback: {critique}"
        )
    return output

• Single model: The same LLM serves as generator, critic, and refiner
• No training: Operates purely at inference time via prompting
• Task-agnostic: Works across diverse tasks with task-specific critique templates
• Multi-aspect feedback: Critique evaluates multiple dimensions (correctness, style, completeness)

Two mechanisms determine when to stop:

• Self-assessment: The model states no further improvements are possible
• Fixed budget: A maximum iteration count (typically 2-4), balancing quality vs. latency

Self-Refine was evaluated on 7 diverse tasks with improvements of 5-40% over direct generation:

Results hold across GPT-3.5 and GPT-4, with quality improving monotonically over 2-3 rounds. Human evaluators consistently prefer Self-Refine outputs.

• Performance depends on the model's self-critique ability — weaker models may not identify their own errors
• Each iteration adds latency (roughly 3x tokens per refinement round)
• Diminishing returns after 2-3 iterations on most tasks
• Occasional regression where refinement introduces new errors

Self-Refine demonstrates that significant quality improvements are achievable without any additional training — purely through structured inference-time computation. This suggests that frontier model capabilities are underutilized by single-pass generation, and iterative refinement is a general-purpose method for extracting better performance.

• Madaan et al. "Self-Refine: Iterative Refinement with Self-Feedback" (2023)
• Shinn et al. "Reflexion: Language Agents with Verbal Reinforcement Learning" (2023)
• Chen et al. "Teaching Large Language Models to Self-Debug" (2023)

• Chain-of-Verification (CoVe)
• Constitutional AI
• Cognitive Architectures for Language Agents