Self-Refine

Self-Refine, introduced by Madaan et al. (2023), is a framework that improves LLM outputs through iterative self-feedback: the same model generates an initial output, critiques it for weaknesses, and refines it based on that feedback. No external training, reinforcement learning, or additional models are required, the approach works purely at inference time with a single LLM.¹⁾

Humans rarely produce perfect first drafts, we revise through iterative critique and improvement. Self-Refine brings this natural revision process to LLMs. While a model's initial output may contain errors, the same model can often identify those issues when explicitly prompted to critique, and fix them when prompted to revise. This leverages an asymmetry: evaluation is easier than generation.

Self-Refine operates in three steps, iterated until convergence:

1. Generate: The LLM produces an initial output $y_0$ given a task prompt $x$
2. Critique: The LLM provides multi-aspect feedback $f_t$ on the current output $y_t$, identifying specific weaknesses
3. Refine: The LLM generates an improved output $y_{t+1}$ conditioned on $x$, $y_t$, and $f_t$

$$y_0 = \text{LLM}(x), \quad f_t = \text{LLM}_{\text{fb}}(x, y_t), \quad y_{t+1} = \text{LLM}_{\text{refine}}(x, y_t, f_t)$$

The loop repeats for $T$ iterations or until the model indicates no further improvements are possible.

def self_refine(model, task_prompt, max_iters=3):
    """Iterative self-refinement loop."""
    output = model.generate(task_prompt)
 
    for i in range(max_iters):
        critique = model.generate(
            f"Review and identify specific issues:\n"
            f"Task: {task_prompt}\nOutput: {output}"
        )
        if "no improvements needed" in critique.lower():
            break
        output = model.generate(
            f"Improve based on feedback:\n"
            f"Task: {task_prompt}\nOutput: {output}\n"
            f"Feedback: {critique}"
        )
    return output

• Single model: The same LLM serves as generator, critic, and refiner
• No training: Operates purely at inference time via prompting
• Task-agnostic: Works across diverse tasks with task-specific critique templates
• Multi-aspect feedback: Critique evaluates multiple dimensions (correctness, style, completeness)

Two mechanisms determine when to stop:

• Self-assessment: The model states no further improvements are possible
• Fixed budget: A maximum iteration count (typically 2-4), balancing quality vs. latency

Self-Refine was evaluated on 7 diverse tasks with improvements of 5-40% over direct generation:²⁾

Results hold across GPT-3.5 and GPT-4, with quality improving monotonically over 2-3 rounds. Human evaluators consistently prefer Self-Refine outputs.

• Performance depends on the model's self-critique ability, weaker models may not identify their own errors
• Each iteration adds latency (roughly 3x tokens per refinement round)
• Diminishing returns after 2-3 iterations on most tasks
• Occasional regression where refinement introduces new errors

Self-Refine demonstrates that significant quality improvements are achievable without any additional training, purely through structured inference-time computation. This suggests that frontier model capabilities are underutilized by single-pass generation, and iterative refinement is a general-purpose method for extracting better performance.

• Generation vs Iterative Refinement in AI Design
• Recursive Self-Improvement
• CRITIC: LLMs Can Self-Correct with Tool-Interactive Critiquing
• Self-Improving AI
• Self-Modifying Software

• Madaan et al. "Self-Refine: Iterative Refinement with Self-Feedback" (2023)
• Shinn et al. "Reflexion: Language Agents with Verbal Reinforcement Learning" (2023)³⁾
• Chen et al. "Teaching Large Language Models to Self-Debug" (2023)⁴⁾