Analogical Prompting

Analogical Prompting is a prompt engineering technique introduced by Yasunaga et al. (2023) from Google DeepMind and Stanford University that instructs large language models to self-generate relevant examples through analogical reasoning before solving a target problem. Inspired by how humans recall past experiences when facing new challenges, this method eliminates the need for manually labeled few-shot exemplars while adapting generated demonstrations to each specific problem.

Existing chain-of-thought (CoT) prompting methods face a trade-off:

• Zero-shot CoT (“think step by step”) is convenient but provides generic guidance insufficient for complex tasks
• Few-shot CoT provides detailed exemplars but requires labor-intensive manual labeling, and fixed exemplars may not be relevant to every problem

Analogical Prompting achieves the best of both worlds: automatically generated, problem-specific exemplars with no manual labeling required.

The approach follows three steps within a single LLM call:

Present the target problem to the LLM.

Instruct the model to recall or generate 3-5 relevant problems (with solutions) that are structurally similar to the target. The prompt explicitly asks for distinct and relevant examples.

The LLM uses its self-generated exemplars as context to solve the original problem.

# Analogical Prompting implementation
 
def analogical_prompt(problem, llm, n_exemplars=3):
    prompt = (
        f"Your task is to solve the following problem.\n\n"
        f"Problem: {problem}\n\n"
        f"Before solving, recall {n_exemplars} relevant and distinct problems "
        f"you have encountered before. For each:\n"
        f"1. State the problem\n"
        f"2. Explain the solution step by step\n"
        f"3. Identify the key principle or technique used\n\n"
        f"After generating these exemplars, solve the original problem using "
        f"insights from the analogies above.\n"
    )
    response = llm.generate(prompt)
    return response
 
 
# Self-Generated Knowledge + Exemplars variant (for code generation)
def analogical_prompt_with_knowledge(problem, llm, n_exemplars=3):
    prompt = (
        f"Your task is to solve the following problem.\n\n"
        f"Problem: {problem}\n\n"
        f"First, identify the core concepts and techniques relevant to this problem.\n"
        f"Provide a brief tutorial or key takeaways for each concept.\n\n"
        f"Then, recall {n_exemplars} relevant and distinct problems. For each:\n"
        f"1. State the problem\n"
        f"2. Explain the solution step by step\n\n"
        f"Finally, solve the original problem using the knowledge and exemplars above.\n"
    )
    response = llm.generate(prompt)
    return response

The paper introduces two complementary approaches:

Generates relevant exemplar problems and solutions. Works well for mathematical reasoning and general problem-solving tasks.

For complex tasks like code generation, the model may over-rely on low-level exemplar patterns. This variant adds an instruction to first identify core concepts and provide high-level tutorials before generating exemplars. This mitigates overfitting to surface-level similarities.

The key advantage is adaptability: generated exemplars are tailored to each problem's specific structure, providing more relevant guidance than any fixed set of demonstrations.

The approach draws from analogical reasoning in cognitive psychology (Vosniadou & Ortony, 1989):

• Humans routinely solve new problems by recalling structurally similar past experiences
• The quality of analogical transfer depends on structural alignment between source and target
• LLMs, trained on vast corpora, have implicit access to enormous problem-solving experience that can be activated through appropriate prompting

Evaluated with GPT-3.5-turbo and GPT-4 across diverse reasoning benchmarks:

Key findings:

• Outperforms both 0-shot CoT and manual few-shot CoT across tasks
• 3-5 self-generated exemplars is the optimal range
• The Knowledge + Exemplars variant is critical for code generation tasks
• Single-pass generation (exemplars + solution in one call) is both efficient and effective

A minimal template for analogical prompting:

[Insert problem here]

Instruction: Before solving, recall 3 relevant and distinct problems
as exemplars. For each, describe the problem and solution. Then solve
the initial problem step by step.

• Yasunaga et al. "Large Language Models as Analogical Reasoners" (2023). arXiv:2310.01714
• Google DeepMind publication page
• ICLR 2024 proceedings

• Chain of Thought
• Few-Shot Prompting
• Self-Consistency
• Generated Knowledge Prompting