Chain of Abstraction (CoA) is a reasoning method for large language models introduced by Gao et al. (2024) that decouples abstract reasoning from domain-specific knowledge grounding. Rather than generating concrete values inline (as in chain-of-thought), CoA trains LLMs to first produce reasoning chains with abstract placeholders, then calls external tools in parallel to fill in specific values. This yields more robust reasoning strategies and faster inference.

Tool-augmented LLMs like Toolformer can access external knowledge through API calls, but face two key limitations in multi-step reasoning:

• Sequential bottleneck: Each tool call blocks generation until the response returns
• Fragile reasoning: Models that interleave concrete values with reasoning steps are sensitive to shifts in domain knowledge

CoA addresses both problems by separating the what to compute from the how to compute it, enabling the LLM to focus on high-level reasoning structure while delegating specifics to tools.

CoA operates in two phases:

The LLM is fine-tuned to produce reasoning chains using abstract placeholders (e.g., y1, y2, y3) instead of concrete values. For example, given a math word problem, the model might output:

y1 = [CALC: cost_per_unit * quantity]
y2 = [CALC: y1 * tax_rate]
y3 = [CALC: y1 + y2]

This forces the model to learn general reasoning strategies independent of specific numerical results.

A supervisor orchestrates parallel calls to domain-specific tools (equation solvers, knowledge retrievers) to replace each placeholder with its actual value. Because placeholders are independent of each other at the abstract level, multiple tool calls execute simultaneously.

# Conceptual illustration of Chain-of-Abstraction
 
def chain_of_abstraction(problem, llm, tools):
    # Phase 1: Generate abstract reasoning chain
    abstract_chain = llm.generate(
        prompt=f"Solve using abstract placeholders:\n{problem}",
        mode="abstract"
    )
    # abstract_chain might be: "y1 = [CALC: 5*12], y2 = [CALC: y1*0.08], answer = [CALC: y1+y2]"
 
    # Phase 2: Parse placeholders and ground via tools (parallel)
    placeholders = parse_placeholders(abstract_chain)
    grounded_values = {}
    for name, tool_call in placeholders.items():
        # Tool calls can execute in parallel when independent
        grounded_values[name] = tools.execute(tool_call, context=grounded_values)
 
    # Replace placeholders with grounded values
    final_answer = substitute(abstract_chain, grounded_values)
    return final_answer

• Placeholder naming: Using multi-character tokens like y1, y2 outperforms single-character variables (x, y)
• Training data: Abstract chains are derived from existing CoT datasets by replacing concrete values with placeholders
• Tool integration: Any domain tool (calculator, search engine, database) can serve as a grounding backend

Evaluated on mathematical reasoning (GSM8K) and open-domain QA (Wiki QA):

Given a problem $p$, CoA generates an abstract chain $a = (a_1, a_2, \ldots, a_n)$ where each $a_i$ is either a reasoning step or a placeholder $y_i = [\text{TOOL}: f_i(\cdot)]$. The grounding function $G$ maps:

$$G(a) = (g_1, g_2, \ldots, g_n), \quad g_i = \begin{cases} a_i & \text{if } a_i \text{ is a reasoning step} \\ \text{TOOL}(f_i) & \text{if } a_i \text{ is a placeholder} \end{cases}$$

The final answer is derived from the fully grounded chain $G(a)$.

• Gao et al. "Efficient Tool Use with Chain-of-Abstraction Reasoning" (2024). arXiv:2401.17464
• COLING 2025 proceedings
• Schick et al. "Toolformer: Language Models Can Teach Themselves to Use Tools" (2023)

• Chain of Thought
• Toolformer
• ReAct Prompting
• Tree of Thoughts