Parallel Function Calling

Parallel function calling enables LLMs to generate and invoke multiple tool calls simultaneously in a single response, reducing latency compared to sequential execution. When independent tool calls are needed (e.g., checking weather and time for the same city), parallel execution can reduce latency from seconds to milliseconds.

Traditional function calling follows a strict serial pattern:

1. LLM generates one tool call
2. System executes the tool and returns result
3. LLM generates next tool call (or final answer)
4. Repeat for each needed tool

For N independent tool calls each taking L seconds, sequential execution costs N x L seconds. Parallel execution reduces this to max(L_1, …, L_N) – a dramatic improvement when calls are independent.

OpenAI's GPT-4 and GPT-4o support parallel tool calls via the parallel_tool_calls=True API parameter. The model outputs multiple tool_calls in a single response message:

# OpenAI parallel function calling
import openai
 
tools = [
    {"type": "function", "function": {
        "name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}}}},
    {"type": "function", "function": {
        "name": "get_time", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}}}}
]
 
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Weather and time in Tokyo?"}],
    tools=tools,
    parallel_tool_calls=True  # Enable parallel calling
)
 
# Response contains multiple tool_calls in one message
for tool_call in response.choices[0].message.tool_calls:
    # Execute each call concurrently
    print(f"{tool_call.function.name}({tool_call.function.arguments})")

Anthropic's Claude models support parallel tool use natively. When the model determines multiple independent tools are needed, it emits multiple tool_use blocks in a single response. The client executes all calls concurrently and returns all results in the next message.

Models served via NVIDIA NIM (e.g., Mistral-7B-Instruct) support parallel tool calls through the same OpenAI-compatible API pattern.

SimpleTool investigates the design space of parallel function calling, analyzing how LLMs learn to emit multiple tool calls and the failure modes that arise:

• Training strategies: Models must be trained on examples with multiple simultaneous tool calls to learn the pattern reliably
• Independence detection: The model must identify which calls are truly independent vs which have data dependencies
• Ordering constraints: Some tool calls depend on results of others and must remain sequential
• Error handling: When one parallel call fails, the model must gracefully handle partial results

Several frameworks optimize how parallel tool calls are generated and orchestrated:

Models data relations (def-use chains) and control dependencies (mutual exclusion) between tool calls:

• Constructs a dependency DAG from the LLM's tool plan
• Assigns independent calls to parallel processors
• Respects data-flow ordering for dependent calls
• Open-source, works with both open and closed models

Extends LLMCompiler with processor load balancing:

• Automates parallel calling by modeling def-use relations
• Balances work across available processors
• Prevents overloads during concurrent execution bursts

Uses selective fusion to group similar tool operations at runtime:

• Inspired by hardware MAD (Multiply-Add) fusion
• Achieves 4x more parallel calls
• 40% reduction in token costs
• 12% lower latency on Copilot-scale benchmarks

In a single LLM generation, models output tool calls as an array. The execution layer:

1. Receives the full array of tool calls
2. Identifies independent calls (no data dependencies)
3. Executes independent calls concurrently (thread pool, async I/O)
4. Waits for all to complete
5. Returns all results to the LLM in one message

This eliminates round-trips: instead of N sequential LLM-tool-LLM cycles, one cycle handles all N calls.

LangChain and similar frameworks leverage parallel function calling for structured extraction – extracting multiple entity types from text in parallel:

• Define separate tools for Person, Location, Organization extraction
• Model calls all extractors simultaneously on the input text
• Results are merged into a unified structured output
• Simpler prompts and fewer errors than sequential extraction

• Dependency detection accuracy: Models sometimes parallelize calls that actually depend on each other
• Model support: Not all LLMs are trained for parallel calling
• Token budget: Multiple tool call specifications consume output tokens
• Error cascading: Failures in parallel calls require coordinated recovery
• Rate limiting: External APIs may throttle concurrent requests

• arXiv:2603.00030 - SimpleTool: Parallel Function Calling Analysis
• arXiv:2405.17438 - LLM-Tool Compiler: Selective Fusion
• arXiv:2504.14872 - LLMOrch: Parallel Tool Orchestration
• LLMCompiler - GitHub

• Speculative Tool Execution - Predicting and pre-executing future tool calls
• Agentic Reinforcement Learning - Training agents for tool use
• Agent RLVR - RL training for agent tool interactions