Gorilla: LLM Connected with Massive APIs

Gorilla is a large language model by Patil et al. (2023) specifically trained for accurate API calling through retriever-aware training. By conditioning the model on retrieved API documentation at both training and inference time, Gorilla surpasses GPT-4 on API call accuracy while dramatically reducing hallucinations — the fabrication of non-existent API endpoints or incorrect parameters. The accompanying APIBench benchmark provides a standardized evaluation for API-calling capabilities.

Gorilla's core innovation is integrating a document retriever directly into the training pipeline rather than relying on the model's parametric memory:

• Training data format: Each example pairs a natural language query with the top-1 retrieved API documentation snippet, formatted as [Query] [Retrieved Docs] → API Call
• Retriever: Uses Contriever, a dense bi-encoder model that embeds queries and documents into 768-dimensional vectors for semantic matching
• Joint optimization: The retriever is fine-tuned alongside the LLM on API-specific queries using InfoNCE loss, achieving 95% top-1 retrieval accuracy
• Base model: Fine-tuned on LLaMA-7B using supervised instruction tuning on approximately 10K query-document pairs

This approach grounds the model's API calls in actual documentation, preventing the hallucination of non-existent endpoints that plagues general-purpose LLMs.

APIBench is a comprehensive benchmark for evaluating end-to-end API calling:

• Scale: 1,541 tasks across 62 diverse real-world APIs (AccuWeather, Hugging Face, Twilio, etc.)
• Splits: Seen APIs (used during training) and unseen APIs (held out for generalization testing)
• Metrics: Exact-match JSON accuracy (correct API name + all parameters), partial credit, and hallucination rate
• Test conditions: Zero-shot prompting with and without retrieval, multi-step API chains, authentication edge cases

# Gorilla-style API call generation with retrieval
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
 
# Load Gorilla model (fine-tuned LLaMA)
model = AutoModelForCausalLM.from_pretrained("gorilla-llm/gorilla-7b-hf-v1")
tokenizer = AutoTokenizer.from_pretrained("gorilla-llm/gorilla-7b-hf-v1")
 
def generate_api_call(query: str, retrieved_doc: str) -> str:
    """Generate a grounded API call from query + retrieved documentation."""
    prompt = (
        f"### User Query: {query}\n"
        f"### Retrieved API Documentation:\n{retrieved_doc}\n"
        f"### API Call:\n"
    )
    inputs = tokenizer(prompt, return_tensors="pt")
    outputs = model.generate(**inputs, max_new_tokens=256)
    return tokenizer.decode(outputs[0], skip_special_tokens=True)
 
# Example: grounded API call generation
query = "Get the current weather forecast for San Francisco"
doc = "GET /forecast?city={city}&units={units} - Returns weather data"
api_call = generate_api_call(query, doc)
# Output: {"api_call": "GET /forecast?city=San Francisco&units=metric"}

Gorilla reduces API hallucinations by 85-90% compared to general-purpose LLMs by grounding generation in retrieved documentation.

A critical advantage of Gorilla's architecture is version-agnostic inference:

• At inference time, the retriever fetches from an up-to-date documentation index (e.g., via FAISS)
• API specification changes are automatically reflected without model retraining
• Experiments swapping documentation versions mid-evaluation show Gorilla maintains 90%+ accuracy
• GPT-4 fails on version-changed APIs due to stale parametric knowledge from training data

The retriever-aware training objective combines language modeling loss with retrieval relevance:

<latex>\mathcal{L} = \mathcal{L}_{LM}(y | x, d^*) + \lambda \mathcal{L}_{ret}(d^* | x)</latex>

where <latex>x</latex> is the user query, <latex>d^*</latex> is the retrieved documentation, <latex>y</latex> is the target API call, and <latex>\lambda</latex> balances the retrieval and generation losses.

• Patil et al. "Gorilla: Large Language Model Connected with Massive APIs" (arXiv:2305.15334)
• Gorilla GitHub Repository
• Gorilla Project Website (UC Berkeley)

• SWE-agent — Agent-Computer Interface for tool use in software engineering
• Agent Distillation — Compressing tool-using agents into smaller models
• Voyager — Code-generating embodied agent with skill library