====== GAIA Benchmark ======

GAIA (General AI Assistants) is a benchmark developed by **Meta** in collaboration with **Hugging Face** and **AutoGPT** for evaluating AI assistants on real-world tasks requiring multi-step reasoning, tool use, and web browsing. Unlike expert-knowledge benchmarks, GAIA tests tasks that are conceptually simple for humans (92% accuracy) but extremely challenging for AI systems.

===== Overview =====

GAIA was designed to address a fundamental question: can AI assistants reliably perform the kind of everyday multi-step tasks that humans handle routinely? Rather than testing specialized expertise, it evaluates fundamental abilities like information retrieval, multi-modal reasoning, and coordinated tool use across 5 to 50 sequential steps.

The benchmark consists of **466 questions** split into a public development set (166 questions) for tuning and a private test set (300 questions) for leaderboard evaluation. Answers in the private set are withheld to prevent data contamination.

===== Difficulty Levels =====

GAIA organizes tasks into three progressively harder levels:

^ Level ^ Description ^ Steps Required ^ Focus ^
| **Level 1** | Basic tool use, solvable by strong LLMs | ~5 steps | Single-tool retrieval and reasoning |
| **Level 2** | Multi-step coordination across tools | 10-30 steps | Planning and sequential execution |
| **Level 3** | High autonomy, strict execution chains | Up to 50 steps | Advanced planning, error recovery |

Level 3 tasks are designed to indicate major capability jumps, requiring sophisticated planning that current LLMs find particularly difficult.

===== Evaluation Methodology =====

GAIA uses **exact-match grading** on short factual answers. Systems receive full tool access (web search, file handling, code execution) with no predefined API restrictions. This contrasts with benchmarks like MMLU that test knowledge recall, instead prioritizing robust real-world interaction.

<code python>
# Example GAIA task structure (simplified)
task = {
    "question": "What is the total population of all countries "
                "that border the country where the Eiffel Tower is located?",
    "level": 2,
    "expected_answer": "332456000",  # Exact match required
    "tools_available": ["web_search", "calculator", "file_reader"],
    "annotator_steps": [
        "Identify Eiffel Tower location (France)",
        "Find all countries bordering France",
        "Look up population of each",
        "Sum the populations"
    ]
}
</code>

===== Results =====

The benchmark reveals a substantial human-AI gap of approximately 27%:

^ System ^ Accuracy ^
| **Humans** | 92% |
| **h2oGPTe (H2O.ai)** | 65% |
| **AutoGen multi-agent** | 40% |
| **GPT-4 with plugins** | 15% |
| **GPT-4-Turbo (standalone)** | <7% |

The dramatic difference between GPT-4 standalone (<7%) and agent-augmented systems (40-65%) demonstrates that tool use and multi-step orchestration are critical capabilities that raw model intelligence alone cannot provide.

===== Significance =====

GAIA established several important principles for AI evaluation:

  * **Real-world grounding** - Tasks reflect actual information needs, not synthetic puzzles
  * **Tool-agnostic evaluation** - Systems choose their own tools and strategies
  * **Human baseline calibration** - 92% human accuracy provides a meaningful ceiling
  * **Resistance to shortcuts** - Multi-step chains prevent pattern-matching solutions

===== References =====

  * [[https://arxiv.org/abs/2311.12983|GAIA: a benchmark for General AI Assistants (arXiv:2311.12983)]]
  * [[https://ai.meta.com/research/publications/gaia-a-benchmark-for-general-ai-assistants/|Meta AI - GAIA Publication]]
  * [[https://huggingface.co/spaces/gaia-benchmark/leaderboard|GAIA Leaderboard on Hugging Face]]
  * [[https://proceedings.iclr.cc/paper_files/paper/2024/file/25ae35b5b1738d80f1f03a8713e405ec-Paper-Conference.pdf|ICLR 2024 Conference Paper]]

===== See Also =====

  * [[terminal_bench]] - CLI/DevOps agent benchmark from Stanford and Laude Institute
  * [[humanitys_last_exam]] - Expert-level question benchmark for frontier models
  * [[computer_use_benchmark]] - GUI interaction benchmarks for agents
  * [[agent_observability]] - Monitoring agent performance in production