====== Agent Evaluation ======

**Agent evaluation** encompasses the benchmarks, metrics, and methodologies used to assess the capabilities of AI agents across domains including software engineering, web navigation, code generation, tool use, and general reasoning. As of 2025, standardized benchmarks have become critical for comparing agent frameworks and tracking progress in autonomous AI capabilities.

===== SWE-Bench =====

**SWE-Bench** tests AI agents on real-world software engineering tasks derived from GitHub issues. Agents must edit codebases, run tests, and resolve bugs in repositories like Django, SymPy, and scikit-learn. The agent interacts via bash tools in Dockerized environments.

**SWE-Bench Verified** is a curated subset of 500 tasks with human-verified fixes for stricter evaluation, addressing concerns about ambiguous or flawed test cases in the original benchmark.

^ Metric ^ Value ^
| Task Source | Real GitHub issues and PRs |
| Environment | Dockerized repository snapshots |
| Top Scores (2025) | >60% resolution rate |
| Key Innovation | End-to-end coding + testing |

Top-performing agents achieve over 60% resolution through high-level planners, specialized training, and memory-augmented architectures.

===== GAIA =====

**GAIA** (General AI Assistants) assesses zero-shot reasoning across question-answering, tool use, and multi-step planning with real-world tasks. It includes 466 tasks across three difficulty levels, requiring agents to integrate web search, code execution, and interpretation without task-specific training data.

^ Level ^ Description ^ Top Scores (2025) ^
| Level 1 | Simple factual questions | ~70-80% |
| Level 2 | Multi-step reasoning | ~60-70% |
| Level 3 | Complex multi-tool tasks | ~50-60% |

===== WebArena =====

**WebArena** benchmarks web-browsing agents in realistic simulations of e-commerce sites, social forums, and content management systems. It contains 804 tasks across four categories: Web Shopping, Web Search, Social Interaction, and Content Editing.

Agents use browser tools for navigation, form-filling, and decision-making. Early GPT-4 agents scored approximately 14%, improving to over 60% by 2025. **IBM CUGA leads at 61.7%** as of early 2025.

===== AgentBench =====

**AgentBench** is a comprehensive suite testing language agents on decision-making, reasoning, and tool usage across 8 diverse environments:

  * Operating system interaction
  * Database querying
  * Web browsing
  * Knowledge graph navigation
  * Lateral thinking puzzles
  * Digital card games
  * Household simulation
  * Web shopping

The benchmark includes 2,000+ tasks with success measured by goal completion rates across all environments.

===== HumanEval =====

**HumanEval** evaluates code generation by prompting models to complete 164 Python functions from docstrings. Scoring uses **pass@k** — the probability that at least one of k generated solutions passes all unit tests.

While originally designed for LLM evaluation rather than agents, HumanEval has been adapted for tool-augmented coding scenarios. Top 2025 models exceed 90% pass@1.

===== Other Notable Benchmarks =====

  * **CUB** (Computer Use Benchmark) — 106 end-to-end workflows across 7 industries for GUI agents; top score 10.4%
  * **OSWorld** — Realistic operating system environment for multimodal desktop agents
  * **Mind2Web** — 2,350 tasks on 137 live websites for web agent evaluation
  * **BFCL v4** (Berkeley Function-Calling Leaderboard) — Multi-step tool use evaluation
  * **Terminal-Bench** — Terminal-based task completion
  * **tau-Bench** — Multi-turn workflow evaluation
  * **ALFWorld** — Household simulation tasks

===== Leaderboard Summary (2025) =====

^ Benchmark ^ Top Performer ^ Score ^ Notes ^
| SWE-Bench Verified | Advanced planners | >60% | End-to-end software engineering |
| WebArena | IBM CUGA | 61.7% | Web browsing autonomy |
| GAIA Level 3 | Leading LLMs | ~50-60% | General reasoning |
| HumanEval | Top LLMs | >90% pass@1 | Code generation |
| CUB | Writer Action Agent | 10.4% | Computer use (very challenging) |
| AgentBench | Domain-specific | ~50-70% avg | Multi-environment |

===== Code Example =====

<code python>
# Simple evaluation harness pattern
import json
from typing import Callable

def evaluate_agent(
    agent_fn: Callable,
    benchmark: list[dict],
    metric_fn: Callable
) -> dict:
    """Evaluate an agent against a benchmark dataset."""
    results = []
    for task in benchmark:
        prediction = agent_fn(task['input'])
        score = metric_fn(prediction, task['expected'])
        results.append({
            'task_id': task['id'],
            'score': score,
            'prediction': prediction
        })
    
    total = len(results)
    passed = sum(1 for r in results if r['score'] >= 1.0)
    return {
        'total_tasks': total,
        'passed': passed,
        'pass_rate': passed / total,
        'results': results
    }

# Example usage
scores = evaluate_agent(
    agent_fn=my_coding_agent,
    benchmark=swe_bench_tasks,
    metric_fn=test_pass_metric
)
print(f'Pass rate: {scores["pass_rate"]:.1%}')
</code>

===== References =====

  * [[https://arxiv.org/abs/2310.06770|Jimenez et al., 2023 — SWE-Bench: Can Language Models Resolve Real-World GitHub Issues?]]
  * [[https://arxiv.org/abs/2311.12983|Mialon et al., 2023 — GAIA: A Benchmark for General AI Assistants]]
  * [[https://arxiv.org/abs/2307.13854|Zhou et al., 2023 — WebArena: A Realistic Web Environment for Building Autonomous Agents]]
  * [[https://arxiv.org/abs/2308.03688|Liu et al., 2023 — AgentBench: Evaluating LLMs as Agents]]
  * [[https://arxiv.org/abs/2107.03374|Chen et al., 2021 — Evaluating Large Language Models Trained on Code (HumanEval)]]
  * [[https://www.swebench.com|SWE-Bench Leaderboard]]
  * [[https://webarena.dev|WebArena Leaderboard]]

===== See Also =====

  * [[computer_use]] — Computer Use and GUI agents
  * [[devin]] — Devin AI software engineer (SWE-Bench participant)
  * [[multi_agent_systems]] — Multi-agent system architectures
  * [[claude_code]] — Claude Code CLI