AI Agent Knowledge Base

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== WebArena Benchmark ======

**WebArena** is a realistic, standalone, self-hosted web environment and benchmark for evaluating autonomous web-browsing agents on high-level natural language tasks that mimic everyday internet activities. It was introduced by Shuyan Zhou, Frank F. Xu, and colleagues, providing 812 diverse tasks across fully functional web applications.

===== Self-Hosted Web Environments =====

WebArena runs entirely self-hosted to ensure reproducibility, avoiding CAPTCHAs, content changes, or external dependencies on live websites. The environment includes four core web applications populated with real-world data:

  * **E-commerce (ShoppingAdmin)** — A fully functional online store with product catalogs, orders, and admin interfaces
  * **Content Management (CMS / LolaRun)** — A content management system for creating and editing web content
  * **Forums / Social Platform (Alpas)** — A discussion forum with user profiles, threads, and moderation features
  * **Knowledge / Tools (OpenStreetMap)** — Map-based point-of-interest lookup with calculator and scratchpad utilities

Additionally, knowledge resources including Wikipedia and user manuals support information-seeking tasks.

===== Task Types =====

The 812 benchmark tasks are diverse, long-horizon, and human-like, categorized into three main types:

  * **Information-seeking** — User-centric queries requiring multi-page navigation (e.g., "When was the last time I bought shampoo?"), distinct from open-domain QA because answers exist within the specific web environment
  * **Site navigation** — Using search functionality, links, and menus to reach specific sections or information within the self-hosted applications
  * **Content / configuration operations** — Creating or editing content, performing transactions, and adjusting settings (e.g., updating order status, purchasing items, adjusting product prices)

Agents interact via multi-tab observations (URL, open tabs, focused tab content) and flexible actions like ''click[ID]'', ''type[ID][text]'', and ''scroll'', with element selection via coordinates or DOM/accessibility tree IDs.

===== Evaluation Methodology =====

WebArena measures **end-to-end functional correctness** rather than partial progress. Success requires complete task completion, evaluated via exact-match or functional equivalence checks on the final state.

Agents are typically limited to approximately 15 steps per task. Baseline results with GPT-4 agents achieved only 14.41% task success rate compared to 78.24% for humans, highlighting the significant challenge of multi-step web reasoning and verification.

By 2025, top agents like SteP improved to 46% on OpenStreetMap and 31% on ShoppingAdmin subsets through trace analysis and failure debugging, yielding 16% gains. However, the benchmark remains largely unsaturated, unlike QA benchmarks such as MMLU.

===== WebArena Verified =====

**WebArena Verified** is a filtered subset of approximately 428 feasible tasks. The original 812 tasks include some that are impossible to complete due to environment limitations (e.g., "Fork all repositories from Facebook" fails because search results are paginated). The Verified subset removes these impossible tasks, providing a stricter and fairer evaluation of agent reliability.

===== VisualWebArena =====

**VisualWebArena** extends the base benchmark by incorporating visual inputs (screenshots) alongside HTML content. This tests multimodal agents on the same task suite, enabling more realistic browser emulation where agents must interpret rendered page layouts, images, and visual cues rather than relying solely on DOM structure.

===== MindSearch =====

**MindSearch** is a multi-agent search framework that decomposes complex web search queries into sub-questions, processes them in parallel using a graph-based planning approach, and synthesizes results. When tested on WebArena-style tasks, it demonstrates advanced reasoning capabilities for information-seeking web tasks that require consulting multiple sources.

<code python>
# Simplified WebArena agent interaction loop
class WebAgent:
    def __init__(self, llm, browser_env):
        self.llm = llm
        self.env = browser_env

    def solve_task(self, task_description, max_steps=15):
        """Execute a web task in the self-hosted environment."""
        observation = self.env.reset(task_description)
        for step in range(max_steps):
            # Get current page state (URL, tabs, DOM/accessibility tree)
            state = self.env.get_observation()
            # LLM decides next action based on task and state
            action = self.llm.predict_action(
                task=task_description,
                observation=state,
                history=self.env.action_history
            )
            # Execute action in browser environment
            observation, done = self.env.step(action)
            if done:
                break
        return self.env.get_final_state()
</code>

===== References =====

  * [[https://arxiv.org/abs/2307.13854|WebArena: A Realistic Web Environment for Building Autonomous Agents (arXiv:2307.13854)]]
  * [[https://webarena.dev|WebArena Official Website]]
  * [[https://github.com/web-arena-x/webarena|WebArena GitHub Repository]]
  * [[https://invariantlabs.ai/blog/what-we-learned-from-analyzing-web-agents|What We Learned from Analyzing Web Agents — Invariant Labs]]

===== See Also =====

  * [[swe_bench|SWE-bench]]
  * [[agent_as_a_judge|Agent-as-a-Judge]]
  * [[agent_index|AI Agent Index]]
AI Agent Knowledge Base

User Tools

Site Tools

Page Tools
