The evaluation of artificial intelligence agents presents a fundamental challenge in the field: performance metrics in controlled laboratory environments often diverge significantly from outcomes in uncontrolled, realistic settings. This phenomenon reflects broader limitations in benchmark design and the difficulty of capturing real-world complexity in standardized test suites.

Sandbox evaluations represent controlled testing frameworks where agents operate within predefined constraints and simplified task environments. These environments isolate AI agents in controlled conditions to prevent harmful real-world consequences while testing their capabilities ¹⁾. These environments typically feature:

* Deterministic behavior: Task outcomes follow consistent, predictable patterns * Clean state management: Agents begin each evaluation with a reset environment free of artifacts from previous interactions * Simplified interfaces: Reduced complexity compared to real-world systems * Bounded action spaces: Limited set of possible actions and outcomes * Accessible task specifications: Clear, explicit goal definitions provided to agents

Common sandbox benchmarks include WebShop ²⁾, a simulated e-commerce environment where agents navigate product pages to complete shopping tasks, and similar task-specific environments designed to test particular agent capabilities. In these controlled settings, modern language model-based agents frequently achieve success rates between 60-75% ³⁾, depending on task complexity and agent architecture.

Real-world evaluations involve agents operating on actual websites, systems, and environments without modification for testing purposes. Key characteristics include:

* Environmental variability: Pages, layouts, and content change dynamically * Noisy signals: Irrelevant information, advertisements, and distractions present alongside task-relevant content * Unexpected failure modes: System timeouts, network issues, and interface inconsistencies * Unconstrained action spaces: Agents must navigate genuine branching decision trees with countless possible paths * Adversarial complexity: Real systems often include safeguards, CAPTCHAs, and anti-bot measures

Research demonstrates that agent performance degrades substantially in real-world settings. This gap reflects fundamental differences between benchmark design assumptions and actual deployment conditions ⁴⁾, as sandboxed evaluations are fundamentally limited because they cannot fully replicate the real-world messiness and unexpected situations that agents encounter in production environments.

The disparity between sandbox and real-world performance stems from multiple interconnected factors:

Generalization limitations: Models trained and evaluated on narrow task distributions struggle when encountering novel interface patterns, unconventional layouts, and unexpected system behaviors. Spurious correlations learned during benchmark optimization do not transfer to diverse real environments.

Error accumulation: Agents operating in open-ended settings face compounding failures. A minor perception error in sandbox environments—where states are explicit and clean—becomes catastrophic in real-world settings where errors propagate through subsequent decision-making steps.

Context window constraints: Real websites contain substantially more visual and textual information than sandbox tasks. Agents must selectively attend to relevant information while managing limited context windows, a capability rarely required in simplified evaluation environments ⁵⁾.

Hallucination and confabulation: Language models may generate plausible but false claims about interface elements or task completion status. Sandbox evaluations with explicit state representations make hallucinations immediately apparent; real-world evaluations allow false claims to propagate undetected.

Behavioral drift: Fine-tuning or prompting approaches that optimize for benchmark performance may inadvertently degrade performance on out-of-distribution tasks characteristic of real-world use.

The sandbox-to-real-world gap carries important implications for AI system deployment:

Evaluation methodology: Organizations developing agent systems must incorporate real-world testing alongside or instead of relying solely on benchmark performance. Reported sandbox performance should not be treated as reliable indicators of production viability.

Model selection: High benchmark performance may reflect overfitting to task-specific patterns rather than robust agent capabilities. Comparative evaluation across multiple environments—both controlled and realistic—provides more informative assessment.

Safety and robustness: The increased complexity of real-world environments amplifies risks associated with agent errors. Systems must include error detection, human override mechanisms, and controlled deployment strategies.

Research directions: Bridging this gap requires advances in robust perception, hierarchical planning, uncertainty quantification, and failure recovery. Techniques from the reinforcement learning literature—such as distribution shift adaptation and uncertainty estimation—show promise for improving real-world performance ⁶⁾.

Recent work has begun addressing real-world evaluation more systematically. Researchers are developing evaluation frameworks that capture realistic complexity while maintaining sufficient structure for reproducible assessment. This includes studying agent behavior on live websites with appropriate safeguards, analyzing failure modes in uncontrolled settings, and developing adaptation mechanisms that allow agents to improve through interaction with real environments.

The recognition that sandbox performance provides limited guidance for real-world deployment has motivated the development of more sophisticated evaluation methodologies that balance experimental control with environmental realism.

• Sandbox vs. Live Web Benchmarks
• Agent Evaluation
• Agent Sandbox Security
• Open-World Evaluations
• Terminal-Bench