HiL-Bench is a benchmark developed by Scale AI Labs designed to evaluate whether AI agents possess the capability to recognize incomplete specifications and appropriately request clarifying questions during task execution ¹⁾. The benchmark addresses a critical gap in agent evaluation by testing pragmatic reasoning and communication competencies rather than solely task completion accuracy.

HiL-Bench measures a specific and increasingly important capability in autonomous agent systems: the ability to identify ambiguities, incompleteness, or conflicting requirements in specifications before proceeding with execution. This human-in-the-loop (HiL) oriented benchmark recognizes that practical AI systems must operate effectively in real-world scenarios where requirements are rarely fully specified upfront.

Traditional agent benchmarks focus primarily on whether systems can successfully complete tasks given clear objectives. HiL-Bench shifts evaluation criteria to assess whether agents demonstrate the maturity to halt execution, recognize specification gaps, and engage in clarification dialogue with human operators. This capability represents a fundamental shift from autonomous operation toward collaborative human-AI workflows ²⁾.

The benchmark constructs test scenarios with intentionally incomplete, ambiguous, or underspecified requirements. Rather than penalizing agents for failure to complete tasks under these conditions, HiL-Bench evaluates whether agents:

* Recognize limitations in provided specifications * Identify critical missing information that would affect task execution quality * Formulate appropriate clarifying questions * Request human intervention at suitable decision points * Avoid making arbitrary assumptions about ambiguous requirements

This evaluation approach contrasts with traditional success metrics that reward task completion regardless of specification quality. HiL-Bench instead treats specification-aware behavior and appropriate escalation as primary success indicators.

The emergence of HiL-Bench reflects broader industry recognition that next-generation AI agents require collaborative capabilities beyond autonomous task execution. As autonomous agents become integrated into complex organizational workflows, the ability to communicate uncertainty and request clarification becomes operationally critical ³⁾.

Scale AI Labs' focus on this capability suggests growing market demand for agents that can operate reliably in ambiguous environments through human-AI collaboration rather than requiring perfect specification. This benchmark contributes to establishing standards for evaluating pragmatic agent reasoning and communication skills alongside technical task performance.

HiL-Bench relates to broader research in agent evaluation frameworks, human-AI interaction design, and prompt engineering for clarification. Similar benchmarks assess agent capabilities in reasoning under uncertainty, multi-turn dialogue, and collaborative problem-solving. The benchmark aligns with contemporary research emphasizing interpretability and controllability in autonomous systems.

• MLE-Bench
• SWE-Bench
• CORE-Bench
• Terminal-Bench
• Web UI Bench