Terminal-Bench (also referred to as TerminalBench or Terminal-Bench 2.0) is a benchmark for evaluating AI agents and large language models on realistic command-line interface (CLI) and DevOps tasks¹⁾. Developed by Stanford University and the Laude Institute, it tests whether AI agents can reliably perform multi-step terminal operations in sandboxed environments, including compiling code, configuring systems, running tools, navigating filesystems, writing scripts, and executing complex command-line operations.

Traditional AI benchmarks focus on text generation or question answering. Terminal-Bench instead measures end-to-end operational reliability in real terminal environments. Each task provides a unique environment with human-written solutions and automated verification tests, ensuring reproducible and meaningful evaluation.

The benchmark targets agent systems and large language models, evaluating the complete pipeline of perception, reasoning, planning, and execution in CLI contexts. It reflects the growing importance of automated system administration in enterprise environments, where large language models increasingly serve as autonomous agents handling infrastructure tasks.

Unlike traditional benchmarks that focus on theoretical knowledge or isolated problem-solving, Terminal-Bench emphasizes real-world applicability by testing systems on actual terminal-based workflows, addressing a critical gap in AI evaluation where language models achieve high performance on academic benchmarks yet require distinct capabilities to operate effectively within command-line environments where much software development, system administration, and data engineering work occurs²⁾.

The benchmark provides a measurable foundation for comparing different approaches to agentic harness design and optimization, enabling researchers and practitioners to systematically evaluate improvements in agent engineering across iterations and variants.

Terminal-Bench 2.0 is the harder successor to the original benchmark³⁾, featuring 89 curated real-world tasks across multiple domains:

• Software Engineering - Building, debugging, and deploying applications
• System Administration - Configuring servers, managing services, networking
• Scientific Workflows - Data processing pipelines, computational tasks
• Security - Input validation, vulnerability assessment (SQL injection, CSRF/SSRF per CWE standards)

Terminal-Bench 2.0 provides a quantitative measurement framework for evaluating LLM performance on terminal-focused coding tasks. The benchmark is particularly relevant for assessing models' capabilities in generating code that interacts with operating system interfaces, command-line tools, and system administration tasks. This class of benchmarks has become increasingly important as LLMs are deployed in software development environments where terminal proficiency is a critical requirement⁴⁾.

The benchmark enables comparative analysis of model performance improvements across different optimization techniques and architectural approaches.

Terminal-Bench 2.0 measures model performance through task completion rates on terminal-based programming challenges. A notable case study demonstrates significant performance improvements through systematic optimization: GPT-5.2-Codex improved from 52.8% to 66.5% task completion through harness and middleware optimization techniques⁵⁾.

This 13.7 percentage point improvement illustrates how targeted infrastructure and execution optimization can substantially enhance LLM capabilities on specialized task domains. The improvement was achieved not through model retraining but through optimization of the execution environment and system integration layers that mediate between the model and terminal operations.

The improvements demonstrated in Terminal-Bench 2.0 evaluations utilized two primary optimization approaches:

Harness Optimization involves refining the evaluation and execution framework to more efficiently present tasks and capture model outputs. An optimized harness can reduce overhead in task representation and improve the clarity of instruction communication.

Middleware Optimization refers to improvements in the software layer that manages communication between the LLM and the terminal environment. Middleware components handle instruction translation, output parsing, error recovery, and state management. Enhanced middleware can reduce the cognitive load on the model by handling routine system-level operations.

• Terminal-Bench 2 Benchmark
• Codex-CLI
• HiL-Bench
• SWE-Bench
• MLE-Bench