This comparison examines the performance differences between automatically evolved agent harnesses and traditional hand-crafted baseline approaches for autonomous system integration. The distinction represents a significant methodological shift in how agent interfaces and interaction protocols are engineered, moving from manual optimization to observability-driven automatic evolution techniques.

Agent harnesses represent the interface layer through which language models interact with external tools, APIs, and system environments. Traditionally, these harnesses have been manually designed by engineers who craft specific protocols, error handling mechanisms, observation formats, and interaction patterns based on domain expertise and iterative testing. The emergence of automatic harness evolution represents an alternative paradigm where systems generate optimized harnesses through automated search and refinement processes guided by empirical performance metrics ¹⁾

Empirical evaluation on Terminal-Bench 2, a standardized benchmark for command-line task execution, reveals substantial performance differentials between approaches. The automatically evolved harness achieved 77.0% success rate, compared to the hand-crafted Codex-CLI baseline at 71.9%, representing a 5.1 percentage point improvement. Additionally, the evolved harness outperformed two other automated baseline approaches by 4–8 percentage points ²⁾

This performance gap suggests that automatically discovered harness configurations may capture interaction patterns and error recovery mechanisms that exceed the scope of manual engineering efforts. The margin of improvement aligns with typical gains observed in hyperparameter optimization and automated machine learning (AutoML) contexts.

Observability-driven automatic evolution operates by instrumenting agent interactions to collect detailed performance telemetry, then using this observational data to guide systematic modifications to harness parameters and protocols. This approach differs fundamentally from hand-crafted baselines in several dimensions:

* Search Space Exploration: Automated evolution can systematically explore configurations that would be impractical for manual enumeration, discovering non-intuitive combinations of parameters and protocols * Empirical Guidance: Rather than relying on intuition or domain assumptions, evolution operates directly from observed success and failure patterns * Rapid Iteration: Automated methods can evaluate candidate configurations at scale, enabling faster convergence toward optimal solutions * Transfer Learning: Evolved harnesses developed for one task domain may partially transfer to related domains, whereas hand-crafted approaches typically require re-engineering

The observability component is critical—detailed logs of agent decisions, environmental responses, error conditions, and task progress provide the signal necessary for optimization algorithms to identify which harness modifications improve performance.

The Codex-CLI baseline, despite representing professional engineering effort, exhibits inherent limitations. Manual harness design typically optimizes for a developer's specific assumptions about likely failure modes and recovery strategies. This creates several constraints:

* Bounded Search Space: Engineers naturally focus on configurations within their conceptual model, potentially missing superior alternatives outside this mental space * Static Adaptation: Hand-crafted harnesses are designed for general-purpose use and may not adapt well to specific task distributions encountered during deployment * Engineering Effort: Manual optimization requires substantial expertise and time, creating a one-shot design rather than continuous improvement * Cognitive Biases: Design choices may reflect engineering assumptions rather than empirical requirements

The 5.1 percentage point gap between Codex-CLI and the evolved approach suggests these limitations have measurable impact on real-world task completion.

These results indicate that automated harness engineering represents a viable alternative to manual optimization, particularly for complex multi-step interaction scenarios. The approach appears especially valuable for:

* Complex tool interactions requiring sophisticated error handling and context management * Domain-specific deployment where harness configuration directly impacts performance * Resource-constrained environments where manual re-engineering is infeasible * Continuous improvement pipelines where harnesses can be refined as new data becomes available

However, the comparison does not eliminate the role of hand-crafted baselines, which may offer advantages in interpretability, debuggability, and controlled deployment contexts where engineers require full understanding of interaction logic.

The emergence of evolved harness approaches raises several open questions for ongoing investigation: the transferability of evolved harnesses across different agent architectures and task domains; the scalability of observability-driven evolution to increasingly complex interaction protocols; the interpretability challenges that arise when optimal harnesses are discovered without explicit human design; and the theoretical foundations explaining why automatic evolution consistently outperforms manual engineering in this context.

• Evolved System Prompt Alone vs Evolved Full Harness
• Observability-Driven Automatic Evolution
• Agent Harness
• Harness Portability Across Model Providers
• Agent-Native Architecture