Agent Cost Optimization

Agent cost optimization is the discipline of managing token economics, inference costs, and compute budgets for production LLM agent systems. Agents make 3-10x more LLM calls than simple chatbots, a single user request can trigger planning, tool selection, execution, verification, and response generation, easily consuming 5x the token budget of a direct chat completion. An unconstrained coding agent can cost $5-8 per task in API fees alone.¹⁾²⁾

Standard LLM pricing appears simple (pay per input/output token), but agents introduce compounding cost multipliers:

• Multi-turn loops: A ReAct loop running 10 cycles can consume 50x the tokens of a single linear pass
• Context accumulation: A 10-turn interaction costs 5x more with naive history appending
• Tool overhead: Each tool call adds tokens for the tool schema, the call, and the result parsing
• Multi-agent coordination: Orchestrator-worker patterns multiply token usage across agents

Beyond API token costs, production agents incur additional expenses from security hardening, autonomous behavior verification, and compliance reviews that must be factored into total cost of ownership calculations.³⁾

Real production cost data:⁴⁾

Prompt caching reuses previously computed key-value (KV) attention tensors for repeated prompt prefixes. For agents that resend the same system prompt, tool definitions, and conversation history across dozens of API calls, caching eliminates 40-90% of redundant computation.⁵⁾

Provider-specific caching:

Cache-friendly architecture: Keep static content (system prompt, tool definitions, few-shot examples) at the beginning of the prompt. Append dynamic content at the end to maximize prefix overlap.

$$C_{effective} = C_{uncached} \times (1 - hit\_rate \times discount)$$

For a 90% cache hit rate with Anthropic's 90% discount: $C_{effective} = C_{uncached} \times 0.19$, an 81% reduction.

Not every agent step requires a frontier model. Model routing classifies tasks by complexity and directs them to the cheapest sufficient model.

Model routing for agent cost optimization
from dataclasses import dataclass
from enum import Enum
 
class ModelTier(Enum):
    CHEAP = "gpt-4o-mini"       # $0.15/$0.60 per M tokens
    MID = "[[claude|claude]]-3.5-haiku"    # $0.80/$4.00 per M tokens
    PREMIUM = "[[claude|claude]]-3.5-sonnet"   # $3.00/$15.00 per M tokens
    FRONTIER = "[[claude|claude]]-opus"    # $15.00/$75.00 per M tokens
 
@dataclass
class RoutingDecision:
    model: ModelTier
    reason: str
 
class AgentRouter:
    def __init__(self, classifier):
        self.classifier = classifier
 
    def route(self, task, context):
        complexity = self.classifier.classify(task)
        if complexity == "simple":
            return RoutingDecision(ModelTier.CHEAP, "Routine subtask")
        elif complexity == "moderate":
            return RoutingDecision(ModelTier.MID, "Standard reasoning")
        elif complexity == "complex":
            return RoutingDecision(ModelTier.PREMIUM, "Complex reasoning")
        else:
            return RoutingDecision(ModelTier.FRONTIER, "Max capability needed")
 
    def estimate_savings(self, task_distribution):
        costs = {
            ModelTier.CHEAP: 0.15, ModelTier.MID: 0.80,
            ModelTier.PREMIUM: 3.00, ModelTier.FRONTIER: 15.00
        }
        routed = sum(costs[self.route(t, {}).model] * pct
                     for t, pct in task_distribution.items())
        return 1 - (routed / 15.00)

A typical task distribution (60% simple, 25% moderate, 12% complex, 3% frontier) with routing yields up to 80% cost reduction versus routing everything through a frontier model. Real-world implementations using specialized stacks demonstrate substantial cost reductions; for example, replacing a monolithic approach that routes every call through a 400B+ parameter model with a specialized stack can reduce per-interaction costs from $1.50 to $0.15.⁶⁾

Critically, scaling agent quantity provides diminishing returns on cost optimization. Research shows that increasing agent count from 64 to 256 agents yields no meaningful quality improvement despite proportional cost increases; model quality and protocol design account for the vast majority of performance variation.⁷⁾ This emphasizes that quality model selection should take priority over quantity-based scaling strategies.

Prompt compression reduces token count while preserving semantic content:

• LLMLingua-2: Compresses prompts up to 5x by identifying and removing redundant tokens⁸⁾
• Incremental summarization: Replace full conversation history with rolling summaries
• Observation masking: Strip verbose tool outputs to essential fields
• Schema pruning: Include only relevant tool definitions per step, not the full catalog

$$C_{optimized} = C_{base} \times compression\_ratio \times (1 - cache\_hit \times cache\_discount)$$

Semantic caching stores LLM responses for similar queries in vector databases, eliminating API calls entirely for 20-40% of repetitive traffic. Unlike exact-match caching, semantic caching uses embedding similarity:

• “What are your business hours?” matches “When are you open?”
• Threshold tuning balances hit rate against response accuracy

Tools: Redis with vector search, GPTCache, Pinecone-based solutions.

Production agent cost management requires instrumentation from day one:

• Per-step cost tracking: Log token usage, model used, and cost for every LLM call
• Budget guardrails: Set per-request and per-user token limits with graceful degradation. Viewing context as a budget rather than a log—where every token spent on tool outputs or history is a cost limiting new user requests—leads to more robust agent architectures that proactively manage token usage.⁹⁾
• Anomaly detection: Alert on cost spikes from infinite loops or unexpected tool chains
• Batch processing: Route non-interactive tasks to batch APIs for 50% token savings
• Cost-aware evaluation: Evaluation methodologies that track token consumption, inference cost, and runtime efficiency alongside accuracy metrics are critical as agent coding can consume 1000x more tokens than traditional chat approaches.¹⁰⁾

Tooling: LangSmith, Braintrust, Helicone, and custom dashboards built on provider usage APIs.¹¹⁾

Combining all techniques can reduce agent costs by 70-90% compared to naive implementations.

• Cost-Aware Agent Evaluation
• Competitive Programming Agents
• Caching Strategies for Agents
• Per-Request vs Token-Based Pricing
• Agent Runtime Economics