The security landscape of AI agent systems encompasses two fundamentally different attack paradigms that exploit distinct vulnerabilities in automated decision-making and human oversight mechanisms ¹⁾. Agent-centered attacks directly compromise the computational behavior and decision logic of autonomous systems, while human-in-the-loop attacks circumvent technical safeguards entirely by targeting human reviewers and approval workflows. Understanding these divergent threat models is critical for designing effective defensive strategies in systems that combine autonomous operation with human oversight.

Agent-centered attacks operate by directly manipulating or corrupting the decision-making processes of autonomous agents themselves. These attacks target the agent's reasoning mechanisms, prompt understanding, tool selection logic, or output generation at the computational level ²⁾.

Common agent-centered attack vectors include prompt injection, where malicious instructions are embedded in input data to override the agent's intended behavior; model poisoning, which corrupts the learned parameters or training data underlying agent decision-making; and adversarial examples that exploit mathematical vulnerabilities in neural network processing. These attacks assume that defeating the agent's technical safeguards is the primary objective. Defense mechanisms typically focus on input validation, model hardening, constraint enforcement through constitutional AI frameworks, and adversarial training.

The effectiveness of agent-centered attacks depends heavily on the attacker's technical sophistication and understanding of the specific agent architecture, model family, and control mechanisms in place. Agents equipped with robust instruction-following capabilities, strong safety training (such as reinforcement learning from human feedback), and layered verification systems present significantly higher barriers to these direct compromise attempts.

Human-in-the-loop attacks represent a fundamentally different threat model that deliberately avoids direct agent compromise in favor of targeting the human operators responsible for reviewing and approving agent actions ³⁾. These attacks exploit psychological and organizational vulnerabilities in oversight processes rather than technical vulnerabilities in the agent itself.

The primary mechanism of human-in-the-loop attacks involves approval fatigue, a well-documented phenomenon where reviewers operating under high volume, time pressure, or cognitive load experience degraded decision-making quality. Attackers leveraging this approach may create numerous seemingly benign requests to exhaust reviewer attention, then slip through genuinely malicious actions disguised in the volume of routine approvals. Additionally, these attacks exploit trust in summaries — when agents generate clean-looking, well-formatted, or favorable summaries of their intended actions, human reviewers may approve actions based on the summary alone without thoroughly examining the underlying implementation details or full context ⁴⁾.

A particularly effective variant involves generating diffs (change summaries) that appear innocuous or necessary while obscuring the true scope or implications of proposed actions. The attacker relies on human reviewers' natural tendency to trust well-presented information and their limited bandwidth to thoroughly audit every action, rather than on technical deception of the agent itself.

The distinction between these attack paradigms reflects different assumptions about system architecture and human-AI interaction patterns:

Technical vs. Psychological Focus: Agent-centered attacks require overcoming technical security measures, while human-in-the-loop attacks exploit human cognitive limitations and organizational processes.

Attacker Skill Requirements: Agent-centered attacks typically require deeper technical understanding of model internals and security mechanisms. Human-in-the-loop attacks leverage social engineering and understanding of human decision-making under pressure.

Detection Mechanisms: Agent-centered attacks can be detected through model monitoring, anomaly detection in reasoning traces, and behavioral analysis of agent outputs. Human-in-the-loop attacks are harder to detect automatically because the agent's behavior may remain technically sound — the compromise occurs in the approval process itself.

Defense Strategies: Protection against agent-centered attacks emphasizes technical hardening, robust training, and constraint enforcement. Defense against human-in-the-loop attacks requires organizational controls such as limiting approval volume per reviewer, requiring detailed justification rather than summaries alone, multiple independent review processes, and audit trails that expose the full decision context.

Systems implementing human oversight must recognize that the human component is not an infallible security layer but rather a potential failure point if not properly structured. The most effective defenses combine technical safeguards in the agent with organizational practices that protect human reviewers from exploitation. This includes distributing review workload to prevent fatigue, requiring substantive review beyond summary examination, implementing staggered or multi-person approval for high-impact decisions, and maintaining detailed audit logs of both agent behavior and human approval rationale.

The rise of increasingly capable and autonomous agents makes understanding these threat models essential. Organizations deploying agent systems with human oversight must explicitly address both attack vectors in their security architecture, rather than assuming that human-in-the-loop processes inherently mitigate risks.

• Agent Loop Taxonomy
• Agent Security Hardening
• AI Agents
• Agent Governance
• Devin for Security