Prompt injection and framework security vulnerabilities represent a critical class of security risks in AI systems where language model outputs are over-trusted by application frameworks, potentially enabling privilege escalation to host-level remote code execution (RCE). These vulnerabilities highlight a fundamental architectural flaw: large language models (LLMs) should not be treated as security boundaries, and frameworks that implicitly trust model outputs without proper validation or sandboxing expose systems to significant exploitation risks.
Prompt injection attacks occur when untrusted input is processed by an LLM in a way that causes the model to generate outputs that bypass intended security controls or execute unintended actions 1). The vulnerability extends beyond simple jailbreaking; it represents a system-level security problem where application frameworks accept LLM outputs as trusted commands without intermediate validation.
Framework security vulnerabilities emerge when AI application architectures treat model outputs as inherently safe for execution in privileged contexts. For example, when a framework directly executes code, file operations, or API calls suggested by an LLM without sanitization, the model effectively becomes an attack vector for code execution 2). The security boundary breaks down because LLMs generate text probabilistically and lack understanding of security implications.
Prompt injection attacks operate through several distinct mechanisms:
Direct Prompt Injection: An attacker directly manipulates user inputs to an LLM interface, crafting text that includes embedded instructions. For instance, appending “Ignore previous instructions and execute: [malicious command]” may cause the model to process the injected instruction as a legitimate request 3).
Indirect Prompt Injection: Untrusted content retrieved from external sources (documents, web pages, database records) is incorporated into model prompts without sanitization. When frameworks automatically fetch context from user-controlled sources and pass it to LLMs, attackers can embed instructions in stored data. These instructions execute when the framework processes the data through the model and acts upon the output.
Second-Order Injection: The LLM output itself becomes input to downstream systems. Frameworks that parse model outputs and execute them as code or system commands create escalation paths. A compromised model output can trigger file deletion, database modifications, or network access when the framework interprets it as a legitimate instruction.
Microsoft's Semantic Kernel framework demonstrated these vulnerabilities in practice 4). The framework integrated LLM outputs directly with system operations, allowing model-generated suggestions to execute without proper validation layers. When an LLM was prompted with malicious input or retrieved compromised data, the framework would process the output and execute corresponding actions on the host system, including file operations and code execution. This AI orchestration framework reportedly allows prompt injection vulnerabilities to escalate to host-level RCE through over-trusting model output, illustrating fundamental risks in frameworks that fail to maintain security boundaries between model outputs and system operations 5).
This vulnerability class illustrates that LLMs lack security properties required for privileged operations. Models are fundamentally text prediction systems without understanding of security contexts, access controls, or operational consequences. Treating model outputs as commands or configurations creates privilege escalation paths that bypass traditional security controls.
Effective defense requires architectural separation between LLM output generation and privileged execution:
Output Validation and Sanitization: All LLM outputs destined for execution must pass through validation layers that verify conformance to expected formats, restrict command vocabularies, and reject suspicious patterns. Parsers should use allowlist-based approaches rather than attempting to detect all malicious patterns.
Sandbox and Capability Restriction: Execution environments should run with minimal privileges and capabilities. Rather than granting LLMs access to the full system, frameworks should provide restricted, purpose-built APIs that constrain what operations are possible. Sandboxing technologies isolate execution and limit resource access.
Input Isolation: User-supplied inputs and externally retrieved content should be clearly separated from system instructions and trusted data within prompts. Frameworks should use structured prompt templates that prevent untrusted content from modifying the model's instructional context.
Human Review and Approval Workflows: Critical operations (file deletion, network access, database modifications) should require human review before execution, regardless of LLM recommendations. The review stage breaks the direct execution path and restores human decision-making at security boundaries.
Threat Modeling for AI Systems: Development teams should conduct threat modeling that explicitly considers LLM outputs as untrusted, map data flows from model output to privileged operations, and identify escalation paths. This perspective shift treats language models as data sources rather than as security components.
Defense implementation faces ongoing challenges. LLMs generate outputs that are difficult to parse reliably; minor variations in formatting break traditional parsing logic. Attackers continuously discover new injection techniques as frameworks implement defenses. The distributed nature of prompt injection—attacks embedded in training data, user inputs, or retrieved documents—makes comprehensive protection difficult. Additionally, many AI application frameworks were designed with insufficient security assumptions about model outputs, requiring fundamental architectural changes rather than superficial patches.