LLM Security Threat Model and Attack Surface Analysis
This article outlines a comprehensive threat model for Large Language Model (LLM) applications, emphasizing the unique security challenges arising from LLMs' unified processing of instructions and data. It maps common vulnerabilities across the LLM pipeline, from input to output and supply chain, and distinguishes between model-centric and agent-centric attacks to guide effective defense strategies.
Read original on ByteByteGoThe core security challenge in LLM applications stems from the model's inability to distinguish between instructions and data, processing both as a single sequence of tokens. This contrasts sharply with traditional software like parameterized database queries, where code and data are structurally separated. This fundamental property leads directly to prompt injection vulnerabilities, where malicious instructions can be embedded in user input or retrieved content, altering the model's intended behavior.
The LLM Attack Surface Map
The article presents an attack surface map aligned with the OWASP Top 10 for LLM Applications, illustrating how threats manifest across the LLM pipeline:
- Input Stage: Direct prompt injection and unbounded consumption (denial of wallet).
- Retrieval Stage (RAG): Indirect injection, vector/embedding weaknesses, and poisoned RAG attacks, where malicious data in the knowledge base can corrupt answers.
- Model Stage: Training-data leakage, data/model poisoning, and system prompt leakage.
- Tools Stage: Excessive agency, where agents possess more permissions than necessary for their tasks.
- Output Stage: Improper output handling (lack of sanitization) and misinformation.
- Supply Chain: Compromise of any component (model, adapter, vector store, tool) before runtime defenses can activate.
Prompt Injection Explained
Prompt injection occurs when instruction-like text is supplied to an LLM, causing the model to prioritize these instructions over the operator's intent. This can happen directly through chat input or indirectly via retrieved content (e.g., malicious email in an inbox being summarized by Copilot).
The Lethal Trifecta: Excessive Agency
The most significant risk for LLM applications lies in
An agent with all three capabilities can be directed by injected instructions to exfiltrate private data to an attacker. Mitigating this risk usually involves removing one of these capabilities, with restricting outbound channels or narrowing data access often being the most cost-effective strategies.
Model vs. Agent Attacks: Prioritizing Defenses
While model-level attacks (e.g., model theft, training-data extraction, poisoning) are real, they are often expensive and may be mitigated by model providers for most users. For teams hosting open models or fine-tuning sensitive data, these become critical. However, for many, the greater immediate risk lies in attacks exploiting an agent's excessive agency, which are more common and can lead to direct material damage.