10 LLM Security Risks Every Team Should Understand
Examine the top 10 LLM security risks affecting modern applications and learn concrete defensive controls to safeguard your data, models, and infrastructure.
Understanding the Security Boundary of Large Language Models
Treat every piece of text fetched from external websites, databases, and uploaded user files as untrusted user input before passing it to an LLM.
Teams integrating machine learning models into production systems encounter security boundaries distinct from traditional web applications. Traditional software separates user data from program logic. Large language models process instructions and data within the exact same context window. Because a model treats every token as natural language text, an attacker can craft inputs that override the original instructions of the developer. This architectural ambiguity makes traditional perimeter defenses insufficient on their own.
Whether you build internal tools or buy commercial software, evaluating llm security risks is essential before deployment. Frameworks such as the OWASP Top 10 for Large Language Model Applications and MITRE ATLAS document how malicious actors exploit these systems. Applying controls early helps you identify systemic weaknesses before unauthorized actors manipulate automated workflows or compromise sensitive business data.
1. Direct and Indirect Prompt Injection
Prompt injection occurs when an attacker manipulates the context of a model to override system instructions. In direct prompt injection, a user enters crafted prompts directly into a chat interface to bypass guardrails. In indirect prompt injection, the attack payload resides in external content, such as a website, email, or uploaded document that the model reads during processing. When the system ingests this data, the model executes the embedded malicious commands instead of the user request.
Defending against prompt injection requires strict architectural separation between instructions and untrusted data. Use structured message roles where supported by model APIs to isolate developer directives from user-supplied inputs. For systems processing third-party text, implement deterministic input filters and output guardrails to block unauthorized commands. If you are deploying a secure AI chatbot, isolate untrusted data feeds and run automated red-teaming tests against injection payloads.
2. Sensitive Information Disclosure
Models can unintentionally reveal proprietary source code, confidential business records, or personal data. This exposure happens when sensitive information enters training datasets, fine-tuning corpuses, or system prompts. If user access controls are absent, an unauthorized user can craft queries that cause the model to echo protected records. The model acts as an unintended extraction channel, bypassing downstream permission layers that normally protect underlying databases.
Preventing information disclosure starts with comprehensive data sanitization. Strip personal data and internal secrets before fine-tuning models or populating retrieval indexes. Implement role-based access control on documents retrieved during generation so users only query information they are authorized to view. Documenting these handling limits within an organizational AI acceptable use policy ensures development teams do not feed restricted corporate data into public model endpoints.
3. Insecure Output Handling
Insecure output handling arises when downstream systems trust model responses without validation. If an application blindly renders model text into a web browser, sends it to an SQL interpreter, or executes it in a system shell, an attacker can trigger classic injection attacks. A model manipulated through prompt injection can generate cross-site scripting strings or malicious database commands that compromise the host environment.
Never pass raw model outputs directly to sensitive execution sinks. Treat all generated text as untrusted input. Apply context-aware output encoding before rendering responses in web browsers. Use parameterized queries for all database operations instead of assembling dynamic strings from model outputs. When models generate code or structured configuration data, validate the output against strict schemas before running downstream scripts or processes.
4. Excessive Agency
Excessive agency occurs when developers grant autonomous capabilities, system privileges, or tool access beyond what the model requires to complete its task. Many modern applications allow models to read mailboxes, query internal APIs, and execute server commands. If an indirect prompt injection attack succeeds against an over-privileged agent, the model can delete databases, transfer funds, or exfiltrate private internal communications without human oversight.
Mitigate excessive agency by enforcing the principle of least privilege across all integrated tools. Grant the model read-only access where write access is unnecessary, and avoid granting universal service account permissions. Implement human-in-the-loop approvals for irreversible or high-impact actions, such as sending external messages or modifying infrastructure records. Log every automated tool invocation so security analysts can audit agent decisions during incident investigations.
5. Training Data and Knowledge Base Poisoning
Data poisoning involves an attacker corrupting the data used to train, fine-tune, or supply knowledge to a model. When attackers manipulate public repositories, forums, or internal documentation, they introduce subtle biases, backdoors, or factual inaccuracies. If your retrieval-augmented generation pipeline ingests these poisoned sources, the model generates flawed outputs or triggers hidden behaviors when exposed to specific trigger phrases.
Securing data pipelines requires strict validation of all ingestion sources. The NCSC guidelines for secure AI system development recommend verifying supply chain integrity across data sources and maintaining cryptographic hashes of training datasets. For retrieval architectures, enforce access controls on document repositories, monitor change logs for unauthorized modifications, and vet external data providers before indexing their feeds into production databases.
6. Supply Chain Dependencies and Vulnerabilities
AI applications rely on sprawling software dependencies, including pre-trained foundation weights, vector search extensions, and open-source orchestration packages. Malicious actors frequently publish compromised Python libraries or backdoored model weights on public repositories. Loading an untrusted model file serialized with unsafe formats can trigger arbitrary code execution immediately upon import, compromising the entire host machine.
Audit the provenance of every model checkpoint and software library before deployment. Avoid deserializing model weights using unsafe formats such as standard Python pickle files; use safer formats like safetensors instead. Pin third-party dependencies to explicit versions and scan all container images for known vulnerabilities. Track these components systematically inside your cyber risk register to evaluate software bills of materials across your AI infrastructure.
7. System Prompt Leakage
System prompt leakage occurs when an unauthorized user extracts the hidden instructional preamble configured by developers. Attackers use jailbreaks and social engineering prompts to command the model to ignore formatting rules and print its setup text verbatim. If your system prompt contains sensitive internal details, intellectual property, API keys, or operational instructions, leaking the prompt undermines both security and business differentiation.
Assume that any text passed into the context window can be extracted by an adversarial user. Never store API secrets, database connection strings, or sensitive passwords inside system prompts or instructions. Treat system prompts as instructions that guide style and boundaries rather than confidentiality controls. Monitor user interactions for extraction patterns such as requests to repeat previous sentences or recite system directives.
8. Resource Exhaustion and Denial of Service
Large language models demand substantial computational power and memory to generate completions. Attackers exploit this design by sending complex, recursive, or high-token prompts engineered to consume maximum server resources. Resource exhaustion attacks can degrade application performance, render internal tools unavailable, or inflate cloud computing bills significantly when using pay-per-token API infrastructure.
Protect model endpoints with strict rate limiting, context window caps, and processing timeouts. Limit the maximum number of tokens a user can submit in a single request and cap the maximum completion length returned by the model. Implement queue management to ensure single users cannot monopolize processing clusters. Monitor GPU utilization and API consumption metrics continuously to spot anomalous usage spikes before costs escalate.
9. Insecure Plugin and Tool Integrations
When an AI system connects to external plugins and services, insecure integrations introduce traditional application vulnerabilities. Models often construct parameters for API calls dynamically from user requests. If the destination API fails to validate input formats or relies entirely on the model to sanitize data, attackers can trigger server-side request forgery, path traversal, or remote command execution on connected microservices.
Enforce strict input validation on all parameters received from model tool calls. Treat the model as an untrusted client that must adhere to standard API authentication and parameter typing rules. Run plugins inside isolated network segments with restricted egress rules to limit the blast radius if an integration gets hijacked. Require explicit authentication tokens for every downstream transaction rather than using shared administrative credentials.
10. Vector Database and Retrieval Insecurities
Retrieval-augmented generation architectures rely on vector databases to find documents semantically related to a query. If access controls are misconfigured at the vector database layer, the retrieval process can fetch documents belonging to different tenants or unauthorized clearance levels. An attacker can also execute retrieval poisoning attacks by inserting crafted documents whose embeddings match common queries, deliberately steering the model toward malicious instructions.
Apply tenant isolation and document-level authorization filters directly within vector search queries. Ensure the retrieval layer validates that the querying user has explicit permissions to read each retrieved document before returning it to the context window. Monitor similarity score distributions for anomalous clusters that may indicate an attacker is attempting to flood vector indexes with misleading or adversarial text.
How to Prioritize Your AI Defense Controls
Securing AI deployments requires prioritizing risks based on your architectural design. Organizations building conversational interfaces on top of third-party APIs face different primary exposures than engineering teams fine-tuning internal open-source models. Conducting a structured cybersecurity risk assessment allows you to map model integrations against business impact and select proportionate technical controls.
Use the defensive decision table below to identify initial baseline controls. Applying defensive depth across design, development, and operation ensures your application handles unforeseen input variations safely while protecting underlying enterprise assets.
Risk Area | Primary Defense Strategy | Verification Method
---------------------|----------------------------------------|-----------------------------------
Prompt Injection | Context separation, structured roles | Red-team prompt injection tests
Output Handling | Parameterized queries, HTML encoding | Automated dynamic AST scanning
Excessive Agency | Least privilege, human-in-the-loop | Role permission boundary review
Data Exposure | Source PII redaction, search ACLs | Retrieval authorization auditing
Resource Drain | Token quotas, rate limits, timeouts | Load testing and quota alertsFrequently Asked Questions
What is the difference between direct and indirect prompt injection?+
Direct prompt injection occurs when a user enters adversarial instructions directly into the model chat window to bypass guardrails. Indirect prompt injection happens when the model retrieves external content, such as a website or email, containing hidden instructions that hijack the processing flow without the user knowing.
Can traditional web firewalls stop LLM security risks?+
Traditional firewalls stop common network attacks and basic web exploits, but they cannot interpret natural language context. Because prompt injection relies on linguistic semantics rather than standard exploit syntax, organizations must implement AI-specific guardrails, context separation, and downstream output encoding alongside standard firewalls.
How can teams prevent sensitive data leakage through LLM applications?+
Prevent sensitive leakage by sanitizing datasets before training or indexing, enforcing strict role-based access controls within retrieval-augmented generation pipelines, and preventing employees from pasting restricted corporate information into public tools through a clear acceptable use policy.
Why is excessive agency considered a severe AI threat?+
Excessive agency gives an automated model unnecessary privileges or autonomous tools. If an attacker tricks the model through prompt injection, they can force the model to delete records, send unauthorized messages, or exfiltrate private files using the broad permissions granted to the AI agent.