What is data poisoning in AI?
Short answer
Data poisoning is when an attacker manipulates the data used to train, fine-tune or ground an AI model, so the model learns harmful behavior, hidden backdoors or wrong answers. It also applies to retrieval systems that pull documents into a model’s context at run time.
The details
Defenses include controlling and recording where training and retrieval data come from, restricting who can add content to knowledge bases, and testing model behavior after each data change.
Terms explained
- Data poisoning
- Data poisoning is the manipulation of data used to train, fine-tune or ground an AI model so that it behaves in a way the attacker wants.
- MITRE ATLAS
- MITRE ATLAS is a public knowledge base of adversary tactics and techniques against AI-enabled systems, modeled on MITRE ATT&CK.
Sources
Related questions
Related Guides
GuideAI Security
How to Deploy a Secure AI Chatbot for Small Business
Learn how to configure, test, and run a secure AI chatbot for your small business. Protect customer data, prevent prompt injection, and limit model access.
Read more GuideAI Security
9 Rules for Safe Use of Generative AI at Work
Discover nine practical rules for the safe use of generative AI at work. Protect company data, verify model outputs, and establish clear operational controls.
Read more GuideAI Security
AI Agent Security: Implementing Least Privilege for Tools
Learn how to enforce least privilege for AI agent security. Control tool calling, scope data access, and prevent unauthorized actions in production.
Read more GuideAI Security
How to Conduct LLM Security Testing for Web Applications
Learn how to perform LLM security testing to identify prompt injection, sensitive data leakage, and insecure output handling before production deployment.
Read more