What is AI red teaming?
Short answer
AI red teaming is structured, authorized testing of an AI system by people trying to make it fail: bypassing its safety rules, extracting sensitive data, triggering harmful output or abusing its tools. The findings feed fixes and ongoing monitoring before and after launch.
The details
Useful references for planning tests include the OWASP Top 10 for LLM Applications and MITRE ATLAS. Test only systems you own or have written permission to test.
Terms explained
- MITRE ATLAS
- MITRE ATLAS is a public knowledge base of adversary tactics and techniques against AI-enabled systems, modeled on MITRE ATT&CK.
- Penetration testing
- Penetration testing is an authorized, simulated attack on a system to find and demonstrate exploitable weaknesses before real attackers do.
- OWASP Top 10
- The OWASP Top 10 is an awareness list of the most critical security risks to web applications, published by the Open Worldwide Application Security Project.
Sources
Related questions
Related Guides
GuideAI Security
How to Conduct LLM Security Testing for Web Applications
Learn how to perform LLM security testing to identify prompt injection, sensitive data leakage, and insecure output handling before production deployment.
Read more GuideAI Security
9 Rules for Safe Use of Generative AI at Work
Discover nine practical rules for the safe use of generative AI at work. Protect company data, verify model outputs, and establish clear operational controls.
Read more GuideAI Security
AI Agent Security: Implementing Least Privilege for Tools
Learn how to enforce least privilege for AI agent security. Control tool calling, scope data access, and prevent unauthorized actions in production.
Read more GuideAI Security
10 LLM Security Risks Every Team Should Understand
Examine the top 10 LLM security risks affecting modern applications and learn concrete defensive controls to safeguard your data, models, and infrastructure.
Read more