This collection is for engineers who need practical ways to defend LLM systems against adversarial inputs, jailbreaks, prompt injection, tool misuse, and unsafe model behaviour. It focuses on resources that move beyond abstract safety discussion into concrete methods: red-team test design, adversarial evaluation datasets, system-level mitigations, permission boundaries, monitoring, and repeatable CI-style safety checks.
| Resource | Date published | Summary |
|---|---|---|
| Google — Evaluate model and system for safety | Unknown | Practical Google checklist for building safety evals, adversarial datasets, red-team exercises, and held-out assurance testing across the model lifecycle. (Google AI for Developers) |
| Google — Google’s Approach for Secure AI Agents | 2025 | Best high-level Google framing of agent security: permissions, containment, trusted/untrusted input separation, monitoring, and non-prompt controls. (Google Cloud Storage) |
| Google DeepMind — Advancing Gemini’s security safeguards | 2025 | Short, accessible overview of how Google uses automated red teaming to harden Gemini against indirect prompt injection during tool use. (Google DeepMind) |
| Google DeepMind — Lessons from Defending Gemini Against Indirect Prompt Injections | 2025 | Most directly relevant Gemini technical report for understanding indirect prompt injection threat modelling, adaptive attacks, automated red-teaming, and defensive evaluation. (arXiv) |
| Google DeepMind — Gemini 1 Technical Report | 2023 | Useful broader context on Gemini’s multimodal model design, safety evaluation, red-teaming, dogfooding, and external evaluation process. (arXiv) |
| OpenAI — Advancing red teaming with people and AI | 2024 | Read this for OpenAI’s process view on combining human red-teamers, AI-assisted attack generation, and reusable safety benchmarks. (OpenAI) |
| OpenAI — External Red Teaming for AI Models and Systems | 2025 | Best OpenAI resource on how to structure external red teaming so findings turn into repeatable evals, risk assessment, and mitigation work. (arXiv) |
| OpenAI — Preparedness Framework v2 | 2025 | Read this for OpenAI’s frontier-risk governance model, including capability thresholds, adversarial testing, red-teaming, and deployment gating. (OpenAI) |
| OpenAI — GPT-4o System Card | 2024 | Good concrete example of how OpenAI reports model-level safety evaluations, red-team findings, and residual risks for a deployed model. |
| OpenAI — GPT-5 System Card | 2025 | Useful example of system-level prompt-injection evaluation across connectors, tools, and in-product safeguards rather than model-only testing. (OpenAI) |
| Anthropic — Responsible Scaling Policy | 2026 | Read this for Anthropic’s formal safety-governance approach: risk tiers, safety cases, deployment constraints, and assurance expectations. (Anthropic) |
| Anthropic — RSP v3 announcement | 2026 | Shorter companion to the full policy, useful for understanding how Anthropic’s frontier-safety posture has evolved. (Anthropic) |
| Anthropic — Constitutional Classifiers / jailbreak defenses | 2025 | Read this for Anthropic’s approach to defending against jailbreaks using model-based classifiers rather than relying only on prompts or refusal text. (Anthropic) |
| Microsoft — AI Red Team guidance | Unknown | Strong practical enterprise guide for organising AI red teams, testing AI products, and translating findings into security controls. (Microsoft Learn) |
| Microsoft — Lessons from Red Teaming 100 Generative AI Products | 2025 | Excellent applied paper for real-world failure modes, including prompt injection, cross-prompt injection, multimodal attacks, and the limits of automated testing. (arXiv) |
| NIST — AI RMF Generative AI Profile | 2024 | Read this for a standards-based taxonomy of GenAI risks, including internal/external evals, red-teaming, impact assessment, and governance. (NIST) |
| NIST — Adversarial Machine Learning Taxonomy and Terminology | 2025 | Useful for precise terminology across adversarial ML: attacker goals, knowledge, capabilities, lifecycle stage, and attack classes. (NIST) |
| OWASP — Top 10 for LLM Applications | Unknown | Essential practitioner taxonomy for LLM app security risks: prompt injection, data leakage, tool misuse, supply chain, insecure output handling, and overreliance. (OWASP Gen AI Security Project) |
| OWASP — LLM01 Prompt Injection | Unknown | Best single reference for distinguishing prompt injection, jailbreaks, direct attacks, indirect attacks, and mitigation patterns. (OWASP Gen AI Security Project) |
| Meta — Purple Llama / CyberSecEval | 2023 | Read this for open tooling and benchmarks around cybersecurity evals, prompt injection defenses, Llama Guard, and Prompt Guard. (AI Meta) |
| NVIDIA — garak LLM vulnerability scanner | 2024 | Practical open-source tool for automated LLM red-teaming across jailbreaks, prompt injection, data leakage, hallucination, toxicity, and other failure modes. (arXiv) |
| promptfoo — LLM red teaming guide | Unknown | Useful hands-on guide for turning adversarial test cases into repeatable CI-style red-team evaluations for LLM apps. (Promptfoo) |
| Rauh et al. — Gaps in the Safety Evaluation of Generative AI | 2024 | Read this for a critical view of why model-only benchmarks are insufficient and why safety evals need real-world, sociotechnical context. (AAAI Publications) |
| Ofcom — Red Teaming for GenAI Harms | 2024 | Good non-lab guide for planning red-team exercises, assembling teams, defining harms, and reporting results in a policy/regulatory context. (ofcom.org.uk) |