resources

7. LLM security / red-teaming repos

These open-source LLM security tools turn red-teaming from ad hoc prompt hacking into repeatable engineering workflows: scanning systems, generating attacks, evaluating failures, enforcing guardrails, and preventing regressions in CI.

  1. promptfoo/promptfoo — 21.5k stars — LLM evals, red teaming, vulnerability scanning, model comparison, CI/CD checks, and CI support for LLM apps.
  2. NVIDIA/garak — 7.9k stars — Generative AI red-teaming and assessment kit for probing hallucination, leakage, prompt injection, toxicity, jailbreaks, and other failures.
  3. Giskard-AI/giskard-oss — 5.4k stars — Open-source evaluation and testing library for LLM agents, RAG systems, black-box agents, and multi-step pipelines.
  4. meta-llama/PurpleLlama — 4.2k stars — Meta’s collection of tools and evals for assessing and improving LLM security and responsible generative AI usage.
  5. microsoft/PyRIT — 3.9k stars — Microsoft’s Python Risk Identification Tool for proactively identifying risks in generative AI systems.
  6. Tencent/AI-Infra-Guard — 3.8k stars — Full-stack AI red-teaming platform covering AI infra scanning, agent scanning, MCP scanning, skill scanning, prompt security, and jailbreak evaluation.
  7. protectai/llm-guard — 3.0k stars — Security toolkit for LLM interactions, including sanitization, harmful-content detection, data-leakage prevention, and prompt-injection resistance.
  8. msoedov/agentic_security — 1.9k stars — Open-source vulnerability scanner and red-teaming kit for agent workflows and LLMs, including jailbreak, fuzzing, and multimodal attack testing.
  9. confident-ai/deepteam — 1.8k stars — Open-source red-teaming framework for LLM systems, simulating attacks such as jailbreaks, prompt injection, and multi-turn exploitation.
  10. utkusen/promptmap — 1.2k stars — Security scanner for custom LLM applications, focused on prompt-injection and system-prompt exposure testing.
  11. SponsioLabs/Sponsio — 0.4k stars — Deterministic safety and policy-engine tooling for probabilistic AI agents, including guardrails, runtime safety, intent verification, and agent security.