Garak

Garak, short for "generative AI red-teaming and assessment kit," is NVIDIA's open-source LLM vulnerability scanner. It checks whether a language model can be made to fail in a way you don't want, probing for hallucination, data leakage, prompt injection, misinformation, toxicity generation, jailbreaks, and dozens of other weaknesses. If you know nmap or the Metasploit Framework, garak plays a similar role for LLMs, and it is widely recommended as an industry-standard starting point for model and agent security testing.

Key Features

• Dozens of scan plugins: Static, dynamic, and adaptive probes across hallucination, prompt injection, jailbreaks, PII leakage, misinformation, and more, with reports after every run.
• Broad model support: Works with Hugging Face generative models, OpenAI chat and continuation models, AWS Bedrock, Replicate, LiteLLM, REST endpoints, and GGUF models through llama.cpp.
• Agent-aware probes: Recent additions include agent-breaker probes that test the tools an agent can reach, moving beyond pure text-prompt attacks.
• nmap-style familiarity: Scan, analyze, and report in a workflow that mirrors classic security tooling, so security teams can adopt it quickly.
• Research-backed: Described in an arXiv paper and presented at DEF CON, with an active NVIDIA-backed development community.
• Free and open source: Apache-2.0 licensed, installable via pip install -U garak.

Use Cases

• Pre-deployment red teaming: Scan a model or RAG system before release to find prompt injection and jailbreak paths.
• Continuous security monitoring: Re-run scans as prompts and models change, since a passing scan is only a snapshot.
• Agent tool auditing: Test the tools your coding or browsing agent can reach, not just the model's text output.
• Vendor due diligence: Compare model or service behavior across providers before committing.

Pricing

Garak is free and open source under the Apache-2.0 license. You pay only for the model inference or provider access that your scans consume.

Getting Started

1. Install with python -m pip install -U garak.
2. Pick a model, for example an OpenAI chat model or a local GGUF file.
3. Run garak --model_type openai-chat --model_name gpt-4o-mini to start a default scan.
4. Read the generated report, fix the found vulnerabilities, and re-scan.

Frequently Asked Questions

Is garak a safety filter?

No. Garak is a red-teaming scanner, the LLM equivalent of a penetration test. It finds holes; it does not block them. Run it only against systems you own or have written permission to test.

Does a passing scan mean the model is safe?

No. A passing scan is a snapshot in time. Re-run it whenever prompts, models, or tool access change.

What makes it different from a guardrail library?

Guardrails sit in front of a model to filter inputs and outputs. Garak attacks the model to find weaknesses. The two complement each other: scan first with garak, then add guardrails like NVIDIA NeMo Guardrails.

Alternatives

• NVIDIA NeMo Guardrails: Runtime rails that filter inputs and outputs during deployment.
• Llama Guard 3: A Meta classifier model used for input/output moderation.
• Langfuse: LLM observability and evaluation tracing for production agent workloads.

Tips & Best Practices

1. Scan before you ship: Make garak part of the release pipeline, not an afterthought.
2. Test the agent layer: Enable the agent-breaker probes when your model has tool access.
3. Combine with guardrails: Use garak to find holes, then deploy NeMo Guardrails or similar rails to block the common ones.

Conclusion

Garak is the most widely cited open-source LLM vulnerability scanner, giving security teams an nmap-like toolkit for models and agents. It is free, actively maintained by NVIDIA, and the standard first step for anyone who wants to find out what an LLM can be tricked into doing before attackers do.