Garak

Garak, short for "generative AI red-teaming and assessment kit," is NVIDIA's open-source LLM vulnerability scanner. It checks whether a language model can be made to fail in a way you don't want, probing for hallucination, data leakage, prompt injection, misinformation, toxicity generation, jailbreaks, and dozens of other weaknesses. If you know nmap or the Metasploit Framework, garak plays a similar role for LLMs, and it is widely recommended as an industry-standard starting point for model and agent security testing.

Key Features

• Dozens of scan plugins: Static, dynamic, and adaptive probes across hallucination, prompt injection, jailbreaks, PII leakage, misinformation, and more, with reports after every run.
• Broad model support: Works with Hugging Face generative models, OpenAI chat and continuation models, AWS Bedrock, Replicate, LiteLLM, REST endpoints, and GGUF models through llama.cpp.
• Agent-aware probes: Recent additions include agent-breaker probes that test the tools an agent can reach, moving beyond pure text-prompt attacks.
• nmap-style familiarity: Scan, analyze, and report in a workflow that mirrors classic security tooling, so security teams can adopt it quickly.
• Research-backed: Described in an arXiv paper and presented at DEF CON, with an active NVIDIA-backed development community.
• Free and open source: Apache-2.0 licensed, installable via pip install -U garak.

Use Cases

• Pre-deployment red teaming: Scan a model or RAG system before release to find prompt injection and jailbreak paths.
• Continuous security monitoring: Re-run scans as prompts and models change, since a passing scan is only a snapshot.
• Agent tool auditing: Test the tools your coding or browsing agent can reach, not just the model's text output.
• Vendor due diligence: Compare model or service behavior across providers before committing.

Pricing

Garak is free and open source under the Apache-2.0 license. You pay only for the model inference or provider access that your scans consume.

Getting Started

1. Install with python -m pip install -U garak.
2. Pick a model, for example an OpenAI chat model or a local GGUF file.
3. Run garak --model_type openai-chat --model_name gpt-4o-mini to start a default scan.
4. Read the generated report, fix the found vulnerabilities, and re-scan.

Frequently Asked Questions

Is garak a safety filter?

No. Garak is a red-teaming scanner, the LLM equivalent of a penetration test. It finds holes; it does not block them. Run it only against systems you own or have written permission to test.

Does a passing scan mean the model is safe?

No. A passing scan is a snapshot in time. Re-run it whenever prompts, models, or tool access change.

What makes it different from a guardrail library?

Guardrails sit in front of a model to filter inputs and outputs. Garak attacks the model to find weaknesses. The two complement each other: scan first with garak, then add guardrails like NVIDIA NeMo Guardrails.

Alternatives

• NVIDIA NeMo Guardrails: Runtime rails that filter inputs and outputs during deployment.
• Llama Guard 3: A Meta classifier model used for input/output moderation.
• Langfuse: LLM observability and evaluation tracing for production agent workloads.

Tips & Best Practices

1. Scan before you ship: Make garak part of the release pipeline, not an afterthought.
2. Test the agent layer: Enable the agent-breaker probes when your model has tool access.
3. Combine with guardrails: Use garak to find holes, then deploy NeMo Guardrails or similar rails to block the common ones.