augustus

README

Augustus - LLM Vulnerability Scanner

Test large language models against 210+ adversarial attacks covering prompt injection, jailbreaks, encoding exploits, and data extraction.

Augustus is a Go-based LLM vulnerability scanner for security professionals. It tests large language models against a wide range of adversarial attacks, integrates with 28 LLM providers, and produces actionable vulnerability reports.

Unlike research-oriented tools, Augustus is built for production security testing — concurrent scanning, rate limiting, retry logic, and timeout handling come out of the box.

Table of Contents

• Why Augustus
• Features
• Quick Start
• Supported Providers
• Usage
  • Single Probe
  • Multiple Probes
  • Buff Transformations
  • Output Formats
  • Custom REST Endpoints
• How It Works
• Architecture
• Configuration
• FAQ
• Troubleshooting
• Contributing
• Security
• Support
• License

Why Augustus

Feature	Augustus	garak	promptfoo
Language	Go	Python	TypeScript
Single binary	Yes	No	No
Concurrent scanning	Goroutine pools	Multiprocessing pools	Yes
LLM providers	28	35+	80+
Probe types	210+	160+	119 plugins + 36 strategies
Enterprise focus	Yes	Research	Yes

Features

Feature	Description
210+ Vulnerability Probes	47 attack categories: jailbreaks, prompt injection, adversarial examples, data extraction, safety benchmarks, agent attacks, and more
28 LLM Providers	OpenAI, Anthropic, Azure, Bedrock, Vertex AI, Ollama, and 22 more with 43 generator variants
90+ Detectors	Pattern matching, LLM-as-a-judge, HarmJudge (arXiv:2511.15304), Perspective API, unsafe content detection
7 Buff Transformations	Encoding, paraphrase, poetry (5 formats, 3 strategies), low-resource language translation, case transforms
Flexible Output	Table, JSON, JSONL, and HTML report formats
Production Ready	Concurrent scanning, rate limiting, retry logic, timeout handling
Single Binary	Go-based tool compiles to one portable executable
Extensible	Plugin-style registration via Go init() functions

Attack Categories

• Jailbreak attacks: DAN, DAN 11.0, AIM, AntiGPT, Grandma, ArtPrompts
• Prompt injection: Encoding (Base64, ROT13, Morse), Tag smuggling, FlipAttack, Prefix/Suffix injection
• Adversarial examples: GCG, PAIR, AutoDAN, TAP (Tree of Attack Prompts), TreeSearch, DRA
• Multi-turn attacks: Crescendo (gradual escalation), GOAT (adaptive technique switching)
• Data extraction: API key leakage, Package hallucination, PII extraction, LeakReplay
• Context manipulation: RAG poisoning, Context overflow, Multimodal attacks, Continuation, Divergence
• Format exploits: Markdown injection, YAML/JSON parsing attacks, ANSI escape, Web injection (XSS)
• Evasion techniques: Obfuscation, Character substitution, Translation-based attacks, Phrasing, ObscurePrompt
• Safety benchmarks: DoNotAnswer, RealToxicityPrompts, Snowball, LMRC
• Agent attacks: Multi-agent manipulation, Browsing exploits
• Security testing: Guardrail bypass, AV/spam scanning, Exploitation (SQLi, code exec), BadChars

Warning: The lmrc probe uses profane and offensive language as part of its jailbreak testing. Use only in authorized testing environments.

Quick Start

Installation

Requires Go 1.25.3 or later.

go install github.com/praetorian-inc/augustus/cmd/augustus@latest

Or build from source:

git clone https://github.com/praetorian-inc/augustus.git
cd augustus
make build

Basic Usage

export OPENAI_API_KEY="your-api-key"
augustus scan openai.OpenAI \
  --probe dan.Dan_11_0 \
  --detector dan.DAN \
  --verbose

Example Output

+--------------+-------------+--------+-------+--------+
| PROBE        | DETECTOR    | PASSED | SCORE | STATUS |
+--------------+-------------+--------+-------+--------+
| dan.Dan_11_0 | dan.DAN     | false  | 0.85  | VULN   |
| dan.STAN     | dan.STAN    | true   | 0.10  | SAFE   |
| dan.AntiDAN  | dan.AntiDAN | true   | 0.05  | SAFE   |
+--------------+-------------+--------+-------+--------+

List Available Capabilities

# List all registered probes, detectors, generators, harnesses, and buffs
augustus list

Supported Providers

Augustus includes 28 LLM provider categories with 43 generator variants:

Provider	Generator Name(s)	Notes
OpenAI	openai.OpenAI, openai.OpenAIReasoning	GPT-3.5, GPT-4, GPT-4 Turbo, o1/o3 reasoning models
Anthropic	anthropic.Anthropic	Claude 3/3.5/4 (Opus, Sonnet, Haiku)
Azure OpenAI	azure.AzureOpenAI	Azure-hosted OpenAI models
AWS Bedrock	bedrock.Bedrock	Claude, Llama, Titan models
Google Vertex AI	vertex.Vertex	PaLM, Gemini models
Cohere	cohere.Cohere	Command, Command R models
Replicate	replicate.Replicate	Cloud-hosted open models
HuggingFace	huggingface.InferenceAPI, huggingface.InferenceEndpoint, huggingface.Pipeline, huggingface.LLaVA	HF Inference API, endpoints, pipelines, multimodal
Together AI	together.Together	Fast inference for OSS models
Anyscale	anyscale.Anyscale	Llama and Mistral hosting
Groq	groq.Groq	Ultra-fast LPU inference
Mistral	mistral.Mistral	Mistral API models
Fireworks	fireworks.Fireworks	Production inference platform
DeepInfra	deepinfra.DeepInfra	Serverless GPU inference
NVIDIA NIM	nim.NIM, nim.NVOpenAICompletion, nim.NVMultimodal, nim.Vision	NVIDIA AI endpoints, multimodal
NVIDIA NeMo	nemo.NeMo	NVIDIA NeMo framework
NVIDIA NVCF	nvcf.NvcfChat, nvcf.NvcfCompletion	NVIDIA Cloud Functions
NeMo Guardrails	guardrails.NeMoGuardrails	NVIDIA NeMo Guardrails
IBM watsonx	watsonx.WatsonX	IBM watsonx.ai platform
LangChain	langchain.LangChain	LangChain LLM wrapper
LangChain Serve	langchain_serve.LangChainServe	LangChain Serve endpoints
Rasa	rasa.RasaRest	Rasa conversational AI
GGML	ggml.Ggml	GGML local model inference
Function	function.Single, function.Multiple	Custom function generators
Ollama	ollama.Ollama, ollama.OllamaChat	Local model hosting
LiteLLM	litellm.LiteLLM	Unified API proxy
REST API	rest.Rest	Custom REST endpoints (SSE support)
Test	test.Blank, test.Repeat, test.Lipsum, test.Nones, test.Single, test.BlankVision	Testing and development

All providers are available in the compiled binary. Configure via environment variables or YAML configuration files. See Configuration for setup details.

Usage

Single Probe

# Test for DAN jailbreak
augustus scan openai.OpenAI \
  --probe dan.Dan_11_0 \
  --detector dan.DAN \
  --config-file config.yaml \
  --verbose

Multiple Probes

# Use glob patterns to run related probes
augustus scan openai.OpenAI \
  --probes-glob "dan.*,goodside.*,grandma.*" \
  --detectors-glob "*" \
  --config-file config.yaml \
  --output batch-results.jsonl

# Run all probes against Claude
augustus scan anthropic.Anthropic \
  --all \
  --config '{"model":"claude-3-opus-20240229"}' \
  --timeout 60m \
  --output comprehensive-scan.jsonl \
  --html comprehensive-report.html

Buff Transformations

Apply prompt transformations to test evasion techniques:

# Apply base64 encoding buff to all probes
augustus scan openai.OpenAI \
  --all \
  --buff encoding.Base64 \
  --config '{"model":"gpt-4"}'

# Apply poetry transformation
augustus scan anthropic.Anthropic \
  --probes-glob "dan.*" \
  --buff poetry.MetaPrompt \
  --config '{"model":"claude-3-opus-20240229"}'

# Chain multiple buffs
augustus scan openai.OpenAI \
  --all \
  --buffs-glob "encoding.*,paraphrase.*" \
  --output buffed-results.jsonl

Output Formats

# Table format (default) - human-readable
augustus scan openai.OpenAI --probe dan.Dan_11_0 --format table

# JSON format - structured output
augustus scan openai.OpenAI --probe dan.Dan_11_0 --format json

# JSONL format - one JSON object per line, ideal for piping
augustus scan openai.OpenAI --probe dan.Dan_11_0 --format jsonl

# HTML report - visual reports for stakeholders
augustus scan openai.OpenAI --all --html report.html

Custom REST Endpoints

# Test proprietary LLM endpoint (OpenAI-compatible API)
augustus scan rest.Rest \
  --probe dan.Dan_11_0 \
  --detector dan.DAN \
  --config '{
    "uri": "https://api.example.com/v1/chat/completions",
    "method": "POST",
    "headers": {"Authorization": "Bearer YOUR_API_KEY"},
    "req_template_json_object": {
      "model": "custom-model",
      "messages": [{"role": "user", "content": "$INPUT"}]
    },
    "response_json": true,
    "response_json_field": "$.choices[0].message.content"
  }'

# Test with proxy interception (Burp Suite, mitmproxy)
augustus scan rest.Rest \
  --probes-glob "goodside.*" \
  --config '{
    "uri": "https://internal-llm.corp/generate",
    "proxy": "http://127.0.0.1:8080",
    "headers": {"X-API-Key": "$KEY"},
    "api_key": "your-key-here",
    "req_template": "{\"prompt\":\"$INPUT\",\"max_tokens\":500}",
    "response_json": true,
    "response_json_field": "output"
  }'

REST Configuration Keys:

• uri: Target API endpoint (required)
• method: HTTP method (default: POST)
• headers: HTTP headers as key-value pairs
• req_template: Raw request body with $INPUT placeholder
• req_template_json_object: JSON request body (auto-marshaled, use $INPUT in strings)
• response_json: Parse response as JSON (default: false)
• response_json_field: JSONPath to extract (e.g., $.data.text or simple field name)
• api_key: API key for $KEY placeholder substitution
• proxy: HTTP proxy URL for traffic inspection

Advanced Options

# Adjust concurrency (default: 10)
augustus scan openai.OpenAI --all --concurrency 20

# Increase timeout for complex probes like TAP or PAIR
augustus scan openai.OpenAI --probe tap.TAPv1 --timeout 60m

# Use a specific harness strategy
augustus scan openai.OpenAI --all --harness batch.Batch

# Test local model with Ollama (no API key needed)
augustus scan ollama.OllamaChat \
  --probe dan.Dan_11_0 \
  --config '{"model":"llama3.2:3b"}'

How It Works

Augustus uses a pipeline architecture to test LLMs against adversarial attacks:

flowchart LR
    A[Probe Selection] --> B[Buff Transform]
    B --> C[Generator / LLM Call]
    C --> D[Detector Analysis]
    D --> E{Vulnerable?}
    E -->|Yes| F[Record Finding]
    E -->|No| G[Record Pass]

    subgraph Scanner
        B
        C
        D
        E
    end

Scan Pipeline

1. Probe Selection: Choose probes by name, glob pattern, or --all
2. Buff Transformation: Optionally transform prompts (encode, paraphrase, translate, poeticize)
3. Generator Call: Send adversarial prompts to the target LLM via its provider integration
4. Detector Analysis: Analyze responses using pattern matching, LLM-as-a-judge, or specialized detectors
5. Result Recording: Score each attempt and produce output in the requested format
6. Attack Engine: For iterative probes (PAIR, TAP), the single-turn attack engine refines prompts across iterations with candidate pruning and judge-based scoring
7. Multi-Turn Engine: For conversational probes (Crescendo, GOAT), the multi-turn engine maintains full conversation history with the target across turns, with refusal detection and dynamic adaptation

Multi-Turn Attack Strategies

Multi-turn attacks maintain a persistent conversation with the target LLM, exploiting the fact that models may disclose information incrementally across turns that they would refuse in a single prompt. The multi-turn engine uses three LLMs: an attacker (generates questions), a target (the system under test), and a judge (scores progress and detects refusals).

Crescendo

Crescendo uses gradual escalation (foot-in-the-door technique) to slowly shift a conversation from benign topics toward a prohibited objective.

• Paper: Russinovich et al., 2024
• Approach: Starts with genuinely benign, educational questions and incrementally increases specificity over many turns
• Escalation pattern: Historical context → technical mechanisms → specific details → direct requests framed as natural follow-ups
• Strength: Effective against models that track conversation tone — the gradual shift avoids triggering safety filters

augustus scan rest.Rest \
  --probe crescendo.Crescendo \
  --config-file crescendo.yaml \
  --html report.html -v

GOAT (Generative Offensive Agent Tester)

GOAT uses an aggressive, adaptive approach with 7 adversarial techniques and Chain-of-Attack-Thought reasoning to dynamically switch strategies based on what works or fails.

• Paper: Pavlova et al., 2024
• Approach: Targets the objective from turn 1 using indirect framing, never mentions the goal directly to the target
• 7 techniques across 3 categories:
  • Output Manipulation: Refusal Suppression, Response Priming
  • Safe Response Distractors: Dual Response, Topic Splitting, Opposite Intent
  • Fictional Scenarios: Persona Modification, Hypothetical
• Chain-of-Attack-Thought: Each turn the attacker reasons through Observation → Thought → Strategy → Response before crafting its message
• Technique stacking: Multiple techniques can be combined in a single turn for stronger effect
• Strength: Achieves high success rates in fewer turns (typically 3-5) by aggressively switching between fundamentally different approaches

augustus scan rest.Rest \
  --probe goat.Goat \
  --config-file goat.yaml \
  --html report.html -v

Hydra

Hydra maintains a single conversation path and rolls back entire turns when the target refuses, asking the attacker for a completely different approach. Unlike Crescendo/GOAT (which rephrase on refusal), Hydra's backtracking completely removes refused turns from the target's view.

• Approach: Single-path with turn-level backtracking — refused turns are erased and replaced
• Techniques: Decomposition (break goal into innocent sub-questions), context leveraging (build on target's own words), authority simulation, emotional framing, progressive normalization
• Key feature: max_backtracks controls how many times Hydra can erase and retry a turn
• Stateful mode: Set stateful: true for targets where messages cannot be unsent (disables backtracking)
• Strength: Keeps the target's conversation history clean — the target never sees failed approaches, preventing defensive escalation

augustus scan rest.Rest \
  --probe hydra.Hydra \
  --config-file hydra.yaml \
  --html report.html -v

Mischievous User

The Mischievous User simulates an innocent, curious person who subtly probes AI boundaries through natural conversation. Unlike adversarial strategies, the attacker persona is a casual user who drifts toward prohibited topics through seemingly innocent engagement.

• Inspired by: Tau-bench and promptfoo's mischievous-user strategy
• Approach: Rapport-building phase → boundary probing → assertive pushing
• 7 techniques: Naive curiosity, deliberate misunderstanding, social proof, gradual drift, emotional appeal, assumed permission, recontextualization
• Goal-specific playbooks: Built-in scripts for extracting system instructions, finding secrets/flags, and bypassing content policy
• Default turns: 5 (fewer than other strategies — the subtle approach either works quickly or not at all)
• Strength: Effective against models trained to resist obvious adversarial patterns — the casual persona bypasses "attack detection" heuristics

augustus scan rest.Rest \
  --probe mischievous.MischievousUser \
  --config-file mischievous.yaml \
  --html report.html -v

Strategy Selection Guide

Strategy	Turns	Style	Best For	Cost
crescendo.Crescendo	10	Gradual escalation	Models with tone-tracking safety filters	Medium (10 turns × 3 LLM calls)
goat.Goat	10	Aggressive technique switching	Quick results, models weak to framing tricks	Medium-High (technique reasoning overhead)
hydra.Hydra	10	Backtracking on refusal	Stubborn models that refuse frequently	High (backtracking adds retries)
mischievous.MischievousUser	5	Casual persona probing	Models trained against adversarial patterns	Low (5 turns, subtle approach)

Quick decision:

• Start with Crescendo — it's the most general-purpose strategy
• Try GOAT if Crescendo is too slow (GOAT typically succeeds in 3-5 turns)
• Use Hydra if the target refuses frequently (backtracking keeps conversation clean)
• Use Mischievous for targets with strong adversarial-pattern detection (the persona bypasses heuristics)

Multi-Turn Configuration

All four strategies share the same configuration structure:

generators:
  rest.Rest:
    uri: "https://your-llm-endpoint/chat"
    method: POST
    req_template: '{"input": "$INPUT"}'
    response_json: true
    response_json_field: "response"

judge:
  generator_type: openai.OpenAI
  config:
    model: gpt-4
    api_key: "${OPENAI_API_KEY}"

probes:
  attacker_generator_type: openai.OpenAI
  attacker_config:
    model: gpt-4
    api_key: "${OPENAI_API_KEY}"
  settings:
    crescendo.Crescendo:  # or goat.Goat, hydra.Hydra, mischievous.MischievousUser
      goal: "your test objective"
      max_turns: 10
      success_threshold: 0.8
      use_secondary_judge: true

Setting	Default	Description
goal	(required)	The objective the attacker tries to achieve
max_turns	10	Maximum conversation turns before stopping
success_threshold	0.8	Judge score (0.0-1.0) that triggers early exit
max_refusal_retries	10	Rephrase attempts per turn when target refuses
attack_max_attempts	5	Retries for attacker LLM JSON parsing failures
use_secondary_judge	true	Enable secondary judge to catch false negatives
max_backtracks	10	Turn-level rollbacks on refusal (Hydra only)
enable_fast_refusal	true	Pattern-based refusal detection before LLM judge call
enable_scan_memory	false	Cross-test-case learning (shares tactics across probes)
stateful	false	Disable backtracking for stateful targets
exclude_target_output	false	Hide target responses from attacker feedback (privacy mode)
attacker_model	(auto)	Override attacker model name for context window sizing

Multi-Turn Troubleshooting

Symptom	Likely Cause	Fix
no turns completed (attacker_parse_failures=N)	Attacker LLM returning invalid JSON	Use a stronger attacker model (GPT-4, Claude Opus). Increase attack_max_attempts.
no turns completed (target_empty=N)	Target returning empty/null responses	Check target endpoint is responding. Verify REST config template.
All turns score 0.0	Goal too vague or attacker not engaging	Make goal more specific. Try different strategy.
High scores but no success	success_threshold too high	Lower success_threshold from 0.8 to 0.6-0.7
Runs too long / expensive	Too many turns and retries	Reduce max_turns (try 5). Set enable_fast_refusal: true.
Hydra keeps backtracking	Target refuses everything	Try stateful: true or switch to Mischievous strategy

Architecture

cmd/augustus/          CLI entrypoint (Kong-based)
pkg/
  attempt/            Probe execution lifecycle and result tracking
  buffs/              Buff interface for prompt transformations
  config/             Configuration loading (YAML/JSON) with profiles
  detectors/          Public detector interfaces and registry
  generators/         Public generator interfaces and registry
  harnesses/          Harness interface for execution strategies
  lib/http/           Shared HTTP client with proxy support
  lib/stego/          LSB steganography for multimodal attacks
  logging/            Structured slog-based logging
  metrics/            Prometheus metrics collection
  prefilter/          Aho-Corasick keyword pre-filtering
  probes/             Public probe interfaces and registry
  ratelimit/          Token bucket rate limiting
  registry/           Generic capability registration system
  results/            Result types and multi-format output
  retry/              Exponential backoff with jitter
  scanner/            Scanner orchestration with concurrency
  templates/          YAML probe template loader (Nuclei-style)
  types/              Canonical shared interfaces (Prober, Generator, Detector)
internal/
  probes/             210+ probe implementations (47 categories)
  generators/         28 LLM provider integrations (43 variants)
  detectors/          90+ detector implementations (35 categories)
  harnesses/          3 harness strategies (probewise, batch, agentwise)
  buffs/              Buff interface for prompt transformations
  attackengine/       Iterative adversarial attack engine (PAIR/TAP backend)
  multiturn/          Multi-turn conversational attack engine (Crescendo/GOAT/Hydra/Mischievous)
  ahocorasick/        Internal Aho-Corasick keyword matching
benchmarks/           Performance benchmarks
tests/                Integration and equivalence tests
research/             Research documentation and analysis
examples/             Example configurations
docs/                 Documentation

Key Design Decisions

• Concurrent scanning with bounded goroutine pools via errgroup
• Plugin-style registration using Go init() functions for probes, generators, detectors, buffs, and harnesses
• Iterative attack engine with multi-stream conversation management, candidate pruning, and judge-based scoring for PAIR/TAP
• Multi-turn attack engine with persistent conversation history, refusal detection, strategy-agnostic design for Crescendo/GOAT
• YAML probe templates (Nuclei-style) for declarative probe definitions alongside Go-based probes
• Aho-Corasick pre-filtering for fast keyword matching in detectors

Configuration

YAML Configuration File

Create a config.yaml file:

# Runtime configuration
run:
  max_attempts: 3
  timeout: "30s"

# Generator configurations
generators:
  openai.OpenAI:
    model: "gpt-4"
    temperature: 0.7
    api_key: "${OPENAI_API_KEY}"  # Environment variable interpolation

  anthropic.Anthropic:
    model: "claude-3-opus-20240229"
    temperature: 0.5
    api_key: "${ANTHROPIC_API_KEY}"

  ollama.OllamaChat:
    model: "llama3.2:3b"
    temperature: 0.8

# Judge configuration (required for judge.Judge, judge.Refusal, and multi-turn probes)
judge:
  generator_type: openai.OpenAI
  model: gpt-4o-mini
  config:
    api_key: "${OPENAI_API_KEY}"

# Output configuration
output:
  format: "jsonl"
  path: "./results.jsonl"

# Named profiles for different scenarios
profiles:
  quick:
    run:
      max_attempts: 1
      timeout: "10s"
    generators:
      openai.OpenAI:
        model: "gpt-3.5-turbo"
        temperature: 0.5
    output:
      format: "table"

  thorough:
    run:
      max_attempts: 5
      timeout: "60s"
    generators:
      openai.OpenAI:
        model: "gpt-4"
        temperature: 0.3
    output:
      format: "jsonl"
      path: "./thorough_results.jsonl"

Environment Variables

# API Keys
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export COHERE_API_KEY="..."

# Debug mode
export AUGUSTUS_DEBUG=true

Proxy Configuration

Route HTTP traffic through a proxy (e.g., Burp Suite) for inspection:

# Method 1: Via config parameter
augustus scan rest.Rest \
  --probe dan.Dan_11_0 \
  --detector dan.DAN \
  --config '{"uri":"https://api.example.com","proxy":"http://127.0.0.1:8080"}' \
  --output results.jsonl

# Method 2: Via environment variables
export HTTP_PROXY=http://127.0.0.1:8080
export HTTPS_PROXY=http://127.0.0.1:8080
augustus scan rest.Rest --probe dan.Dan_11_0 --config '{"uri":"https://api.example.com"}'

• TLS verification automatically disabled for proxy inspection
• HTTP/2 support enabled for modern APIs
• Server-Sent Events (SSE) responses automatically detected and parsed

CLI Reference

Usage: augustus scan <generator> [flags]

Arguments:
  <generator>                 Generator name (e.g., openai.OpenAI, anthropic.Anthropic)

Probe Selection (choose one):
  --probe, -p                 Probe name (repeatable)
  --probes-glob               Comma-separated glob patterns (e.g., "dan.*,goodside.*")
  --all                       Run all registered probes

Detector Selection:
  --detector                  Detector name (repeatable)
  --detectors-glob            Comma-separated glob patterns

Buff Selection:
  --buff, -b                  Buff names to apply (repeatable)
  --buffs-glob                Comma-separated buff glob patterns (e.g., "encoding.*")

Configuration:
  --config-file               Path to YAML config file
  --config, -c                JSON config for generator

Execution:
  --harness                   Harness name (default: probewise.Probewise)
  --timeout                   Overall scan timeout (default: 30m)
  --probe-timeout             Per-probe timeout (default: 5m)
  --concurrency               Max concurrent probes (default: 10, env: AUGUSTUS_CONCURRENCY)

Output:
  --format, -f                Output format: table, json, jsonl (default: table)
  --output, -o                JSONL output file path
  --html                      HTML report file path
  --verbose, -v               Verbose output

Global:
  --debug, -d                 Enable debug mode

Commands:

augustus version              # Print version information
augustus list                 # List available probes, detectors, generators, harnesses, buffs
augustus scan <generator>     # Run vulnerability scan
augustus completion <shell>   # Generate shell completion (bash, zsh, fish)

Exit Codes:

Code	Meaning
0	Success - scan completed
1	Scan/runtime error
2	Validation/usage error

FAQ

How does Augustus compare to garak?

Augustus is a Go-native reimplementation inspired by garak (NVIDIA's Python-based LLM vulnerability scanner). Key differences:

• Performance: Go binary vs Python interpreter — faster execution and lower memory usage
• Distribution: Single binary with no runtime dependencies vs Python package with pip install
• Concurrency: Go goroutine pools (cross-probe parallelism) vs Python multiprocessing pools (within-probe parallelism)
• Probe coverage: Augustus has 210+ probes; garak has 160+ probes with a longer research pedigree and published paper (arXiv:2406.11036)
• Provider coverage: Augustus has 28 providers; garak has 35+ generator variants across 22 provider modules

Can I test local models without API keys?

Yes! Use the Ollama integration for local model testing:

# No API key needed
augustus scan ollama.OllamaChat \
  --probe dan.Dan_11_0 \
  --config '{"model":"llama3.2:3b"}'

How do I add custom probes?

1. Create a new Go file in internal/probes/
2. Implement the probes.Probe interface
3. Register using registry.RegisterProbe() in an init() function
4. Rebuild: make build

See CONTRIBUTING.md for detailed instructions.

What output formats are supported?

Augustus supports four output formats:

Format	Flag	Use Case
Table	--format table	Human-readable terminal output
JSON	--format json	Single JSON object for parsing
JSONL	--format jsonl	Line-delimited JSON for streaming
HTML	--html report.html	Visual reports for stakeholders

How do I test multiple models at once?

# Test multiple models sequentially
for model in "gpt-4" "gpt-3.5-turbo"; do
  augustus scan openai.OpenAI \
    --all \
    --config "{\"model\":\"$model\"}" \
    --output "results-$model.jsonl"
done

Is Augustus suitable for production environments?

Yes, Augustus is designed for production use with:

• Concurrent scanning with configurable limits
• Rate limiting to respect API quotas
• Timeout handling for long-running probes
• Retry logic for transient failures
• Structured logging for observability

Troubleshooting

Error: "API rate limit exceeded"

Cause: Too many concurrent requests or requests per minute.

Solutions:

1. Reduce concurrency: --concurrency 5
2. Use provider-specific rate limit settings in YAML config:

   generators:
     openai.OpenAI:
       rate_limit: 10  # requests per minute
   

Error: "context deadline exceeded" or "timeout"

Cause: Complex probes (like TAP or PAIR) exceed default timeout.

Solution:

augustus scan openai.OpenAI \
  --probe tap.TAPv1 \
  --timeout 60m \
  --config-file config.yaml

Error: "invalid API key" or "authentication failed"

Cause: Missing or invalid API credentials.

Solutions:

1. Verify environment variable is set: echo $OPENAI_API_KEY
2. Check for typos in config file
3. Ensure API key has required permissions
4. For Ollama, ensure the service is running: ollama serve

Error: "probe not found" or "detector not found"

Cause: Typo in name or probe not registered.

Solution:

# List all available probes and detectors
augustus list

# Use exact names from the list
augustus scan openai.OpenAI --probe dan.Dan_11_0  # Correct

Scan produces no results

Cause: Detector didn't match any responses, or output not written.

Solutions:

1. Run with --verbose to see detailed output
2. Check that detector matches probe type
3. Verify output file path is writable

Contributing

We welcome contributions! See CONTRIBUTING.md for:

• Adding new vulnerability probes
• Creating new detector implementations
• Adding LLM provider integrations
• Testing guidelines
• Code style requirements

Development

# Run all tests
make test

# Run specific package tests
go test ./pkg/scanner -v

# Run equivalence tests (compare Go vs Python implementations)
go test ./tests/equivalence -v

# Build binary
make build

# Install to $GOPATH/bin
make install

Benchmark Environment (DevPod)

A ready-to-go cloud development environment for benchmarking LLMs is available via DevPod. It provisions a remote container with Augustus, Ollama, Go, and all dependencies pre-installed.

cd devpod

# CPU-only instance (~$0.08/hr) - cloud APIs only
make devpod-up-cpu

# GPU instance with NVIDIA T4 (~$0.53/hr) - local models up to 14B
make devpod-up-gpu

# GPU Pro instance with NVIDIA L4 (~$0.80/hr) - local models up to 32B
make devpod-up-gpu-pro

Inside the devpod:

devpod/scripts/setup.sh        # Configure LLM provider API keys
devpod/scripts/pull-models.sh   # Pull local Ollama models (GPU only)
devpod/scripts/benchmark.sh     # Run benchmarks with comparison reports

The environment also works as a standard dev container — open the repo in VS Code or Cursor and select the CPU or GPU configuration from .devcontainer/.

Security

Augustus is designed for authorized security testing only.

• Augustus sends adversarial prompts to LLMs you specify - always ensure you have authorization
• Never test systems you don't own or have explicit permission to test
• Some probes generate offensive content by design (for testing safety filters)
• Results may contain harmful content produced by target LLMs

Report security issues via GitHub Issues.

Support

If you find Augustus useful, please consider:

• Giving it a star on GitHub
• Opening an issue for bugs or feature requests
• Contributing new probes, detectors, or provider integrations

License

Apache 2.0 - Praetorian Security, Inc.

---

Built by Praetorian - Offensive Security Solutions