secretscan-go

README

secretscan-go

Secret detection engine for git repositories and files.

Inspired by gitleaks (25.6K stars) — reimplemented from scratch with zero dependencies, parallel scanning, severity levels, and JSON rule configuration.

Why This Exists

gitleaks is the gold standard for secret detection, but it relies on 7+ external dependencies (cobra, viper, lipgloss, aho-corasick, etc.) and uses TOML configuration. secretscan-go reimplements the core detection engine with:

• Zero external dependencies — only Go stdlib
• JSON rule configuration — no TOML parser needed
• Severity levels — critical/high/medium/low/info per rule
• Parallel scanning — configurable worker pools for both git and file scanning
• 25 built-in rules — AWS, GitHub, GitLab, Google, Stripe, Slack, and more

Features

Core Detection Engine

• Regex pattern matching — compiled Go regexps with secret group extraction
• Shannon entropy detection — per-rule entropy thresholds filter low-entropy false positives
• Keyword prefiltering — skip expensive regex matching when keywords are absent
• Fingerprint deduplication — SHA-256 based fingerprints prevent duplicate findings

Git History Scanning

• Walk entire commit history with git log -p
• Scan staged changes with git diff --staged
• Extract commit metadata (SHA, author, email, date, message)
• Parallel diff processing with configurable workers
• Allowlist specific commits

File Scanning

• Recursive directory scanning
• Binary file detection (null-byte check + extension filtering)
• Automatic skip of .git, node_modules, vendor, __pycache__
• Configurable max file size
• Parallel file processing

Output Formats

• Text — human-readable, sorted by severity
• JSON — machine-readable, full finding details
• SARIF v2.1.0 — GitHub Code Scanning compatible

Allowlisting

• Path patterns — skip files matching regex (e.g., vendor/.*, test/.*)
• Regex patterns — suppress findings matching specific patterns (e.g., example keys)
• Stop words — skip findings containing known false-positive words
• Commit SHAs — skip specific commits
• AND/OR conditions — combine allowlist criteria
• Per-rule allowlists — different rules can have different allowlists

25 Built-in Rules

Category	Rules	Severity
AWS	Access Key ID, Secret Access Key	Critical
GitHub	PAT, OAuth, App Token, Fine-Grained PAT	Critical/High
GitLab	Personal Access Token	Critical
Google	API Key, OAuth Client Secret	High
Slack	Bot Token, Webhook URL	High/Medium
Stripe	Secret Key, Publishable Key	Critical/Low
Private Keys	RSA, EC, DSA, OPENSSH	Critical
Generic	Password, API Key, Secret, Database URL	High/Medium
NPM	Access Token	High
PyPI	Upload Token	High
Heroku	API Key	High
Twilio	API Key	High
SendGrid	API Key	High
Mailgun	API Key	Medium
JWT	JSON Web Token	Medium
Azure	Storage Account Key	Critical

Installation

go install github.com/JSLEEKR/secretscan-go@latest

Or build from source:

git clone https://github.com/JSLEEKR/secretscan-go.git
cd secretscan-go
go build -o secretscan-go .

Usage

Scan a Git Repository

# Scan current directory (auto-detects git)
secretscan-go scan

# Scan a specific repository
secretscan-go scan /path/to/repo

# Scan staged changes only (pre-commit)
secretscan-go scan --staged

# Scan commits since a specific SHA
secretscan-go scan --since abc123def

Scan Files

# Scan specific files
secretscan-go scan-file config.yaml .env credentials.json

# Scan a non-git directory
secretscan-go scan /path/to/directory

Output Formats

# Human-readable text (default)
secretscan-go scan

# JSON output
secretscan-go scan -f json

# SARIF for GitHub Code Scanning
secretscan-go scan -f sarif -o report.sarif

# Save to file
secretscan-go scan -o report.json -f json

List Rules

# Text table
secretscan-go list-rules

# JSON format
secretscan-go list-rules -f json

Configuration

# Use custom rules
secretscan-go scan -c custom-rules.json

# Adjust workers for performance
secretscan-go scan -w 8

# Redact secrets in output (keep first 4 chars)
secretscan-go scan --redact 4

# Set scan timeout
secretscan-go scan --timeout 10m

# Custom exit code
secretscan-go scan --exit-code 2

Custom Rules (JSON)

Create a rules.json file:

{
  "title": "My Custom Rules",
  "rules": [
    {
      "id": "internal-api-key",
      "description": "Internal API Key",
      "regex": "INTERNAL-[A-Z0-9]{32}",
      "keywords": ["INTERNAL-"],
      "severity": "high",
      "tags": ["internal", "api"],
      "entropy": 3.5,
      "allowlist": {
        "regexes": ["INTERNAL-EXAMPLE"],
        "paths": ["test/.*"],
        "condition": "OR"
      }
    }
  ],
  "allowlist": {
    "paths": ["vendor/.*", "go\\.sum"],
    "stopWords": ["example", "test", "fake"]
  }
}

Rule Fields

Field	Type	Description
id	string	Unique rule identifier
description	string	Human-readable description
regex	string	Go regexp pattern
path	string	File path filter (regex)
secretGroup	int	Capture group index for secret extraction
entropy	float	Minimum Shannon entropy threshold
keywords	[]string	Fast-path keywords (skip regex if absent)
tags	[]string	Categorization tags
severity	string	critical, high, medium, low, info
allowlist	object	Per-rule allowlist

Architecture

main.go                  Entry point
cmd/root.go              CLI argument parsing and command dispatch
config/config.go         JSON rule configuration and compilation
detect/detect.go         Core detection engine (regex + entropy + allowlist)
detect/finding.go        Finding type and fingerprint generation
entropy/entropy.go       Shannon entropy calculation
rules/builtin.go         25 built-in detection rules
scanner/git.go           Git history and staged change scanning
scanner/file.go          File system scanning with binary detection
report/report.go         Text and JSON output formatters
report/sarif.go          SARIF v2.1.0 output formatter

Detection Flow

Input (git repo / directory / file)
  |
  v
Scanner (git log / file walk)
  |
  v
Fragment (content + metadata)
  |
  v
Detector
  |-- Keyword prefilter (skip if no keyword match)
  |-- Path filter (skip if path doesn't match)
  |-- Allowlist check (path, commit)
  |-- Regex matching (all rules in parallel)
  |-- Secret group extraction
  |-- Entropy check (if threshold set)
  |-- Finding allowlist (regex, stopword)
  |-- Fingerprint generation (SHA-256)
  |-- Deduplication
  |
  v
Findings → Reporter (text / JSON / SARIF)

Comparison with gitleaks

Feature	gitleaks	secretscan-go
Language	Go	Go
Dependencies	7+ (cobra, viper, lipgloss, etc.)	0 (stdlib only)
Config format	TOML	JSON
Rule count	150+	25 (extensible via JSON)
Severity levels	No	Yes (5 levels)
Prefiltering	Aho-Corasick trie	Keyword substring match
Entropy detection	Per-rule	Per-rule
Git scanning	Subprocess	Subprocess
SARIF output	Yes	Yes
Allowlisting	Yes (paths, commits, regexes)	Yes (paths, commits, regexes, stopwords)
Parallel scanning	Semaphore	Worker pool (configurable)
Binary	~15MB	~5MB

Trade-offs

• gitleaks has 150+ rules covering more niche services — secretscan-go has 25 core rules
• gitleaks uses Aho-Corasick for prefiltering — secretscan-go uses simpler keyword matching
• gitleaks has pre-commit hook integration — secretscan-go focuses on scanning

Testing

# Run all tests
go test ./... -v

# Run with race detector
go test ./... -race

# Run specific package
go test ./detect/ -v

# Run benchmarks
go test ./entropy/ -bench=.

Test coverage: 123 tests across 7 packages

• config/ — JSON parsing, compilation, validation, allowlists
• detect/ — Pattern matching, entropy, allowlists, deduplication, concurrency
• entropy/ — Shannon entropy calculation, charset filtering
• rules/ — Built-in rule validation, compilation, uniqueness
• report/ — JSON, text, SARIF output formatting
• scanner/ — Git history scanning, file scanning, binary detection
• cmd/ — CLI argument parsing, command dispatch

CI/CD Integration

GitHub Actions

- name: Secret Scan
  run: |
    go install github.com/JSLEEKR/secretscan-go@latest
    secretscan-go scan -f sarif -o results.sarif

- name: Upload SARIF
  uses: github/codeql-action/upload-sarif@v2
  with:
    sarif_file: results.sarif

Pre-commit Hook

#!/bin/sh
secretscan-go scan --staged
exit $?

License

MIT

---

Built as part of JSLEEKR/daily-challenge V2 Pipeline: Reimplement & Compare.