README
¶
triage
Lightweight, dependency-free security analysis of byte strings for Go: Shannon entropy scoring, content classification, and secret/credential detection. Point it at any string and it tells you whether it looks like a URL, an IP, a file path, an opaque high-entropy blob, or a leaked credential.
Extracted from txtr, where it powers the
--triage/--secrets modes for firmware and malware analysis.
Install
go get github.com/richardwooding/triage
Usage
package main
import (
"fmt"
"github.com/richardwooding/triage"
)
func main() {
res := triage.Classify([]byte("AKIAIOSFODNN7EXAMPLE"), 4.5)
fmt.Printf("entropy: %.2f\n", res.Entropy) // entropy: 3.68
fmt.Printf("category: %q\n", res.Category) // category: "Base64"
for _, s := range res.Secrets {
fmt.Printf("[%s] %s: %s\n", s.Severity, s.Rule, s.Match)
// [HIGH] AWS Access Key: AKIAIOSFODNN7EXAMPLE
}
fmt.Println(res.Interesting()) // true
}
Classify takes the bytes to analyze and a minEntropy threshold (bits/byte) above which an
opaque token is flagged as high-entropy. Pass 0 to disable high-entropy flagging.
Each SecretFinding carries the byte offsets of the match (Start/End, where
str[Start:End] == Match), so you can highlight or redact the exact span.
The raw entropy function is exported too:
triage.Entropy([]byte("hello")) // 1.92
Redaction
Use Redact to scrub secrets out of text (logs, crash dumps, anything you're
about to store or forward) before it leaves your process:
clean := triage.Redact([]byte(`db: "AKIAIOSFODNN7EXAMPLE"`))
// db: "[REDACTED]"
Redact never modifies its input and returns a new slice. RedactWith lets you
supply a custom mask. Overlapping matches are merged so each region is masked
exactly once, and the fixed mask avoids leaking the original secret's length.
Custom rules and allowlists
Classify and Redact use the built-in rules. For custom detectors, an
allowlist of known-benign values, or a fixed entropy threshold, build a
Scanner:
s := triage.NewScanner(
triage.WithExtraRules(triage.Rule{
Name: "Acme Key",
Severity: triage.SeverityHigh,
Pattern: regexp.MustCompile(`\bacme_[a-z0-9]{8}\b`),
}),
triage.WithAllowlist("AKIAIOSFODNN7EXAMPLE"), // suppress the docs key
triage.WithMinEntropy(4.5),
)
res := s.Classify([]byte("acme_abcd1234"))
clean := s.Redact([]byte("acme_abcd1234"))
Options compose in order:
| Option | Effect |
|---|---|
WithMinEntropy(f) |
High-entropy threshold for Scanner.Classify |
WithRules(...) |
Replace the rule set entirely (use DefaultRules() as a base) |
WithExtraRules(...) |
Append custom rules to the built-ins |
WithoutRules(names...) |
Disable specific built-ins by name |
WithAllowlist(values...) |
Suppress findings whose match equals a listed value |
WithRedactMask(mask) |
Replacement used by Scanner.Redact |
A Scanner is read-only and safe for concurrent use after construction.
What it detects
Content categories (first match wins, structural classification):
| Category | Example |
|---|---|
URL |
https://example.com/path |
Email |
user@example.com |
IPv4 / IPv6 |
10.0.0.5, 2001:db8::1 (validated via net.ParseIP) |
Domain |
api.github.com |
WinPath / UnixPath |
C:\Windows\System32, /usr/bin/txtr |
Hex / Base64 |
deadbeefcafebabe, TWFuIGlz... |
UUID |
550e8400-e29b-41d4-a716-446655440000 |
Secret rules (precompiled, with optional entropy gating):
AWS access keys, PEM private keys, GitHub tokens, Slack tokens, Stripe keys, Google API keys,
GitLab personal access tokens, npm access tokens, Anthropic & OpenAI API keys, SendGrid API
keys, Telegram bot tokens, Twilio API keys, Square access tokens, JSON Web Tokens, and generic
key=value secret assignments.
High-entropy blobs: long opaque tokens above the entropy threshold that don't classify as benign structural content (so file paths and cipher lists don't get flagged, but base64/hex key material does).
Design
- Zero dependencies — standard library only.
- ReDoS-safe — all patterns use Go's
regexp(RE2), which guarantees linear-time matching; patterns are compiled once at package init, never per call. - Low false positives — IPs are validated with
net.ParseIP; the generic secret rule requires an actual:/=assignment (not prose); domains require a lowercase TLD; high entropy is suppressed for recognized structural content. - Fast — strings shorter than the shortest possible secret skip the secret scan; entropy is a single O(n) pass.
License
MIT — see LICENSE.
Documentation
¶
Overview ¶
Package triage provides lightweight security analysis of byte strings: Shannon entropy scoring, content classification (URL, email, IPv4/IPv6, domain, file path, hex, base64, UUID), and secret/credential detection (AWS keys, PEM private keys, GitHub/Slack/Stripe tokens, Google API keys, JWTs, and generic key=value assignments).
It depends only on the standard library, making it easy to embed in scanners, CI secret gates, log scrubbers, and forensics tooling. The entry point is Classify; Entropy is exposed separately for reuse.
Index ¶
Examples ¶
Constants ¶
const DefaultRedactMask = "[REDACTED]"
DefaultRedactMask is the replacement written over each detected secret by Redact. A fixed token (rather than length-preserving asterisks) avoids leaking the original secret's length.
Variables ¶
This section is empty.
Functions ¶
func Entropy ¶
Entropy returns the Shannon entropy of data in bits per byte, a value in the range [0, 8]. Higher values indicate more randomness: natural-language text typically scores ~4.0-4.5, base64-encoded data ~6, and cryptographic key material approaches 8.
Example ¶
package main
import (
"fmt"
"github.com/richardwooding/triage"
)
func main() {
fmt.Printf("%.2f\n", triage.Entropy([]byte("hello")))
}
Output: 1.92
func Redact ¶ added in v0.3.0
Redact returns a copy of str with every detected secret replaced by DefaultRedactMask. It is intended for scrubbing logs, crash dumps, and other text before it is stored or forwarded. The input is never modified.
Detection uses the same rules as Classify; bytes that are not part of a secret are preserved verbatim. Overlapping matches are merged so each region is masked exactly once.
Example ¶
package main
import (
"fmt"
"github.com/richardwooding/triage"
)
func main() {
fmt.Println(string(triage.Redact([]byte("aws_key = AKIAIOSFODNN7EXAMPLE"))))
}
Output: aws_key = [REDACTED]
func RedactWith ¶ added in v0.3.0
RedactWith behaves like Redact but replaces each detected secret with the supplied mask instead of DefaultRedactMask.
Types ¶
type Category ¶
type Category string
Category is a human-readable classification of a string's content.
const ( CategoryURL Category = "URL" CategoryEmail Category = "Email" CategoryIPv4 Category = "IPv4" CategoryIPv6 Category = "IPv6" CategoryUUID Category = "UUID" CategoryPathWindows Category = "WinPath" CategoryPathUnix Category = "UnixPath" CategoryDomain Category = "Domain" CategoryHex Category = "Hex" CategoryBase64 Category = "Base64" )
Classification categories.
type Option ¶ added in v0.5.0
type Option func(*Scanner)
Option configures a Scanner. Options are applied in order, so later options observe the effects of earlier ones (e.g. WithoutRules after WithRules).
func WithAllowlist ¶ added in v0.5.0
WithAllowlist suppresses findings whose matched text exactly equals one of values. It is the way to silence known-benign tokens such as the documentation keys (e.g. "AKIAIOSFODNN7EXAMPLE"). Repeated calls accumulate.
func WithExtraRules ¶ added in v0.5.0
WithExtraRules appends custom rules to the scanner's current rule set.
func WithMinEntropy ¶ added in v0.5.0
WithMinEntropy sets the threshold (bits/byte) above which an opaque token is flagged as high-entropy by Scanner.Classify. Pass 0 to disable the flag.
func WithRedactMask ¶ added in v0.5.0
WithRedactMask sets the replacement written over each secret by Scanner.Redact. The slice is used as-is; do not mutate it afterwards.
func WithRules ¶ added in v0.5.0
WithRules replaces the scanner's rule set entirely. Pass DefaultRules as a base if you want to extend rather than replace the built-ins.
func WithoutRules ¶ added in v0.5.0
WithoutRules removes any rules whose Rule.Name matches one of names. Use it to disable specific built-ins, e.g. WithoutRules("Generic Secret").
type Result ¶
type Result struct {
// Entropy is the Shannon entropy (bits/byte) of the trimmed string.
Entropy float64
// HighEntropy is true when Entropy meets the configured threshold and the
// string looks like an opaque token rather than natural-language text.
HighEntropy bool
// Category is the content classification (first match wins by priority);
// the empty string when nothing matched.
Category Category
// Secrets holds any detected secrets/credentials.
Secrets []SecretFinding
}
Result holds the triage analysis of a single string.
func Classify ¶
Classify analyzes a string and returns its entropy, content category, and any detected secrets, using the built-in secret rules. minEntropy is the threshold (bits/byte) above which an opaque token is flagged as high-entropy; pass 0 to disable that flag.
For custom rules, an allowlist, or a fixed entropy threshold, build a Scanner with NewScanner and call Scanner.Classify.
Example ¶
package main
import (
"fmt"
"github.com/richardwooding/triage"
)
func main() {
res := triage.Classify([]byte("AKIAIOSFODNN7EXAMPLE"), 4.5)
fmt.Printf("entropy=%.2f category=%q\n", res.Entropy, res.Category)
for _, s := range res.Secrets {
fmt.Printf("[%s] %s: %s\n", s.Severity, s.Rule, s.Match)
}
}
Output: entropy=3.68 category="Base64" [HIGH] AWS Access Key: AKIAIOSFODNN7EXAMPLE
func (Result) Interesting ¶
Interesting reports whether the result is worth surfacing in --secrets mode: it contains a detected secret or qualifies as a high-entropy blob.
type Rule ¶ added in v0.5.0
type Rule struct {
// Name identifies the rule in a [SecretFinding] and is the key used by
// [WithoutRules] to disable it.
Name string
// Severity is reported on every finding the rule produces.
Severity Severity
// Pattern is the compiled detector. Required.
Pattern *regexp.Regexp
// MinEntropy, when > 0, gates the rule: a candidate match whose Shannon
// entropy is below this threshold is rejected as a likely false positive.
MinEntropy float64
}
Rule is a precompiled detector for one class of secret. Callers can supply custom rules to a Scanner via WithRules or WithExtraRules.
Patterns should be compiled once (e.g. with regexp.MustCompile) and reused, never recompiled per string. Go's regexp engine (RE2) guarantees linear-time matching, so patterns are inherently free of catastrophic-backtracking (ReDoS) risk.
func DefaultRules ¶ added in v0.5.0
func DefaultRules() []Rule
DefaultRules returns a fresh copy of the built-in secret detectors, suitable for passing to WithRules as a base to extend or trim. The patterns are shared (they are immutable and safe for concurrent use); only the slice is copied, so appending to the result never affects the package defaults.
type Scanner ¶ added in v0.5.0
type Scanner struct {
// contains filtered or unexported fields
}
Scanner performs triage analysis with a configurable set of secret rules, a fixed entropy threshold, an allowlist of known-benign values, and a redaction mask. Create one with NewScanner; the zero value is not usable.
After construction a Scanner is read-only and safe for concurrent use. For one-off analysis with the built-in rules, the package-level Classify and Redact functions are simpler.
Example ¶
A Scanner adds custom rules and an allowlist on top of the built-in detectors. Here a project-specific "Acme Key" rule is added and the AWS documentation key is allowlisted so it is not reported.
package main
import (
"fmt"
"regexp"
"github.com/richardwooding/triage"
)
func main() {
s := triage.NewScanner(
triage.WithExtraRules(triage.Rule{
Name: "Acme Key",
Severity: triage.SeverityHigh,
Pattern: regexp.MustCompile(`\bacme_[a-z0-9]{8}\b`),
}),
triage.WithAllowlist("AKIAIOSFODNN7EXAMPLE"),
)
res := s.Classify([]byte("acme_abcd1234 and AKIAIOSFODNN7EXAMPLE"))
for _, f := range res.Secrets {
fmt.Printf("%s: %s\n", f.Rule, f.Match)
}
}
Output: Acme Key: acme_abcd1234
func NewScanner ¶ added in v0.5.0
NewScanner returns a Scanner configured by opts. With no options it uses the built-in rules (DefaultRules), a disabled high-entropy threshold (0), no allowlist, and DefaultRedactMask.
func (*Scanner) Classify ¶ added in v0.5.0
Classify analyzes str using the scanner's rules, allowlist, and entropy threshold. See Classify for the meaning of each Result field.
func (*Scanner) DetectSecrets ¶ added in v0.5.0
func (s *Scanner) DetectSecrets(str []byte) []SecretFinding
DetectSecrets returns the secrets found in str using the scanner's rules and allowlist, without computing entropy or category.
type SecretFinding ¶
type SecretFinding struct {
Rule string `json:"rule"`
Severity Severity `json:"severity"`
Match string `json:"match"`
// Start and End are the byte offsets of the match within the scanned
// string: str[Start:End] == Match. End is exclusive. They make it possible
// to highlight or redact the exact span (see [Redact]).
Start int `json:"start"`
End int `json:"end"`
}
SecretFinding describes a single secret detected inside a string.
Source Files