goredact

README

goredact

goredact is a pure-Go, bounded-memory redaction filter: it reads a byte stream, writes the same stream back with secrets replaced, and never holds more than a fixed window of input. It is built for the data path — agent sessions, log shippers, environment dumps, upload pipelines — where plaintext must not be buffered whole, spilled to a temporary file, or delayed until a scan finishes.

What it is, and what it is not

The distinction that matters is not "faster scanner". It is filter versus auditor, and the two are complements:

	goredact	Gitleaks / Betterleaks
Shape	Redact(ctx, dst, src) — bytes in, redacted bytes out	scan a repository, emit a findings report
Output	the sanitized stream itself	JSON/SARIF/CSV findings; --redact masks the report, never the content
Memory	fixed: ChunkSize + rule-set window, independent of input size and finding count	grows with findings; the reporter needs the complete result set
Latency	first output after one rule window of input, then continuous	nothing until the scan completes
Rules	67 built-in, literal triggers plus bounded Go validators, no regex on the scan path	several hundred regexes, plus decoding, archives, git history, live credential validation
Scope	one stream	repositories, commit history, archives, remote forges

Use goredact when secrets must be removed from data in flight, with a latency and memory budget: filtering agent or LLM output before it is persisted, sanitizing logs before they leave a host, redacting a body before it is uploaded or compressed, or scrubbing a stream inside a request handler.

Use Gitleaks or Betterleaks when you are auditing artifacts at rest and want maximum coverage rather than a byte-output path: pre-commit hooks, CI gates, git-history forensics, archive and multi-layer encoding recursion, baseline diffing, and checking whether a discovered credential is live. GoRedact does none of those things and is not trying to.

Running both is a reasonable posture: goredact in the pipeline, a repository auditor in CI.

The shape

engine, err := goredact.New(goredact.Config{
	Profile: goredact.ProfileBalanced,
})
if err != nil {
	log.Fatal(err)
}

stats, err := engine.Redact(ctx, os.Stdout, os.Stdin)
if err != nil {
	log.Fatal(err)
}
log.Printf("redactions=%d bytes=%d", stats.Findings, stats.BytesRead)

The zero configuration selects the balanced profile. An engine is immutable after construction and safe for concurrent use; every call consumes an io.Reader and writes redacted bytes to an io.Writer without retaining the whole input.

By default each secret is replaced with one [REDACTED] marker (Config.Marker). Config.MaskStrategy can instead select MaskLengthPreserving (* per redacted byte, output length equals input length) or MaskFormatPreserving (per-character-class substitution that also keeps separators, so token shapes and fixed-width records survive) — see the MaskStrategy docs for the disclosure trade-offs.

Performance profile

GoRedact's cost is a function of bytes, not of how many secrets it finds. That property, rather than any single multiplier, is the reason it fits the data path. One-core, same host, same protocol, against Betterleaks 1.7.4:

Corpus	GoRedact	Betterleaks	Ratio
No trigger literals, 512 MiB	648 MiB/s, 10 MiB RSS	106 MiB/s, 41 MiB RSS	6.1x
Trigger literals, no secrets, 512 MiB	296 MiB/s, 10 MiB RSS	80 MiB/s, 41 MiB RSS	3.7x
A secret in every record, 32 MiB	229 MiB/s, 10 MiB RSS	1 MiB/s, 4,829 MiB RSS	242x

Read the shape of the table, not the last number. GoRedact's memory is flat at ~10 MiB across a 16x change in input size and every candidate density, and its throughput degrades gracefully (648 → 296 → 229 MiB/s) because each candidate is validated inside that rule's bounded window. An auditor's cost is driven by confirmed findings instead, since each one pays for submatch extraction, entropy scoring, context extraction, filtering, and permanent residency in the report.

The two tools are not doing identical work: on the third corpus they confirm nearly the same number of findings but from disjoint rules, and neither is a superset of the other. Nothing here measures relative recall or precision. Gitleaks 8.30.1-1 numbers, the private 976-file session corpus, per-profile runs, host details, scanner flags, the corpus defect behind the third row, and every reproduction command are in docs/BENCHMARKS.md.

Streaming latency

Output lags input by at most one rule-set window, regardless of how long the stream runs — the engine emits every byte it can prove is not part of a growing match. Measured on a trickle producer, first output byte arrives after 16,566 bytes of input on the balanced profile, and the bound is tunable by dropping the rules that need the longest lookahead:

Active rules	Lag bound
balanced, all 66	16,566 B
minus pem-private-key, putty-private-key	8,278 B
minus also jwt, supabase-service-role-key	4,393 B
minus also aws-bedrock-short-lived-api-key	774 B

The default 16 KiB comes entirely from the two private-key block rules; the median built-in rule window is 86 bytes. Trade block-key detection for latency with Config.DisableRules when the stream is interactive.

Install and use

go get github.com/baum-ai/goredact@v0.2.0
go install github.com/baum-ai/goredact/cmd/goredact@v0.2.0
goredact stream --profile balanced < session.jsonl > session.redacted.jsonl
goredact stream --zstd --stats - < session.jsonl > session.redacted.jsonl.zst
goredact dir --report-format sarif --report-path findings.sarif ./workspace

Run goredact --help to discover commands, goredact <command> --help for command-specific flags and examples, or goredact completion --help to set up shell completion. The stream command also accepts --input and --output; it removes incomplete output after a failed scan. The dir command recursively scans regular files and writes JSON, CSV, JUnit, or SARIF findings without including matched secret values. By default, progress, statistics, and reports contain metadata and counts only. See docs/CLI_AND_UPLOAD.md for reports, compression, and multipart upload integration.

Directory reports omit matched values by default. dir --show-secrets includes them when explicitly requested; its output must be handled as credential material.

Detection and guarantees

The built-in rules cover provider tokens (source control, cloud, AI, payment, messaging, and project management), authentication headers, contextual assignments, cookies, credentials in URLs, and private-key blocks. Fast, balanced, and deep profiles trade detector coverage for cost; deep adds a broader-keyword, lower-confidence generic assignment heuristic on top of everything balanced already catches. See docs/PROFILES.md for the complete rule table.

For every profile, memory is bounded by the configured chunk and rule windows. Matched bytes are excluded from findings, statistics, and package-generated errors. The same input, rule-set version, and configuration produce the same output. Detection is deliberately not a proof that an input contains no secrets; unsupported encodings and obfuscation are documented in docs/THREAT_MODEL.md.

Development and release evidence

go test ./...
go test -race ./...
go vet ./...
make bench-ci

• Architecture: docs/DESIGN.md
• Accuracy and differential harness: docs/ACCURACY.md
• Benchmark suite and report procedure: docs/BENCHMARKS.md
• Rule authoring: docs/RULE_AUTHORING.md
• Security reporting: SECURITY.md
• Versioning: docs/VERSIONING.md
• Changes: CHANGELOG.md
• Code of Conduct: CODE_OF_CONDUCT.md

The core library remains stdlib-only and has no cgo path. The optional reference CLI uses the pure-Go klauspost/compress module for zstd.

Licence

MIT. Dependency and source provenance is recorded in THIRD_PARTY.md.

Documentation

Overview

Package goredact provides high-throughput streaming detection and redaction of secrets from large text streams such as agent session logs, build output, and environment dumps.

The library is pure Go (no cgo) and processes input through fixed-size buffers, so memory usage is bounded regardless of input size. Input is consumed from an io.Reader and redacted output is written to an io.Writer; no temporary files are ever created.

Guarantees

• Bounded memory: memory usage does not grow with input size.
• No plaintext persistence: unredacted data is never written to temporary storage.
• No secret leakage: matched secret values never appear in errors, findings, statistics, or any diagnostic output produced by this package.
• Determinism: the same input, configuration, and rule-set version always produce identical output.
• Reuse: an Engine is immutable after construction and safe for concurrent use across goroutines; each Redact call maintains its own scan state.

Errors

Errors returned by this package never contain bytes from the scanned input. Errors that originate in the caller-supplied reader or writer are wrapped in ReadError or WriteError respectively, so callers can distinguish I/O failures from configuration or internal failures; the wrapped error is produced by the caller's own io implementation and is returned unmodified. Invalid configurations are reported by New as errors wrapping ErrInvalidConfig.

Out of scope for v0.1

• General-purpose PII detection.
• Live credential verification against provider APIs.
• Recursive decoding of arbitrary encoded or compressed content (base64-wrapped secrets, custom obfuscation, encrypted payloads).
• Detection of secrets split across multiple encodings or interleaved streams.

See docs/THREAT_MODEL.md for the complete threat model.

Index

• Constants
• Variables
• type Confidence
  • func (c Confidence) String() string
• type Config
• type CustomRule
• type Engine
  • func New(cfg Config) (*Engine, error)
  • func (e *Engine) ActiveRules() []RuleInfo
  • func (e *Engine) Redact(ctx context.Context, dst io.Writer, src io.Reader) (Stats, error)
  • func (e *Engine) RuleSetVersion() string
• type Finding
• type MaskStrategy
  • func (m MaskStrategy) String() string
• type Profile
  • func (p Profile) String() string
• type ReadError
  • func (e *ReadError) Error() string
  • func (e *ReadError) Unwrap() error
• type RuleInfo
  • func BuiltinRules() []RuleInfo
• type Stats
• type WriteError
  • func (e *WriteError) Error() string
  • func (e *WriteError) Unwrap() error

Constants

const (
	// ProfileFast enables only deterministic, high-confidence detectors:
	// provider token prefixes, authentication headers, credential-bearing
	// URLs, and private-key markers.
	ProfileFast = Profile(core.ProfileFast)

	// ProfileBalanced is the default. It extends ProfileFast with
	// contextual assignment detection, entropy validation, cookies, and
	// common generic credential patterns.
	ProfileBalanced = Profile(core.ProfileBalanced)

	// ProfileDeep extends ProfileBalanced with more expensive or
	// lower-confidence detectors and selected decoding. In v0.1 the only
	// deep-only built-in rule is a broad, low-confidence generic secret
	// assignment heuristic (keyed on words like "key", "secret", "auth",
	// "access", "credential", "creds"); every other rule that runs in
	// ProfileDeep also runs in ProfileBalanced. See docs/PROFILES.md.
	ProfileDeep = Profile(core.ProfileDeep)
)

const (
	// ConfidenceLow marks heuristic detections (e.g. the deep profile's
	// broad generic-assignment rule) that are expected to over-match.
	ConfidenceLow = Confidence(core.ConfidenceLow)

	// ConfidenceMedium marks contextual detections confirmed by
	// surrounding structure and entropy rather than an exact token shape.
	ConfidenceMedium = Confidence(core.ConfidenceMedium)

	// ConfidenceHigh marks detections confirmed by an exact, well-known
	// token shape or structure.
	ConfidenceHigh = Confidence(core.ConfidenceHigh)
)

const (
	// MaskFixedMarker replaces each confirmed span with exactly one copy
	// of Config.Marker, regardless of the span's length. This is the
	// default and discloses nothing about the redacted value, not even
	// its length.
	MaskFixedMarker = MaskStrategy(core.MaskFixedMarker)

	// MaskLengthPreserving replaces each redacted byte with '*'. Output
	// length equals input length, so fixed-width records and offsets in
	// surrounding text survive redaction. The length of each secret is
	// disclosed; its contents are not.
	MaskLengthPreserving = MaskStrategy(core.MaskLengthPreserving)

	// MaskFormatPreserving replaces each redacted byte with a stand-in
	// of the same coarse character class: uppercase letters become 'X',
	// lowercase letters 'x', digits '9'; the separator bytes
	// - _ . : / + = @ pass through verbatim; every other byte becomes
	// '*'. Output length equals input length and token shapes (JWT dots,
	// UUID dashes, base64 padding) survive, which keeps
	// structured/fixed-format consumers parsing. The preserved shape and
	// separators are information about the original secret — choose this
	// only when downstream consumers need format fidelity.
	MaskFormatPreserving = MaskStrategy(core.MaskFormatPreserving)
)

const DefaultChunkSize = core.DefaultChunkSize

DefaultChunkSize is the input buffer size used when Config.ChunkSize is zero.

const DefaultMarker = core.DefaultMarker

DefaultMarker is the replacement text used when Config.Marker is empty.

Variables

var ErrInvalidConfig = core.ErrInvalidConfig

ErrInvalidConfig is returned (wrapped, with detail that never includes input data) by New when the configuration is invalid.

Types

type Confidence

type Confidence core.Confidence

Confidence expresses how certain a rule is that a match is a real secret.

func (Confidence) String

func (c Confidence) String() string

String returns the confidence name.

type Config

type Config struct {
	// Profile selects the detection profile. Defaults to ProfileBalanced.
	Profile Profile

	// ChunkSize is the size in bytes of the fixed input buffer. Values
	// below the minimum required by the compiled rule set are rejected by
	// New. Defaults to DefaultChunkSize.
	ChunkSize int

	// Marker is the fixed replacement text written in place of each
	// redacted span under MaskFixedMarker. Defaults to DefaultMarker.
	// The other mask strategies ignore it.
	Marker string

	// MaskStrategy selects how confirmed spans are replaced in the
	// output. The zero value, MaskFixedMarker, writes Marker once per
	// span; MaskLengthPreserving and MaskFormatPreserving instead
	// replace each redacted byte one-for-one, so output length equals
	// input length. See the MaskStrategy constants for the disclosure
	// trade-offs. Masking never changes what is detected or what
	// Findings/Stats report, and every strategy stays
	// allocation-bounded on the streaming path.
	MaskStrategy MaskStrategy

	// EnableRules, when non-empty, restricts detection to exactly the
	// listed rule IDs (an allowlist), overriding Profile: a named rule
	// runs even if its MinProfile is more detailed than Profile requests.
	// Unknown IDs are rejected by New, as is a resulting empty rule set
	// (e.g. every enabled ID also appearing in DisableRules).
	EnableRules []string

	// DisableRules removes the listed rule IDs from the active set (a
	// denylist, applied after EnableRules). Unknown IDs are rejected by
	// New, as is a resulting empty rule set.
	DisableRules []string

	// CustomRules adds caller-defined rules to the active set. Custom rule
	// IDs must not collide with built-in rule IDs.
	CustomRules []CustomRule

	// OnFinding, when non-nil, is invoked synchronously for each confirmed
	// finding during Redact, in input order. The Finding never contains
	// matched input bytes. The callback must not retain references past
	// its return and must be fast; it runs on the scanning path. When one
	// Engine serves concurrent Redact calls, OnFinding is invoked
	// concurrently from each call's own goroutine — it must be safe for
	// concurrent use if the Engine is shared.
	OnFinding func(Finding)

	// RecordAligned, when true, makes the engine prefer emitting output
	// aligned to newline record boundaries when it is safe to do so: a
	// mid-stream emission is held back to the last buffered '\n' rather
	// than the raw safe limit, so downstream consumers of structured,
	// newline-delimited formats (e.g. JSONL session logs) observe whole
	// records instead of a line split across two writes. It never
	// changes what is redacted, only where emission boundaries fall
	// while streaming, and it never relaxes any streaming guarantee
	// (bounded memory, no temporary files, determinism): alignment is a
	// best-effort output-shaping preference that falls back to the raw
	// limit whenever no newline is buffered, so the stream still makes
	// progress. Defaults to false.
	RecordAligned bool
}

Config configures an Engine. The zero value is a valid configuration using ProfileBalanced, DefaultChunkSize, and DefaultMarker.

type CustomRule

type CustomRule struct {
	// ID is the stable rule identifier. Required, must be unique.
	ID string

	// Triggers are the literal byte strings that activate the rule.
	// Required, each must be non-empty.
	Triggers []string

	// CaseFold, when true, matches triggers with ASCII case folding.
	CaseFold bool

	// Confidence reported for findings confirmed by this rule.
	Confidence Confidence

	// MaxLookbehind and MaxLookahead bound the validation window in bytes
	// before the trigger start and after the trigger end.
	MaxLookbehind int
	MaxLookahead  int

	// Validate inspects window, in which the trigger occupies
	// window[trigStart:trigEnd], and reports the half-open byte range
	// [start, end) within window to redact. Implementations must not
	// retain window, log it, or include its contents in any output.
	//
	// A panic inside Validate does not propagate out of Redact: it is
	// caught and returned as an error naming this rule, so a faulty
	// validator cannot crash a process that redacts untrusted input.
	// The panic value never appears in the error (errors never contain
	// input bytes), the reported span is defensively clamped to window,
	// and output already written before the panic remains valid redacted
	// output.
	//
	// When one Engine serves concurrent Redact calls, Validate is
	// invoked concurrently from each call's own goroutine — like
	// Config.OnFinding, it must be safe for concurrent use if the
	// Engine is shared. Pure functions of the window are naturally safe.
	Validate func(window []byte, trigStart, trigEnd int) (start, end int, ok bool)
}

CustomRule is a caller-supplied detection rule.

A CustomRule is located by literal trigger strings. When a trigger is found, Validate is invoked with a bounded window of surrounding bytes and decides whether a secret is present and which bytes to redact.

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine detects and redacts secrets from byte streams.

An Engine is immutable after construction and safe for concurrent use by multiple goroutines; each Redact call maintains independent scan state.

func New

func New(cfg Config) (*Engine, error)

New compiles the active rule set for the configuration and returns a reusable Engine.

func (*Engine) ActiveRules

func (e *Engine) ActiveRules() []RuleInfo

ActiveRules returns the rules compiled into e — the effective result of applying Config.Profile, EnableRules, DisableRules, and CustomRules — sorted by ID. The returned slice is a copy owned by the caller; mutating it does not affect e, and no internal engine state is exposed.

func (*Engine) Redact

func (e *Engine) Redact(ctx context.Context, dst io.Writer, src io.Reader) (Stats, error)

Redact copies src to dst, replacing every confirmed secret according to the configured MaskStrategy (by default, one copy of Config.Marker per secret), and returns statistics about the scan.

Redact streams: it never buffers more than a bounded window of input and never writes unredacted data to temporary storage. It returns the first error encountered: ctx.Err() on cancellation, a *ReadError on source failure, a *WriteError on destination failure, or an error when a custom rule's Validate panics (see CustomRule). On error, output already written remains valid redacted output; no unredacted confirmed secret has been emitted.

ctx is polled once per read iteration: cancellation cannot interrupt a call already blocked inside src.Read.

func (*Engine) RuleSetVersion

func (e *Engine) RuleSetVersion() string

RuleSetVersion reports the version identifier of the compiled built-in rule set, for inclusion in caller diagnostics.

type Finding

type Finding struct {
	// RuleID is the stable identifier of the rule that confirmed the
	// secret, e.g. "aws-access-key-id".
	RuleID string

	// Confidence is the confidence level of the confirming rule.
	Confidence Confidence

	// Start is the byte offset of the first redacted byte in the input.
	Start int64

	// End is the byte offset one past the last redacted byte in the input.
	End int64
}

Finding describes one confirmed and redacted secret occurrence.

A Finding never contains the matched secret value, only its location and the rule that confirmed it. Offsets refer to byte positions in the original (unredacted) input stream.

type MaskStrategy

type MaskStrategy core.MaskStrategy

MaskStrategy selects how each confirmed secret span is replaced in the output. The zero value is MaskFixedMarker, so the zero Config keeps the documented fixed-marker behavior.

func (MaskStrategy) String

func (m MaskStrategy) String() string

String returns the strategy name.

type Profile

type Profile core.Profile

Profile selects a predefined trade-off between scanning throughput and detection depth.

func (Profile) String

func (p Profile) String() string

String returns the profile name.

type ReadError

type ReadError struct{ Err error }

ReadError wraps an error returned by the source io.Reader, so callers can distinguish source I/O failures from configuration or destination failures (see the package documentation's Errors section). The wrapped error is produced by the caller's own reader and is preserved unmodified for errors.Is/As; goredact itself never puts scanned input bytes in an error.

func (*ReadError) Error

func (e *ReadError) Error() string

func (*ReadError) Unwrap

func (e *ReadError) Unwrap() error

type RuleInfo

type RuleInfo struct {
	// ID is the stable rule identifier, e.g. "aws-access-key-id".
	ID string

	// Name is the human-readable rule name.
	Name string

	// Confidence is the confidence level reported by findings this rule
	// confirms.
	Confidence Confidence

	// MinProfile is the least detailed profile that includes this rule.
	// For custom rules (Custom == true) this is always the zero Profile
	// value: custom rules are not tiered by profile — they are active
	// whenever they are part of an Engine's configuration, regardless of
	// Config.Profile — so MinProfile carries no meaning for them.
	MinProfile Profile

	// Custom reports whether this is a caller-supplied rule (added via
	// Config.CustomRules) rather than a built-in.
	Custom bool
}

RuleInfo describes one built-in or custom detection rule for introspection and diagnostics. It never contains matched input bytes or any secret material — only stable identifiers and metadata.

func BuiltinRules

func BuiltinRules() []RuleInfo

BuiltinRules returns every built-in rule this version of goredact ships, regardless of profile, sorted by ID. Use (*Engine).ActiveRules to see the rules compiled into a specific Engine.

type Stats

type Stats struct {
	// BytesRead is the total number of input bytes consumed.
	BytesRead int64 `json:"bytes_read"`

	// BytesWritten is the total number of output bytes produced.
	BytesWritten int64 `json:"bytes_written"`

	// Candidates is the number of trigger/rule pairs submitted to a
	// validator. It is useful for operational and benchmark telemetry and
	// never exposes input contents.
	Candidates int64 `json:"candidates"`

	// Findings is the number of confirmed, redacted secret spans.
	Findings int64 `json:"findings"`

	// RedactedBytes is the total number of input bytes replaced by
	// redaction markers.
	RedactedBytes int64 `json:"redacted_bytes"`

	// ByRule maps rule IDs to the number of confirmed findings attributed
	// to that rule. It is nil when there are no findings.
	ByRule map[string]int64 `json:"by_rule,omitempty"`
}

Stats summarises one Redact call. It never contains matched input bytes. The JSON field names use snake_case, matching the CLI's directory-report schema.

type WriteError

type WriteError struct{ Err error }

WriteError wraps an error returned by the destination io.Writer, so callers can distinguish destination I/O failures from configuration or source failures (see the package documentation's Errors section). The wrapped error is produced by the caller's own writer and is preserved unmodified for errors.Is/As; goredact itself never puts scanned input bytes in an error.

func (*WriteError) Error

func (e *WriteError) Error() string

func (*WriteError) Unwrap

func (e *WriteError) Unwrap() error