scanner

Documentation

Index

• func MaskText(text string, findings []Finding, placeholder string) string
• type BatchScorer
• type DetectedSpan
• type Detector
• type Finding
  • func SafeForOutput(findings []Finding) []Finding
  • func Scan(text string, threshold float64, entropyThreshold float64, ctxChars int, ...) ([]Finding, error)
  • func (f Finding) MarshalJSON() ([]byte, error)
• type Options
• type RevealedFinding
  • func (r RevealedFinding) MarshalJSON() ([]byte, error)
• type Scanner
  • func New(opts Options) (*Scanner, error)
  • func (s *Scanner) BatchScore(triples []SpanTriple) ([]float64, error)
  • func (s *Scanner) Close() error
  • func (s *Scanner) Redact(text string, placeholder string) (string, []Finding, error)
  • func (s *Scanner) ScanReader(r io.Reader) ([]Finding, error)
  • func (s *Scanner) ScanString(text string) ([]Finding, error)
  • func (s *Scanner) UseDetector(d Detector)
• type Scorer
• type ScorerFactory
• type SpanTriple

Functions

func MaskText

func MaskText(text string, findings []Finding, placeholder string) string

MaskText replaces each finding's span with a placeholder. Non-overlapping findings expected (extractor dedupes). Returns masked text.

SAFETY: each finding's [Start,End] is snapped outward to the regex extractor's match at that position before masking. This prevents the case where the model returns a narrow span (e.g. just the username of a connection-string credential) and the placeholder leaves the leading or trailing bytes of the actual secret visible in the output.

Types

type BatchScorer

type BatchScorer interface {
	BatchScore(triples []SpanTriple) ([]float64, error)
}

BatchScorer is an optional extension to Scorer. When the concrete scorer implements it, scanner.Scan collects all candidates first and runs one batched transformer forward pass instead of N sequential Score calls. Falls back to per-candidate Score otherwise.

type DetectedSpan

type DetectedSpan struct {
	Start int
	End   int
	Type  string
	Score float32
}

DetectedSpan is the minimal shape a Detector returns. Mirrors native.Span without importing pkg/native (that would form a cycle: pkg/native already imports pkg/scanner).

type Detector

type Detector interface {
	Detect(text string) ([]DetectedSpan, error)
}

Detector is the v2 NER pipeline contract. Any type that implements Detect(text) -> []DetectedSpan satisfies it; pkg/native.Detector does (via a thin adapter wrapped by callers). Kept minimal so pkg/scanner stays independent of any concrete model runtime.

type Finding

type Finding struct {
	File       string  `json:"file,omitempty"`
	Line       int     `json:"line"`
	Column     int     `json:"column"`
	Rule       string  `json:"rule"`
	Span       string  `json:"span,omitempty"`
	Redacted   string  `json:"redacted"`
	Start      int     `json:"start"`
	End        int     `json:"end"`
	Entropy    float64 `json:"entropy"`
	Confidence float64 `json:"confidence,omitempty"`
}

Finding is the output unit: one detected secret or PII span.

SECURITY NOTE: Span holds the raw secret/PII value. Library callers who serialize findings to logs, dashboards, CI artifacts, telemetry, or any external sink should ALWAYS pass through SafeForOutput first (or set Span = "" manually). Otherwise hush leaks the very thing it was supposed to find. The hush CLI emits findings without Span by default; --output-reveal-secrets opts back in.

func SafeForOutput

func SafeForOutput(findings []Finding) []Finding

SafeForOutput returns a copy of the findings with Span cleared, so the raw secret/PII value never leaks via JSON serialization. Use this before writing findings to any external sink. Pass-through equivalent to setting f.Span = "" on each item.

Note: as of v0.1.10, Finding.MarshalJSON also omits Span by default, so SafeForOutput is now belt-and-suspenders. It still clears Span on the in-memory struct, which matters for callers that build their own non-JSON output paths (text templates, struct printers, etc).

func Scan

func Scan(text string, threshold float64, entropyThreshold float64, ctxChars int, scorer Scorer) ([]Finding, error)

Scan finds candidates, optionally filters with the model, returns findings.

PII candidates (RuleType == "pii") bypass the model entirely: regex precision is enough on those, and the current shipped classifier was trained on credentials so it would over-suppress real PII findings (emails, SSNs, credit cards). PII findings are reported with confidence 1.0 from the regex match.

func (Finding) MarshalJSON

func (f Finding) MarshalJSON() ([]byte, error)

MarshalJSON serializes a Finding without the raw Span value. This is the safe default: any pipeline that pipes findings to logs/dashboards/CI artifacts via json.Marshal cannot accidentally leak the secret it just detected. Callers that genuinely need the raw value (rotation, revocation) should read f.Span directly and emit it through a private sink.

type Options

type Options struct {
	MinConfidence    float64
	EntropyThreshold float64
	CtxChars         int
	ModelOff         bool
	IntraOpThreads   int

	// DetectorPrefilter, when true, runs the regex+entropy extractor first
	// in v2 NER mode and only invokes the detector on text regions
	// containing candidates. Cuts cost on clean files from ~2.6s/4KB to
	// near zero by skipping the model entirely when no regex/entropy hit
	// fires. The detector still has final say over what the spans are.
	// Has no effect when no detector is wired.
	DetectorPrefilter bool

	// UseDetector, when true, signals the higher-level hush package to
	// auto-wire the embedded v2 NER detector. Implies ModelOff=true (the
	// v1 sequence classifier is not loaded). pkg/scanner itself does not
	// act on this field — see hush.New.
	UseDetector bool
}

Options configures a Scanner.

Zero values are sensible defaults — these match the CLI:

MinConfidence = 0.5    keep findings scoring >= 0.5
EntropyThreshold = 4.0 shannon entropy floor for generic candidates
CtxChars = 256         chars of left/right context passed to the scorer
ModelOff = false       classifier enabled; set true to skip ML filtering
IntraOpThreads = 0     ORT default (one thread per CPU)

type RevealedFinding

type RevealedFinding struct{ F Finding }

RevealedFinding wraps a Finding so json.Marshal includes the raw Span value. Use this only when the sink is private and the caller has explicitly opted in (e.g. CLI --output-reveal-secrets). Default Finding.MarshalJSON intentionally omits Span.

func (RevealedFinding) MarshalJSON

func (r RevealedFinding) MarshalJSON() ([]byte, error)

MarshalJSON on RevealedFinding emits the raw Span. Callers who reach for this type are explicitly opting in to a leak risk.

type Scanner

type Scanner struct {
	// contains filtered or unexported fields
}

Scanner is a stateful, ergonomic facade over Scan + MaskText.

It owns the classifier lifecycle so callers do not have to wire one up by hand. Safe for concurrent use.

func New

func New(opts Options) (*Scanner, error)

New returns a ready to use Scanner. When ModelOff is false it loads the embedded classifier via the default factory (see DefaultScorerFactory). Callers must Close() the scanner to release the ONNX session.

func (*Scanner) BatchScore

func (s *Scanner) BatchScore(triples []SpanTriple) ([]float64, error)

BatchScore scores many candidate spans in a single transformer forward pass when the underlying scorer supports it. Falls back to per-candidate Score when it doesn't. Returns probabilities in input order.

Useful when a caller already has a list of (left, span, right) triples from their own extraction pipeline and wants to skip hush's regex stage.

func (*Scanner) Close

func (s *Scanner) Close() error

Close releases the classifier. Safe to call multiple times.

func (*Scanner) Redact

func (s *Scanner) Redact(text string, placeholder string) (string, []Finding, error)

Redact scans text and returns a masked copy with each finding replaced. placeholder supports the %s verb, which is substituted with the rule name. An empty placeholder falls back to the default [REDACTED_<RULE>_<N>].

func (*Scanner) ScanReader

func (s *Scanner) ScanReader(r io.Reader) ([]Finding, error)

ScanReader scans an arbitrarily large reader without slurping the full stream into memory. As of v0.1.11 it reads in 1 MB chunks with a 4 KB trailing overlap and dedupes findings whose absolute offset matches across chunks. Constant memory regardless of input size (#15).

func (*Scanner) ScanString

func (s *Scanner) ScanString(text string) ([]Finding, error)

ScanString scans a string, returning findings.

When UseDetector has installed an NER detector (v2 path), the regex extractor and Scorer are bypassed and the detector emits spans directly. Otherwise the legacy regex + Scorer pipeline runs.

func (*Scanner) UseDetector

func (s *Scanner) UseDetector(d Detector)

UseDetector switches the scanner to v2 NER mode. When set, the regex extractor and Scorer are bypassed; the detector emits spans directly. Pass nil to revert to the regex+scorer pipeline.

type Scorer

type Scorer interface {
	Score(left, span, right string) (float64, error)
}

Scorer returns a probability [0,1] that a candidate is a real secret. Nil scorer means no ML filtering; all candidates become findings.

type ScorerFactory

type ScorerFactory func(intraOpThreads int) (Scorer, func() error, error)

ScorerFactory returns a Scorer plus a closer callback. The closer is invoked by Scanner.Close(). Set DefaultScorerFactory from a package that depends on the classifier (e.g. pkg/classifier's init or a small shim in cmd/hush) so pkg/scanner stays decoupled from ORT.

var DefaultScorerFactory ScorerFactory

DefaultScorerFactory is nil by default. Applications that want the embedded classifier should import a package that sets this.

type SpanTriple

type SpanTriple struct {
	Left, Span, Right string
}

SpanTriple is one candidate to batch-score: left context, span, right context. Re-exported from native semantics for scanner callers that only import pkg/scanner.