Documentation
¶
Overview ¶
Package scan is OpticTrace's safety net.
The rest of the system masks what you NAME: a redaction rule covers $.credit_card.number because you wrote it down. The failure that actually bites in production is the field you FORGOT — a new endpoint ships, nobody adds a rule, and secrets land in the payload store.
scan inverts the model. It reads records that already passed governance and looks for values that LOOK sensitive regardless of what the rules say, then tells you which redaction rule would have caught them.
Design constraints, in priority order:
- Never print a secret. Findings carry a masked sample only — a scanner that echoes the credential it found has just leaked it again, into your CI logs this time.
- Prefer precision over recall. A detector that cries wolf gets muted, and a muted detector protects nothing. Patterns here are structural (issuer prefixes, checksums, framing) rather than "looks random".
Index ¶
Constants ¶
const ( SevCritical = "critical" // a credential: exploitable as-is if leaked SevHigh = "high" // regulated personal data (PCI, national IDs) SevMedium = "medium" // personal data, lower blast radius )
Severity ranks how urgently a finding should be acted on.
const MaxPatternLength = 512
MaxPatternLength bounds a user-supplied pattern. scan runs over every recorded body, so a pathological regex is an availability problem rather than merely a noisy one.
Variables ¶
var Detectors = []Detector{ { Kind: "private-key", Severity: SevCritical, Why: "a PEM private key block was stored verbatim", // contains filtered or unexported fields }, { Kind: "aws-access-key-id", Severity: SevCritical, Why: "an AWS access key ID, usually paired with a secret nearby", // contains filtered or unexported fields }, { Kind: "github-token", Severity: SevCritical, Why: "a GitHub personal access / app token", // contains filtered or unexported fields }, { Kind: "slack-token", Severity: SevCritical, Why: "a Slack API token", // contains filtered or unexported fields }, { Kind: "stripe-secret-key", Severity: SevCritical, Why: "a live Stripe secret key", // contains filtered or unexported fields }, { Kind: "google-api-key", Severity: SevCritical, Why: "a Google API key", // contains filtered or unexported fields }, { Kind: "jwt", Severity: SevCritical, Why: "a JSON Web Token — bearer credentials are replayable until they expire", // contains filtered or unexported fields }, { Kind: "credit-card", Severity: SevHigh, Why: "a Luhn-valid card number — PCI-DSS scope", // contains filtered or unexported fields }, { Kind: "iban", Severity: SevHigh, Why: "an IBAN bank account number", // contains filtered or unexported fields }, { Kind: "us-ssn", Severity: SevHigh, Why: "a US Social Security Number", // contains filtered or unexported fields }, { Kind: "email", Severity: SevMedium, Why: "an email address — personal data under GDPR and similar regimes", // contains filtered or unexported fields }, }
Detectors is the built-in set, ordered most to least severe. Every pattern is anchored on structure a real credential has — an issuer prefix, a checksum, a framing line — so ordinary prose does not trip them.
var Verifiers = map[string]func(string) bool{
"luhn": luhnValid,
"iban": ibanPlausible,
"us_ssn": ssnPlausible,
"verhoeff": verhoeffValid,
}
Verifiers are the named checksum routines a user-defined detector can borrow with `verify:`.
This registry exists because of the design rule the built-in set follows: every pattern is anchored on structure a real credential has — an issuer prefix, a checksum, a framing line — so ordinary prose does not trip it. A user-supplied regex cannot be trusted to hold that line on its own, and a scanner that cries wolf gets switched off. Exposing the checksums by name lets an org-specific detector be as precise as the built-ins.
Functions ¶
func KnownVerifier ¶ added in v0.8.0
KnownVerifier reports whether name is a registered verifier, so a bad name fails `optictrace validate` rather than at scan time in production.
func Mask ¶
Mask renders a value safe to display: first and last two characters with the middle replaced, and long values truncated. The point of a finding is "there is a card number in this field", never the number itself.
func VerifierNames ¶ added in v0.8.0
func VerifierNames() []string
VerifierNames lists the registered verifiers, sorted, for error messages.
Types ¶
type Detector ¶
type Detector struct {
Kind string
Severity string
Why string // what a reader should understand about the risk
// contains filtered or unexported fields
}
Detector recognizes one class of sensitive value.
func NewDetector ¶ added in v0.8.0
NewDetector builds a user-defined detector, validating everything that can be validated ahead of time. verify may be empty for no checksum.
type Finding ¶
type Finding struct {
Kind string `json:"kind"`
Severity string `json:"severity"`
Why string `json:"why"`
Method string `json:"method"`
Route string `json:"route"`
Location string `json:"location"` // request_body | response_body | request_headers | response_headers | app_log
Field string `json:"field"` // JSON path or header name
Count int `json:"count"`
FirstAt time.Time `json:"first_seen"`
LastAt time.Time `json:"last_seen"`
Sample string `json:"masked_sample"` // never the raw value
// Suggest is the optic.yaml fragment that would have prevented this.
Suggest string `json:"suggested_rule"`
}
Finding groups every occurrence of one sensitive-value class at one place in the API — a route plus a field path. Grouping is what makes the report actionable: "17 hits of credit-card at $.payment.pan on POST /api/v1/orders/**" maps to exactly one line of optic.yaml.
type Match ¶
type Match struct {
Kind string
Severity string
Why string
Masked string // safe to print: never the raw value
}
Match is one detector hit inside a single value.
type Report ¶
type Report struct {
Scanned int `json:"records_scanned"`
// LinesScanned counts application log lines examined. Reported separately
// from records because "0 findings" means something very different when
// no log lines were looked at than when thousands were.
LinesScanned int `json:"log_lines_scanned"`
// SpansScanned counts inner spans examined, reported separately for the
// same reason: "0 findings" means something very different when no span
// attributes were looked at than when thousands were.
SpansScanned int `json:"spans_scanned"`
Since time.Time `json:"since"`
Findings []Finding `json:"findings"`
}
Report is a complete scan result.
func Records ¶
Records scans a slice of stored telemetry. Equivalent to feeding each record to a Scanner; kept for callers that already hold the records.
func (*Report) HasAtLeast ¶
HasAtLeast reports whether any finding meets or exceeds a severity — the CI gate. Medium (personal data) is common enough that gating on critical or high is the sane default.
type Scanner ¶ added in v0.8.0
type Scanner struct {
// contains filtered or unexported fields
}
Scanner accumulates findings one record at a time.
The streaming shape exists because the callers used to load every record in the window — full bodies included — into a slice before scanning any of them. At the default 64 KiB capture limit and a 20,000-row window that is gigabytes resident before the first detector runs, on an endpoint that is reachable without authentication. Folding incrementally costs one record.
func NewScanner ¶ added in v0.8.0
NewScanner scans with the built-in detector set.
func NewScannerWith ¶ added in v0.8.0
NewScannerWith appends org-specific detectors to the built-ins. Custom detectors run last so a built-in's checksum-anchored match wins the severity when both fire on the same value.
func (*Scanner) AddAppLog ¶ added in v0.11.0
AddAppLog scans one application log line.
This surface matters more than the payloads, not less. A payload is structured and can be masked by JSON path; a log line is free text written by whoever was debugging that day, and it routinely carries tokens and whole request bodies inside stack traces. A leak detector that only reads payloads is looking where the data is easiest to protect rather than where it escapes.
Findings group by service and level rather than by route, because that is what a log line has: the code that wrote it is identified by the service it ran in, not by the request that happened to trigger it.
func (*Scanner) AddSpan ¶ added in v0.15.1
AddSpan scans one inner span's attributes and error text.
This surface is easy to forget and easy to leak through. A payload has a JSON path a rule can name; a statement is free text that a driver assembled, and the parameter it interpolated is the customer's. A card number sitting in `db.statement` was invisible to this scanner until it looked here, which meant the leak detector reported clean on the one surface nobody had written a rule for yet.
Source Files