dlp

package
v0.0.0-...-19b86f1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 31, 2026 License: MIT Imports: 26 Imported by: 0

Documentation

Overview

Package dlp implements the layered Data Loss Prevention (DLP) pipeline described in ARCHITECTURE.md section 2. The pipeline is: 

classifier → Aho-Corasick prefix scan → regex validation
→ hotword proximity → entropy → exclusion → scoring → threshold

Privacy invariant: scan content stays in process memory only. No domain names, URLs, IP addresses, or matched substrings are ever written to disk, logged, or persisted in SQLite. Only anonymous integer counters (dlp_scans_total, dlp_blocks_total) cross the SQLite boundary.

Index

Constants

View Source 
const (
	ActionAllow  = "allow"
	ActionBlock  = "block"
	ActionRedact = "redact"
)

Redaction action values for RedactionResult.Action.

View Source 
const (
	DestinationKindAIChat   = "ai_chat"
	DestinationKindAICode   = "ai_code"
	DestinationKindCodeHost = "code_host"
	DestinationKindPasteBin = "paste_bin"
	DestinationKindSocial   = "social"
	DestinationKindEmail    = "email"
	DestinationKindUnknown  = "unknown"
)

Destination-kind enum values used by the global bias table and (typically) by the browser extension when populating SourceContext.DestinationKind. Strings, not iota, because they cross the JSON wire boundary.

View Source 
const (
	ElementKindInput           = "input"
	ElementKindTextarea        = "textarea"
	ElementKindContentEditable = "contenteditable"
	ElementKindFormSubmit      = "form_submit"
	ElementKindNetworkBody     = "network_body"
	ElementKindDragDrop        = "drag_drop"
	ElementKindClipboard       = "clipboard"
)

Element-kind enum values matching what each browser-extension interceptor populates (one per content-script type).

View Source 
const AllowlistScopeAny = "*"

AllowlistScopeAny is the scope value meaning "any destination".

View Source 
const CanaryNamePrefix = "Canary:"

CanaryNamePrefix is prepended to a canary's user label to form the Pattern.Name. The block-notification path keys on this prefix to render canary hits distinctively and to attribute a trigger back to the canary's label.

View Source 
const CategoryUncategorized = "uncategorized"

CategoryUncategorized is the fallback category for patterns that did not declare one in dlp_patterns.json. It is exposed so callers (UI, tests) can refer to it without a magic string.

View Source 
const ConcurrentEvalThreshold = 10 * 1024

ConcurrentEvalThreshold is the byte threshold above which the pipeline evaluates per-pattern groups concurrently. Below this the sequential path is cheaper because of goroutine setup overhead.

View Source 
const LargeContentThreshold = 50 * 1024

LargeContentThreshold is the byte threshold above which Pipeline drops low-severity patterns from the candidate set. Configurable per pipeline via SetLargeContentThreshold; the default matches the `large_content_threshold` setting in config.yaml.

View Source 
const ScanCacheCapacity = 1024

ScanCacheCapacity bounds the number of entries the LRU holds. Each entry is small (32-byte digest + ScanResult + a doubly-linked list node) so the default of 1024 fits comfortably in a few hundred KiB.

View Source 
const ScanCacheTTL = 5 * time.Second

ScanCacheTTL is the default lifetime of a cache entry. Five seconds is short enough that operators do not need to manually invalidate the cache after a rule update and long enough to deduplicate the rapid-fire scans that hit the agent when an extension's paste, form-submit, and fetch interceptors all fire on the same content.

Variables

This section is empty.

Functions

func CanaryName 

func CanaryName(label string) string

CanaryName returns the Pattern.Name for a canary with the given label.

func CheckHotwords 

func CheckHotwords(content string, match Match, pattern Pattern) bool

CheckHotwords returns true if any of pattern.Hotwords appears within pattern.HotwordWindow bytes of the match. The match itself is excluded from the haystack so the hotword cannot accidentally be the matched substring. Matching is case-insensitive.

A pattern with no hotwords or hotword_window == 0 returns false (no boost). Callers (the scorer) decide what to do with that result — patterns with require_hotword=true should be filtered out higher up.

func GenerateCanaryToken 

func GenerateCanaryToken(label string) (string, error)

GenerateCanaryToken returns a fresh, unguessable canary token of the form PGCRY<8-hex label hash><16-char random>. The label hash makes the token weakly self-describing for debugging without revealing the label; the random suffix (80 bits) makes collisions and guessing infeasible. Returns an error only if the system CSPRNG fails.

func IsCanaryName 

func IsCanaryName(name string) (label string, ok bool)

IsCanaryName reports whether a Pattern.Name (or a ScanResult's PatternName) belongs to a canary, and returns the user label.

func IsPlaceholderShape 

func IsPlaceholderShape(value string) bool

IsPlaceholderShape reports whether value is a recognisable template placeholder.

func IsPublicExample 

func IsPublicExample(value string) bool

IsPublicExample reports whether value is a known public example value. Matched values are excluded from DLP block decisions.

Returns false for empty input.

func LoadOrGenerateSalt 

func LoadOrGenerateSalt(path string) ([]byte, error)

LoadOrGenerateSalt reads the salt at path. If the file does not exist, a fresh 32-byte random salt is generated, persisted with 0600 perms, and returned. If the file exists but is empty or malformed, an error is returned (the caller should NOT silently overwrite — that would invalidate every existing allowlist entry).

func NormalizeContent 

func NormalizeContent(content string) string

NormalizeContent returns the canonical form of content for DLP scanning. The result may be longer than the input when base64 blocks were decoded and appended.

func ScoreBiasFromContext 

func ScoreBiasFromContext(pat Pattern, source SourceContext) int

ScoreBiasFromContext computes the score delta described in the file-level comment.

func ScoreMatch 

func ScoreMatch(in ScoreInput) int

ScoreMatch returns the aggregate score for one match.

func ShannonEntropy 

func ShannonEntropy(s string) float64

ShannonEntropy returns the per-byte Shannon entropy of s. Empty strings have zero entropy. Result is in bits-per-byte, so a uniform 256-byte random sequence approaches 8.0. Typical thresholds are 3.0–4.0 bits/byte for "looks random enough to be a secret".

Types

type Allowlist 

type Allowlist struct {
	// contains filtered or unexported fields
}

Allowlist owns the dlp_allowlist SQLite table plus an in-RAM snapshot for O(1) lookups on the scan hot path.

func NewAllowlist 

func NewAllowlist(ctx context.Context, db *sql.DB, salt []byte, now func() time.Time) (*Allowlist, error)

NewAllowlist constructs an Allowlist backed by db and salted with salt. The current contents of the dlp_allowlist table are read into the in-RAM cache so subsequent Allows() calls are O(1).

Pass a non-nil time.Now-equivalent (or nil for time.Now) for deterministic TTL tests.

func (*Allowlist) Add 

func (a *Allowlist) Add(ctx context.Context, value, scope, patternName string, ttl time.Duration) (string, error)

Add records a new allowlist entry. value is hashed before insert. scope == "" is normalised to "*" (any destination). ttl == 0 means never expire. patternName is optional display metadata for the settings UI.

func (*Allowlist) Allows 

func (a *Allowlist) Allows(value, destinationKind string) bool

Allows reports whether value is on the allowlist for the given destination kind. The match key is SHA-256(salt || normalize(value)) where normalize mirrors the public-example bloom helper (whitespace, dashes, underscores stripped; ASCII-lowercased). Scope semantics: "*" wins on any destination, otherwise the scope string must equal the destination kind exactly. Expired entries (expires_at > 0 AND expires_at <= now) are ignored and not eligible to fire.

Allows updates the entry's last_hit timestamp in the in-RAM cache only (the DB write is best-effort, fire-and-forget via the touchAsync helper) so it stays on the scan hot path.

func (*Allowlist) Cleanup 

func (a *Allowlist) Cleanup(ctx context.Context) (int, error)

Cleanup removes expired entries from both the DB and the in-RAM cache. Returns the number of rows removed. Safe to call periodically (e.g. once an hour from main.go); the in-RAM Allows check honours expiry on its own so this is purely housekeeping.

func (*Allowlist) List 

func (a *Allowlist) List(ctx context.Context) ([]AllowlistEntry, error)

List returns a snapshot of all entries, sorted by created_at descending (newest first) so the settings UI shows the most recent "never block this" actions at the top.

func (*Allowlist) Remove 

func (a *Allowlist) Remove(ctx context.Context, saltedHash string) (bool, error)

Remove deletes the entry by salted_hash. Returns true if a row was actually removed.

func (*Allowlist) Size 

func (a *Allowlist) Size() int

Size reports how many active entries are loaded in the in-RAM cache. Used by tests and the /api/status endpoint.

func (*Allowlist) SortedHashesForTest 

func (a *Allowlist) SortedHashesForTest() []string

SortedHashesForTest returns the in-RAM entry keys in deterministic order. Used by tests; not exported through the API.

type AllowlistEntry 

type AllowlistEntry struct {
	SaltedHash  string `json:"salted_hash"`
	Scope       string `json:"scope"`
	PatternName string `json:"pattern_name"`
	ExpiresAt   int64  `json:"expires_at"`
	CreatedAt   int64  `json:"created_at"`
	LastHit     int64  `json:"last_hit"`
}

AllowlistEntry is the public shape returned by Allowlist.List(). SaltedHash is hex-encoded SHA-256 (64 chars).

type Automaton 

type Automaton struct {
	// contains filtered or unexported fields
}

Automaton wraps the underlying Aho-Corasick matcher and the slice of patterns it was built from. patternsByPrefixIdx[i] is the list of patterns that share dictionary entry i — patterns can share prefixes (e.g. multiple patterns with prefix "api"), in which case all sharing patterns are treated as candidates and validated by their own regex.

func BuildAutomaton 

func BuildAutomaton(patterns []*Pattern) *Automaton

BuildAutomaton constructs an Automaton from the given patterns. Patterns with an empty Prefix bypass the Aho-Corasick scan and are emitted as candidates with offset 0 for every Scan call.

Matching is case-insensitive at the prefix level (we lowercase both the dictionary and the content). The per-pattern regex still honours its own flags during validation.

func (*Automaton) PrefixCount 

func (a *Automaton) PrefixCount() int

PrefixCount returns the number of distinct lowercase prefixes that were fed to the underlying matcher. Useful for tests.

func (*Automaton) Scan 

func (a *Automaton) Scan(content string) []Candidate

Scan performs a single Aho-Corasick pass over content and returns the set of (offset, pattern) candidates that the regex step should validate. Empty content returns nil.

type Candidate 

type Candidate struct {
	Offset  int
	Pattern *Pattern
}

Candidate is a (offset, pattern) pair emitted by the Aho-Corasick scanner. Offsets are byte offsets into the scanned content.

type Correlator 

type Correlator struct {
	// contains filtered or unexported fields
}

Correlator tracks recent paste tails per session.

func NewCorrelator 

func NewCorrelator(ttl time.Duration, tailLen, maxSess int) *Correlator

NewCorrelator returns a Correlator with default tunables (30s TTL, 256-byte tail, 4096 max simultaneous sessions). Pass non-zero values to override; zero falls back to defaults.

func (*Correlator) ActiveSessions 

func (c *Correlator) ActiveSessions() int

ActiveSessions returns the current number of tracked sessions. Intended for tests + metrics; not safe to use as a security signal.

func (*Correlator) Combine 

func (c *Correlator) Combine(sessionID, content string) (string, bool)

Combine updates the session state with content and returns the concatenated view (prior tail + current content). If this is the first paste in the session, returns (content, false). Otherwise returns (priorTail + sep + content, true).

sessionID == "" disables correlation — the function returns (content, false) and does not touch any state. Safe to call with empty session ID.

Safe for concurrent use.

func (*Correlator) Forget 

func (c *Correlator) Forget(sessionID string)

Forget removes the session entry. Useful when an upstream signals session end (e.g. browser tab closed).

type DictionaryMatchType 

type DictionaryMatchType string

DictionaryMatchType describes how Exclusion.Words are evaluated. 

const (
	// ExactMatch — the Match.Value must equal one of Words.
	ExactMatch DictionaryMatchType = "exact"
	// ProximityMatch — any of Words must appear within Window
	// bytes of the match (default mode when not specified).
	ProximityMatch DictionaryMatchType = "proximity"
)

type Exclusion 

type Exclusion struct {
	AppliesTo string              `json:"applies_to"`
	Type      ExclusionType       `json:"type"`
	Words     []string            `json:"words,omitempty"`
	Pattern   string              `json:"pattern,omitempty"`
	Window    int                 `json:"window,omitempty"`
	MatchType DictionaryMatchType `json:"match_type,omitempty"`

	// Suppress, when true on a regex exclusion, fully drops the match
	// instead of subtracting ExclusionPenalty. Use for known-doc
	// patterns such as AIza...EXAMPL... that should never count even
	// if all other signals (hotwords, entropy) line up.
	Suppress bool `json:"suppress,omitempty"`

	// Compiled is populated by LoadExclusions for regex exclusions.
	Compiled *regexp.Regexp `json:"-"`
}

Exclusion is a rule that suppresses or penalises matches that look like known false positives (e.g. "AKIAIOSFODNN7EXAMPLE", emails on @example.com, the literal word "placeholder" within 50 chars).

func LoadExclusions 

func LoadExclusions(path string) ([]Exclusion, error)

LoadExclusions reads exclusions from path and compiles each regex exclusion. Dictionary exclusions need no compilation.

func MergeExclusionsFromDir 

func MergeExclusionsFromDir(bundledPath, localDir string) ([]Exclusion, error)

MergeExclusionsFromDir does the same as MergePatternsFromDir for exclusions. Exclusion identity is (Type, AppliesTo, Match, Pattern); ties replace the bundled entry, mismatches append.

func ParseExclusions 

func ParseExclusions(raw []byte) ([]Exclusion, error)

ParseExclusions is the in-memory equivalent of LoadExclusions.

type ExclusionResult 

type ExclusionResult struct {
	// Hit indicates whether at least one exclusion applies. The
	// scorer applies the exclusion_penalty when Hit is true.
	Hit bool

	// SuppressEntirely is true when an exact-match dictionary
	// exclusion fired — e.g. AKIAIOSFODNN7EXAMPLE for AWS Access
	// Key. The pipeline drops the match entirely in this case
	// instead of just penalising it.
	SuppressEntirely bool
}

ExclusionResult is the outcome of CheckExclusion.

func CheckExclusion 

func CheckExclusion(content string, match Match, xs []Exclusion) ExclusionResult

CheckExclusion evaluates every exclusion in xs against (content, match). Order matters only for SuppressEntirely: as soon as an exact dictionary hit is found the function returns immediately. Otherwise it accumulates Hit across all matching exclusions.

type ExclusionType 

type ExclusionType string

ExclusionType is the discriminator for Exclusion entries. 

const (
	ExclusionDictionary ExclusionType = "dictionary"
	ExclusionRegex      ExclusionType = "regex"
)

type Match 

type Match struct {
	Pattern *Pattern
	Start   int
	End     int
	Value   string
}

Match is a regex-validated hit: a Pattern matched the content at [Start, End). The raw matched substring is held only in memory and must never be persisted.

func ValidateCandidates 

func ValidateCandidates(content string, candidates []Candidate) []Match

ValidateCandidates runs each candidate pattern's compiled regex over a window around its Aho-Corasick offset and returns deduplicated Match results. Patterns whose Compiled field is nil are silently skipped — callers are expected to have run LoadPatterns first.

Match values reference the original content slice and must NOT be persisted; they live only for the duration of the scan call.

type Pattern 

type Pattern struct {
	Name           string   `json:"name"`
	Regex          string   `json:"regex"`
	Prefix         string   `json:"prefix"`
	Severity       Severity `json:"severity"`
	ScoreWeight    int      `json:"score_weight"`
	MinMatches     int      `json:"min_matches,omitempty"`
	Hotwords       []string `json:"hotwords"`
	HotwordWindow  int      `json:"hotword_window"`
	HotwordBoost   int      `json:"hotword_boost"`
	RequireHotword bool     `json:"require_hotword"`
	EntropyMin     float64  `json:"entropy_min"`

	// Disabled marks a pattern as off by default. The pattern is
	// still loaded and available for override files or API toggles,
	// but LoadPatterns strips it from the active set unless an
	// override explicitly re-enables it.
	Disabled bool `json:"disabled,omitempty"`

	// Category groups patterns for selective enable/disable
	// ("PII", "cloud", "auth", …). Patterns loaded without a
	// category default to CategoryUncategorized so the toggle UI
	// still sees them.
	Category string `json:"category,omitempty"`

	// ContextBias is the source-context score. Per-pattern additive score
	// delta keyed by SourceContext.DestinationKind. Empty / missing
	// = no bias (default scoring behaviour). Example:
	//   "context_bias": { "code_host": -2, "paste_bin": +1 }
	// Pattern-level entries override the global scorer rules in
	// agent/internal/dlp/scorer.go.
	ContextBias map[string]int `json:"context_bias,omitempty"`

	// Compiled is populated by LoadPatterns; nil until compiled.
	Compiled *regexp.Regexp `json:"-"`
}

Pattern is a single DLP pattern loaded from rules/dlp_patterns.json. The compiled regex is filled in by LoadPatterns; callers should not mutate Pattern values after loading.

func CanaryPattern 

func CanaryPattern(token, label string) *Pattern

CanaryPattern builds the exact-match DLP Pattern for a canary. The regex is the literal token (QuoteMeta'd), severity is critical, and the score weight is high enough to block on a single match with no hotword required — a canary in a prompt is unambiguous exfil.

The regex is pre-compiled: unlike file-loaded patterns (which LoadPatterns compiles), a canary pattern is built directly, so it must arrive at the pipeline with a non-nil Compiled field or matchRegex skips it. QuoteMeta output is always a valid regexp, so MustCompile cannot panic here.

func LoadPatterns 

func LoadPatterns(path string) ([]*Pattern, error)

LoadPatterns reads patterns from path and compiles each pattern's regex. The slice is shared by the Aho-Corasick scanner and the regex validator — every Pattern.Compiled is non-nil on return.

func MergePatternsFromDir 

func MergePatternsFromDir(bundledPath, localDir string) ([]*Pattern, error)

MergePatternsFromDir loads bundled patterns from bundledPath and (if present) merges in patterns from <localDir>/dlp_patterns_override.json. Merge semantics: an entry in the override file with the same Name replaces the bundled entry; otherwise it is appended. An empty localDir or missing override file leaves the bundled set untouched.

func ParsePatterns 

func ParsePatterns(raw []byte) ([]*Pattern, error)

ParsePatterns is the in-memory equivalent of LoadPatterns.

type Pipeline 

type Pipeline struct {
	// contains filtered or unexported fields
}

Pipeline ties all the DLP pipeline steps together.

func NewPipeline 

func NewPipeline(weights ScoreWeights, threshold *ThresholdEngine) *Pipeline

NewPipeline returns an empty pipeline. Callers must call Rebuild before Scan returns useful results — Scan on an empty pipeline always returns Blocked=false.

func (*Pipeline) Allowlist 

func (p *Pipeline) Allowlist() *Allowlist

Allowlist returns the active allowlist, or nil if H is disabled.

func (*Pipeline) Cache 

func (p *Pipeline) Cache() *ScanCache

Cache returns the attached ScanCache, or nil if caching is disabled.

func (*Pipeline) Categories 

func (p *Pipeline) Categories() []string

Categories returns the sorted list of distinct pattern categories currently loaded. Used by the Electron UI's Rules page.

func (*Pipeline) Correlator 

func (p *Pipeline) Correlator() *Correlator

Correlator returns the active multi-piece correlator, or nil if correlation is disabled.

func (*Pipeline) DisabledCategories 

func (p *Pipeline) DisabledCategories() []string

DisabledCategories returns the current set of disabled categories. The returned slice is a snapshot and safe for the caller to mutate.

func (*Pipeline) EnableAllowlist 

func (p *Pipeline) EnableAllowlist(a *Allowlist)

EnableAllowlist turns on per-user feedback suppression. Pass nil to disable. Safe to call concurrently with Scan.

func (*Pipeline) EnableCache 

func (p *Pipeline) EnableCache(c *ScanCache)

EnableCache attaches a ScanCache to the pipeline. Passing nil disables caching (useful for tests). Safe to call concurrently with Scan.

func (*Pipeline) EnableCorrelator 

func (p *Pipeline) EnableCorrelator(c *Correlator)

EnableCorrelator turns on multi-piece detection. Pass nil to disable. Safe to call concurrently with Scan.

func (*Pipeline) LargeContentThreshold 

func (p *Pipeline) LargeContentThreshold() int

LargeContentThreshold returns the current threshold in bytes.

func (*Pipeline) Patterns 

func (p *Pipeline) Patterns() []*Pattern

Patterns returns a snapshot of the loaded patterns. Used for tests and introspection only — callers must not mutate the slice.

func (*Pipeline) Rebuild 

func (p *Pipeline) Rebuild(patterns []*Pattern, exclusions []Exclusion)

Rebuild atomically swaps in a new pattern set, automaton, and exclusion list. Called on agent startup and whenever rule files are updated (POST /api/rules/update or future automatic rule sync).

func (*Pipeline) Redact 

func (p *Pipeline) Redact(ctx context.Context, content, sessionID string, source SourceContext) RedactionResult

Redact runs the same pipeline as ScanWithContext and, when the verdict is to block, attempts to remove the offending secrets in place rather than rejecting the whole payload.

Safety contract (see docs/redaction-design.md §7) — all enforced in code below, every edge fails CLOSED to block:

  1. Relocate each secret verbatim in the ORIGINAL content. Match offsets are in normalized space, so a secret that only exists after normalization (homoglyph / base64 obfuscation) cannot be located here → block, never a partial splice.
  2. Merge overlapping / adjacent spans so no un-redacted sliver can survive between two patterns covering the same secret.
  3. Token-collision guard: the token chosen for a span is guaranteed not to already occur in the original text.
  4. Verify-pass: re-scan the redacted output; if anything still trips, discard it and block. A leak now needs two independent failures.

Pure function: no I/O, no globals (preserves the engine-portability invariant). source may be the zero value.

func (*Pipeline) ResetCache 

func (p *Pipeline) ResetCache()

ResetCache drops every cached scan result. Exposed so callers that mutate state outside the pipeline's setters — most notably Threshold().Set() on the embedded ThresholdEngine — can keep the cache in sync with the live policy. A nil cache is a no-op.

func (*Pipeline) Scan 

func (p *Pipeline) Scan(ctx context.Context, content string) ScanResult

Scan runs the full DLP pipeline on content and returns the highest scoring match's decision. ctx is honoured at the entry point only; individual pipeline steps are bounded and complete quickly.

func (*Pipeline) ScanSession 

func (p *Pipeline) ScanSession(ctx context.Context, content, sessionID string) ScanResult

ScanSession is the session-aware variant of Scan. When sessionID is non-empty and a correlator is enabled (see EnableCorrelator), content is combined with the session's prior paste tail so that secrets split across consecutive pastes can be detected. Pass an empty sessionID to behave identically to Scan.

func (*Pipeline) ScanWithContext 

func (p *Pipeline) ScanWithContext(
	ctx context.Context,
	content, sessionID string,
	source SourceContext,
) ScanResult

ScanWithContext is the source-context entrypoint. It accepts the SourceContext the caller (typically the browser extension) collected about WHERE the scan is happening — destination host / kind, DOM element type, code-fence flag, language hint, and content-derived hashes. The scorer biases verdicts by this context (the scoring stage wires source through but the consult it). Pass an empty SourceContext{} to behave identically to ScanSession.

func (*Pipeline) SetCanaryPatterns 

func (p *Pipeline) SetCanaryPatterns(canaries []*Pattern)

SetCanaryPatterns swaps in the user's canary tripwire patterns and re-applies the active set, preserving the current rule-file patterns. Pass nil/empty to clear all canaries. Safe to call concurrently with Scan; the scan cache is reset so verdicts produced before a canary existed are not served stale.

func (*Pipeline) SetDisabledCategories 

func (p *Pipeline) SetDisabledCategories(categories []string)

SetDisabledCategories replaces the set of disabled pattern categories. Pass an empty slice to re-enable all categories. The agent UI uses this to let operators turn off PII or low-severity pattern groups without editing the rule file. The scan cache is reset alongside the update so verdicts produced before a category was disabled (or re-enabled) cannot survive the change.

func (*Pipeline) SetLargeContentThreshold 

func (p *Pipeline) SetLargeContentThreshold(n int)

SetLargeContentThreshold updates the byte threshold above which the pipeline switches to "critical/high only" scanning. Zero or negative values restore the default LargeContentThreshold. Safe to call concurrently with Scan. The scan cache is reset alongside the update so cached verdicts produced under the previous threshold cannot leak past the change.

func (*Pipeline) SetWeights 

func (p *Pipeline) SetWeights(w ScoreWeights)

SetWeights atomically updates the scoring weights. The scan cache is reset alongside the update so verdicts produced under the previous weights cannot leak past the change — otherwise a PUT /api/dlp/config that raises a hotword/entropy boost would only take effect after the cache TTL expired.

func (*Pipeline) Threshold 

func (p *Pipeline) Threshold() *ThresholdEngine

Threshold returns the threshold engine. Used by the API to expose GET /api/dlp/config and PUT /api/dlp/config.

type Redaction 

type Redaction struct {
	PatternName string `json:"pattern_name"`
	Token       string `json:"token"`
	Start       int    `json:"start"`
	End         int    `json:"end"`
}

Redaction is one applied replacement. Start/End are byte offsets into the RETURNED RedactedContent (not the original). The matched value is deliberately absent — it must never leave this process.

type RedactionResult 

type RedactionResult struct {
	ScanResult
	Action          string      `json:"action"`
	RedactedContent string      `json:"redacted_content,omitempty"`
	Redactions      []Redaction `json:"redactions,omitempty"`
}

RedactionResult is what Redact returns. It embeds the underlying ScanResult so callers that only care about block/allow can read Blocked/PatternName/Score directly.

type ScanCache 

type ScanCache struct {
	// contains filtered or unexported fields
}

ScanCache is a small fixed-size LRU keyed on SHA-256 of the scanned content. It is safe for concurrent use.

func NewScanCache 

func NewScanCache(capacity int, ttl time.Duration) *ScanCache

NewScanCache constructs an LRU cache with the supplied capacity and TTL. Zero or negative values fall back to ScanCacheCapacity / ScanCacheTTL respectively.

func (*ScanCache) Lookup 

func (c *ScanCache) Lookup(content string) (ScanResult, bool)

Lookup returns the cached ScanResult for content, or (zero, false) on a miss or stale entry. A stale entry is also evicted as a side effect so subsequent Lookup calls return a miss until the next Put.

func (*ScanCache) Put 

func (c *ScanCache) Put(content string, result ScanResult)

Put stores the ScanResult for content. If the cache is at capacity the oldest entry is evicted. Putting an existing digest refreshes its TTL and moves it to the front of the LRU.

func (*ScanCache) Reset 

func (c *ScanCache) Reset()

Reset drops every cached entry. Used by tests and on rule reload.

func (*ScanCache) Stats 

func (c *ScanCache) Stats() ScanCacheStats

Stats returns a snapshot of the cache counters.

type ScanCacheStats 

type ScanCacheStats struct {
	Size      int    `json:"size"`
	Capacity  int    `json:"capacity"`
	Hits      uint64 `json:"hits"`
	Misses    uint64 `json:"misses"`
	Evictions uint64 `json:"evictions"`
}

Stats returns anonymous counters used by /api/status. They are intentionally not exposed per-entry — the agent must never reveal which content was scanned.

type ScanResult 

type ScanResult struct {
	Blocked     bool   `json:"blocked"`
	PatternName string `json:"pattern_name"`
	Score       int    `json:"score"`
}

ScanResult is what Pipeline.Scan returns. PatternName is empty when Blocked is false. Score is the highest score across all matches.

type ScoreInput 

type ScoreInput struct {
	Pattern        Pattern
	Match          Match
	HotwordPresent bool
	Entropy        float64
	NumMatches     int // total number of matches for this Pattern
	ExclusionHit   bool
	Weights        ScoreWeights
}

ScoreInput is everything the scorer needs to score a single match.

type ScoreWeights 

type ScoreWeights struct {
	HotwordBoost     int
	EntropyBoost     int
	EntropyPenalty   int
	ExclusionPenalty int
	MultiMatchBoost  int
}

ScoreWeights holds the per-instance scoring multipliers loaded from the dlp_config SQLite table.

func DefaultScoreWeights 

func DefaultScoreWeights() ScoreWeights

DefaultScoreWeights mirrors the defaults seeded into dlp_config.

type Severity 

type Severity string

Severity is the per-pattern severity level. Each severity has its own configurable threshold in the dlp_config SQLite table. 

const (
	SeverityCritical Severity = "critical"
	SeverityHigh     Severity = "high"
	SeverityMedium   Severity = "medium"
	SeverityLow      Severity = "low"
)

type SourceContext 

type SourceContext struct {
	DestinationKind string `json:"destination_kind,omitempty"`
	DestinationHost string `json:"destination_host,omitempty"`
	ElementKind     string `json:"element_kind,omitempty"`
	InCodeFence     bool   `json:"in_code_fence,omitempty"`

	// LanguageHint is a code-block language tag (e.g. "yaml",
	// "javascript", "go"). Populated from a `<code
	// class="language-…">` ancestor on the focused element.
	LanguageHint string `json:"language_hint,omitempty"`

	// PathHint is the path-hint axis. A coarse classification of the
	// file path the user is interacting with on code-host
	// destinations. Values: "test" / "fixture" / "spec" / "mock" /
	// "docs" / "src" / "" (unknown). Derived by the extension from
	// the URL pathname so the agent never sees the raw path.
	//
	// The scorer adds a -1 bias when destination_kind == code_host
	// AND PathHint ∈ {test, fixture, spec, mock} — stacks with the
	// global code_host destination delta to suppress the dominant
	// real-world FP class (committed test fixtures).
	PathHint string `json:"path_hint,omitempty"`

	SurroundingHash string `json:"surrounding_hash,omitempty"`
	PageURLHash     string `json:"page_url_hash,omitempty"`
}

SourceContext describes WHERE a scan request originates so the scorer can bias verdicts by destination (e.g. the same AKIA-shaped value pasted to chat.openai.com is exfil; the same value in a *_test.go on github.com is a fixture).

Every field is optional. The browser extension fills what it can per interceptor (see extension/src/content/scan-client.ts); the agent uses what it gets. A zero-value SourceContext behaves identically to the plain Scan path.

Privacy invariant: SurroundingHash and PageURLHash are SHA-256 truncated to 64 bits (16 hex chars). No raw URL, no surrounding text, and no user identifier ever crosses this boundary.

func (SourceContext) IsZero 

func (s SourceContext) IsZero() bool

IsZero reports whether source carries no information. Equivalent to `source == SourceContext{}` but reads better at call sites.

type ThresholdEngine 

type ThresholdEngine struct {
	// contains filtered or unexported fields
}

ThresholdEngine holds the current Thresholds value and protects it with a read-mostly mutex so /api/dlp/config writes are safe under concurrent /api/dlp/scan reads.

func NewThresholdEngine 

func NewThresholdEngine(t Thresholds) *ThresholdEngine

NewThresholdEngine returns an engine seeded with t.

func (*ThresholdEngine) Get 

func (e *ThresholdEngine) Get() Thresholds

Get returns a copy of the current thresholds.

func (*ThresholdEngine) Set 

func (e *ThresholdEngine) Set(t Thresholds)

Set replaces the current thresholds atomically.

func (*ThresholdEngine) ShouldBlock 

func (e *ThresholdEngine) ShouldBlock(score int, severity string) bool

ShouldBlock returns true when score meets or exceeds the threshold for the given severity. Unknown severities fall back to the "low" threshold so unknown values err on the side of allowing — blocking only on a high score keeps surprise blocks rare.

type Thresholds 

type Thresholds struct {
	Critical int
	High     int
	Medium   int
	Low      int
}

Thresholds maps each severity to the minimum score that triggers a block. Values mirror the dlp_config SQLite singleton.

func DefaultThresholds 

func DefaultThresholds() Thresholds

DefaultThresholds mirrors the defaults seeded into dlp_config.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.