Documentation
¶
Overview ¶
Package analyzer is the detection framework for alcatraz: the recognizer contract, regex pattern recognizers, the registry and the engine that runs them. It is pure Go and dependency-free.
The framework is deliberately separate from the concrete recognizers (which live in the recognizers package) so callers can build a custom engine with only the recognizers they want, or add their own via the Recognizer interface. Most callers should use the top-level alcatraz package, which wires the framework together with the full default recognizer set.
Index ¶
- Constants
- type ArtifactRecognizer
- type BatchNlpEngine
- type ContextValidator
- type Engine
- func (e *Engine) Analyze(text string, o Options) []Result
- func (e *Engine) AnalyzeBatch(texts []string, o Options) [][]Result
- func (e *Engine) Languages() []string
- func (e *Engine) SetNlpEngine(n NlpEngine)
- func (e *Engine) SetThreshold(t float64)
- func (e *Engine) SupportedEntities(language string) []string
- type Match
- type Matcher
- type NerSpan
- type NlpArtifacts
- type NlpEngine
- type Options
- type Pattern
- type PatternRecognizer
- func (pr *PatternRecognizer) Analyze(text string, entities []string) []Result
- func (pr *PatternRecognizer) Context() []string
- func (pr *PatternRecognizer) Name() string
- func (pr *PatternRecognizer) SupportedEntities() []string
- func (pr *PatternRecognizer) SupportedLanguage() string
- func (pr *PatternRecognizer) WithContext(words ...string) *PatternRecognizer
- func (pr *PatternRecognizer) WithContextValidator(v ContextValidator) *PatternRecognizer
- func (pr *PatternRecognizer) WithValidator(v Validator) *PatternRecognizer
- type Recognizer
- type Registry
- type Result
- type Token
- type Validator
Constants ¶
const ( MinScore = 0.0 MaxScore = 1.0 )
Score bounds for a detection. A score is a confidence in [MinScore, MaxScore].
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type ArtifactRecognizer ¶
type ArtifactRecognizer interface {
Recognizer
// AnalyzeWithArtifacts is Analyze with the shared artifacts of the
// current Analyze call. The entities parameter has the same filter
// semantics as Recognizer.Analyze. Returned offsets are byte indices.
AnalyzeWithArtifacts(text string, entities []string, artifacts *NlpArtifacts) []Result
}
ArtifactRecognizer is an optional extension of Recognizer for detectors that consume precomputed NlpArtifacts. The engine detects it by type assertion: when an NlpEngine is configured (Engine.SetNlpEngine) and at least one applicable recognizer implements this interface, the engine runs the NLP pipeline once and calls AnalyzeWithArtifacts instead of Analyze.
type BatchNlpEngine ¶
type BatchNlpEngine interface {
NlpEngine
// ProcessTexts runs the NLP pipeline over all texts and returns one
// NlpArtifacts per text, in input order.
ProcessTexts(texts []string, language string) ([]*NlpArtifacts, error)
}
BatchNlpEngine is an optional extension of NlpEngine for backends that can process several texts in one inference call, which amortizes per-call tokenization and graph overhead. Engine.AnalyzeBatch detects it by type assertion and falls back to per-text ProcessText calls when absent.
type ContextValidator ¶
ContextValidator inspects a match together with its surroundings: the full text and the match's byte span [start, end). It is a filter — returning true keeps the match at its current score, returning false drops it — and is the pure-Go way to express lookahead/lookbehind ("keep this only if preceded or followed by X"). A nil context validator is a no-op.
type Engine ¶
type Engine struct {
// contains filtered or unexported fields
}
Engine runs a registry of recognizers over text and reconciles their results. It is safe for concurrent use: Analyze does not mutate engine state.
func NewEngine ¶
NewEngine builds an engine over the given registry. languages records the engine's configured languages for reference; analysis language is chosen per call via Options.
func (*Engine) Analyze ¶
Analyze detects entities in text. The pipeline: run every applicable recognizer, de-duplicate overlapping same-type spans, apply the score threshold, then the allow list. Matched substrings are filled into each result's Text field.
func (*Engine) AnalyzeBatch ¶
AnalyzeBatch is Analyze over several texts at once, returning one result slice per text in input order. When the configured NLP backend implements BatchNlpEngine, the model runs a single inference call for the whole batch instead of once per text — the per-text results are identical to Analyze, only faster. Recognizers, threshold and allow list apply per text exactly as in Analyze.
func (*Engine) SetNlpEngine ¶
SetNlpEngine attaches an NLP backend. When set, Analyze runs it at most once per call and only when an applicable recognizer implements ArtifactRecognizer and shares the resulting NlpArtifacts with every such recognizer. Without it, ArtifactRecognizers fall back to their plain Analyze method. Call during setup, before the engine is used concurrently.
func (*Engine) SetThreshold ¶
SetThreshold sets the default score threshold applied when Options.Threshold is nil.
func (*Engine) SupportedEntities ¶
SupportedEntities returns the entity types detectable for a language.
type Match ¶
type Match struct {
Groups [][2]int
}
Match is a single regex match expressed in byte offsets. Groups[0] is the whole match; Groups[i] is the i-th capture group. A group that did not participate in the match has both offsets set to -1.
type Matcher ¶
type Matcher interface {
// FindAll returns every non-overlapping match, left to right.
FindAll(text string) []Match
// String returns the source pattern, for diagnostics.
String() string
}
Matcher finds all non-overlapping matches of a compiled pattern in text, reporting byte-offset spans. The default implementation (stdMatcher) wraps the standard library's RE2 engine, which is linear-time and dependency-free but does not support lookaround or backreferences.
Matcher is the extension point for alternative engines: supply one to NewPatternMatcher to back a Pattern with, for example, a backtracking engine that supports lookahead/lookbehind. Implementations MUST report byte offsets (not rune/code-point indices) so spans line up with the analyzed string.
type NerSpan ¶
type NerSpan struct {
// EntityType is the canonical entity name, e.g. "PERSON".
EntityType string
// Start and End are byte offsets into the analyzed text: [Start, End).
Start int
End int
// Score is the detection confidence in [MinScore, MaxScore].
Score float64
}
NerSpan is one model-detected entity span. EntityType is a canonical entities.* name (label mapping is the NlpEngine implementation's job) and Score is on the engine's [MinScore, MaxScore] scale.
type NlpArtifacts ¶
type NlpArtifacts struct {
// Tokens holds the tokenization of the analyzed text, when the engine
// provides one. It may be nil for engines that only do NER.
Tokens []Token
// Ents holds the model-detected entity spans.
Ents []NerSpan
}
NlpArtifacts is the output of one NLP pass over a text: tokens and model-detected entity spans. It is computed at most once per Analyze call and handed to every ArtifactRecognizer.
type NlpEngine ¶
type NlpEngine interface {
// ProcessText runs the NLP pipeline over text. The language is the
// engine-level analysis language (ISO 639-1).
ProcessText(text, language string) (*NlpArtifacts, error)
}
NlpEngine produces NlpArtifacts for a text. Implementations wrap a model runtime and live outside the core so the core stays dependency-free.
type Options ¶
type Options struct {
// Entities, when non-nil, restricts detection to these entity types.
Entities []string
// Language selects the recognizer set; defaults to "en" when empty.
Language string
// Threshold, when non-nil, drops results scoring below it.
Threshold *float64
// AllowList suppresses results whose matched text is allow-listed.
AllowList []string
// AllowListRegex treats AllowList entries as regular expressions (joined
// with "|") instead of exact strings.
AllowListRegex bool
}
Options tunes a single Analyze call. The zero value is valid: it analyzes English text with every recognizer and no score threshold.
type Pattern ¶
type Pattern struct {
Name string
Regex string
Score float64
// Group selects which capture group is reported as the entity span.
// 0 (the whole match) is the default.
Group int
// contains filtered or unexported fields
}
Pattern is a named, compiled regular expression with a base confidence score.
By default a match's whole span (group 0) is reported as the entity. Set Group (via WithGroup) to report a capture group instead — the idiomatic RE2 way to emulate lookbehind/lookahead: match the surrounding context but emit only the captured entity.
func MustPattern ¶
MustPattern is like NewPattern but panics on an invalid regex. It is meant for package-level recognizer definitions where the patterns are constants known to compile.
func NewPattern ¶
NewPattern compiles a pattern with the standard library RE2 engine, returning an error if the regex is invalid.
func NewPatternMatcher ¶
NewPatternMatcher builds a pattern from an arbitrary Matcher. This is the extension point for engines that support features the RE2 default lacks — most notably lookahead/lookbehind (see the alcatraz/lookaround module). The Regex field is populated from m.String() for diagnostics.
type PatternRecognizer ¶
type PatternRecognizer struct {
// contains filtered or unexported fields
}
PatternRecognizer detects a single entity type using one or more regex patterns, with optional structural and context validators. It is the workhorse behind almost every built-in recognizer.
func NewPatternRecognizer ¶
func NewPatternRecognizer(name, entity, language string, patterns []*Pattern) *PatternRecognizer
NewPatternRecognizer creates a recognizer for a single entity type.
func (*PatternRecognizer) Analyze ¶
func (pr *PatternRecognizer) Analyze(text string, entities []string) []Result
Analyze implements Recognizer. Every pattern is run over the text; the configured capture group of each match is taken as the entity span, scored at the pattern's base score, then promoted to MaxScore or dropped by the structural validator, and finally filtered by the context validator. (A promotes/drops; the context validator only filters.) Overlapping same-entity matches are de-duplicated before returning.
func (*PatternRecognizer) Context ¶
func (pr *PatternRecognizer) Context() []string
Context returns the recognizer's context words.
func (*PatternRecognizer) Name ¶
func (pr *PatternRecognizer) Name() string
Name implements Recognizer.
func (*PatternRecognizer) SupportedEntities ¶
func (pr *PatternRecognizer) SupportedEntities() []string
SupportedEntities implements Recognizer.
func (*PatternRecognizer) SupportedLanguage ¶
func (pr *PatternRecognizer) SupportedLanguage() string
SupportedLanguage implements Recognizer.
func (*PatternRecognizer) WithContext ¶
func (pr *PatternRecognizer) WithContext(words ...string) *PatternRecognizer
WithContext attaches context words that hint at the entity nearby. They are retained for future context-aware scoring (which requires an NLP backend) and are inert in the pattern-only engine. Returns the recognizer for chaining.
func (*PatternRecognizer) WithContextValidator ¶
func (pr *PatternRecognizer) WithContextValidator(v ContextValidator) *PatternRecognizer
WithContextValidator attaches a context-aware filter that sees the full text and the match span. Use it for lookaround-style rules ("only if preceded by 'SSN:'"). Returns the recognizer for chaining.
func (*PatternRecognizer) WithValidator ¶
func (pr *PatternRecognizer) WithValidator(v Validator) *PatternRecognizer
WithValidator attaches a structural validator (e.g. a checksum). Returns the recognizer for chaining.
type Recognizer ¶
type Recognizer interface {
// Name returns a stable identifier for the recognizer.
Name() string
// SupportedEntities lists the entity types this recognizer can emit.
SupportedEntities() []string
// SupportedLanguage returns the ISO 639-1 language code the recognizer
// is registered under.
SupportedLanguage() string
// Analyze scans text and returns detections. When entities is non-nil it
// is a filter: the recognizer should only run if it supports at least one
// of the requested types. Returned offsets are byte indices into text.
Analyze(text string, entities []string) []Result
}
Recognizer detects entities of one or more types in text. Pattern-based recognizers are provided by PatternRecognizer; the interface is exported so callers can plug in custom logic (including future ML/NER backends) and add it to a Registry or pass it to the engine.
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry holds recognizers keyed by language.
func NewRegistry ¶
NewRegistry creates a registry for the given languages.
func (*Registry) Add ¶
func (r *Registry) Add(language string, rec Recognizer)
Add registers a recognizer under the given language. The language is the key analysis is performed against; it is supplied explicitly (rather than taken from the recognizer) because structured-identifier recognizers are language-independent and are typically registered under every analyzed language.
func (*Registry) Recognizers ¶
func (r *Registry) Recognizers(language string, entities []string) []Recognizer
Recognizers returns the recognizers for a language. When entities is non-nil only recognizers that support at least one requested type are returned.
func (*Registry) SupportedEntities ¶
SupportedEntities returns the sorted, de-duplicated set of entity types a language's recognizers can emit.
type Result ¶
type Result struct {
// EntityType is the canonical entity name, e.g. "EMAIL_ADDRESS".
EntityType string
// Start and End are byte offsets into the analyzed text: [Start, End).
Start int
End int
// Score is the detection confidence in [MinScore, MaxScore].
Score float64
// Text is the matched substring. It is populated by the engine after
// filtering; recognizers leave it empty.
Text string
// RecognizerName identifies which recognizer produced the result.
RecognizerName string
}
Result is a single detected entity: its type, byte-offset span in the analyzed text and a confidence score. Offsets are byte indices (Go's regexp engine reports bytes), so text[Start:End] yields the matched substring.
func RemoveDuplicates ¶
RemoveDuplicates collapses overlapping detections of the same entity type, keeping the highest-scoring span and dropping any span contained within a kept one. Zero-score results are discarded. Detections of different entity types never suppress each other. The result is sorted by score (descending), then start offset (ascending), then length (descending).
func (Result) ContainedIn ¶
ContainedIn reports whether r's span is fully inside other's span.
func (Result) EqualIndices ¶
EqualIndices reports whether r and other cover exactly the same span.
func (Result) Intersects ¶
Intersects returns the number of overlapping bytes between r and other, or 0 when they do not overlap.
type Token ¶
type Token struct {
// Text is the token's surface form.
Text string
// Start and End are byte offsets into the analyzed text: [Start, End).
Start int
End int
}
Token is a single token of the analyzed text with its byte span. text[Start:End] yields the token.
type Validator ¶
Validator inspects a matched substring and decides whether it is a true positive. It is used for entities with a verifiable structure (checksums, range rules). Returning true promotes the match to MaxScore; returning false drops it. A nil validator leaves the pattern's base score untouched.
Source Files