README
¶
scan
A Go library for matching many regular expressions against blocks of bytes. It compiles patterns into shared literal filters and regex automata, with Hyperscan and VectorScan as engine references and gohs as the API reference. There is no cgo, native library, or third-party module dependency.
The library is experimental. The API and compiled database format may change.
Installation
Requires Go 1.26 or later.
go get github.com/git-pkgs/scan
Usage
Compile a trusted pattern set once, then reuse the database and scratch space across inputs. Compilation has no CPU or memory budget; applications accepting untrusted rule configurations must impose their own limits.
package main
import (
"fmt"
"log"
"github.com/git-pkgs/scan"
)
func main() {
db, err := scan.Compile(
&scan.Pattern{Expression: `token=[a-z0-9]{8}`, ID: 1, Flags: scan.SingleMatch | scan.SomLeftMost},
&scan.Pattern{Expression: `code=[0-9]{4}`, ID: 2, Flags: scan.SingleMatch | scan.SomLeftMost},
)
if err != nil {
log.Fatal(err)
}
scratch := scan.NewScratch(db)
err = db.Scan([]byte("token=abc12345 code=1234"), scratch, func(match scan.Match) error {
fmt.Printf("%d %d:%d\n", match.ID, match.From, match.To)
return nil
})
if err != nil {
log.Fatal(err)
}
}
This prints 1 0:14 and 2 15:24. IDs come from the patterns; offsets are byte offsets, with an exclusive end.
Use Match when only a boolean result is needed:
matched, err := db.Match(data, scratch)
Databases can be shared between goroutines. Each concurrent scan needs its own scratch space; scratch.Clone() creates an independent scratch for another worker. Passing nil instead of scratch allocates it internally; reusing scratch retains its buffers between calls and grows them when a larger database needs more space.
Embed compiled patterns
Marshal serializes the compiled tables. A build-time generator can write them to a file:
compiled, err := db.Marshal()
if err != nil {
log.Fatal(err)
}
if err := os.WriteFile("patterns.scan", compiled, 0o644); err != nil {
log.Fatal(err)
}
Embed that file in the application and load it once at startup:
import (
_ "embed"
"github.com/git-pkgs/scan"
)
//go:embed patterns.scan
var compiledPatterns []byte
func loadPatterns() (*scan.Database, error) {
return scan.UnmarshalDatabase(compiledPatterns)
}
Generate the file before running go build. Loading restores the compiled tables without parsing or recompiling expressions. Artifacts are versioned, so regenerate them when upgrading to an incompatible format. Only load artifacts produced by a trusted compiler; the checksum detects corruption but does not authenticate the contents.
Matching behaviour
Patterns use Go's regexp/syntax parser. Look-around and backreferences are unsupported. Matching is byte-oriented by default; UTF8 enables rune-oriented matching, while reported offsets remain byte offsets.
Scan reports matching end offsets in ascending order, including overlapping matches. It reports one event per pattern and end offset, without capture groups. SomLeftMost includes the earliest matching start in From; without it, From is zero. SingleMatch limits each pattern to one event per scan.
Caseless, DotAll, and MultiLine control case folding, whether dot matches newlines, and line anchors. Expressions that can match empty input require AllowEmpty. The Prefilter flag is reserved and currently does not relax matching semantics.
Scan collects and sorts events before invoking the handler. Returning an error stops further callbacks and returns that error, but does not avoid the matching work already done. Match avoids event collection and sorting and stops residual matching after its first match.
Only block scanning is supported. Each call treats its input as a separate block; matches do not span calls. There are no streaming or vectored scan APIs, and this package is not a drop-in replacement for gohs.
Performance
A paired run on an Apple M1 Pro scanned a 38.7 MB repository corpus with 416 patterns in 44.9 to 45.0 ms with scan, versus 47.4 to 47.5 ms with VectorScan. Each backend had three 750 ms benchmark runs on one CPU. These are warmed block-scanner timings, excluding compilation, database loading, and Git I/O. Scan allocated zero bytes per warmed pass.
The corpus includes a 34 MB binary, where scan performs well. Individual source and config inputs still take roughly two to three times the native scanning time. The aggregate result does not imply a general speed advantage or end-to-end parity when scanning Git history.
Testing
CGO_ENABLED=0 go test ./...
go test ./internal/engine -run '^$' -bench . -benchmem
go test ./internal/engine -run '^$' -fuzz '^FuzzSecretsScan$' -fuzztime=30s -parallel=1
Corpus tests, benchmarks, and the input fuzz target use the adjacent ../secrets checkout, or the directory set by SECRETS_ROOT. They skip when the corpus is unavailable. Other tests run without that checkout.
The public API and examples are at the module root. Scanner implementation and its tests are in internal/engine. The separate bench module contains the native VectorScan comparisons and the cmd/e2e CLI comparison tool; its cgo dependency is excluded from the library module.
License
MIT. fdr.go and teddy_diagnostic_test.go contain code derived from Intel's Hyperscan under BSD-3-Clause. Their copyright and license notices are retained in those files.
Documentation
¶
Overview ¶
Package scan compiles many regular expressions into one block scanner.
Index ¶
Examples ¶
Constants ¶
const ( Caseless = engine.Caseless DotAll = engine.DotAll MultiLine = engine.MultiLine SingleMatch = engine.SingleMatch AllowEmpty = engine.AllowEmpty UTF8 = engine.UTF8 Prefilter = engine.Prefilter SomLeftMost = engine.SomLeftMost )
Variables ¶
var ( ErrInvalid = engine.ErrInvalid ErrScanTerminated = engine.ErrScanTerminated ErrScratchInUse = engine.ErrScratchInUse )
Functions ¶
This section is empty.
Types ¶
type CompileStats ¶
type CompileStats = engine.CompileStats
CompileStats describes the routing decisions made while compiling a database.
type Database ¶
Database is an immutable block-mode multi-pattern database.
func Compile ¶
Compile builds one block database from patterns. Patterns must be trusted; compilation does not enforce resource budgets.
func UnmarshalDatabase ¶
UnmarshalDatabase loads compiled tables produced by a trusted compiler.
func (*Database) Marshal ¶
Marshal encodes the compiled database for storage or go:embed.
Example ¶
package main
import (
"fmt"
"log"
"github.com/git-pkgs/scan"
)
func main() {
db, err := scan.Compile(&scan.Pattern{Expression: `token=[a-z0-9]{8}`})
if err != nil {
log.Fatal(err)
}
compiled, err := db.Marshal()
if err != nil {
log.Fatal(err)
}
loaded, err := scan.UnmarshalDatabase(compiled)
if err != nil {
log.Fatal(err)
}
matched, err := loaded.Match([]byte("token=abc12345"), scan.NewScratch(loaded))
if err != nil {
log.Fatal(err)
}
fmt.Println(matched)
}
Output: true
func (*Database) Match ¶
Match reports whether any expression matches data.
Example ¶
package main
import (
"fmt"
"log"
"github.com/git-pkgs/scan"
)
func main() {
db, err := scan.Compile(&scan.Pattern{Expression: `token=[a-z0-9]{8}`})
if err != nil {
log.Fatal(err)
}
scratch := scan.NewScratch(db)
matched, err := db.Match([]byte("token=abc12345"), scratch)
if err != nil {
log.Fatal(err)
}
fmt.Println(matched)
}
Output: true
func (*Database) Scan ¶
func (db *Database) Scan(data []byte, scratch *Scratch, handler MatchHandler) error
Scan searches data and reports matching end offsets in ascending order. Returning an error stops callbacks, after matching work has completed.
Example ¶
package main
import (
"fmt"
"log"
"github.com/git-pkgs/scan"
)
func main() {
db, err := scan.Compile(
&scan.Pattern{Expression: `token=[a-z0-9]{8}`, ID: 1, Flags: scan.SingleMatch | scan.SomLeftMost},
&scan.Pattern{Expression: `code=[0-9]{4}`, ID: 2, Flags: scan.SingleMatch | scan.SomLeftMost},
)
if err != nil {
log.Fatal(err)
}
scratch := scan.NewScratch(db)
err = db.Scan([]byte("token=abc12345 code=1234"), scratch, func(match scan.Match) error {
fmt.Printf("%d %d:%d\n", match.ID, match.From, match.To)
return nil
})
if err != nil {
log.Fatal(err)
}
}
Output: 1 0:14 2 15:24
func (*Database) Stats ¶
func (db *Database) Stats() CompileStats
Stats returns counts useful when tuning a pattern set.
type MatchHandler ¶
type MatchHandler = engine.MatchHandler
MatchHandler receives matches in engine order. Returning an error stops callbacks.
type Pattern ¶
Pattern associates an expression and compile flags with an application ID.
func NewPattern ¶
NewPattern returns a pattern with the supplied expression and flags.
type Scratch ¶
Scratch contains mutable scan state. Concurrent scans need separate scratches.
func (*Scratch) Clone ¶
Clone returns independent scan state with the same capacity.
Example ¶
package main
import (
"fmt"
"log"
"github.com/git-pkgs/scan"
)
func main() {
db, err := scan.Compile(&scan.Pattern{Expression: `token=[a-z0-9]{8}`})
if err != nil {
log.Fatal(err)
}
scratch := scan.NewScratch(db)
inputs := [][]byte{[]byte("token=abc12345"), []byte("no match")}
results := make(chan bool, len(inputs))
for _, input := range inputs {
workerScratch := scratch.Clone()
go func() {
matched, err := db.Match(input, workerScratch)
if err != nil {
log.Fatal(err)
}
results <- matched
}()
}
matchedInputs := 0
for range inputs {
if <-results {
matchedInputs++
}
}
fmt.Println(matchedInputs)
}
Output: 1
Source Files
Directories