hgmLibre2

README

hgmLibre2

A self-contained native RE2 regular-expression library for Go. It vendors RE2's C++ source and exposes it through cgo, so it needs no abseil, no CMake, and downloads nothing at build time.

The listed methods use the same names and signatures as the standard library regexp package (see Supported API), so the two are easy to read interchangeably. Both the string and the []byte families are provided, and they share one matching core — passing a []byte costs no copy (see Byte-slice methods). It is not a drop-in replacement for *regexp.Regexp, and not meant to be: the io.Reader variants, SubexpIndex, LiteralPrefix, Longest, marshal/unmarshal, etc. are not provided, and some semantics differ from stdlib on purpose — most notably ReplaceAllString substitutes a literal repl (no $1 / ${name} / $$ expansion) — plus a few edge cases; see Differences from stdlib. Which of the two engines to use for a given call site is answered — with numbers — in doc/与标准库regexp怎么选.md; the short version is in Why.

Why

Use this library by default. Reach for the standard library regexp only in the three cases listed below — they are all recognizable by inspection, so no experiment is needed to tell them apart.

Go's regexp is RE2-derived in syntax and in its linear-time guarantee, but its matcher is a from-scratch NFA simulation (onepass / bitstate backtracking / one-pass NFA). It gets a fast path only when a literal prefix can be extracted; when it cannot, it restarts at every position. This library runs the real native RE2 lazy DFA: one linear pass, essentially independent of input shape. That single difference is where the numbers come from — it is not "C beats Go".

On identical patterns and identical corpora the native engine did not lose a single throughput measurement (compile cost and the per-call floor are separate, and are the exceptions below): 1.1× at worst, and 11–85× wherever the standard library cannot extract a literal prefix — a leading (?i), \b, character class or alternation is enough to lose it. Matching also happens entirely on the C side, so the steady state is 0 B/op on the Go heap, which keeps the GC heap goal (and therefore the process peak) from moving at all.

Full numbers, method, corpora and every exception: doc/与标准库regexp怎么选.md ("choosing between this library and stdlib regexp", Chinese). Every number in it is reproduced by go test -run TestStdlibCompare -v . (stdlib_compare_test.go), so re-run it after changing the library or moving to another machine.

Prefer the standard library regexp when:

• The pattern is compiled at run time and not cached. Compiling costs 1.2–4.6× more here (native RE2 does more work up front), and a freshly compiled pattern used once cannot earn that back: measured 9.2 µs (stdlib) vs 21.6 µs here for "compile (?i)\b<word>\b, then match one 90-byte sentence". Hoist the pattern to a package-level variable if you can — or, if it is a whole keyword table, compile it into one RegexpSet instead, which puts you back on this library.
• The call sits at the cgo bridge-cost floor. One crossing costs ~67 ns here versus ~2 ns for a stdlib call (Ryzen 5900X · linux/amd64 · go1.26.5). That only decides the outcome when the match itself is cheaper than the crossing — i.e. a few bytes of input and a pattern simple enough for onepass. Note that "short input" alone is not the criterion: on a 161-byte string with six backtracking-shaped patterns this library is 24× faster and allocation-free.
• You are compiling somebody else's pattern — user- or config-supplied — and the accepted syntax must match stdlib byte for byte. The two engines disagree at the edges (\C, nesting-depth limit, a few escapes; see Differences). That is a semantic contract, not a performance question.

One more thing that is not about speed: sharing a single *Regexp across goroutines does not scale linearly here (every search takes a read lock on the DFA state cache), so on tiny inputs under heavy concurrency the two engines come out even. On body-sized input the lock is irrelevant and this library is still ~100× ahead at 1000 goroutines. See Concurrency.

• the pattern can match the empty string and you FindAll over a whole document — e.g. (?m)^[ \t]*$ or (?m)^\s*(?://.*)?$. An empty-matchable pattern succeeds on every line, so FindAll is forced to emit one match per line and pay the advance-and-restart path for each; on 115 KiB of line-shaped text that is 0.23x (10.4 ms vs 2.4 ms). This is a pattern shape problem, not an engine problem: making the same intent non-empty-matchable (* → +) flips it to 8.45x in this library's favour, and is faster under the standard library too. Try rewriting the pattern before you settle for the standard library here. Numbers: go test -run TestEmptyWidthMultiline -v .

Beyond speed, this library also avoids the costs of the usual ways to get native RE2 into Go:

• No wazero / WASM runtime. Wrappers like go-re2 run RE2 inside a wazero WebAssembly runtime, which probes stdio handles at startup. In environments with no standard handles (e.g. a Windows SCM service) that probing can fail. hgmLibre2 links RE2 natively, so there is no runtime to instantiate.
• No abseil / CMake. The vendored RE2 is the last pre-abseil release (tag 2023-03-01), which is plain self-contained C++11. cgo compiles the .cc files directly; there is no separate build system to drive.
• Single static binary, cross-compilable. Because it is just C++11 + cgo, it cross-compiles with zig as the C/C++ toolchain.

The one hard requirement is cgo: it must stay enabled, and a C++11 compiler (clang, gcc or zig c++) must be available. A pure-Go / CGO_ENABLED=0 build cannot use this library at all.

Install

go get github.com/hgmGoLib/hgmLibre2

Requires Go 1.19+. cgo must be enabled (the default) and a C++11 compiler must be available. Any of clang, gcc, or zig c++ works.

Usage

package main

import (
	"fmt"

	"github.com/hgmGoLib/hgmLibre2"
)

func main() {
	re := hgmLibre2.MustCompile(`(?P<key>\w+)=(?P<num>\d+)`)

	fmt.Println(re.MatchString("a=1"))                 // true
	fmt.Println(re.FindStringSubmatch("port=8080"))    // [port=8080 port 8080]
	fmt.Println(re.ReplaceAllString("x=1 y=2", "*"))   // * *  (repl is literal)
}

Supported API

The listed methods share their names and signatures with regexp. Matching is leftmost-first, the same as regexp.Compile (RE2's default Perl mode), not leftmost-longest — e.g. (a|aa) on "aa" yields "a", just like stdlib. UTF-8 input.

• Compile, MustCompile, QuoteMeta
• CompileMaxMem / MaxMem — not in stdlib; the RE2 memory budget for one pattern (Compile uses RE2's 8 MB default); see Scanning backwards for when it matters
• String, NumSubexp, SubexpNames
• MatchString
• FindString, FindStringIndex, FindStringSubmatch, FindStringSubmatchIndex
• FindAllString, FindAllStringIndex, FindAllStringSubmatch, FindAllStringSubmatchIndex
• ReplaceAllString (repl is literal — no $1 / ${name} / $$ expansion, unlike stdlib), ReplaceAllStringFunc
• Split
• The []byte counterparts of all of the above: Match, Find, FindIndex, FindSubmatch, FindSubmatchIndex, FindAll, FindAllIndex, FindAllSubmatch, FindAllSubmatchIndex, ReplaceAll, ReplaceAllFunc — see Byte-slice methods below
• FindReplaceWithin / FindReplaceWithinBytes / AppendFindReplaceWithin — not in stdlib; replace inside every match of another pattern in one C-side pass; see FindReplaceWithin and AppendFindReplaceWithin below
• RegexpSet (NewRegexpSet, NewRegexpSetMaxMem, GetPatternLen, Match, MatchAny, MatchBytes, MatchAnyBytes) — not in stdlib; one DFA answering "which of these N patterns hit" in a single scan; MatchAny drops the indices and returns at the first hit position instead of scanning to the end; see RegexpSet below
• RegexpSet.MatchStats / MatchStatsBytes (ScanStats) and RegexpSet.MemInfo (SetMemInfo) — not in stdlib; per-scan and per-Set DFA counters (flushes, states built, budget left); see Measuring a Set below
• RegexpSet.Attrib (AttribInfo, PatternCost) — not in stdlib; a diagnostic build answers which patterns are building all those DFA states; see Attribution
• RegexpSet.FindAllIndex / FindAllIndexBytes (NewFindAllIndexAlloc, RegexpSet_FindAllIndex_Alloc_t, RegexpSet_FindAllIndex_Run_t) — not in stdlib; one scan reporting where each pattern of the set can end, handed back in batches; see FindAllIndex below
• RegexpSet.NewMatchScanner (MatchScanner, SetMatch, Scan, SetModes, MatchScanMode_t, Hit, HitIDs, Stats, MatchScanStats_t, Close) — not in stdlib; the finished form of the above: one pass giving the hit table and de-duplicated match spans, replacing Match + one FindAllStringIndex per hit pattern; see Where a set matched below
• RegexpSetReverse.NewMatchScanner (MatchScannerReverse, same method set) — not in stdlib; the same thing from the other end: one backwards pass giving de-duplicated spans under rightmost-longest, for tables whose patterns are cheap in reverse and explosive forwards; see Where a reverse set matched below
• RegexpSet.ResolveSpan / ResolveSpanWithin / ResolveSpanBytes (the same three on RegexpSetReverse, in the opposite direction) — not in stdlib; complete one end point into a whole span with a single anchored question whose cost does not depend on input length; see ResolveSpan below
• PatternLenRange / RegexpSet.PatternLenRange / PatLenUnbounded — not in stdlib; the byte-length range a pattern can match; see PatternLenRange below
• RegexpSet.ViableOneStats — not in stdlib; how much the lazily built per-pattern reverse sets (used to recover left edges) cost in states and bytes
• CompileLongest / CompileLongestMaxMem / MustCompileLongest — not in stdlib as constructors (the stdlib spelling is the re.Longest() mutator); compile one pattern under leftmost-longest (POSIX) semantics instead of the default leftmost-first. Both pick the same start; they differ only on the end. Needed whenever you want "the longest match starting here" in a single search instead of two, and required if you feed the span into a checksum — a greedy short end truncates the hit
• Regexp.FindStringIndexAtWithin (and the same method on FindStringIndex_ctx_t) — not in stdlib; a search anchored at from and not crossing bound, with offsets in the original string so \b / ^ / $ still see the real neighbours. Paired with a longest-mode object this is ResolveSpan for a single pattern — but over RE2::Match, so it inherits the NFA fallback that the set path does not have
• RegexpReverse.ResolveSpanWithin — not in stdlib; the single-pattern twin of RegexpSetReverse.ResolveSpanWithin: given a match end, the leftmost start, bounded. Implemented as the very call RE2::Match makes internally to find a match's left edge (reverse program + kAnchored + kLongestMatch)
• RegexpReverse.MemInfo — not in stdlib; the DFA cache high-water mark of that pattern's reverse program, Built=false if it has never been walked (querying never builds it)
• DFAStats / DFAStatsZero (DFAStats_t) — not in stdlib; process-wide counters for DFA state-cache flushes; the per-Set counters above are usually what you want instead; see DFA cache thrashing below
• FindStringIndex_ctx_t (NewFindStringIndex_ctx, FindStringIndex, FindIndex) — not in stdlib; a scratch-reusing FindStringIndex that is steady-state allocation-free
• AppendAllStringIndexFlat — not in stdlib; FindAllStringIndex without the intermediate [][]int; see AppendAllStringIndexFlat below
• ReplaceAllStringFunc_ctx_t (NewReplaceAllStringFunc_ctx, AppendReplaceAllStringFunc) — not in stdlib; ReplaceAllStringFunc appending into a caller-owned []byte, steady-state allocation-free, reporting changed rather than matched; see AppendReplaceAllStringFunc below
• RegexpReverse (CompileReverse, CompileReverseMaxMem, MustCompileReverse, MatchString, Match, MatchStats) — not in stdlib; a separate object that gives the same yes/no answer as a forward Regexp, with the DFA walking the original buffer back to front; see Scanning backwards below
• RegexpSetReverse (NewRegexpSetReverseMaxMem, GetPatternLen, Match, MatchBytes, MatchAny, MatchAnyBytes, MatchStats, MemInfo, Attrib, plus the FindAllIndex and ResolveSpan families above) — not in stdlib; the reverse-compiled twin of RegexpSet, and a separate type: it used to be a *RegexpSet carrying a Reverse() bool, which made two opposite meanings share one method name; see Scanning backwards below
• FreeC — not in stdlib; see Resource management

Byte-slice methods

The []byte methods are a second facade over the same matching core, not a separate implementation. Matching only ever needs a byte pointer plus a length on the C side, so a []byte is handed to RE2 directly: no string(b) copy at any input size, and results (Find, FindSubmatch, FindAll, …) are sub-slices of your input sharing its backing array, exactly like stdlib's bytes family. Two consequences worth knowing:

• The input must not be mutated while a call is in flight (same rule as stdlib).
• On the no-change path, ReplaceAll / ReplaceAllFunc / FindReplaceWithinBytes return the original src slice with zero allocation (matching this library's lazy-materialization style) rather than a fresh copy as stdlib does — so do not write to the returned slice. Copy it if you need an independently writable buffer. Everything else, including the nil-vs-empty result conventions, matches stdlib and is pinned by differential tests in bytes_test.go.

Naming: where stdlib has the method, the stdlib name is used (FindIndex ↔ FindStringIndex); this library's own methods take a Bytes suffix (FindReplaceWithinBytes, RegexpSet.MatchBytes).

RegexpSet

RegexpSet compiles N patterns into one DFA and answers "which of these N hit" in a single unanchored scan, instead of (?:re1)|(?:re2)|… as a coarse pre-filter followed by a second per-pattern pass. Match returns the hit indices (into the patterns slice you passed); it deliberately does not return positions — callers needing an offset run FindStringIndex on the few hit patterns afterwards.

set, err := hgmLibre2.NewRegexpSet(patterns)         // RE2 default budget: 8 MB
set, err := hgmLibre2.NewRegexpSetMaxMem(patterns, 32<<20) // explicit budget

var buf []int32                       // reuse across calls → zero alloc
for _, text := range corpus {
    for _, idx := range set.Match(text, buf) {
        ...                           // idx indexes `patterns`
    }
}

Sizing maxMem

maxMem is RE2's RE2::Options::max_mem. Prog::CompileSet splits it once, so the single knob raises two different ceilings:

• Compile time — an instruction-count ceiling for the whole set ((maxMem-sizeof(Prog))/4/sizeof(Inst), capped at 2²⁴). Too many patterns, or patterns that are individually heavy (case-folded classes, {n,m} repeats), overrun it and RE2::Set::Compile() fails. That is the only thing that makes NewRegexpSet/NewRegexpSetMaxMem return an out-of-memory error, and the error text carries the pattern count and the budget in force.
• Run time — whatever is left funds the DFA's state cache. Overrunning this is not an error: the DFA flushes its cache and keeps going with the same result. It is, however, a cliff and not a slope — see DFA cache thrashing — and it is invisible unless you count the flushes (set.MemInfo().FlushesTotal, or DFAStats process-wide; see Measuring a Set). See also Why Match has no error return.

So: if you got "set compile failed (out of memory)", do not split the table into two sets — two sets means two scans over every input, which costs far more than the memory. Double maxMem until it compiles (8 → 16 → 32 MB). It is a build-time, once-per-process cost, and the set is read-only and concurrency-safe afterwards. Capacity scales roughly linearly with the budget: on one real-world 509-pattern table, 1 MB held 132 of them, 2 MB held 204, and 4 MB held all 509.

⚠️ "It compiles" is not the same as "the budget is big enough": that only clears the compile-time ceiling. Whether the run-time half is big enough is a separate question with a separate answer, and the only way to know is to count flushes — see below. And once a set is genuinely thrashing, splitting it does become the better trade (two warm scans beat one thrashing scan by a wide margin), so the "do not split" advice above applies to the compile-time ceiling only.

DFA cache thrashing

When the DFA's state cache cannot hold the state set the current input is walking, RE2 does not evict LRU-style: DFA::ResetCache throws the whole cache away and rebuilds from scratch. The answer stays correct, so nothing in the API changes — which is exactly why this can quietly cost you an order of magnitude for a long time.

Two properties make it worth counting rather than reasoning about:

• It is a cliff, not a slope. A working set 1% over budget does not cost 1%. Every flush is followed by rebuilding thousands of states, each one an NFA closure computation. Synthetic kwN[\s\S]{0,8}tgtN set, 60 patterns, 16 distinct 64 KB bodies (the helpers in dfastats_test.go), on one box:

  maxMem	flushes	throughput
  1 MB	79	8.1 MB/s
  8 MB	3	18.6 MB/s
  64 MB	0	164.7 MB/s

  Three flushes over a megabyte of input are worth a 9× slowdown.

• A single-shape benchmark cannot see it. Scanning the same body N times warms the cache on the first pass and never allocates a state again, so every budget looks identical. Production scans a different body per request, and each new shape pushes the state set outward. Benchmark with a rotation of distinct bodies, and look at flush counts, not just at throughput.

DFAStats() returns a snapshot of process-wide counters; DFAStatsZero() zeroes them for segmented measurement:

hgmLibre2.DFAStatsZero()
for _, body := range distinctBodies {   // single-threaded, one warm-up pass first
    set.Match(body, buf)
}
st := hgmLibre2.DFAStats()
// st.Resets == 0  ⇒  this budget holds this pattern table on this corpus.
// st.Resets  > 0  ⇒  double maxMem, rebuild the set, measure again.
// st.LastStateBudget / st.LastCacheStates report the budget and the state count
// in force at the most recent flush — the working set's lower bound.

SearchFailures counts the other failure mode, where the DFA gives up on a search entirely; a lone Regexp falls back to the NFA (correct, an order of magnitude slower), while RE2::Set never gets there (see below).

The counters are process-wide: RE2's hook carries no context, and set matching has none at all, so a delta only attributes to a specific scan if you measure single-threaded. Under concurrency, read them as a rate. For anything finer, use the per-Set / per-scan counters below.

The flush is not the whole cliff

Counting flushes is how you find out whether the budget is big enough, but once it is big enough the cost does not disappear — it moves. Measured per call on one 223-pattern table over 32 distinct 80 KB bodies, splitting the calls into "this call hit a flush" and "this call did not":

• A call that hits a flush is only 1.3–2.8× slower than one that does not. The flush itself is not an order of magnitude.
• The order of magnitude is in building states. At a budget with zero flushes: first pass over brand-new text, 23.7 ms per body; second pass over the same bodies (not a single new state), 0.36 ms per body. That is 66×, with no flush anywhere in sight.

So the cache only ever pays off on repeated text. Novel text keeps asking for new states and does not converge: scanning 1657 distinct real records, the new-states-per-KB rate fell from 84.7 to 16.2 and then stopped falling.

Practical consequence: sizing maxMem to zero flushes buys you "do not fall off the cliff" — it is necessary, and it is cheap. It does not buy the 66×. If every request carries fresh text, the first-pass number is your ceiling, and getting under it means making the table build fewer states (see What drives state explosion) or scanning less text (a cheap literal pre-filter in front of the big set).

Measuring a Set

DFAStats is process-wide. Two finer counters attribute to one RegexpSet and to one call, with no globals and no thread_local — so they work under concurrency and in a process holding many sets.

// Per scan: pass a stack-allocated ScanStats. Passing nil (i.e. using Match)
// costs nothing — the C side does not count at all.
var st hgmLibre2.ScanStats
hits := set.MatchStats(text, buf, &st)
// st.Flushes     whole-table cache wipes during THIS call (>0 = part of this
//                call ran two orders of magnitude slow, and it took a write
//                lock that stalled every other goroutine scanning this set)
// st.Grows       cache growth events — these lose NO states; not thrashing
// st.StatesBuilt states created during this call (→ 0 in steady state)
// st.Bytes       input size; Bytes/StatesBuilt = bytes of text per new state
// st.StatesEnd, st.StateBudget, st.MemLeft   water level at the end

// Per set: cumulative, read-only, never builds a DFA just to answer.
mi := set.MemInfo()
// mi.FlushesTotal  == 0 after a run over distinct bodies ⇒ this budget fits
// mi.StatesBuiltTotal  the direct "how expensive is this table" number
// mi.States, mi.Used(), mi.StateBudget, mi.ArenaCap

mi.ArenaCap is the memory actually obtained from the system, as opposed to StateBudget, which is only a ceiling. It is normally far below the budget: raising maxMem does not, by itself, cost resident memory.

Caveat on StatesBuilt: it is the delta of a counter on the shared DFA, so a concurrent scan of the same set is counted in too. Attribute single-threaded.

What drives state explosion

Measured by rewriting one real 223-pattern table and rescanning the same corpus at a budget with zero flushes, so the only variable is the pattern source:

• Tolerance windows dominate, roughly linearly. Narrowing every {0,N} in the table to {0,2} took 51844 states down to 4366 (11.9×), about +700 states per unit of window width. An automaton has no cheap way to remember "how far in am I", so X{0,W}Y is expanded W times.
• Shape splits the table cleanly. The 84 patterns containing a {0,N} gap accounted for 76.8% of the states; the other 139 accounted for 3.4%.
• Combining patterns is super-additive, but only mildly. Two halves scanned separately built 32573 states; merged into one set, 51844 — 1.59×, not a product. Pattern count is the second-order term; window width is the first.
• Input entropy is not a driver. Random full-byte noise never enters a window and builds almost nothing. Minified JavaScript is the cheapest corpus measured: 87 KB of jquery.min.js built 947 states where 82 KB of real text built 3906. Per-offset attribution shows why — the four hottest 1 KB segments of that file are the only natural English in it (the licence comment, an exception string, a MIME-type string, un-minified property names).
• Character-class-driven branches are the trap. A branch like (?-i:\([A-Z]{2,5}\)) has no literal anchor, so its cost tracks whatever alphabet the corpus happens to use — it was the one pattern that cost more on minified JS than on real text. The same missing literal anchor is what makes such a pattern unfilterable. Give every branch a real literal.
• Narrowing windows does not make a state cheaper. State "width" (how many NFA threads are live at once) is the price of building one state. Narrowing the whole table to {0,2} cut the state count 11.9× while average width moved 37.3 → 35.7 (4%). Window width controls how many states exist, not what one costs.

Attribution: which patterns build the states

Not compiled in by default. Rebuild with the macro and the Attrib accessor starts returning data (without it, Enabled is false and everything is zero — there are no fields and no branches in the default build):

CGO_CXXFLAGS="-O2 -DRE2_DFA_ATTRIB=1" go build ./...

a := set.Attrib()
for _, p := range a.Pats[:20] {   // already sorted, most expensive first
    fmt.Println(p.Index, p.Excess)   // p.Index indexes your patterns slice
}
// a.NInstHist / a.NInstMax  distribution of state width = per-state build cost
// a.BirthHist[64]           states built per 1/64 of the input; flat = the
//                           whole text is building states, spiky = go read
//                           those offsets and you will see what triggers it

Sort by Excess (= Insts - States), which is what Pats is already sorted by. Do not sort by States: under an unanchored search the DFA must consider a match starting at every position, so every pattern's entry instruction sits in nearly every state and States saturates at 100% for most of the table.

The ranking has been validated by ablation: dropping the top 20 by Excess took one table from 51844 states to 9606 (5.4×), where dropping 20 patterns at random only reached 36810 (1.4×); top 39 gave 11.5× against 1.03× for 39 random. It is also stable across corpora — three corpora of completely different shape agreed on 10–11 of the top 12 — so one calibration run on any real corpus is enough. Absolute state counts are not portable that way; they move 4–8× with the corpus.

Ablation is a diagnostic, not a deployment: removing patterns changes what you detect. Use the ranking to decide which few patterns to rewrite, or which few to isolate into their own set — and if you do split, split by that ranking or by shape, never round-robin by index. Splitting barely reduces the total state count (51844 → 44573 at k=8); what it reduces is the largest shard, which is what the budget has to cover. A round-robin k=8 split measured 3× slower than not splitting at all, because the input is scanned eight times.

Why Match has no error return

RE2::Set::Match reports DFA failure by returning false — indistinguishable from "nothing matched" unless you ask for its ErrorInfo. That would be a silent-miss hazard for a detector, so it is worth being precise about when it can actually happen: once Compile() has succeeded, a DFA out-of-memory on the match path is not reachable, which is why Match and friends return no error.

Two facts in the vendored RE2 make that hold:

• The DFA's "cache thrashing, bail out to the NFA" branch is explicitly disabled for set matching — re2_dfa.cc guards it with kind_ != Prog::kManyMatch ("RE2::Set cannot fall back, so we just have to keep on keeping on"). A set scan therefore flushes and rebuilds its state cache indefinitely rather than failing. The remaining failure paths need a single state not to fit in a freshly flushed cache, and the DFA constructor already refuses to initialize unless the budget holds at least 20 states.
• That constructor check is exercised at compile time, not first use: Compiler::CompileSet finishes by running a "hello, world" DFA search (Prog::kManyMatch) and returns NULL if it fails, precisely because a set has no NFA to fall back on. A too-small budget therefore fails Compile(), which you get as an error from the constructor.

Verified empirically as well as by reading the source: the 509-pattern corpus above, compiled at its minimum viable budget (3.71 MB, i.e. the least runtime cache RE2 will accept for it), scanned against adversarial inputs (random full-byte, random ASCII, near-miss fragments, byte-cycle, CJK, single- character runs; 256 KB–8 MB each, with and without needles appended). One 8 MB scan alone forced 5 679 state-cache flushes. Across every run: zero kOutOfMemory from ErrorInfo, zero hits from RE2's own hooks::DFASearchFailure, and the hit set matched a per-pattern MatchString oracle exactly, every time. The same DFA-failure hook does fire, as expected, for single-Regexp matching under a starved budget — and there it is harmless, because a lone RE2 falls back to the NFA and still returns the correct answer.

Where a set matched: MatchScanner

RegexpSet.Match answers which of the N patterns hit. Learning where they hit used to cost a second pass per hit pattern: Match once to narrow the table down, then FindAllStringIndex on a forward Regexp for each of the k patterns that matched — 1 + k passes over the whole input, and those k are the expensive unanchored kind (the .*? prefix makes every offset a candidate start, which is the state-explosion fuse described in What drives state explosion).

Those positions were already computed by the first pass and then thrown away: RE2's kManyMatch DFA notes every byte at which some pattern can end, and drops the note when the scan ends. This library keeps it. On a 6.4 MB body, Match alone costs 18.5 ms and Match plus collecting every end point costs 18.4 ms — within noise, so the positions are free.

MatchScanner is the finished form of that: one pass over the input, giving both the hit table and de-duplicated match spans, in batches, with a fixed memory footprint.

ms, unsupported, err := set.NewMatchScanner()  // reusable workspace: build once, keep it, Close it
defer ms.Close()
// unsupported: pattern indices that cannot produce spans at all (today: they match the
// empty string). Fixed at construction, independent of any input — settle them here.

ms.SetModes(modes)                // optional: what each pattern needs (default: spans, auto-tiered)

err = ms.Scan(body, func(batch []hgmLibre2.SetMatch) {
    for _, m := range batch {     // body[m.Lo:m.Hi] is a real match of pattern m.Index
        handle(m.Index, body[m.Lo:m.Hi])
    }
})                                // err != nil ⟹ the whole pass is void; redo it with FindAll
ids := ms.HitIDs()                // the same hit table Set.Match would have returned
// ms.Hit(i) is the O(1) form of the same answer
st := ms.Stats()                  // Walks / Cands / Tries / Emits for that pass
// st.Tries/st.Walks is the number to watch: 1.00 means the first candidate start was
// always the answer. Measured 1.00 on every production table. The first three counters
// cover variable-length patterns only (fixed-length ones never look back); Emits counts
// every span, so Tries/Emits is NOT "tries per look-back".

What it buys. Two measurements, on different tables and corpora — do not compare them across.

On a production-shaped set (90 patterns, 52 of them wanting spans, 7.03 MB), with all three modes measured in the same run:

mode	whole leg	vs. the old path
old path (Match, then FindAll over the whole body per hit pattern)	78.2 ms	1.00×
span (default, leftmost-longest)	43.8 ms	1.79×
spanFast (removed 2026-08-28)	24.6 ms	3.18×
gate only (everything boolOnly)	21.9 ms	3.57×

All three hand back the identical 10 956 spans here, because every pattern in that table is anchored at both ends.

On a real 155-pattern table over a 6.4 MB body (47 patterns hitting, steady state, 64 MB budget) the whole leg drops from 369.3 ms (Match + per-pattern FindAll) to 24.6 ms — 15.0× — and Go-heap allocation from 4.0 MB / 2252 objects to ~0 / 146. By input size on that corpus: ≤ 8 KB break-even (1.0×), 32 KB 3.1×, 512 KB 6.1×, 2 MB 14×. The worst case measured is 0.94× (6% slower): a synthetic string with a hit every 38 bytes, where nearly every byte is inside a match and the two cgo crossings per hit have nothing to amortise against. 🔴 Both sets of numbers predate the 2026-08-28 path change; they answer a different question — how much better is this layer than not having it (versus the old "gate Match plus one FindAll per hit pattern") — and that ratio only improved. Post-change numbers are in doc/补起点换路的实测账_20260828.txt.

Rules that matter, all pinned by tests (matchscan_test.go, spanscan_*_test.go):

• The batch slice is the internal buffer itself, overwritten in place by the next batch. Keep what you need by appending it elsewhere. This is why the API is batched and not "give me one array at the end": run counts scale with the input (~30 741 runs/MB on that table), so accumulating would make memory track document size. Scan itself keeps a fixed 12 KB of output buffer plus the 48 KB run buffer underneath, whatever the input length.
• Results are interleaved across patterns, ordered by when a span closes. Within one pattern they are ascending by Lo and non-overlapping. Grouping by pattern is one append on your side; doing it in the library would mean accumulating, which is exactly what the batching avoids.
• Passing nil as batchFn is legal — you then get only the hit table, i.e. Set.Match semantics, with no span work done at all.
• The workspace is not concurrency-safe: one MatchScanner per goroutine. The RegexpSet behind it is read-only and shared freely, so several scanners can run against one set concurrently.
• text is only referenced during Scan (the left edge is recovered by reading it); the scanner does not retain it afterwards.

SetModes: what each pattern needs

ms.SetModes(modes) declares, per pattern index, one of two things (indices beyond a short slice take the zero value; nil = all default):

mode	what it means
MatchScanMode_span (zero value)	give me spans. Leftmost-longest, unconditionally — no pattern shape escapes it.
MatchScanMode_boolOnly	I only need "did it hit". No span is ever closed, no endpoint recovered.

🔴 Until 2026-08-28 there was a third mode, MatchScanMode_spanFast, forcing a cheap cursor path that did not guarantee leftmost-longest and required the caller to fuzz out a per-pattern clearance first. It is gone. The path that replaced it is both strictly leftmost-longest and cheaper than spanFast was, so the trade it offered no longer exists — see How the three paths became one below.

boolOnly is the one that pays: patterns left out of span work still appear in the hit table, they just never pay for endpoint recovery, which is the expensive half. Many entries in a big table are pure booleans ("does this class of content appear at all") and nobody ever asks where they hit; on the table above, two such patterns alone accounted for 57% of all runs. This is static information — you know at build time which branches consult a position — so set it once, right after building the scanner.

It is one three-state table rather than two masks ("wants a span" × "which path") because wants-no-span-but-use-path-B is not a meaningful combination; two masks would eventually contradict each other.

SetModes returns an error if a pattern that can match the empty string is given anything other than boolOnly — every offset would be a zero-length hit and the cursor cannot advance past it. That is rejected up front rather than degrading silently at scan time.

What Scan guarantees

Long-form, in Chinese: doc/MatchScanner的leftmost-longest保证.md.

In the default mode (MatchScanMode_span) the spans handed to you satisfy:

1. text[Lo:Hi] is a real match of that pattern;
2. spans of one pattern are non-overlapping and ascending by Lo;
3. the semantics are leftmost-longest — i.e. re.Longest().FindAllStringIndex.

Three things are easy to misread there:

• (2) is per pattern. Two different patterns still overlap freely; that is not duplication, it is two questions each wanting an answer.
• (3) is not "same as FindAllStringIndex". The stdlib default is leftmost-first (greedy); the two disagree wherever the greedy first hit at a start is shorter than the longest one. Compare against Longest(), or you get a false red.
• Empty-capable patterns are rejected by SetModes, not silently degraded.

How the endpoint is recovered, per pattern:

pattern shape	mode	how
min == max (fixed length)	any	Lo = Hi - min, one subtraction, the regex engine is never entered. Both paths agree here, so the mode does not apply.
variable	span (default)	two steps. ① A reverse pass from the end e with every live state seeded, walking left until the machine dies, collecting all viable-prefix starts in [cursor, e) (RegexpSetReverse.ViableStarts, on a reverse set holding just that one pattern). ② Try those candidates in ascending order with an anchored longest forward search; the first one that verifies is the answer. Ascending ⟹ leftmost; longest-mode ⟹ longest end. Strictly leftmost-longest for any pattern shape, and no maxLen is needed — the look-back's lower bound is wherever the reverse machine died. Cost: the look-back windows are pairwise disjoint (bounded by one extra pass over the text) plus however many false candidates get verified — measured 1.00 tries per look-back on every production table.
—	—	if the single-pattern object will not compile, maxMem is too small and Scan fails the whole pass.

One rule governs all of it (2026-08-27): the pass over the text uses the set; every endpoint-completion call after it uses that pattern's own single-pattern object, never the set. Three reasons:

1. A single-pattern object goes through RE2::Match, which falls back DFA → OnePass/BitState/NFA. The set's anchored resolve is a kManyMatch DFA and nothing else — upstream included (re2_set.cc:216: dfa_failed ⟹ return false) — so "the DFA gave up" there can only fail the whole pass. After the change only the reverse look-back can still hit that, because RE2 itself has no other way to find a match's left edge.
2. Endpoint traffic no longer churns the big shared DFA cache of the whole table.
3. Smaller states: a single pattern does not carry kManyMatch's per-state id list.

The answers were byte-for-byte unchanged (TestMatchScanPathsSameAsSetRoute: 300 AST-generated corpora per path, 54 000 spans, zero differences), and dense-hit corpora got 27–36% cheaper. 🔴 That test was removed on 2026-08-28 along with the two paths it replayed; the same AST-generated corpus is now pinned in matchscan_astfuzz_test.go against stdlib's Longest(), which is a harder oracle than a replica of our own old code.

How the three paths became one (2026-08-28)

For the fixed-length tier the equality is provable, not merely measured: an end e has exactly one possible start e-min, so starts are monotone in ends and "greedy leftmost non-overlapping" is precisely what the cursor produces. It is cross-checked against FindAllStringIndex on 60 000 random fixed-length patterns (TestMatchScanStrictVsFindAll).

For the variable-length tier the pairing has to be rebuilt on the Go side, and how you rebuild it decides what semantics you get. This scan yields the set of end offsets, with no start/end pairing and no priority information in it — RE2's greediness lives in the NFA instruction priority order, which kManyMatch (the only mode that reports all ends) discards. Until 2026-08-28 three rebuilds coexisted:

	how	semantics	expensive because
path A (spanFast)	reverse machine seeded with accept only → one start, then an anchored longest end	a third kind — neither leftmost-first nor leftmost-longest	two calls per hit, look-back windows overlap
path B (old default)	one forward unanchored longest search from max(cursor, e-maxLen)	leftmost-longest	patterns with no length bound must walk the gaps (up to 2.00× the text)
path D2 (separate type MatchScanner2)	what the table above now describes	leftmost-longest	verifying false candidates (never happens on real tables)

Path A's defect is structural: seeding accept only sees the starts where a match ends exactly at e. \b(?:ab cd ef|cd)\b against "ab cd ef" — the smallest end the set reports is the one for "cd" (offset 5), so the look-back can only reach 3, while the real leftmost start is 0 (text[0:5)="ab cd" is not a match, but it is a viable prefix: append " ef" and it becomes one). Only seeding every live state sees that candidate, which is exactly what step ① above does.

The evidence for collapsing them — 11 corpora at the 100 MB scale (console build output · eight credential-dense generators · product source + manuals + endpoint ELF · a real local Claude history) × 9 production gate tables = 99 cells. Raw reports in doc/补起点换路的实测账_20260828.txt.

• Semantics. Compared span by span, after sorting into a canonical (pattern, Lo, Hi) order: D2 vs path B — zero differences across 161.9 million spans. D2 vs path A — 37 differences, all in the source/manual/ELF corpus, and every one of them is path A truncating the left edge: a UAE Emirates ID came out 1985-1234567-1 where the real answer is 784-1985-1234567-1; a prompt-injection marker came out <SYS> where the real answer is <<SYS>>. 🔴 That is precisely the failure mode this document keeps warning about — feed a shifted span to a check digit (mod-10 / Luhn / mod-97) and it fails, so the consumer discards a real hit and you get a silent miss. Switching paths did not just save time; it fixed 37 wrong boundaries on real text. 🔴 Do not compare in emission order: the guarantees only cover ordering within one pattern, so a stream comparison measures "different permutation", not "different spans" (it flagged all 85 000 spans of an 8 MB corpus as mismatched; sorted, zero differed).
• Cost. Summed over each gate chain, D2 was the fastest in all 11 — 0.48–0.91× path A, 0.6–1.0× path B — and Tries/Walks was 1.00 in all 99 cells.
• Memory. D2 needs a per-pattern reverse set (vp1); path A needed a per-pattern reverse object (rev1). Same order of magnitude: on the largest table (158 patterns, 89 of them actually asked for a position) 9.6 MB vs 7.6 MB. Against path B it is a net add, since B built no reverse object at all. Measure it with ViableOneStats().

Removed along with the two paths: MatchScanMode_spanFast, the guard rejecting it in MatchScannerReverse.SetModes, the MatchScanner2 type, RegexpSet.reverseOne / ReverseOneStats / rev1, and the tests that existed only to compare paths.

Anchoring used to remove path A's problem entirely — wrapping a variable pattern in word boundaries (\b(?:…)\b) pins the start, so "which start to pick" never arose: in a 120 000-span comparison the 60 differing spans of the bare patterns became 0. That is now a property of the patterns rather than a requirement on the caller: fixed-length spans were always safe to slice with, and variable-length spans are too, since they are strictly leftmost-longest.

"I could not give you everything": exactly two ways to say it

🔴 There is no "I bailed halfway through, these patterns are yours to finish" middle state (it was removed on 2026-08-27). Only two:

where	what it says
NewMatchScanner's unsupported	[]int32 — pattern indices that cannot produce spans at all. Today there is exactly one reason: the pattern matches the empty string (PatternLenRange's min <= 0), so every position is a zero-length hit the cursor cannot pin down.
Scan's err	this pass is void — the batches you already received do not count either. Redo the whole input with FindAll.

unsupported does not depend on the input: it is decided when the workspace is built and never changes, so you can pin it in a regression test (put an a* in the set and the index comes back). Configure those patterns as boolOnly — they still appear in the hit table — or run them the old way. Asking for spans on one makes SetModes fail immediately; if you never call SetModes they are handled as boolOnly, since you were already told at construction.

Scan's err has three causes, all of them "maxMem is too small": the underlying FindAllIndex pass failed; one of the two single-pattern objects needed to fill in the other end would not compile; the reverse look-back gave up.

🔴 "The reverse set would not compile" is deliberately not given a fallback. Measured byte-exactly on 2026-08-28: of 590 production patterns only 16 cost more in reverse than forward (all open-ended {n,} shapes, in one table), at a maximum ratio of 1.021×. For "the forward set compiles but the reverse single-pattern set does not" to actually happen, the set must hold essentially one pattern and maxMem must land inside a 2%-wide band above that pattern's own threshold — for the worst pattern found (a three-part JWT) that band is 74 bytes wide: forward 3580, reverse 3654. On a table with several patterns the forward set costs orders of magnitude more, so the window cannot exist. Reaching this branch only ever means the caller misconfigured maxMem — which should be reported, not silently papered over by quietly switching implementations. A fourth, "runs did not arrive monotonically in scan order", is a broken invariant inside this library — a bug, reported through the same err rather than swallowed.

Why not partial success? An error code the caller cannot construct has no business being in a return value. It forces this chain: the caller must write a fallback → the fallback never runs → what never runs cannot be tested → untested code is usually wrong → the day it finally matters, execution goes down a path that has never executed. That is worse than having no fallback and failing the whole pass.

And the anchored resolve giving up could not be provoked at all: it runs the small DFA (a single start position, not the whole-text scan). Three shapes (ab, [A-Za-z][A-Za-z0-9]{2,19}key, (?i)[a-z0-9]{3,20}@[a-z0-9.\-]{3,20}) were swept every 100 bytes across the 3 000-byte band just above the wall where they first compile (maxMem 2400 / 5800 / 24400), resolving every 3rd offset of a 60 KB text with CJK in it — zero give-ups. Below the wall, NewRegexpSetMaxMem fails cleanly instead. The program and the DFA are funded from the same maxMem: if the program fits, what is left covers the handful of states one resolve walks; if it does not, you never get a set. There is no window in between.

🔴 And no, it cannot "fall back to the NFA": on the set path there is no NFA. A single RE2 falls back (see the Fall back to NFA below chain in re2_re2.cc), but RE2::Set::Match is DFA-only upstream too (re2_set.cc:216 — dfa_failed just returns false). The NFA interface does not answer which pattern matched, and the kManyMatch ids come out of the id list inside a DFA state. Giving the set's anchored resolve an NFA path means compiling a separate \A(?:pat) RE2 per pattern — which is the whole thing ResolveSpan exists to avoid.

🔴 There is deliberately no "give me one array at the end" entry point (AppendAllMatches was removed on 2026-08-27). Such a convenience form always creeps from tests and measurement into production, and its dst is a ratchet buffer that tracks input length (~0.037 MB per MB of input on the table above) — exactly what the batched interface exists to avoid. Callers who want an array append one line inside their own Scan callback, where the cost is visible to whoever wrote it.

FindAllIndex: the raw end-point runs

MatchScanner is built on top of a lower layer, exported for callers who want to apply their own pairing or overlap policy:

alloc, _ := set.NewFindAllIndexAlloc()   // reusable workspace; not concurrency-safe
defer alloc.Close()

err := set.FindAllIndex(body, alloc, func(runs []hgmLibre2.RegexpSet_FindAllIndex_Run_t) {
    for _, r := range runs {
        // pattern r.ReIndex ends at every offset in r.Lo..r.Hi (both inclusive)
    }
})

• A run is {ReIndex, Lo, Hi}: pattern ReIndex has a match end point at every value in Lo..Hi, a closed interval in original-input byte offsets (Lo <= Hi always). Closed, not half-open, because these are collapsed points, not a span — Hi is itself a real end point.
• Which end is meant is fixed by the direction of the set, not by a field name: a forward RegexpSet reports match ends (exclusive, i.e. text[?:Hi] is a match), a RegexpSetReverse reports match starts (inclusive).
• Both ends of the run are reported because collapsing them would lose matches without any error: ab|c on "abc" ends at 2 and 3, which are consecutive, and keeping only 3 silently drops the [0,2) match.
• This is not FindAllStringIndex semantics. That returns a leftmost-first, non-overlapping sequence; this returns all end points of all patterns, overlaps included (abcd|bc on "abcd" reports both). Choosing between overlapping hits is policy, and the library does not decide it for you — use MatchScanner if you want finished spans.
• Ordering is not globally ascending (a run closes only when its pattern hits again non-consecutively, or at end of input, so patterns interleave), but it is ascending within one pattern, which is what the cursor above relies on.
• Offsets are int32: it is the native width (so the C side writes the buffer directly, with no per-run conversion), it is signed because end - minLen is legitimately negative near the start of the input, and RE2 caps input at 2 GiB anyway.
• alloc may be nil (one is created and thrown away per call, costing one native allocation); on a hot path keep one. It is bound to the set it was created from — using it with another set returns an error rather than a wrong answer — and it is not concurrency-safe. FindAllIndexBytes is the zero-copy []byte twin.
• Batch size is fixed at 4096 runs (48 KB) and is deliberately not a knob. The native side suspends like sqlite3_step — it saves DFA state by value, releases the DFA cache read lock, returns to Go, and resumes on the next call — so no lock is held while your callback runs.
• There is no "stop early" return from the callback. If you only need a yes/no answer, MatchAny stops at the first hit inside RE2, which is earlier than any Go side brake.

ResolveSpan: complete one end point into a span

end, ok, err := set.ResolveSpan(text, start, id)              // forward: start (incl) -> end (excl)
lo, ok2, err2 := rev.ResolveSpan(text, end, id)               // reverse: end (excl) -> start (incl)
pos, ok3, err3 := set.ResolveSpanWithin(text, from, bound, id) // bound = how far to look, <0 = no limit

This is a single anchored question at one offset, not a scan: the cost is how far that one match can extend, and is independent of input length — asking it on a 1 KB input and on a 6.4 MB input costs the same. It returns the longest match at that end point, not the first one found, because stopping at the first gives the shortest span, which truncates the hit and makes downstream fixed-length or checksum validation reject a genuine match. ok == false means the pattern does not match at that end point at all (wrong offset or wrong id). It is read-only and may be called concurrently with scans on other goroutines. ResolveSpanBytes is the []byte twin.

Use it to complete an end point; do not run a reverse FindAllIndex over the whole input to do the same job (a whole-table reverse scan of that 6.4 MB body takes 65 s versus 18 ms forward, and one-pattern-at-a-time reverse scans put back exactly the 1 + k passes this API removes). Equally, do not rebuild it on the Go side: an unanchored re.FindStringIndex(text[from:]) keeps the .*? prefix and scans to the end of the input, while a hand-written \A(?:pat) means a second Regexp object with its own DFA cache and a semantic equivalence you have to maintain by hand. ResolveSpan uses the set's own program and DFA cache, through its anchored entry point.

ResolveSpanWithin's bound limits how far the resolution may look (right limit going forward, left limit going backward). It is what makes patterns that can extend without limit ((?s).*KEY) constant-cost instead of O(input). Matching context is always the whole input, so \b, ^ and $ still see the real neighbouring bytes — a bound can only make the answer shorter, never wrong.

PatternLenRange

min, max := hgmLibre2.PatternLenRange(`[A-Z]\d{3}`)   // 4, 4
min, max = set.PatternLenRange(i)                     // same, from the table built with the set
// max == hgmLibre2.PatLenUnbounded (-1) means "no upper bound"

The byte-length range a pattern can match, computed once at set-build time (155 patterns in under 1 ms) with Go's regexp/syntax — the same grammar, and no change to the vendored RE2. Three tiers drive everything above: min == max means the start is a subtraction; a finite max means a bounded look-back; PatLenUnbounded means there is no window to look back over, and the pattern falls back to the caller. Patterns that RE2 accepts but Go's parser rejects return (0, PatLenUnbounded) — conservative in the safe direction, i.e. it can only push a pattern onto the fallback path, never produce a wrong start.

Where a reverse set matched: MatchScannerReverse

The mirror of MatchScanner: one pass from the end of the input, handing back de-duplicated, non-overlapping spans in batches. The API is identical — open it on a *RegexpSetReverse instead of a *RegexpSet:

rs, _ := hgmLibre2.NewRegexpSetReverseMaxMem(patterns, hgmLibre2.DefaultSetMaxMem)
ms, unsupported, _ := rs.NewMatchScanner()   // reusable workspace; not concurrency-safe
defer ms.Close()

err := ms.Scan(body, func(batch []hgmLibre2.SetMatch) {
    for _, m := range batch { _ = body[m.Lo:m.Hi] }   // a real match of pattern m.Index
})

Two things differ from the forward scanner, and only two:

1. spans come back in descending Lo order (forward: ascending);
2. the de-overlap rule is rightmost-longest (forward: leftmost-longest).

Both directions guarantee the same three things: every span is a real match, spans of the same pattern never overlap, and no region containing a match is silently skipped. The two rules differ only where two real matches actually overlap — everywhere else they agree span for span:

input	pattern	leftmost-longest	rightmost-longest
abab	a|ab	[0,2) [2,4)	[2,4) [0,2) — same set, reversed order
aab	ab|b	[1,3) = "ab"	[2,3) = "b" — genuinely different

If you need to match re.Longest().FindAllStringIndex byte for byte, use the forward scanner. If you just need everything in the text framed (masking, locating, counting), either rule does.

When to reach for it. When the table contains patterns that explode forwards and collapse in reverse — the S B{m,n} L shape of Scanning backwards. Before this layer existed, such a table could only be scanned backwards as a gate: Match said which patterns hit, and getting positions meant a second forward pass over the whole body — which is exactly the 1 + k passes MatchScanner exists to collapse.

Reverse is the easier direction, not the harder one. A forward DFA pass reports match ends, so the start has to be guessed back on the Go side — that guess is what the two-step recovery described above is about. A reverse pass reports match starts, and leftmost/rightmost-longest is defined on starts. So there is no guess here:

• reverse FindAllIndex → match starts, monotone in scan order (right to left);
• forward single-pattern FindStringIndexAtWithin(from: start, bound: cursor) → the longest end that does not cross the cursor.

Hence no candidate-collection step at all, and no maxLen window. Each span costs exactly one anchored search — proportional to the length of that match, not to the length of the input. (That call used to go through a one-pattern set; since 2026-08-27 it uses the pattern's own single-pattern longest-mode object, for the same three reasons listed under the forward scanner.) It also picks up the family the forward default tier cannot take: patterns with no upper length bound (email and friends), which forward needs maxL to bound a look-back window for.

Why scanning right-to-left still keeps nothing. Only because the rule flipped with it. Insisting on leftmost-longest while scanning backwards is what would force buffering: the span in your hand can still be swallowed by one further left that you have not reached yet — bounded patterns could ride a maxL-wide delay buffer, but unbounded ones would have to accumulate until the scan ends, and memory tracking input length is precisely what this layer exists to avoid. Under rightmost-longest the problem does not arise: going right to left, the first start you see is final, because nothing further left can outrank it — the same sentence as the forward scanner's, in a mirror. So the cursor still advances inside the callback and output still goes into the same fixed 12 KB buffer.

How it is pinned. matchscan_reverse_test.go generates its corpus from each pattern's own regexp/syntax AST (random bytes never produce real matches — that is a vacuously green test) and checks against an exhaustive rightmost-longest oracle written against stdlib alone. The first five patterns in that list are exactly the counterexamples that broke the forward spanFast path removed in 2026-08-28 (abc|b, a|ab, x{1,3}[a-c]?(?:ab|cd)?, (?:ab)?[bc]{1,2}, (?:ab)*b{1,3}); reverse diverges on none of them, because there is no guess to get wrong. The oracle carries its own self-check: ab|b against "aab" must produce different answers under the two rules, or the whole comparison is vacuous.

Scanning backwards

S B{m,n} L — a counted repeat whose start class is strictly narrower than its repeat class, ending in a literal — is the shape that blows the state count up. [A-Za-z][A-Za-z0-9]{2,19}key is the canonical one: every letter can open a candidate match, every following alphanumeric keeps it alive, so a DFA state has to remember which of the last 20 offsets are still live. That set is an arbitrary subset, so the state count is exponential in the bound.

No rewrite fixes this. (a|b)*a(a|b)^k needs 2^k states in any DFA, minimal included, so it is a property of the language and not of RE2. But the reverse language, (a|b)^k a(a|b)*, needs k+2. Direction is the lever.

Forward and reverse are two objects, not two methods on one — a pattern's two directions are two programs with two DFA caches, and a pattern normally only ever runs in one of them:

rev, _ := hgmLibre2.CompileReverse(`[A-Za-z][A-Za-z0-9]{2,19}key`)
hit := rev.MatchString(text)          // same answer a forward Regexp would give

revSet, _ := hgmLibre2.NewRegexpSetReverseMaxMem(patterns, hgmLibre2.DefaultSetMaxMem)
idx := revSet.Match(text, buf)       // a *RegexpSetReverse; same hit set as a forward set

The reverse program is built by RE2's own compiler (concatenations reversed, ^/$ swapped, \b unchanged, multi-byte UTF-8 sequences re-encoded back-to-front) and the DFA then walks your buffer from the end. Nothing is copied and nothing is reversed by the caller.

Measured on 120 distinct 8 KB bodies, one pattern per set so the memory is attributable:

	states	state cache	hit set
forward	35 149	5.35 MB	16
reverse	45	0.01 MB	16

What it costs you (single RegexpReverse). A RegexpReverse answers "is there a match", not "where". There is no Find on it: a reverse scan meets the last match in the input first, so a reverse Find could only ever be rightmost — a different semantics from the forward leftmost-first one. Use reverse as the cheap gate and run FindStringIndex on a forward Regexp for the few inputs that hit.

On the set side that objection is answered rather than avoided: rightmost is a perfectly good de-overlap rule as long as it is the declared one, so MatchScannerReverse gives spans under rightmost-longest and says so.

RegexpSetReverse does report positions, in its own direction: FindAllIndex gives match starts (inclusive) where the forward set gives ends, and ResolveSpan turns a known end into the matching start. That second one is what a reverse set is really for. 🔴 Scanning a whole table backwards is a different proposition from scanning one pattern backwards: state counts inside a set multiply, so a 155-pattern table that scans a 6.4 MB body in 18 ms with zero flushes forward takes 65 s in reverse, with the arena pinned at its 254 MB ceiling and still flushing. Measure MemInfo().FlushesTotal before pointing a reverse set at whole documents; to recover the left edge of a hit you already found, use ResolveSpanWithin, whose cost is independent of input length. MatchScanner does exactly this internally — a lazily built one-pattern reverse set per pattern that ever needs a left edge, never used to scan text (see ViableOneStats for what those cost: 32 patterns, 973 states, 2.0 MB on that table).

Direction is a per-pattern decision, not a global switch. The mirror shape loses by the same mechanism it wins by: on a corpus containing no key, (?s).{20}key costs 21 states forward and 1 reverse, while key(?s).{20} costs 1 forward and 21 reverse. Measure both directions on real input — build a one-pattern set each way and compare MemInfo().States — then put each pattern in whichever set matches its cheap direction. Two scans over the input still beat one scan that is thrashing.

This is not "reverse the pattern text yourself". Writing the pattern backwards and reversing the input gets the same answer but not the same cost. RE2's Simplify expands x{2,19} with the mandatory copies first and the optional nest after; reversing the concatenation moves that optional nest to the front of the read order, and the live-start sets then nest inside one another instead of forming arbitrary subsets. Same language, same bytes, different automaton: 17 states through this API versus 25 247 for the hand-rolled version (TestReverseIsNotHandRolledTextReversal). Hand-rolled reversal also splits multi-byte UTF-8 and needs a second copy of the input.

Why a separate type. A reverse scan runs a second Prog (Regexp::CompileToReverseProg), and a DFA state cache belongs to its Prog (Prog::dfa_first_ / dfa_longest_) — so the two directions are two programs and two caches no matter how the Go API is shaped. RegexpReverse makes that visible: one object, one direction, one cache, and the caller can see from the type which direction the pattern is running. Want both directions for the same pattern? Compile both objects.

Budgets and the fallback. The reverse program is compiled lazily on the first scan, with the budget from CompileReverseMaxMem (CompileReverse gives it RE2's 8 MB default). If the reverse DFA gives up mid-scan — RE2 bails out of a Prog search that is building states faster than it consumes input — the object silently falls back to one forward match of its own. The answer is always correct; MatchStats reports FellBack so you can tell that a scan did not get the saving. RegexpSet never bails (RE2 only flushes for kManyMatch), so a reverse set has no fallback path.

The other lever is memory. CompileMaxMem raises the budget for a single pattern the way NewRegexpSetMaxMem does for a set — same knob, same two ceilings (Sizing maxMem). On the pattern above, 60 distinct 16 KB bodies flush the cache 6 times at the 8 MB default and 0 times at 256 MB. Reverse scanning gets to 0 flushes at the default budget, with a peak of 9 states. Raising the budget buys throughput with RAM; scanning backwards buys it with nothing, when the shape allows.

AppendAllStringIndexFlat

re.AppendAllStringIndexFlat(dst, s, n) returns the same matches as re.FindAllStringIndex(s, n), appended to a caller-owned []int as [s0, e0, s1, e1, …] instead of being wrapped in a fresh [][]int.

FindAllStringIndex makes two throwaway allocations per call that both scale with the match count: the flat []int the C side is copied into (2*nmatch ints per match — all groups, even though only group 0 is returned), and the [][]int shell (one 24-byte slice header per match). On a large body with a high hit count that dominates: 190k matches ≈ 40 bytes each ≈ 7.6 MB for a single call, all of it garbage as soon as the caller has walked the matches once. This variant fills group 0 only (nmatch=1, which also shrinks the C-side vector<StringPiece>), and appends into your buffer, so passing buf[:0] makes repeat calls steady-state allocation-free.

var locs []int                          // reuse across calls
for _, text := range corpus {
    locs = re.AppendAllStringIndexFlat(locs[:0], text, -1)
    for i := 0; i+1 < len(locs); i += 2 {
        start, end := locs[i], locs[i+1]
        ...
    }
}

The match set, its order, and the empty-match handling are identical to FindAllStringIndex — both go through the same C loop — and are pinned against it and against stdlib in find_all_flat_test.go. Use FindAllStringSubmatchIndex when you need capture groups.

AppendReplaceAllStringFunc

ctx.AppendReplaceAllStringFunc(dst, re, src, f) produces exactly what re.ReplaceAllStringFunc(src, f) produces, but appends it to a caller-owned []byte and keeps the match-position table on the ctx, so both buffers are reused across calls.

It returns (dst, changed). changed means "the result differs from src", not "the regexp matched" — it is defined to be exactly re.ReplaceAllStringFunc(src, f) != src. Two cases report false, and both leave dst byte-for-byte as you passed it in:

1. nothing matched — fast return, no bytes written;
2. something matched but every f handed the original text straight back, so the result is byte-identical to src — the appended bytes are rolled back.

Case 2 is not a nicety. Replacements written against this API are usually decoders or de-obfuscators whose f carries its own validity check and returns m unchanged when it fails: an HTML numeric entity &#…; whose code point is out of range, a hex run of odd length or one that decodes to non-printable bytes. Those match but change nothing, and a caller that treats matched as changed ends up with a spurious extra copy of the original — one more buffer to hold, one more pass to scan, one more duplicate to reconcile. The check costs essentially nothing: a length mismatch decides it outright, and only an exactly-equal length pays for a memcmp.

The rollback restores the length and the contents, but not the capacity: dst may come back pointing at a larger backing array (the len(src) bytes just reserved). That is a win for the next call — just always use the returned slice rather than the one you passed in.

var ctx hgmLibre2.ReplaceAllStringFunc_ctx_t   // zero value is usable; not goroutine-safe
var buf []byte                                 // reuse across calls
for _, text := range corpus {
    out, changed := ctx.AppendReplaceAllStringFunc(buf[:0], re, text, decode)
    buf = out         // keep the (possibly grown) buffer either way
    if !changed {
        use(text)     // decoding changed nothing, nothing allocated
        continue
    }
    use(string(buf))  // or keep working on the bytes
}

Why it exists: ReplaceAllStringFunc pays two throwaway allocations per call that both scale with the body — the match table (2*(numSubexp+1) ints per match, of which the concatenation loop reads only group 0), and the result buffer. The result buffer used to be a bare strings.Builder growing from nothing: for a large byte slice Go grows by 1.25×, so the cumulative allocation converges to 1/(1-1/1.25) = 5×len(src), with 4× of that also paid in memcpy. Measured on a 64 MB body in a hex-decoding leg: 329 MB of Builder growth, 4.9 bytes allocated per input byte. ReplaceAllStringFunc now sizes that buffer to len(src) in one shot (the same thing stdlib's replaceAll and this library's own ReplaceAllFunc byte facade already did), which is a 1× buffer and no regrow copies. This variant goes one step further and reuses the buffer you already own, which is what a hot loop calling it per segment actually wants.

ReplaceAllStringFunc itself is now a thin shell over this method, so the two share the match set, the order, the call sites of f and the concatenation (both read group 0 out of the same C loop) — and the same lazy materialization as ReplaceAllString: a call that changes no bytes hands the original src back with zero allocation. That, the append contract, the rollback contract, the no-bleed-on-reuse contract and the steady-state-zero-allocation claim are pinned in replace_func_ctx_test.go.

FindReplaceWithin

find.FindReplaceWithin(strip, src, repl) is exactly equivalent to the two-regex idiom

find.ReplaceAllStringFunc(src, func(m string) string {
    return strip.ReplaceAllString(m, repl)
})

— locate each match of find, then run strip→repl within that matched segment — but the whole outer loop and every inner replacement run in one cgo call, instead of one cgo crossing per match plus one per separator. The algorithm is byte-for-byte identical: find can stay zero-capture so it still uses RE2's fastest no-submatch DFA, and strip still only edits inside the located segment.

It is lazy / zero-alloc on the no-change path: the C++ side does not build or copy a result string until the first replacement that actually changes bytes. If src is unchanged (no match, or matched but strip removed nothing), it returns src verbatim with no allocation. Only a genuinely-modified input pays for one result buffer.

One syntax note: here repl is an RE2 rewrite string. (RE2::GlobalReplace is RE2's own built-in replace-all; its rewrite string is RE2's native substitution syntax: \1..\9 expand to the corresponding capture group, \0 to the whole match, \\ is a literal backslash, everything else is literal.) So this differs from both stdlib's $1 / ${name} and this library's literal ReplaceAllString repl — three different conventions. For the common literal repl (e.g. "", which has no \), all three coincide.

Motivating use case: undoing separator obfuscation — find = a separator-tolerant keyword skeleton (i[\s._-]{0,2}g…), strip = the separator class, repl = "", so i.g-n_o r.e is normalized back to ignore. On the common path (ordinary text, nothing obfuscated) it is allocation-free and matches the plain DFA scan throughput; on input full of split keywords it is ~2× faster than the nested-ReplaceAllStringFunc form, with allocations collapsed from O(matches) to one.

AppendFindReplaceWithin

find.AppendFindReplaceWithin(dst, strip, src, repl) ([]byte, bool) is the append-into-your-own-buffer twin, for callers that consume the result once and throw it away (build a decoded view → scan it with a RegexpSet → drop it). Same C kernel, same changed predicate; the only difference is where the result lands: FindReplaceWithin mints a fresh Go string on every changed call (one C.GoStringN copy of the whole result), while this one memcpy's it into the dst you pass — so a reused buffer makes the steady state zero Go-heap allocation.

out, changed := find.AppendFindReplaceWithin(buf[:0], strip, src, "")
// changed ⟺ find.FindReplaceWithin(strip, src, "") != src
// changed ⟹ string(out) == find.FindReplaceWithin(strip, src, "")
if changed {
    buf = out          // always keep the returned slice: it may have re-based
    scanSet.Match(bytesStrView(out), hits)
}

changed=false leaves dst untouched down to its length — the caller should use the original src. The returned bytes are a view into the caller's buffer: appending to that buffer again (or reslicing it to [:0]) invalidates them.

There is no _ctx_t for this one: the outer match loop and the inner replacement both live in C++, so the result itself is the only Go-side allocation that scales with the input — and that one is now the caller's.

The test suite (hgmLibre2_test.go) cross-checks every method against the standard library regexp on a shared corpus of patterns and inputs; results are identical on that corpus (the corpus uses only literal ReplaceAllString repls, see the API difference below). TestReplaceAllStringIsLiteral pins the literal-repl behavior, and review_verify_test.go pins the engine-level differences below as differential tests.

bytes_test.go does the same for the []byte family over the same corpus — each method against both its stdlib counterpart and its own string twin — plus a hand-computed hit/miss pair per method (pinning nil vs empty), and the zero-copy contracts: results share the input's backing array, read-only methods never mutate the input, the no-change path reuses src, and Match([]byte) allocates less than MatchString(string(b)).

go test ./...

Prefilter: which literals must appear, and which patterns can never be filtered

Prefilter exposes RE2's own prefilter machinery (FilteredRE2 / PrefilterTree, both already vendored here). It answers three questions:

p, err := hgmLibre2.NewPrefilter(patterns, 0 /*minAtomLen: 0 = RE2 default*/, 0 /*maxMem*/)
atoms := p.Atoms()              // lowercased, distinct literals that must appear
live  := p.Potentials(found)    // given the atom indices found in the text: which patterns can still match
hard  := p.Unfiltered()         // which patterns need NO literal at all -> they always have to run

Prefilter does no matching of its own. It hands you the atom list; you find those atoms with whatever string matcher you like (an Aho-Corasick automaton, or memmem), then ask which patterns survive. A pattern that does not survive is guaranteed not to match. Matching must be case-insensitive, or done on a lowercased copy of the text, because the atoms are lowercased.

Unfiltered() is the reason this is exposed. "Screen the text with a cheap literal gate first, and only run the big table on what gets through" is the one direction that raises the throughput ceiling (see doc/set性能优化经验.txt §4 G) — but it has a hard cap: patterns with no required literal ([A-Za-z0-9+/=_-]{20,}, (?-i:\([A-Z]{2,5}\))) have to run no matter what the text looks like. Measure that set before building a prefilter stage, not after.

🔴 Only RE2's own prefilter gets this right. A hand-rolled "pull the literals out of the pattern source" extractor answers wrongly on (?:foo|[A-Z]{5}): it contains the literal foo, yet the pattern as a whole is unfilterable, because the other alternative does not need foo. That reasoning lives in an AND-OR tree and is not something you can eyeball. prefilter_test.go pins this case, along with the soundness property the whole idea rests on: every pattern that really matches a text must appear in Potentials() for the atoms found in that text.

minAtomLen is a real trade-off knob, not a tuning detail. Raising it yields fewer, longer atoms — a faster matcher, but more patterns fall into Unfiltered(). Measured on a 112-pattern production table: RE2's default gives 1654 atoms and only 4 unfilterable patterns, but the atoms are so short that they occur in nearly every text, so nothing gets filtered out; minAtomLen=6 gives 216 atoms and 38 unfilterable patterns, which filters much harder but starts from a 34% floor. Measure both ends on your own table.

Tuning for the DFA state cache

doc/set性能优化经验.txt is the long-form version of the performance material above, for a single Regexp as well as for a RegexpSet: the mental model, what to measure and in what order, what actually drives state explosion, the three knobs in benefit order (direction > pattern shape > memory budget), how to split a table, and a list of approaches that were measured and rejected. Read it before tuning a table of hundreds of patterns, or before reaching for CompileMaxMem/CompileReverse; the sections above are the summary.

Differences from stdlib regexp

This is the complete list of concrete behavior differences from Go's standard library regexp. The first two are API-design choices (this library deliberately is not a drop-in); the rest follow from running the native RE2 engine instead of Go's from-scratch reimplementation. All are intentional and covered by tests.

For a migration checklist that pairs each gap (here and in Supported API) with what to use instead — together with the performance side of the same decision — see doc/与标准库regexp怎么选.md §4.

1. ReplaceAllString repl is literal — no $ expansion. stdlib expands $1 / ${name} / $$ in the replacement string; here repl is inserted byte-for-byte with no expansion and no escaping (so "$1" stays "$1", "$$" stays "$$"). This is the one method that is not signature-compatible in behavior. If you need capture-group substitution, use ReplaceAllStringFunc and build the replacement yourself. (FindReplaceWithin is a different, non-stdlib method and uses RE2's \1 rewrite syntax — see its section above.)
2. []byte replace methods reuse src on the no-change path. stdlib's ReplaceAll / ReplaceAllFunc always return a freshly allocated slice; here an input that comes out byte-for-byte unchanged (no match, or a replacement that changes nothing) is returned as the original src slice, allocation-free — so the result must not be written to. The string methods have always behaved this way; strings being immutable, it is only observable in the []byte family. Content-wise the results are identical, including the nil-vs-empty conventions. See Byte-slice methods.
3. Invalid UTF-8 input. stdlib treats each invalid byte as one-byte U+FFFD and lets . match it; native RE2 only matches whole valid runes, so on e.g. []byte{0xff,'a',0xfe} the pattern . finds just the a. If you match on possibly-invalid UTF-8 and need stdlib's behavior, use stdlib.
4. \C is accepted (RE2 "any byte"); stdlib regexp rejects \C at compile time. More generally a handful of escapes are RE2-only or stdlib-only, so a pattern valid in one may be rejected by the other.
5. 2 GiB input limit. Lengths/offsets cross the cgo boundary as 32-bit int, so inputs (and patterns) longer than 2^31-1 bytes are conservatively treated as no match / returned unchanged rather than matched. stdlib has no such limit. (Irrelevant unless you feed multi-gigabyte strings.)
6. Case-folded literals fold over the full Unicode orbit when they are merged into a character class. When a case-folded literal is one branch of an alternation that RE2 factors into a single character class, the class picks up every fold-equivalent rune, not just the ASCII pair: \w|[kK] also matches U+212A KELVIN SIGN, where stdlib matches only k/K. ([sS]|\w has always matched U+017F this way.) This is upstream RE2 behavior and only shows up for non-ASCII fold-equivalents.
7. Capture names may be non-ASCII. (?P<中文>a) compiles here; stdlib has rejected non-ASCII capture names since Go 1.22. Both forms of named group — (?P<name>expr) and (?<name>expr) — are accepted, as in stdlib (Go 1.22+).
8. No nesting-depth limit. stdlib rejects patterns whose parse tree nests deeper than 1000 (expression nests too deeply, Go 1.19+); this library accepts them. 200 000 nested groups compile in ~100 ms with linear memory and no stack growth (parsing, simplification and teardown are all iterative), and 400 000 fails cleanly on the capture-group limit — so this is only a matter of accepting more than stdlib, not a robustness gap. If you compile untrusted patterns and want stdlib's ceiling, check the depth yourself before compiling.

Not a difference, but worth stating: matching is leftmost-first here, which is also stdlib's default (regexp.Compile); stdlib's opt-in leftmost-longest mode ((*Regexp).Longest) is not provided. Capture-group names of any length are returned in full and duplicate named groups are accepted — same as stdlib.

Concurrency: sharing one Regexp is fine (it just doesn't scale linearly)

Share one package-level *Regexp, the way you would with stdlib. This section exists to explain a scaling curve, not to ask you to do anything about it.

A Regexp is safe to use from multiple goroutines, but it does not scale linearly. Every DFA search takes a read lock on that Regexp's DFA state cache (DFA::cache_mutex_, a pthread_rwlock on Linux); the lock exists only so the rare whole-cache flush can run exclusively, yet every single search pays for it. Read locks do not exclude each other, but the reader count is an atomic on one shared cache line, so with enough goroutines that line ping-pongs between cores and the "concurrent" searches serialize.

Measured on a 20-core Ryzen 5900X, non-matching pattern, ns/op:

	14-byte input	4 KB input
one shared *Regexp, 1 goroutine	69–74	453–467
one shared *Regexp, 16 goroutines	42–77	62–69
one *Regexp per goroutine, 16	9.5–13	38–51
stdlib *regexp.Regexp shared, 16	4.3–4.5	67–70

Compiling the read lock out (measurement only) makes the shared case match the per-goroutine case exactly (8.0–8.5 ns at 16 goroutines), so the entire gap is that one lock. Short inputs suffer most, but even 4 KB inputs lose ~1.6×.

This is not a reason to stop sharing. The whole effect is ~33 ns per call at 16 goroutines on 14-byte inputs, and buying it back is a bad trade in most programs: one *Regexp per worker means one compile per worker (microseconds to milliseconds each, and the pattern is usually compiled once at init today), one separate DFA state cache per worker — so the peak native memory and the max_mem budget you tuned both multiply by the worker count — and lifetime management (pooling, FreeC) that a package-level variable doesn't need. A shared Regexp also reuses cached DFA states across goroutines; N private copies each rebuild them.

Only consider per-worker copies if a profile actually points at this lock — i.e. regex matching is a top cost in your program, inputs are short, and the concurrency is high. Otherwise keep the one shared variable. Correctness, RegexpSet, and low concurrency are unaffected either way.

Note what the stdlib column does not say: at 14 bytes a single cgo call (~50 ns) already costs more than the whole match, so stdlib wins there whatever the locking does; that row is a scaling reference, not a throughput comparison. At 4 KB the shared case is level with stdlib and the per-goroutine case is ~1.7× faster. This is upstream RE2 issue #569; the benchmark that produces the table is contention_bench_test.go.

Resource management

A Regexp holds a native RE2 object freed automatically by a finalizer, so for ordinary use you do nothing. When you compile a large number of patterns dynamically and want the native memory reclaimed promptly instead of waiting for GC, call FreeC() to release the C++ object immediately.

FreeC is deliberately minimal and unguarded: it is not safe for concurrent use, and calling any method (or FreeC again with a live match in flight) after the object is freed is a use-after-free. FreeC itself is idempotent (a second call is a no-op). If you don't need prompt reclamation, don't call it and let the finalizer handle cleanup.

The native object is freed exactly once under every call ordering — there is no double-free between FreeC and the finalizer, for two independent reasons:

• FreeC clears the finalizer (runtime.SetFinalizer(re, nil)) in the same call that frees the object. Since you must hold a live reference to re to call FreeC, the finalizer cannot already be scheduled, so clearing it always wins and it never runs afterwards.
• Even if a nil handle ever reached the underlying cre2_free, that function is null-safe (it returns immediately on nullptr).

Note the asymmetry this implies: only the free path tolerates a nil handle. The match/replace methods do not — calling any of them after FreeC dereferences a freed/nil RE2 and crashes. The null-tolerance exists solely so the finalizer can never misfire, not as a guard for post-free use.

Vendored RE2

The RE2 C++ source is vendored in this directory (see VENDOR.txt for the exact layout and how to upgrade). It is pinned to RE2 tag 2023-03-01, the last release before RE2 took an abseil dependency; later releases cannot be compiled this way directly.

A small set of later upstream fixes is backported on top of that tag — the ones that are real fixes rather than abseil churn, most notably a silent false-negative in alternation factoring (0a|0[aA] used not to match "0A"), support for (?<name>expr), and not expanding counted repetitions of zero-width operators (\b{1000}). Each site is tagged [backport re2 <commit>] in the source; VENDOR.txt lists them, together with the upstream commits that were deliberately not taken and why.

Three further fixes come from upstream pull requests that are still open (tagged [backport re2 PR#NNN]), reproduced and cross-checked against stdlib here before being taken. The one that mattered: RE2's "if the DFA is rebuilding its cache this fast, fall back to the NFA" heuristic compared p - resetp, which is negative during the reverse scan that locates a match's start — so the heuristic had never fired in that direction and the reverse DFA would flush itself indefinitely. Fixing it takes (?s)a[a-d]{24}b[a-d]* over 1 MB from 43 flushes / 234 ms to 1 flush / 34 ms, with identical results.

Local changes to the DFA

The vendored DFA is not byte-for-byte upstream. re2_dfa.cc stores each transition-table slot as a 4-byte offset into an arena of states instead of an 8-byte State*, and grows that arena on demand instead of reserving the whole budget up front. Three other vendored files carry small additive changes for the counters above — re2_set.cc and the re2/prog.h / re2/set.h headers gain optional out-parameters and accessors — plus one new header, re2/dfa_stats.h, which is not upstream at all. Every other .cc file is stock. VENDOR.txt lists the same set — that is the list to re-apply when the vendored RE2 is upgraded.

Consequences, all measured on real pattern tables and real corpora:

• The same budget holds 1.74× more states, so the budget at which a given table stops flushing drops by one step (e.g. 128 MB → 64 MB).
• Throughput does not regress on tables that never flush (it is a few percent better), and improves by two orders of magnitude on a table that was flushing at that budget — the win is crossing the cliff, not the encoding.
• Peak RSS falls ~30% on large working sets. On a budget that is genuinely saturated it rises ~10%, because the same bytes now hold 1.74× more states.

Match results are unchanged and this is enforced, not assumed: hit-set digests are compared bit-for-bit against a build of the original 8-byte-pointer code across a matrix of pattern tables, corpora, and budgets. The original encoding is still in the source and can be restored with CGO_CXXFLAGS="-O2 -DRE2_DFA_NEXT_BITS=64 -DRE2_DFA_ARENA=0", which is useful as a control when bisecting a performance question.

The only other build-time macro a caller might want is -DRE2_DFA_ATTRIB=1, which turns on attribution. Both default to off/stock, and the default build carries no fields, branches, or counters for either.

License

BSD 3-Clause, the same license as RE2. See LICENSE and RE2_LICENSE.txt. The vendored RE2 files retain the copyright of the RE2 Authors.

Documentation

Overview

bytes.go — []byte 系方法: 与 string 系【同一套定位内核】的另一个门面, 供正文本来就是 []byte 的调用方 (HTTP body / 文件内容 / 解码缓冲) 免掉 string(b) 的全量拷贝。

零拷贝原理: 匹配本身在 C 侧只吃「字节指针 + 长度」(见 strBytePtr / cre2_match_at), 与 Go 侧是 string 还是 []byte 无关。故本文件不重写任何匹配逻辑, 只把 b 零拷贝成 string 视图喂给既有内核 (findFrom / matchAllFlat / replaceAllLiteralRaw / findReplaceWithinRaw), 再按返回的 index 从 【原 b】切子切片 —— 与 stdlib bytes 系一样, 结果与输入共享底层数组, 全程零字节拷贝。

命名规则: stdlib *regexp.Regexp 有同名 []byte 方法的照搬 stdlib (Find/FindIndex/… ↔ FindString/ FindStringIndex/…); 本库自有的方法加 Bytes 后缀 (FindReplaceWithinBytes / RegexpSet.MatchBytes)。

语义与 stdlib 的两点差异 (与本库 string 系保持一致, 非 stdlib drop-in):

1. Replace 系在【逐字节无改动】时直接返回原 src 切片 (零分配), 不像 stdlib 总返回新副本; 调用方【不得改写】返回值, 需要独立可写副本请自行 copy。
2. 其余 nil / 空切片语义与 stdlib 逐一对齐 (结果为空 → nil), 见各方法注释。

并发/可变性约束 (同 stdlib): 调用期间不得改写传入的 b —— C 侧直接读它的底层数组。

dfastats.go — DFA 状态缓存计数: 把"预算够不够"从猜变成量。

【病灶】RE2 的 DFA 状态缓存装不下当前语料走出来的状态集时, 不是 LRU 淘汰而是【整表清空】 重建 (DFA::ResetCache)。结果仍然正确 —— 所以这件事在调用方眼里没有任何信号 —— 但它是 悬崖不是斜坡: working set 比预算大 1%, 吞吐掉几十倍。RegexpSet (kManyMatch) 更容易撞: 状态里带着"已命中哪几条"的位集, 状态数对 pattern 条数是超线性的。

【为什么单形状 benchmark 量不到】同一份 body 扫 N 遍, 第一遍之后缓存全热, 再不新建状态, 所以"换个预算吞吐没差别"是必然结论 —— 而生产是每个请求一份【互不相同】的 body, 每换一份 就把缓存冲垮一次。要量这条曲线, 语料必须多形状, 且要看 Resets 而不是只看吞吐。

【怎么用】

• 标定 maxMem: 单线程热身跑一批【互不相同】的真 body, 取 Resets 增量; >0 就把预算翻倍 重编重来, 直到增量归零 —— 那个预算才是"够", 编译过得去不算够。
• 产线: 定期采样算速率 (resets/秒 或 resets/次扫描)。稳定 >0 = 正站在悬崖底下。

口径: 【进程级】计数, 不区分是哪个 Regexp/RegexpSet 造成的 (RE2 的钩子不带上下文, Set 匹配 更是完全没有)。并发扫描时取差值只能得到"这段时间内全进程的次数", 要归因到某一次扫描请单线程量。

find_all_flat.go — FindAllStringIndex 的【无中间结构】变体: 把全部匹配的 [start,end) 直接 追加进调用方的可复用 []int, 不产 [][]int 外壳、也不产一次性的 flat 表。

动机: FindAllStringIndex 每次调用要产两笔一次性分配 ——

① matchAllFlat 的 flat []int (每匹配 2*nmatch 个 int, 即便调用方只要 group0);
② res [][]int 外壳 (每匹配一个 24 字节切片头, 指回 flat)。

在"大正文 + 高命中数"的热路径上这两笔按匹配数线性放大: 实测 16MiB body 上 19 万处命中 = 40 字节/命中 = 7.6MB 一次调用, 而调用方拿到 [][]int 之后无非是顺序遍历 loc[0]/loc[1]。

本变体只回填 group0 (nmatch=1, 同 FindStringIndex_ctx): C 侧的 vector<StringPiece> 也随之 从 numSubexp+1 缩到 1, 逐处匹配整循环仍留在 C 内 (一次 cgo 过境, 不是每处一次)。 结果写成 [s0,e0,s1,e1,…] 追加进 dst —— 调用方传 buf[:0] 即可跨调用复用同一块内存, 稳态零分配。语义与 FindAllStringIndex 逐处相同 (推进/空匹配去重都在同一段 C 代码里), 对拍门见 find_all_flat_test.go。

find_at.go —— 「必须从这一点起头」: 锚定在 from 的一次搜索。

── 它和 find_from.go 那一组差在哪 ────────────────────────────────────────── FindStringIndexFrom(s, pos): 【非】锚定 —— 起点不许早于 pos, 但可以比 pos 晚。 FindStringIndexAtWithin(s, from, bound): 锚定 —— 起点【就是】from, 不是就没有匹配。

两者的 C 入口是同一个 RE2::Match, 只差 anchor 一个参数; 而 text/textlen 一律传【整串】, from/bound 只圈定"在哪一段里搜" —— 所以 \b / ^ / $ 看到的是真实邻字节, 调用方不必自己 s[from:bound] 切一刀 (切完两侧就是假邻居, 答案会错)。

── 为什么需要这一组 ──────────────────────────────────────────────────────── 这就是"给一个起点, 求这条 pattern 在这儿能伸到哪"—— set 那侧叫 ResolveSpan。 拿【单条】对象做同一件事有两个好处:

① 走的是 RE2::Match 那条完整的路 (DFA → OnePass/BitState/NFA 逐级回退), DFA 放弃了
   还有下家; set 那侧的锚定解析是 kManyMatch 的 DFA 独一条, 没有下家 ——
   "DFA 放弃"在那边只能整遍失败, 在这边根本不发生。
② 用的是这一条自己的程序和自己那份 DFA 缓存, 不去冲刷整表那份大的。

🔴 要"最长的那个终点"就得配 CompileLongestMaxMem 编出来的对象。普通对象给的是贪心那个

终点 —— 变长 pattern 上那是【把命中截断】, 下游过校验位会把真命中判成假 (见
CompileLongestMaxMem 的红字)。

find_ctx.go — FindStringIndex 的【复用 scratch】变体: 把每次调用都要新分配的 cgo 回填缓冲 与返回切片挂到一个 ctx 上, 单线程顺序反复调稳态零分配 (需调用方持有 ctx 配合)。

动机: (*Regexp).FindStringIndex 每次都 make 一个 []C.int 回填缓冲 + (命中时)一个 []int 返回切片; 在「大正文逐段反复跑同一批正则」的热路径上 (如把正文切成段, 每段都跑一遍规则表), 这两笔分配按匹配 次数线性放大, 是分配次数大头. 本变体把缓冲挪进 ctx, 同一 ctx 反复调只在首次分配.

与 FindStringIndex 语义一致: 只取 group0 ([start,end))、leftmost、非锚定。返回的切片指向 ctx 内部, 仅在【下次用同一 ctx 调用前】有效; 需留存请自行 copy. ctx 非线程安全, 并发各持一个。

find_from.go —— 「从 pos 起找下一处」: 非锚定, 但起点不许早于 pos。

── 为什么单独有这么一组入口 ──────────────────────────────────────────────── 内核早就有这个能力: C 层 cre2_match_at(h, text, textlen, startpos, endpos, ...) 里 text 传的 是【整串】, 只有 [startpos,endpos) 这一段参与搜索, 所以 \b / ^ / $ 在切口处看到的是真实邻 字节。Go 层 findWithin(s, from, bound) 也一直包着, 只是所有导出入口都写死 0/len(s) —— 要"从某处接着找"的调用方只能自己 s[pos:] 切一刀, 而切完 \b / ^ / $ 看到的就是【假邻居】 (切口两边的字节没了), 答案会错。

🔴 所以这组入口的存在理由只有一条: 让"只在某一段里找"这件事不必切片。参数都是【原串上的

偏移】, 整串照样喂给 RE2。

谁在用: ① MatchScanner 的 B 路 (matchscan.go 的 MatchScanMode_t) —— 它要"起点 >= 游标的 最左匹配", 正是这个形状 (拿的是 longest 口径的对象, 所以一趟就是整段区间)。 ② 拿到整段区间之后回头补捕获组偏移的调用方 (下面那个 Within 版)。 🔴 锚定版 (起点必须【就是】pos) 在 find_at.go: FindStringIndexAtWithin。

find_replace_within_append.go — FindReplaceWithin 的【追加进调用方缓冲】变体。

动机 (2026-08-22 · 200MB 语料 memprofilerate=1 实测): 调用方的"去分隔符再匹配"那条腿走 FindReplaceWithin, 而后者在【有改动】那条路上把 C 侧结果整份 C.GoStringN 拷成一个新的 Go string —— 200MB 正文上就是每次调用一份 200MB 的 Go 堆分配, 一次注入检测走两趟 (#5 heal / #7 combo), 单这一条在那份 profile 里排第五 (400MB, 全场 7.5%)。而那两份产物都是【当场喂进 Set 扫一遍就丢】, 没有一个字节需要留存 ⇒ 典型的"该往调用方自己那块复用底上写"的形状。

契约与 AppendReplaceAllStringFunc 一条线 (见 replace_func_ctx.go): 返回 (dst, changed), changed=false 时 dst 一个字节都没动。changed 的定义同 FindReplaceWithin —— 结果与 src 逐字节 不同才叫变了 (C 侧惰性物化: 无匹配 / 命中但删 0 个字符 都报 0, 且 C 侧连缓冲都不开)。

本文件不带 ctx: 这条路的 scratch 全在 C 侧 (外层循环 + 段内替换都在 C++ 里), Go 侧唯一那笔 按正文线性的分配就是结果本身, 而结果由调用方传 dst 进来 ⇒ 没有第二块需要跨调用留着的东西。

findallindex.go —— FindAllIndex: 一遍扫正文, 边扫边【一批一批】把"哪条 pattern 的匹配端点 落在哪一段"交出来。这是本库多正则一侧的【底座】, 上面那层"算出完整区间"(matchscan.go) 就搭在它身上。

── 它替掉的是哪段路 ──────────────────────────────────────────────────────── 调用方今天的写法是两段式: 先 Set.Match 扫一遍拿到"哪几条命中", 再为了知道【在哪】把这几条 各自的 Regexp 对整篇正文跑一遍 FindAllStringIndex。命中 k 条就是 1 + k 遍全文, 而且那 k 遍 用的是【非锚定】正则 —— `.*?` 前缀让"哪个位置能当起点"变成"每个位置都能", 状态数对计数 上界指数增长 (doc/状态数为什么会相乘.txt: 同一条 pattern 加个 \b 就是 967 倍的差距)。

可位置本来就在第一遍里算出来过: kManyMatch 的 DFA 每走到一个能结束匹配的字节都记了一次, 走完把它扔了。FindAllIndex 就是把这一遍记下来的东西接出来 ——【不额外要钱】, 同一份正文 同一份 DFA 缓存, 6.4MB 上实测 Match 18.5ms / 收端点 18.4ms。

── 吐的是【端点游程】, 不是匹配区间 ──────────────────────────────────────── 交出来的每一条是 (ReIndex, Lo, Hi): 第 ReIndex 条 pattern 的匹配端点落在 Lo..Hi 里的 【每一个】值上 (原文字节偏移,【两端都含】, 恒 Lo <= Hi)。

正向 RegexpSet        : 端点 = 匹配【右端】(不含), 即 text[?:Hi] 是一个匹配
反向 RegexpSetReverse : 端点 = 匹配【左端】(含),   即匹配从 text[Lo] 开始

🔴 两端【都给】所以无信息损失: 展开 Lo..Hi 就还原成逐个端点。这一点不能省 —— 只留一端

(比如只留最右那个) 会把两个真实独立的匹配悄悄并成一个: `ab|c` 撞 "abc" 的右端是 2 和 3,
连号, 只留 3 就把 [0,2) 那个匹配弄丢了, 而且【不报错】。

🔴 为什么是【闭区间】而不是 Go 习惯的半开: 这三个数不是一个区间, 是"一串端点"的收敛写法。

Hi 是一个真端点, 不是"末端后一位"。写成半开就得记住 Hi 那个位置到底算不算, 而这正是
最容易错的地方。字段名里也没有 start/end 的暗示 —— 说的是哪一端由【方向】定死
(正向 = 右端, 反向 = 左端), 一个类型里塞两套名字反而会骗人。

🔴 顺序【不保证】全局按位置升序。一段游程要等"这条 pattern 再次命中且与上次不连号"或者

"整篇扫完"才收口, 所以不同 pattern 的游程会交错, 最后一批还会在扫完时集中吐出来。
但【同一条 pattern 内部】按【扫描方向】单调 —— 正向 set 升序, 反向 set 【降序】
(扫的方向本来就是单向的)。上面那两层的游标就靠这条 (matchscan.go / matchscan_reverse.go)。

🔴 语义不是 FindAllStringIndex。FindAll 给的是 leftmost-first 的【不重叠】匹配序列;

这里给的是"所有 pattern 的所有匹配端点", 重叠的也在里面 (`abcd|bc` 撞 "abcd" 两条都报)。
要 FindAll 那个口径的调用方看 matchscan.go —— 取舍规则 (优先级贪心 / 相交即丢 / …)
是调用方的业务, 这一层不替它决定。

── 为什么是"一批一个数组", 不是"一条一个回调" ────────────────────────────── 游程条数没有上界 (真表实测约 30741 条/MB, 200MB 的 body 就是 47MB), 所以【不能】攒成一个 完整数组还给调用方 —— 那等于让内存跟着正文长度走。但反过来"一条调一次回调"也是白扔钱: 那是每条游程一次【不可内联的间接调用】, 6.4MB 上 19.7 万次。

量过 (两条腿交替跑各 40 轮取中位数 —— 顺着跑量不出来, 进程内前后段的漂移就有 ±5%, 比要量的东西还大):

一条一回调  18.47 / 19.11 / 19.90ms      一批一数组  18.29 / 18.81 / 19.62ms
差 +1.0% / +1.6% / +1.4%  ⟹ 每条游程约 1.3ns

一批一个数组把这笔钱摊掉 (4096 条才一次调用), 而且调用方拿到的是一段连续内存, for 循环 能矢量化、边界检查能提到循环外。省下的只有 1% 出头, 但这个 API 的存在理由就是快, 白给的 1% 也不该给。

── alloc 是什么 ──────────────────────────────────────────────────────────── native 那层是 sqlite3_step 式的: 攒满一批就【挂起】(按内容存下当前 DFA 状态, 放掉 DFA 的 缓存读锁, 返回给 Go), 取走之后再进去、重新拿锁、按内容把状态查回来接着扫。挂起期间一把锁 都不持有 —— 反过来说, 如果做成"C 直接回调进 Go", 回调期间还攥着读锁, 谁想 flush 谁就得 等 Go 跑完。那个挂起点 + 那一批缓冲就是 alloc。

为什么不能"缓冲不够就扩容重跑": 重跑要付的正是最贵的那一遍 (新正文现造 DFA 状态, 实测比命中缓存的重复扫慢 66 倍)。

🔴 交给 batchFn 的那段切片是 alloc 里那块缓冲【本身】, 不是拷贝 —— 下一批会原地覆写它。

要留就自己 append 走。

alloc 传 nil 也能用 (当场建一个、用完就扔), 只是每次 FindAllIndex 多一笔 native 分配。 热路径上建一个长期留着。🔴 alloc 【不是并发安全的】: 一个 goroutine 一个。 同一个 set 上开多个 alloc 并发扫是可以的 (set 本身只读)。alloc 认它出生的那个 set, 串用会报错。

── 偏移为什么是 int32 ────────────────────────────────────────────────────── ① 宽度锁死, 32 位/64 位平台上一样宽; native 侧本来就是 int32, 这块缓冲是 C 直接写进去的,

换任何别的宽度都得多一趟【逐条转换】—— 正是上面那 2.4% 要躲的东西。

② 不用 uint32: 上面那层算起点要做 end - minLen, 正文开头几个端点上这是【负数】。

有符号下它一眼可判; 无符号下它回绕成 42 亿, 边界判断会【静默】放行, 然后在
text[42亿:end] 上炸 —— 这类下溢是最难查的一种。

③ RE2 本来就把正文卡在 2GiB (见各处 maxCInt 检查), int32 装得下。

Package hgmLibre2 — 自带 cgo 的原生 RE2 正则库: 不用 go-re2 / 不用 abseil / 不用 cmake, 编译期不下载远程源 (RE2 2023-03-01 源码已 vendored 在本目录, 纯 C++11, zig 可交叉编译).

相比 go-re2 的 wazero 后端: 原生 cgo 路径不实例化 wazero runtime, 也不做 stdio 句柄探测, 因此在无 std 句柄的环境 (如 Windows SCM service) 也能正常用; 同时是单文件静态链接.

API 方法名/签名与 stdlib regexp 的 string 系与 []byte 系方法一致 (Compile/MustCompile + Find/Replace 系列; []byte 门面见 bytes.go, 与 string 系共用同一套匹配内核, 传 []byte 不产生拷贝), 便于互读; 但【不是】*regexp.Regexp 的 drop-in, 也不打算是. 匹配选择是 leftmost-first (同 regexp.Compile, 非 leftmost-longest). 与 stdlib 的有意差异: ReplaceAllString 的 repl 按【字面】 替换 (不展开 $1/${name}/$$, 见该方法注释); 以及原生 RE2 引擎的边角 (非法 UTF-8 上 . 的匹配、\C 任意字节等按 RE2 语义) —— 详见 README 的 "Differences from stdlib regexp" 一节.

match_step.go — 全匹配的【sqlite3_step 式】原语: C 侧一次填一批命中进一块批缓冲, Go 侧取走这批、再 step 下一批, 直到扫完。内存里【从来没有全部命中信息】, 只有一批。

为什么要它 (2026-08-26 · 接 doc/plan12/20260826_213re2.txt): 老路 FindAll* / AppendAllStringIndexFlat 一次调用要在三层各付一笔 ∝ 命中数的账 ——

① C 侧 cre2_match_all 的 std::vector<int> acc 逐处 push_back, 扫完再 malloc 一整块拷过去,
   峰值是整张命中表的【两份】(纯 RSS, Go profile 上根本看不见);
② Go 侧 matchAllFlat 的 flat = make([]int, count*2*nmatch), 把 ① 整块再拷进 Go 堆;
③ 外壳 make([][]int, count) / make([]string, count) / 每处一个 []string。

Append*Flat 形态只干掉了 ②③, 而且是拿【live-max】换的: dst 这块复用缓冲会涨到历史最大命中数 就再也不缩, 挂在 plan/pool 上 × 并发度常驻 —— 累计分配好看了, 常驻反而变成常态。① 一分没省。

step 形态把三层全部干掉: C 直接写进 Go 缓冲 (无 vector · 无 malloc · 全程零次全量拷贝), 缓冲大小固定为一批, 与命中数和正文长度都无关。live-max 从 O(命中数 × per × 并发) 降到 O(一批 × 并发)。

挂起为什么不需要 native 对象: 单条 Regexp 的每处匹配本来就是一次独立的 RE2::Match(full, pos, …) —— DFA 状态与 cache 锁都在那一次 Match 内部生灭。所以"挂起"就是把 pos/prevEnd 两个 int 交回 Go, 下次原样传回来。没有 C 对象、没有 finalizer、没有 Close、 没有 cgo handle 生命周期、没有"这个工作区是不是这条 re 的"检查。 (对比 RegexpSet 的 findallindex.go: 那边挂起的是一个扫到一半的【连续 DFA】, 三个 int 描述不了,

才不得不有 native 挂起点 + finalizer + Close —— 那套复杂度是 set 扫描逼出来的, 不是 step 固有的。)

cgo 过境次数 = ceil(命中数/批容量) + 1; 无匹配就 1 次, 与老路完全相同 (逐处匹配的循环整段留在 C 内)。

── 批缓冲从哪来 (2026-08-26 第二版: 库内 sync.Pool, 调用方不再持有工作区) ────────────── 第一版的形状是 StepAll…(st *MatchStep_t, …) —— 调用方自己持有一块工作区跨调用复用。 它在"调用方本来就有个 per-scan 的壳可以挂"时是零开销, 但代价是【没有壳的调用方会写成 函数内 var st MatchStep_t】, 而那正好是最差的一种: 进 C 之前必须无条件先备好缓冲 (Go 侧拿不到"这次有没有命中"的先验), 于是每次调用白付一笔 ~200B 的 make,【命不命中都付】。 而 FindAll* 在无命中那条路上几乎不花钱 (12B/2 笔) ⟹ 扫描型负载 (一张规则表挨个打同一份正文, 绝大多数调用是 miss) 换成 step 之后字节数反而涨。调用方产品实测: 8.2MB 档 920.4M → 922.7M, 对象数倒是降了 2 万 —— 典型的"对象少了、字节多了"两头不靠。

所以缓冲改成【库内一个 sync.Pool 持有, 每次调用借一块、返回前还回去】:

· 调用方一个工作区都不用持有, API 从三个参数变两个, 也就不存在"写成 var st"这种最差用法;
· 每块的尺寸【与 per / 命中数 / 正文长度全无关】, 恒是 stepBufInts 个 int32 (4KB),
  常驻是 O(4KB × 并发度) —— 与被否掉的 Append*Flat (O(历史最大命中数 × 并发) 且只涨不缩)
  完全不是一回事;
· 一个 Get/Put 来回 9.5ns · 零分配 (BenchmarkX_syncPoolRoundtrip);
· 顺带把嵌套调用变安全了 —— batchFn 里再起一条 step 扫描各借各的块, 老形状共用一个 st 会就地
  互相覆写。

四方对拍与定案理由见 zexp_step_alloc_bench_test.go 顶部 (那里的变体 E 就是现在这条主线)。

🔴 边界: step【不取代】FindAll* —— 那一族的契约就是"一次吐完个数组", 而"先在 C 里数好个数再 一次精确 malloc"正是该契约下的最优解。拿 step 去物化 FindAll* 实测是净亏 (Go append 阶梯累计 收敛到 5N: 20000 处命中 1.45MB/4 笔 → 5.70MB/26 笔, CPU +17%, 见 BenchmarkFindAllSub_matAll_vs_step)。所以分工是: 不需要全部物化 ⇒ step; 就是要数组 ⇒ FindAll*。 被这条边界挤掉的是中间那种【半物化】形态 (AppendAllStringIndexFlat): 它既没省下物化, 又要背一块 ∝ 命中数的常驻 ratchet 缓冲, 两头不靠 —— 所以它被删, 而 FindAll* 不动。

同题实测 (5900X · (\w+)=(\w+) · per=6 · 64 字节一处命中):

1MB 正文 · 子组版: FindAllStringSubmatchIndex 7.55ms / 1,179,668 B / 4 笔
                → StepAllStringSubmatchIndex 7.37ms / 0 B / 0 笔   (CPU -2.4%)
1MB 正文 · group0: AppendAllStringIndexFlat  4.44ms /       873 B / 0 笔
                → StepAllStringIndex          4.35ms / 0 B / 0 笔   (CPU -2.0%)
miss 路径:        FindAllStringSubmatchIndex  598ns /  12 B / 2 笔
                → StepAllStringSubmatchIndex  557ns /   0 B / 0 笔  (CPU -6%)

(这里量的只是 Go 堆; C 侧那份 vector+malloc 的峰值消除不进 Go profile, 要 RSS/massif 才看得见。)

matchscan.go —— 一遍扫正文, 边扫边【一批一批】交出各条 pattern 的不重复命中区间。

🔴 一句话: 【口径无条件是 leftmost-longest】, 没有旋钮, 没有"快而不准"的档。

这句保证怎么兑现的 · 对拍要拿哪个当 oracle:
doc/MatchScanner的leftmost-longest保证.md。

🔴 镜像那一半在 matchscan_reverse.go: RegexpSetReverse.NewMatchScanner —— 从末尾往前扫,

口径 rightmost-longest, 区间按 Lo 【降序】。表里有"正着扫爆状态、反着读塌回线性"的
pattern (S B{m,n} L 那一族) 才用它; 两种口径的差别见那份 doc 第 8 节。

── 它替掉的是哪段路 ──────────────────────────────────────────────────────── 调用方今天的写法是"两段式": 先 Set.Match 扫一遍拿到"哪几条命中"(一张 bool 表), 然后为了知道 【命中在哪】, 把这几条各自的 Regexp 拿出来对整篇正文再跑一遍 FindAllStringIndex。命中 k 条 就是 1 + k 遍全文。

可是位置本来就在第一遍里算出来过 —— kManyMatch 的 DFA 每走到一个能结束匹配的字节都会记一次, 走完把它扔了 (这正是 FindAllIndex 接出来的东西, 且【不额外要钱】: 同一份正文同一份 DFA 缓存, 6.4MB 上实测 Match 18.5ms / 收游程 18.4ms)。MatchScanner 就是把 FindAllIndex 给的 右端补成完整区间 —— 它整个就是【搭在 FindAllIndex 上面的一层】, 自己不碰 native。

SetModes(modes)            【每条要什么】两态: 只要 bool / 要区间 (默认)。全文见下一节。
Scan(text, batchFn)        一遍全文。命中表 (HitIDs / Hit, 与 Set.Match 同解) 边扫边填,
                           命中区间【边扫边收口、攒够一批就交出去】。
                           只返回 err: 要么全给, 要么整遍不算数 —— 见下面那一节。
Stats()                    上一遍的账 (回看了几次 · 收到几个候选 · 验了几次 · 吐了几处)。
                           变长条的钱全在"验了几个假候选"上, 没这几个数就看不见它。

🔴 一批一批交出去是【内存】上的要求, 不是风格。底下 FindAllIndex 本来就是 sqlite3_step

式的: 一批 4096 条游程 (48KB) 装满就挂起、交给 Go、取走再进去接着扫, 缓冲循环复用,
不随正文长度涨。要是这一层把结果全 append 攒起来等扫完再还给调用方, 那个固定缓冲就白
设计了 —— 实测真表上游程约 30741 条/MB, 收口后的输出还有 0.037MB/MB, 200MB 的 body
就是每个并发扫描 7.4MB 常驻, 而且 AppendMatches 那种"再照抄一份给调用方"的接口等于
两份缓冲。现在这一层【一个字节都不留】: 游标推进在回调里当场跑完, 收口出来的区间写进
一块固定的 matchScanBatch 缓冲, 满了就交出去、就地复用。
真表实测: 游程 196744 条 → 输出 74249 处, 而且其中 57% 的游程来自两条【只当 bool 用】的
pattern, boolOnly 一挡就没了 —— 挡掉的是那几条的【端点补全】(真花钱的那步), 不只是内存。

🔴 交给 batchFn 的那段切片是内部缓冲【本身】, 下一批原地覆写。要留就自己 append 走。

🔴 交出来的顺序【不按 pattern 分组】, 各条 pattern 的结果按收口先后交错着来 (同一条 pattern

内部仍按 Lo 升序)。想按条归拢是调用方那边一句 append 的事, 库这边归拢就得攒 = 又是缓冲。

能边扫边收口是因为同一条 pattern 的游程【跨批次也是升序】的 (正向扫本来就从左往右走)。

── "没能全给你"这件事: 一张建工作区就交的名单 + 一个整遍的 err ─────────────

🔴 这一层【没有】"扫到一半反悔、这几条你自己补"的中间态 (2026-08-27 拆掉的; 之前那个

   Scan 返回 unresolved 名单的设计是错的, 原委见下)。现在只有两处交代:

	NewMatchScanner 的 unsupported   走不了区间这条路的那几条 pattern (当下只有一个原因:
	                                 能匹配空串)。【与正文无关】, 建工作区那一刻就定死,
	                                 想写回归测试就写 —— 扔一条 a* 进去必然报出它。
	Scan 的 err                      这一遍不算数, 整篇走老路 FindAll。来由都是 maxMem 配小了
	                                 (底下那遍 FindAllIndex 失败 / 补端点要的单条对象编不出来 /
	                                 反向回推那一趟 DFA 放弃), 另有一种"游程乱序"是本库的 bug。

🔴 为什么不做成"部分成功": 一个调用方【造不出来】的错误码, 不该出现在返回值里。它逼出

的是这么一条链 —— 调用方必须写兜底 → 兜底跑不到 → 跑不到就没法测 → 没法测的代码基本
是错的 → 真出事那天走的是一段从没执行过的路。那比"根本没有兜底、直接整遍失败"更危险。
量过: 锚定解析用的是【小 DFA】(起点唯一), 不是扫全文那个; 三种形状的 pattern 在"刚好
编得出来"的那道墙上面 3000 字节的带子里逐档细扫 (60KB 正文 · 每 3 字节一个起点),
放弃 0 次 —— 墙底下则是 NewRegexpSetMaxMem 当场干净报错。所以这条分支本来就该是
"整遍失败"这种粗粒度的交代, 而不是一套调用方永远验不了的补偿逻辑。

🔴 "补端点要的单条对象编不出来"这一条同理【不兜底】: 2026-08-28 逐字节量过 —— 590 条

生产 pattern 里, "反向单条 set 比正向单条贵"的只有 16 条 (全在同一张凭据表,
全是 {n,} 开放尾巴), 最大倍率 1.021×。要让"正向 set 编得出来而反向单条编不
出来"真的发生, set 里得只装【一条】pattern 且 maxMem 恰好卡进那条 pattern 阈值往上
2% 的带子里 (实测那条 JWT 三段式: 正向 3580 字节 / 反向 3654 字节, 窗口 74 字节)。
多条的表上正向 set 本身就贵出几个量级, 这个窗口从结构上不存在。能走到这里只说明
调用方 maxMem 配错了 —— 那就该报错让他调大, 静默换一条实现只会把配置错误藏起来。

── 两态旋钮 (SetModes): 每条 pattern 要什么 ─────────────────────────────────

MatchScanMode_boolOnly           只要"命中没命中"。一处区间都不收口、一次端点都不补。
                                 门上很多位只当 bool 用 (某某类内容在不在), 从来没人问它
                                 在哪 —— 那几条不该为它花补端点的钱。
MatchScanMode_span               要区间。【零值 = 默认档】, 无条件 leftmost-longest。

🔴 2026-08-28 之前这里是【三态】: 多一个 MatchScanMode_spanFast, 强制走那条便宜但

【不保证】leftmost-longest 的"路 A"(游标启发式), 挂之前要调用方自己拿 fuzz 跑一份
"这一条走 A 也不岔"的凭据。整档删掉了 —— 换上来的这条路 (下面那节) 既是严格口径,
价钱又比 A 还便宜, 那一档就没有存在理由了, 留着只会让人以为还有便宜可占。

── 端点怎么补: 一遍 set 扫完, 之后【全走单条对象】────────────────────────────

🔴 一句话的规矩 (2026-08-27 定的): 【扫正文那一遍走 set, 之后补端点的每一趟都走这一条

   pattern 自己的单条对象, 一趟都不再回 set】。三条理由:
   ① 单条对象走的是 RE2::Match 那条完整的路 (DFA → OnePass/BitState/NFA 逐级回退),
      DFA 放弃了还有下家; set 那侧的锚定解析是 kManyMatch 的 DFA 独一条, 没有下家 ——
      "DFA 放弃"在那边只能整遍失败 (RE2::Set::Match 里 dfa_failed 就直接 return false,
      上游也是这么写的: re2_set.cc:216)。
   ② 补端点的流量不再冲刷整表那份大 DFA 缓存 (真表 155 条那份), 两边互不干扰。
   ③ 状态更小: 单条不必背 kManyMatch 每个状态那张 id 表。

	定长 (min == max)  Lo = Hi - min。【不进正则引擎】, 一句减法, 一趟都不用。起点唯一,
	                   所以它跟下面那条变长路必然同解。

	变长 (min < max)   【两步】:
	                   ① 反向 · 种全部状态 · 从右端 e 往左走到死 —— 把起点落在 [cur, e) 里的
	                      【全部可行前缀起点】一次收齐 (spanviable.go 的 ViableStarts,
	                      走的是 RegexpSet.viableOne 那条"反向 · 只装这一条的 set")。
	                   ② 候选【从小到大】逐个拿正向单条 longest 锚定去验 (forwardOne →
	                      Regexp.FindStringIndexAtWithin), 第一个验过的就是答案。
	                   为什么这就是 leftmost-longest, 见下一节那两条证明。

	要的那个单条对象编译不出来 = maxMem 配小了, Scan 整遍报错 (见上面那一节)。

🔴 反向必须是【一条一个】。整表建一个反向 set 是死路: set 里状态数是相乘的

(doc/状态数为什么会相乘.txt), 155 条的反向表在 6.4MB 正文上实测 65 秒 / arena 顶满 254MB
还在 flush, 而正向同一张表 18ms 零 flush。一条一个就没有这个乘法; 而且这些反向对象
【从不用来扫正文】, 只做锚定回推 —— 起点只有一个, 那套 `.*?` 前缀引起的状态爆炸机制
从根上就不存在。再加上惰性: 只有真被问到的那几条才会被建出来。

── 为什么这条路是对的 (两条, 都要) ─────────────────────────────────────────

① 候选集不漏。设真答案是 [s, E) 且 s ∈ [cur, e)。E 是一个匹配右端且 E > s >= cur,

而 e 是【> cur 的最小右端】⟹ E >= e ⟹ text[s, e) 是 text[s, E) 的前缀 ⟹ 它是
一个可行前缀 ⟹ s 一定在 ViableStarts 给的候选里。∎
升序试 ⟹ 第一个通过的必然是 leftmost; 而 fwd 是 longest 口径编的, 锚定在 s 上给的
就是最长右端 ⟹ 严格 leftmost-longest。

② 一个都没验过 ⟹ [cur, e) 里根本没有起点 ⟹ 游标可以直接推到 e (对①取逆否)。

所以"全军覆没"这一支不是放弃, 是【证明了这一段是空的】。

顺带一条: 验过的那个候选 s 给出的右端 E 必然 >= e (E 是右端且 E > cur ⟹ E >= e), 所以游标每次都真的越过 e —— 各轮的回看窗口 [cur, e] 两两不交且递增, 反向那一趟的 累加封顶 = 多扫一遍正文。这正是它比老路便宜的地方: 老的"路 A"每处命中固定两趟且窗口 【相交】, 老的默认档"路 B"没有上界的条目要走完空隙。

── 对外那条保证, 逐字读 ─────────────────────────────────────────────────────

交出来的区间:

① text[Lo:Hi] 是这条 pattern 的一个真匹配;
② 【同一条 pattern】吐的区间互不相交, 按 Lo 升序;
③ 口径是 leftmost-longest (= stdlib 的 re.Longest().FindAllStringIndex)。

三处容易读错的地方:

🔴 ② 只管【单条】。两条 pattern 在同一片正文上照样重叠 —— 那不是重复, 是两个问题各要
   一个答案 (下面那段"只在同一条 pattern 内部去重"讲的就是这件事)。
🔴 ③ 【不是】"与 FindAllStringIndex 相同"。stdlib 默认的 FindAll 是 leftmost-first
   (贪心), 两者在"同一起点上贪心先撞到的比最长的短"时给不同的右端。要对拍就拿
   Longest() 那个去对, 拿默认那个对会是【假红】。
🔴 能匹配空串的 pattern 只能配 boolOnly, SetModes 当场报错 —— 不是运行时静默退化。

── 重复: 每条 pattern 一个游标, 一次左到右推进 ────────────────────────────── 变长 pattern 在一片正文上会在每个可收的位置各报一个右端 (\p{Han}{2,4} 撞 "张三李四王五" 报 6/9/12/15/18 五个), 它们说的其实是同一片区域。推进的规矩:

① 右端落在已吐出去那一处里面 (Hi <= cur) → 跳过;
② 否则求【起点不早于 cur 的最靠左那个】起点 —— 回看窗口掐在 cur 上, 绝不越过游标;
③ 从这个起点取最长右端吐出去, cur 推到它。

🔴 ② 的窗口掐在 cur 上是【正确性】不是省钱: 不掐的话上例 Hi=18 会回推到 6, 与刚吐出去的

[0,12) 相交而被丢掉, "王五"就无声无息漏了。掐上之后回推到 12, 吐 [12,18)。

🔴 【只在同一条 pattern 内部去重, 跨 pattern 一概不合并】。两条 pattern 在同一片正文上各自

命中不是重复, 是两个问题各要一个答案 (带空格的和不带空格的两条 pattern, 下游正是靠
"这一段里有没有空格"分流的; 合了就是漏检)。
实例: "Passport No: A123456780"
上 \b[A-Z][12]\d{8}\b (台湾身份证) 与 (?i)\b[A-Z]{1,2}\d{7,9}[A-Z]?\b (护照号) 抢
【完全同一段】[13,23)。合并的话下标在前的台湾身份证先占, 而它自己又过不了 mod-10
校验位被消费点毙掉 ⟹ 这段明文护照号一条都不出。"谁赢"只由常量写在第几行决定, 而
"谁能活"要等消费点把校验位跑完才知道 —— 库这层两样都不知道。真语料上被 ≥2 条 pattern
盖住的字节占已盖住字节的 55.6%, 同一字节最多被 8 条盖, 所以这是常态不是边角。

── 老账: 这条路是怎么换上来的 (2026-08-28) ─────────────────────────────────

在这之前补端点有三条路并存, 默认档 B + 一个旋钮 A, 外加一个用来比价的独立类型 MatchScanner2 (路 D2)。现在只剩 D2 这一条, A/B 与 MatchScanner2 这个类型一起删了。

路 A (旧 spanFast)  反向【只种 accept】回推一个起点 + 正向锚定取最长右端。两趟, 窗口
                    【相交】。给的是"第三种口径"—— 既不是 leftmost-first 也不是
                    leftmost-longest: \b(?:ab cd ef|cd)\b 撞 "ab cd ef" 时门给的最小右端
                    是 "cd" 那处, 只种 accept 就只回推得到 "cd" 的左端, 真正的最左起点 0
                    根本不在候选里。这是它的病根, 也正是本路"种全部状态"要解掉的那件事。
路 B (旧默认档)     从 max(cur, e-maxL) 起做一次正向【非锚定】longest 搜索。一趟, 口径严,
                    贵在【没有上界的条目要走完空隙】(maxL < 0 时下界塌回游标)。

换掉的凭据 (2026-08-28, 全部在 100MB 量级的真语料 × 9 张生产真门表上跑; 11 份语料 = console 前端产物 + 凭据二次方八腿 + asc源码/说明书/端点ELF 混合 + 本机 claude 真历史):

口径   11 份语料 × 9 张门 = 99 格, 逐区间按 (条, Lo, Hi) 排序对拍。
       与路 B 【一处不差】—— 合计对账 1.619 亿处区间, 差 0 处。这是"敢把默认档换掉"
       的全部凭据。
       与路 A 差 【37 处】, 全在那份 asc源码/说明书/ELF 的混合语料上, 而且每一处都是
       路 A 把左端截短了 —— 例如阿联酋身份证 A 给 "1985-1234567-1" 而真答案是
       "784-1985-1234567-1", 提示注入标记 A 给 "<SYS>" 而真答案是 "<<SYS>>"。
       🔴 这正是文件里一直警告的那种伤: 区间偏了, 拿去过校验位 (mod-10 / Luhn /
          mod-97) 会失败, 整条真命中被下游自己毙掉 = 无声漏报。所以这次换路不只是
          省钱, 是【真的在真语料上修掉了 37 处错边界】。
价钱   11 条门链合计, 本路【每一条都是最快的】。相对路 A 0.48~0.91×, 相对路 B 视语料
       0.6~1.0×。"试/看"(每次回看正向锚定验了几次) 在 99 格里【全是 1.00】——
       升序第一个候选就是答案, 人造反例 a|[ab]+c 那种 2× 退化在真表上不发生。
内存   本路的常驻是"反向单条 set"(vp1), 路 A 是"反向单条"(rev1), 同一个量级:
       最大的那张 158 条表上 89 条被真问到位置, vp1 9.6MB vs rev1 7.6MB (1.26×)。
       相对路 B 是【净增】—— B 只要正向单条, 一条反向都不建。这一笔是这次换路的
       全部代价, 量它用 (*RegexpSet).ViableOneStats()。

更早那一笔 (2026-08-27): 两条老路的第二趟原先都回整表 set.ResolveSpan, 换成单条对象之后 答案一字不差而价钱降了 27~36% (TestMatchScanPathsSameAsSetRoute)。那次换的是"回不回 set", 这次换的是"起点怎么找回来", 是两件事。

Scan 报了 err 就整篇走老路 FindAll —— 库这边宁可退回去也不给一个"像是对的"答案。 老路要是想从某个偏移接着扫, 【别切片】: (*Regexp).FindStringIndexFrom(text, pos) 参数是 原串偏移, \b / ^ / $ 看到的还是真邻居 (见 find_from.go)。 配了 boolOnly 的那几条从来不参与收口, 也就无所谓补不补。

生命周期同 FindAllIndex 的 alloc: 可复用工作区, 热路径上建一次留着,【不是并发安全的】。 text 只在 Scan 那一遍里被引用 (补左端要读它), Scan 返回之后这一层不再持有它。

matchscan_reverse.go —— 反向 MatchScanner: 从正文【末尾往前】一遍扫, 边扫边一批一批交出 各条 pattern 的不重叠命中区间。口径是 rightmost-longest。

🔴 一句话: 它是 matchscan.go 那个的【镜像】, 两处不一样 ——

① 交出来的区间按 Lo 【降序】(正向是升序);
② 去重叠的口径是 rightmost-longest (正向是 leftmost-longest)。
两种口径没有本质区别, 只是结果不一样: 同一片正文上"谁先占坑"从左边换到了右边。
全文见 doc/MatchScanner的leftmost-longest保证.md 第 8 节。

── 为什么会有这一层 ────────────────────────────────────────────────────────

有一族 pattern 正着扫状态数对计数上界指数增长, 反着读就塌回线性 —— `S B{m,n} L` 里起始类 严格窄于重复类那一族 (doc/状态数为什么会相乘.txt §3, 实测同一条 pattern 正向 66572 状态 / 8.39MB, 反向 42 状态 / 0.07MB)。这种表本来就该反着扫。可在这一层补上之前, 反向 set 只回答 得了"命中没有 / 哪几条命中", 要位置就得命中之后再正向扫一遍全文 —— 而"把 1+k 遍压成 1 遍" 正是 MatchScanner 存在的全部意义。这一层把那一遍补回来。

── 反向【更好】做, 不是更难做 ──────────────────────────────────────────────

正向那一遍 DFA 交出来的是匹配的【右端】, 起点得在 Go 这侧【回推】—— matchscan.go 里那一节 "端点怎么补"讲的就是这件事的代价 (反向种全部状态收候选 + 升序逐个正向锚定验)。 反向交出来的是【左端】= 起点, 而 leftmost/rightmost-longest 这个口径本来就定义在起点上, 所以这一层连那一步都没有:

反向 set FindAllIndex                                  → 匹配左端, 按扫描方向 (从右往左) 单调
正向【单条】FindStringIndexAtWithin(from=左端, bound=游标) → 【最长】右端, 且绝不越过游标

于是: 没有"收候选再逐个验"这一步, 也不需要 maxL 窗口。每处命中恒等于"一趟锚定搜索", 代价 = 这处命中有多长, 与正文长度无关。

🔴 补右端那一趟走的是【这一条 pattern 自己的单条对象】(longest 口径编的), 不是"一条

pattern 的 set" (2026-08-27 换掉的)。理由与正向那侧一字不差: 单条走 RE2::Match 那条
带 NFA 回退的完整路 · 状态更小 · 不去冲刷整表那份 DFA 缓存。见 regexpset.go 的
fwd1/vp1 那段。

── 为什么"从右往左"仍然一个字节都不用攒 ─────────────────────────────────────

因为口径也跟着翻了。要是硬要在反向扫描上给 leftmost-longest, 那就得攒: 手上这一处随时可能 被更靠左、还没扫到的那一处整个吃掉, 有上界的还能靠一个 maxL 宽的延迟缓冲兜住, 无上界的 (邮箱那种) 得攒到整篇扫完 —— 内存跟着正文长, 正是这一层存在的理由被赔掉。 改成 rightmost-longest 这件事就没了: 从右往左走, 【第一个见到的起点就是最终答案】, 左边不可能再来一个把它顶掉 —— 与正向"第一个见到的就是最终答案"是同一句话照镜子。 所以游标在回调里当场推完, 输出写进固定的 matchScanBatch 缓冲 (12KB), 满了就交出去、就地复用。

── 对外那条保证, 逐字读 ─────────────────────────────────────────────────────

① text[Lo:Hi] 是这条 pattern 的一个真匹配;
② 【同一条 pattern】吐的区间互不相交, 按 Lo 【降序】;
③ 口径是 rightmost-longest: 在还没被占掉的那段正文里, 反复取【起点最靠右】的那个匹配,
   同起点取【最长】, 吐出去, 再往左接着找。

三处容易读错的地方:

🔴 ② 的【降序】是这个类型与正向那个最容易写错的差别。要升序是调用方那边一句 reverse 的事,
   库这边翻就得攒 = 又是缓冲 (见上一节)。
🔴 ② 只管【单条】。跨 pattern 一概不合并, 理由与正向完全一样 (matchscan.go 那段护照号/
   身份证抢同一段的实例)。
🔴 能匹配空串的 pattern 只能配 boolOnly, SetModes 当场报错。

── rightmost-longest 与 leftmost-longest 差在哪 ─────────────────────────────

只在【两个真匹配互相交叠】的地方差, 不交叠的正文上两者逐处相同。例:

a|ab  撞 "abab"   leftmost-longest [[0,2) [2,4)]     rightmost-longest [[2,4) [0,2)]  (同一批, 顺序反)
ab|b  撞 "aab"    leftmost-longest [[1,3)]           rightmost-longest [[2,3)]        ← 这里才真差

后一种局面下"谁赢"由方向定, 两边都是【真匹配】, 都不重叠, 都不漏段。选哪个是调用方的事: 要跟 stdlib 的 re.Longest().FindAllStringIndex 逐字节对上就用正向那个; 只是要"把这片正文里 的东西都框出来"(脱敏 · 定位 · 计数), 两个都行。

生命周期同正向那个: 可复用工作区, 热路径上建一次留着,【不是并发安全的】。

patlen.go —— 每条 pattern 的【匹配字节长度区间】(min, max), 建集期算一次。

用来干什么: NewMatchScanner 拿到"这条 pattern 的匹配在第几字节结束"之后, 还得知道它从哪开始。 怎么找开始, 完全由这个区间决定, 三档差着数量级:

min == max        定长 (NRIC 9 字节 · 身份证 18 字节 · 邮编…): start = end - min, 一句减法,
                  【根本不进正则引擎】。
max  > 0          变长有上限 (\p{Han}{2,4} ≤ 12 字节): 从 end 往回最多看 max 字节, 常数代价。
max  < 0          没上限 ([a-z ]{4,} · .* · {n,}): 回看距离 = 正文长度, 这条不划算,
                  交给调用方走老路 (那条 pattern 自己扫一遍全文)。

🔴 为什么用 Go 的 regexp/syntax 解析而不是问 RE2: RE2 的 Regexp 树上没有这个量, 要么改

native 加一趟递归、要么在 Go 这侧解析。两边都是同一套语法 (Go 的 regexp 本来就是 RE2),
而这只在【建集期】跑一次 (155 条 < 1ms), 所以选了不动 native 的那条。
解析不出来 (RE2 支持而 Go 不支持的写法) 一律【当没上限】—— 保守方向: 只会让这条落回老路,
不会给出错误的 start。

prefilter.go — Prefilter: RE2 自己的「必需字面量」推导 (FilteredRE2 / PrefilterTree 的门面)。

回答三个问题:

Atoms()              这批 pattern 想命中, 正文里【必须】先出现哪些字面量?
                     (已小写化、去重 —— 拿它们去正文里找的时候要么大小写不敏感, 要么先折成小写)
Potentials(found)    正文里找到了这几个原子, 那么还有哪几条 pattern 【可能】命中?
                     没进这个名单的 pattern 【保证】不命中, 可以整条跳过。
Unfiltered()         哪几条 pattern 【没有】必需字面量, 因而任何前置粗筛都筛不掉?

── 第三个问题才是接这套出来的动机 ────────────────────────────────────────────

「先用一道便宜的字面量门挡掉大多数正文, 剩下的才进大表」是本库文档里唯一能抬高吞吐上限的方向 (doc/set性能优化经验.txt §4 E)。但这个方向有个天花板: 有些 pattern 根本没有必需字面量 —— 纯字符类驱动的, 形如 `[A-Za-z0-9+/=_-]{20,}` 或 `(?-i:\([A-Z]{2,5}\))` —— 它们无论正文长什么样 都得跑。这批的规模直接决定了粗筛能省下多少, 所以做粗筛之前必须先量它。

🔴 这个数【只有 RE2 自己的 prefilter 算得准】。手写一个"从 pattern 源串里抠字面量"的抽取器 会在 `(?:foo|[A-Z]{5})` 上答错: 它含字面量 foo, 但整条【不可过滤】—— 另一支不需要 foo, 所以 foo 不出现的正文里它照样可能命中。AND-OR 树上的这类推理没法凭直觉做对。

── 与 RegexpSet 的分工 ───────────────────────────────────────────────────────

RegexpSet 是"把 N 条编进一个 DFA, 一遍扫回答哪几条命中"—— 它自己就是答案。 Prefilter 不做匹配, 它只出【筛子】: 原子表给调用方拿去用自己的字符串匹配器 (AC 自动机 / memmem 都行) 找, 找完回来问还剩哪几条。两者是可以叠的: 先 Prefilter 缩小候选, 再拿缩小后的子集建 Set。

生命周期: 构建期一次 (每条 pattern 各编一个 RE2 + 推 AND-OR 树, 不便宜), 之后只读。 AllPotentials 内部只读遍历自己的树, 并发安全。

quotemeta.go — QuoteMeta: 把字符串里的正则元字符转义。与 stdlib regexp.QuoteMeta 逐字符等价 (照搬 stdlib 实现)。原 README 未列此函数, 但调用方常用「QuoteMeta(用户输入) 拼进 pattern 再 Compile」, 为了让 *hgmLibre2.Regexp 能完全替换 *regexp.Regexp(去掉对 stdlib regexp 的依赖)而补上。

regexpset.go — RegexpSet: 多正则「一次扫描·返回哪几条命中」的 litscan 风格 API。

动机: 调用方常把 N 条正则拼成 (?:re1)|(?:re2)|… 做一道"任一命中"快拒门, 命中后还得再逐条 跑一遍才知道是哪条。RE2::Set 把 N 条编进【一个 DFA】, 一遍扫就直接回答"哪几条命中"—— 取代那道粗门, 且把"是哪条"的信息一并拿到, 命中后不必再逐条全跑 (只需要位置的调用方再对 命中条单独取 FindStringIndex)。语义 unanchored/partial (正文任意位置出现即命中)。

⚠ 只回答"哪几条", 不回答"在哪" (无位置)。需要 fragment/offset 的调用方拿到命中 index 后,

对那几条 (通常 0 条) 各跑一次 FindStringIndex 即可。

生命周期: NewRegexpSet 构建期一次 (编译 DFA), 之后只读、并发安全; Match 传入复用 buf 即零分配。

regexpset_reverse.go —— RegexpSetReverse: 反着编译的多正则集合, 与正向的 RegexpSet 是【两个类型】, 不是一个类型上的一个开关。

── 为什么必须拆成两个类型 ────────────────────────────────────────────────── ① 两份完全不同的 DFA 状态缓存。正向和反向是两套程序 (Prog / ReverseProg), 各自挂各自的

状态缓存。混在一个对象里, MemInfo() 报的是哪一份就说不清了 —— 而"状态缓存有多大"
正是这个库唯一真正要盯的成本。

② 两边的贵法差着三个数量级, 而这件事必须写在类型上让人看见。真表实测: 155 条正向 set

扫 6.4MB = 18ms / 零 flush; 同一张表反向 = 65 秒 / arena 顶满 254MB 还在 flush
(set 里的状态数是【相乘】的, 见 doc/状态数为什么会相乘.txt)。藏在一个 Reverse() bool
后面, 调用方不会知道自己刚踩了什么。

③ 两边吐的位置【含义相反】: 正向吐匹配的右端, 反向吐左端。同一个方法名返回意思相反的

数字, 是最容易写错又最难查的那种错。分成两个类型, 参数名就能直说 (endLo/endHi 对
startLo/startHi)。

这也是本库单条正则早就在用的形状 (Regexp / RegexpReverse, 见 reverse.go)。 set 这边一直是个例外, 现在补齐。

── 只做两件事 ──────────────────────────────────────────────────────────────

Match / MatchAny      "哪几条命中" —— 与正向【逐条相同】(同一条语言换个方向读, 不是近似)
FindAllIndex          "各条的匹配【左端】落在哪一段"
ResolveSpan*          给一个右端, 锚定回推出左端 (不扫正文)

反向 set 最主要的用途【不是】扫正文, 是最后那个 ResolveSpan*: 单点、锚定、有界, 代价只跟"回看多远"有关, 与正文长度无关。要拿它扫全文之前先量一遍 MemInfo().Flushes。

replace_func_ctx.go — ReplaceAllStringFunc 的【复用已有分配】变体: 结果直接追加进调用方自己的 []byte 底, 匹配位置表挂在 ctx 上跨调用复用。同一 ctx + 同一块底反复调, 稳态零 Go 堆分配 (f 自己返回的串除外)。

动机 (2026-08-22 · 16/32/64MB 语料 memprofilerate=1 实测): (*Regexp).ReplaceAllStringFunc 每趟 要付两笔按正文线性放大的一次性分配 ——

① matchAllFlat 的 flat []int (每匹配 2*(numSubexp+1) 个 int, 而拼接只读 group0);
② 收尾 strings.Builder 的增长阶梯。裸 Builder 从 0 开始翻倍/1.25 倍地长到 N,
   累计分配收敛到 5N (Go 大切片 1.25 倍增长 ⇒ 1/(1-1/1.25)=5), 拷贝也白付 4N。

实测 engine 的 hex 解码腿在 64MB 正文上 Builder 累计 329MB = 每输入字节 4.9 字节。 (①②在 ReplaceAllStringFunc 里现在也修了 —— 见该方法注释的"一次开到位"; 但那仍是每趟一块新底, 逐段反复调的热路径要的是【连新底都不要】, 那就是本文件。)

契约同 find_ctx.go 那套: ctx 非线程安全, 并发各持一个; 零值即可用 (首次调用惰性长出 scratch)。 结果不放 ctx 里 —— 由调用方传 dst 进来, 因为这类热路径的调用方手上通常已经有一块要往上追加的底 (拿回来的切片生命周期就归调用方, 不存在"下次调用前有效"这种限制)。

reverse.go — 反着扫: 让 DFA 从正文【末尾往前】走原始 buffer。

【为什么需要它】 `S B{m,n} L` 这种【起始类窄于重复类】的计数重复 —— 典型如 `[A-Za-z][A-Za-z0-9]{2,19}key` —— 正向 DFA 的一个状态要记住"当前活跃的起点集合"。起始类 S 严格窄于重复类 B 时, 这个集合不再是 一段连续的后缀 (k+1 种), 而可以是任意子集 (2^k 种), 于是状态数对界 n 指数增长。

这不是实现问题: `(a|b)*a(a|b)^k` 在 Myhill-Nerode 意义上就需要 2^k 个状态, 【任何】保语言的 改写都消不掉。但同一条语言【反过来读】—— `(a|b)^k a(a|b)*` —— 只要 k+2 个状态。 所以真正有效的一招不是改正则, 是改方向。

【为什么由库来做, 而不是调用方自己反转】 调用方自己把 pattern 和正文都按字节反转能凑合出效果, 但有三个坑:

• 字节反转会把多字节 UTF-8 拆散 (rune 的字节序列反了就不是那个 rune);
• pattern 的反转要正确处理 ^ $ \b 与所有 concat 的嵌套, 自己写的反转器只能保证"存在性等价";
• 正文要多复制一份。

RE2 的编译器【本来就会】编反向程序 (它内部用反向 DFA 找匹配左端), concat 反序、^/$ 对调、 多字节 rune 的字节序列反编、\b 不变, 全都是现成的。本文件只是把这条既有能力接出来: 程序反着跑, 正文【原封不动】, 一个字节都不复制。

而且它比手写反转【更省】, 不只是更省事: RE2 的 Simplify 把 `x{2,19}` 展开成"必需拷贝在前、 可选嵌套在后", 编译器反序之后可选嵌套跑到了读取顺序的【前面】—— 各个起点的活跃集合于是 互相嵌套 (只取最外层) 而不构成任意子集, 状态数不炸。手写反转 pattern 文本再正向编, 必需拷贝仍在读取顺序的前面, 照炸不误。实测同一条语言同一串字节: 库的反向 17 个状态, 手写反转 25247 个 (reverse_test.go 的 TestReverseIsNotHandRolledTextReversal)。

【怎么用: 正反是两个对象】 单条走 CompileReverse (类型 RegexpReverse), 整表走 NewRegexpSetReverseMaxMem —— 都是独立对象, 不是正向对象上的一个开关。一条 pattern 的两个方向本来就是两套程序、两份 DFA 状态缓存, 而方向是每条 pattern 各自的决定, 所以哪条走哪个方向, 由调用方建对象的时候定死。

【语义边界 —— 只回答"命中没有 / 哪几条命中", 不回答"在哪"】 不提供 Find 系列, 也暂不打算补: 反向搜索只走到匹配的【左端】就停, 拿不到右端, 而且它先撞上的 是正文里靠后的那处匹配 —— 真做 Find 只能是 rightmost 语义, 与正向的 leftmost-first 是两套东西。 需要位置的调用方: 先用反向这道便宜的门筛, 极少数命中的再走一次正向 FindStringIndex, 位置语义仍然是正向那套。命中【与否】两个方向逐字相同 —— 这是"同一条语言换个方向读", 不是近似, 不缩语义。

【什么时候该用】

• 该用: pattern 里有起始类窄于重复类的计数重复, 且只需要"命中没有"。
• 不该用: 需要位置; 或 pattern 反过来才是那个坏形状 —— 方向是【每条 pattern 各自】的决定, 不是全局开关。镜像的一对: `(?s).{20}key` 正向 21 个状态 / 反向 1 个, `key(?s).{20}` 反过来 (reverse_test.go 的 TestReverseDirectionIsPerPattern 钉着这两个数)。 哪个方向便宜, 拿真语料量一遍 Flushes/StatesEnd 就知道 (见 RegexpReverse.MatchStats)。
• 单条 RegexpReverse 的路径【一定走 DFA】, 不走 RE2 对短文本的 bitstate/onepass 快路径, 所以小正文上它不一定比正向 Regexp.MatchString 快。它换的是状态数, 不是常数。

spanresolve.go —— ResolveSpan: 给一个端点, 锚定推出【另一端】。单点、有界、不扫正文。

── 它和 FindAllIndex 是两件事, 别按对称的样子去用 ────────────────────────── FindAllIndex 是【扫一遍正文】: 代价跟正文一样长。 ResolveSpan 是【在一个点上问一句】: "从这儿起(或到这儿止), 这条 pattern 能伸到哪?"

代价 = 这处命中实际能延伸多远, 与正文长度【无关】。1KB 的正文和 6.4MB 的
正文上问同一句话, 价钱一样。

🔴 所以"补一处命中的另一端"要用它, 不要用反向 FindAllIndex 去扫一遍。反向扫全表在 6.4MB

上是 65 秒 (正向 18ms); 拆成一条一条反着扫倒是便宜, 可命中 k 条就是 k 遍全文 ——
正好是 FindAllIndex 存在的意义 (把 1+k 遍压成 1 遍) 被原样赔回去。

🔴 也不要在 Go 这侧自己补。自己补只能另编一条 \A(?:pat) 的锚定正则 —— 每条 pattern 一个

Regexp 对象、一份独立的 DFA 缓存, 还得手工保证它和 set 里那条语义一致; 而【非锚定】的
那种补法 (拿原正则去扫 text[from:]) 更糟: `.*?` 前缀让每个位置都能当起点, 状态数对
计数上界指数增长 (doc/状态数为什么会相乘.txt 里同形状差 967 倍), 等于把 FindAllIndex
刚省下来的又赔回去。这里走的是 set 自己那份程序、那份 DFA 缓存, 且真锚定。

spanviable.go —— ViableStarts: 给一个匹配【右端】, 把它左边全部【候选起点】收下来。

── 它和 ResolveSpan 差在哪 (只有一处, 但是决定性的) ───────────────────────── 反向 set 的 ResolveSpan : 反向机器只种【accept】—— 回答的是"哪些 s 使 text[s,e) 【正好】

是一个匹配", 而且只给最靠左的那一个。

反向 set 的 ViableStarts : 反向机器种【全部指令】—— 回答的是"哪些 s 起头的匹配【路过】了 e",

即 text[s,e) 是个【可行前缀】(还能被某个后缀补成真匹配)。
后者是前者的超集, 而且【全部】给出来。

🔴 为什么需要这个超集: 门 (正向 set + kManyMatch) 只给右端, 不给"起点终点的配对"。

拿最小的那个右端去只种 accept 地回推, 得到的起点未必是真正的最左起点 ——
  \b(?:ab cd ef|cd)\b 撞 "ab cd ef": 门给的最小右端是 "cd" 那一处的右端 (偏移 5),
  只种 accept 只能回推到 3 ("cd" 的左端); 而真正的 leftmost 起点是 0 ——
  text[0:5) = "ab cd" 【不是】匹配, 但它是可行前缀 (再补 " ef" 就成了)。
种全部状态才看得见 0 这个候选。2026-08-28 之前 MatchScanner 有一档 spanFast 走的正是
"只种 accept"那条路 (老的"路 A"), 上面这个例子就是它那个"第三种口径"的病根;
整档删了, 现在 MatchScanner 补起点【只走本函数这一条路】, 换来的就是严格 leftmost-longest。

🔴 为什么"种全部状态"就等于可行前缀 (证明): 反向 set 的程序 R 认的是 reverse(L)。

这一趟从 e 往左吃字节, 吃进去的串正好是 reverse(text[s,e)); 种子是 R 的【全部活状态】,
终点是 R 的 accept。于是 —— 走到 accept ⟺ 存在某个活状态 q 能吃着 reverse(text[s,e))
走到 accept ⟺ reverse(text[s,e)) 是 reverse(L) 里某个词的【后缀】 ⟺ text[s,e) 是 L 里
某个词的【前缀】= 可行前缀。∎

🔴 这一步与 ResolveSpan 一样【只能在库里做】: 种全部状态要的是 DFA 起始状态的构造权

(re2_dfa.cc 的 start_[kStartViable]), 从外面根本够不着。

代价 = 这处命中能往回够多远 (可行前缀集合空了机器就死, 当场收工), 与正文长度无关 —— 与 ResolveSpan 同一个量级, 同一个道理。

split.go — Split: 按匹配处切分字符串。与 stdlib regexp.Split 逐字符等价 (照搬 stdlib 实现 · 内部走本库 FindAllStringIndex)。原 README 未列, 为让 *hgmLibre2.Regexp 完全替换 *regexp.Regexp 而补上。

zexp_step_alloc.go — 【已定案 · 留作证据】"批缓冲从哪来" 这道题的两个落选方案的实现。 三条路的匹配循环完全相同 (同一段 C 代码或其孪生), 唯一差别是【批缓冲从哪来】:

主线  StepAllStringSubmatchIndex     库内 sync.Pool 持有 Go 缓冲 (match_step.go · 原实验里的变体 E)
B 落选 stepAllCAlloc                  C 侧首次命中时 malloc, 本次调用内缓存, 返回前 free
D 落选 stepAllGoLocal                 每次调用现 make 一块 Go 缓冲 (最朴素的写法)

三者对外形态一样 (re.XXX(s, n, batchFn)), 所以能同题对拍。定案理由与四方数字见 zexp_step_alloc_bench_test.go 顶部 —— 一句话: B 只比主线快 5%, 却要动 C 代码、每条提前返回 路径都得记得 free, 而且契约从"读到旧数据"降级成 use-after-free; D 是四个里最差的那个。 (原来还有个变体 A "调用方持有 *MatchStep_t 工作区" —— 它连同 MatchStep_t 一起已被主线取代,

理由见 match_step.go 头注"批缓冲从哪来"一节。)

Index

• Constants
• func DFAStatsZero()
• func PatternLenRange(pattern string) (min, max int)
• func QuoteMeta(s string) string
• type AttribInfo
• type DFAStats_t
  • func DFAStats() DFAStats_t
• type FindStringIndex_ctx_t
  • func NewFindStringIndex_ctx() *FindStringIndex_ctx_t
  • func (ctx *FindStringIndex_ctx_t) FindIndex(re *Regexp, b []byte) []int
  • func (ctx *FindStringIndex_ctx_t) FindStringIndex(re *Regexp, s string) []int
  • func (ctx *FindStringIndex_ctx_t) FindStringIndexAtWithin(re *Regexp, s string, from, bound int) []int
  • func (ctx *FindStringIndex_ctx_t) FindStringIndexFrom(re *Regexp, s string, pos int) []int
• type MatchScanMode_t
• type MatchScanStats_t
• type MatchScanner
  • func (m *MatchScanner) Close()
  • func (m *MatchScanner) Hit(i int) bool
  • func (m *MatchScanner) HitIDs() []int32
  • func (m *MatchScanner) Scan(text string, batchFn func(ms []SetMatch)) error
  • func (m *MatchScanner) SetModes(modes []MatchScanMode_t) error
  • func (m *MatchScanner) Stats() MatchScanStats_t
• type MatchScannerReverse
  • func (m *MatchScannerReverse) Close()
  • func (m *MatchScannerReverse) Hit(i int) bool
  • func (m *MatchScannerReverse) HitIDs() []int32
  • func (m *MatchScannerReverse) Scan(text string, batchFn func(ms []SetMatch)) error
  • func (m *MatchScannerReverse) SetModes(modes []MatchScanMode_t) error
• type PatternCost
• type Prefilter
  • func NewPrefilter(patterns []string, minAtomLen int, maxMem int64) (*Prefilter, error)
  • func (p *Prefilter) Atoms() []string
  • func (p *Prefilter) FreeC()
  • func (p *Prefilter) GetPatternLen() int
  • func (p *Prefilter) Potentials(atomIdx []int32) []int32
  • func (p *Prefilter) Unfiltered() []int32
• type Regexp
  • func Compile(pattern string) (*Regexp, error)
  • func CompileLongest(pattern string) (*Regexp, error)
  • func CompileLongestMaxMem(pattern string, maxMem int64) (*Regexp, error)
  • func CompileMaxMem(pattern string, maxMem int64) (*Regexp, error)
  • func MustCompile(pattern string) *Regexp
  • func MustCompileLongest(pattern string) *Regexp
  • func (re *Regexp) AppendAllStringIndexFlat(dst []int, s string, n int) []int
  • func (find *Regexp) AppendFindReplaceWithin(dst []byte, strip *Regexp, src, repl string) ([]byte, bool)
  • func (re *Regexp) Find(b []byte) []byte
  • func (re *Regexp) FindAll(b []byte, n int) [][]byte
  • func (re *Regexp) FindAllIndex(b []byte, n int) [][]int
  • func (re *Regexp) FindAllString(s string, n int) []string
  • func (re *Regexp) FindAllStringIndex(s string, n int) [][]int
  • func (re *Regexp) FindAllStringSubmatch(s string, n int) [][]string
  • func (re *Regexp) FindAllStringSubmatchIndex(s string, n int) [][]int
  • func (re *Regexp) FindAllSubmatch(b []byte, n int) [][][]byte
  • func (re *Regexp) FindAllSubmatchIndex(b []byte, n int) [][]int
  • func (re *Regexp) FindIndex(b []byte) []int
  • func (find *Regexp) FindReplaceWithin(strip *Regexp, src, repl string) string
  • func (find *Regexp) FindReplaceWithinBytes(strip *Regexp, src, repl []byte) []byte
  • func (re *Regexp) FindString(s string) string
  • func (re *Regexp) FindStringIndex(s string) []int
  • func (re *Regexp) FindStringIndexAtWithin(s string, from, bound int) []int
  • func (re *Regexp) FindStringIndexFrom(s string, pos int) []int
  • func (re *Regexp) FindStringSubmatch(s string) []string
  • func (re *Regexp) FindStringSubmatchIndex(s string) []int
  • func (re *Regexp) FindStringSubmatchIndexWithin(s string, from, bound int) []int
  • func (re *Regexp) FindSubmatch(b []byte) [][]byte
  • func (re *Regexp) FindSubmatchIndex(b []byte) []int
  • func (re *Regexp) FreeC()
  • func (re *Regexp) Match(b []byte) bool
  • func (re *Regexp) MatchString(s string) bool
  • func (re *Regexp) MaxMem() int64
  • func (re *Regexp) NumSubexp() int
  • func (re *Regexp) ReplaceAll(src, repl []byte) []byte
  • func (re *Regexp) ReplaceAllFunc(src []byte, f func([]byte) []byte) []byte
  • func (re *Regexp) ReplaceAllString(src, repl string) string
  • func (re *Regexp) ReplaceAllStringFunc(src string, f func(string) string) string
  • func (re *Regexp) Split(s string, n int) []string
  • func (re *Regexp) StepAllStringIndex(s string, n int, batchFn func(flat []int32) bool)
  • func (re *Regexp) StepAllStringIndexCAlloc(s string, n int, batchFn func(flat []int32) bool)
  • func (re *Regexp) StepAllStringSubmatchIndex(s string, n int, batchFn func(flat []int32) bool)
  • func (re *Regexp) StepAllStringSubmatchIndexCAlloc(s string, n int, batchFn func(flat []int32) bool)
  • func (re *Regexp) StepAllStringSubmatchIndexGoLocal(s string, n int, batchFn func(flat []int32) bool)
  • func (re *Regexp) String() string
  • func (re *Regexp) SubexpNames() []string
• type RegexpReverse
  • func CompileReverse(pattern string) (*RegexpReverse, error)
  • func CompileReverseMaxMem(pattern string, maxMem int64) (*RegexpReverse, error)
  • func MustCompileReverse(pattern string) *RegexpReverse
  • func (rr *RegexpReverse) FreeC()
  • func (rr *RegexpReverse) Match(b []byte) bool
  • func (rr *RegexpReverse) MatchStats(s string, st *ScanStats) bool
  • func (rr *RegexpReverse) MatchString(s string) bool
  • func (rr *RegexpReverse) MaxMem() int64
  • func (rr *RegexpReverse) MemInfo() SetMemInfo
  • func (rr *RegexpReverse) ResolveSpanWithin(text string, from, bound int32) (pos int32, ok bool, err error)
  • func (rr *RegexpReverse) String() string
• type RegexpSet
  • func NewRegexpSet(patterns []string) (*RegexpSet, error)
  • func NewRegexpSetMaxMem(patterns []string, maxMem int64) (*RegexpSet, error)
  • func (s *RegexpSet) Attrib() AttribInfo
  • func (s *RegexpSet) FindAllIndex(text string, alloc *RegexpSet_FindAllIndex_Alloc_t, ...) error
  • func (s *RegexpSet) FindAllIndexBytes(text []byte, alloc *RegexpSet_FindAllIndex_Alloc_t, ...) error
  • func (s *RegexpSet) GetPatternLen() int
  • func (s *RegexpSet) Match(text string, buf []int32) []int32
  • func (s *RegexpSet) MatchAny(text string) bool
  • func (s *RegexpSet) MatchAnyBytes(text []byte) bool
  • func (s *RegexpSet) MatchBytes(text []byte, buf []int32) []int32
  • func (s *RegexpSet) MatchStats(text string, buf []int32, st *ScanStats) []int32
  • func (s *RegexpSet) MatchStatsBytes(text []byte, buf []int32, st *ScanStats) []int32
  • func (s *RegexpSet) MemInfo() SetMemInfo
  • func (s *RegexpSet) NewFindAllIndexAlloc() (*RegexpSet_FindAllIndex_Alloc_t, error)
  • func (s *RegexpSet) NewMatchScanner() (m *MatchScanner, unsupported []int32, err error)
  • func (s *RegexpSet) PatternLenRange(i int) (min, max int)
  • func (s *RegexpSet) ResolveSpan(text string, from, id int32) (pos int32, ok bool, err error)
  • func (s *RegexpSet) ResolveSpanBytes(text []byte, from, id int32) (pos int32, ok bool, err error)
  • func (s *RegexpSet) ResolveSpanWithin(text string, from, bound, id int32) (pos int32, ok bool, err error)
  • func (s *RegexpSet) ViableOneStats() (n int, states, arenaCap int64)
• type RegexpSetReverse
  • func NewRegexpSetReverseMaxMem(patterns []string, maxMem int64) (*RegexpSetReverse, error)
  • func (r *RegexpSetReverse) Attrib() AttribInfo
  • func (r *RegexpSetReverse) FindAllIndex(text string, alloc *RegexpSet_FindAllIndex_Alloc_t, ...) error
  • func (r *RegexpSetReverse) FindAllIndexBytes(text []byte, alloc *RegexpSet_FindAllIndex_Alloc_t, ...) error
  • func (r *RegexpSetReverse) GetPatternLen() int
  • func (r *RegexpSetReverse) Match(text string, buf []int32) []int32
  • func (r *RegexpSetReverse) MatchAny(text string) bool
  • func (r *RegexpSetReverse) MatchAnyBytes(text []byte) bool
  • func (r *RegexpSetReverse) MatchBytes(text []byte, buf []int32) []int32
  • func (r *RegexpSetReverse) MatchStats(text string, buf []int32, st *ScanStats) []int32
  • func (r *RegexpSetReverse) MatchStatsBytes(text []byte, buf []int32, st *ScanStats) []int32
  • func (r *RegexpSetReverse) MemInfo() SetMemInfo
  • func (r *RegexpSetReverse) NewFindAllIndexAlloc() (*RegexpSet_FindAllIndex_Alloc_t, error)
  • func (r *RegexpSetReverse) NewMatchScanner() (m *MatchScannerReverse, unsupported []int32, err error)
  • func (r *RegexpSetReverse) ResolveSpan(text string, from, id int32) (pos int32, ok bool, err error)
  • func (r *RegexpSetReverse) ResolveSpanBytes(text []byte, from, id int32) (pos int32, ok bool, err error)
  • func (r *RegexpSetReverse) ResolveSpanWithin(text string, from, bound, id int32) (pos int32, ok bool, err error)
  • func (r *RegexpSetReverse) ViableStarts(text string, from, bound, id int32, out []int32) (n int, err error)
• type RegexpSet_FindAllIndex_Alloc_t
  • func (a *RegexpSet_FindAllIndex_Alloc_t) Close()
• type RegexpSet_FindAllIndex_Run_t
• type ReplaceAllStringFunc_ctx_t
  • func NewReplaceAllStringFunc_ctx(nMatchHint int) *ReplaceAllStringFunc_ctx_t
  • func (ctx *ReplaceAllStringFunc_ctx_t) AppendReplaceAllStringFunc(dst []byte, re *Regexp, src string, f func(string) string) ([]byte, bool)
• type ScanStats
• type SetMatch
• type SetMemInfo
  • func (m SetMemInfo) Used() int64

Constants

const DefaultMaxMem int64 = 8 << 20

DefaultMaxMem 是 RE2 的默认内存预算 (RE2::Options::kDefaultMaxMem = 8MB)。 Compile 用的就是它; CompileMaxMem 传 <=0 也回落到它。

const DefaultSetMaxMem int64 = DefaultMaxMem

DefaultSetMaxMem 是 RE2 的默认内存预算 (RE2::Options::kDefaultMaxMem = 8MB)。 NewRegexpSet 用的就是它; NewRegexpSetMaxMem 传 <=0 也回落到它。

const PatLenUnbounded = -1

PatLenUnbounded 是 max 的"没有上限"取值。

Functions

func DFAStatsZero

func DFAStatsZero()

DFAStatsZero 把四个计数归零, 便于分段测量 (标定循环每轮开头调一次)。 进程级共享: 并发跑多组测量时归零会互相踩, 那种场合请改用取差值。

func PatternLenRange

func PatternLenRange(pattern string) (min, max int)

PatternLenRange 算一条 pattern 的匹配字节长度区间。max = PatLenUnbounded 表示没有上限。 pattern 解析不了时返回 (0, PatLenUnbounded) —— 与"没上限"同一档, 调用方照兜底路走即可。

func QuoteMeta

func QuoteMeta(s string) string

QuoteMeta 返回把 s 中所有正则元字符转义后的字符串 (匹配该串字面量的 pattern)。 与 stdlib regexp.QuoteMeta 逐字符一致。元字符全是 ASCII, 故按字节遍历即正确。

Types

type AttribInfo

type AttribInfo struct {
	Enabled bool // 编译时没开 RE2_DFA_ATTRIB
	Built   bool // 这个 Set 还没扫过 (DFA 都没建)
	// StatesTotal 是生涯建过的状态数 (同 SetMemInfo.StatesBuiltTotal)。
	StatesTotal int64
	// SharedInsts 是落在"多条 pattern 共用"指令上的次数。最典型的是非锚定搜索开头
	// 那个 .* 循环 —— 它每条 pattern 都能走到, 归不了因, 所以单列。
	SharedInsts int64
	// NInstSum/StatesTotal = 平均"状态宽度"。宽度就是【单个状态的造价】:
	// 建一个状态是 O(ninst), 所以这两个数回答的是"状态变多了"还是"状态变胖了"。
	NInstSum int64
	NInstMax int64
	// NInstHist[i] = ninst 落在 [2^i, 2^(i+1)) 的状态数。
	NInstHist [16]int64
	// BirthHist[i] = 建状态时读到了正文的第 i/64 段。
	// 平坦 = 全篇都在造状态 (缓存对这种语料没用); 集中在几个桶 = 有特定文本形态在触发。
	BirthHist [64]int64
	// Pats 按 Excess 降序 (Excess 相同再看 Insts), 只含 States>0 的条目。
	Pats []PatternCost
}

AttribInfo 回答"这几万个状态是谁造的、有多贵、在正文哪一段造的"。

🔴 要 CGO_CXXFLAGS=-DRE2_DFA_ATTRIB=1 编译才有数据, 否则 Enabled=false。 默认构建里这套采集代码根本不存在 (零字段零开销), 因为它只在排障时才有用。

⚠ 这是【归因】不是【因果】: "状态里有 #31 独占的指令"不等于"这个状态是 #31 一条造成的"。 共现型 pattern 的状态本来就是多条 pattern 的位集乘出来的, 这里给的是排序, 不是分解。

type DFAStats_t

type DFAStats_t struct {
	// Resets 是 DFA::ResetCache 的累计次数 —— 状态缓存被整表清空重建了几次。
	// 这是 thrash 的直接读数: 一批扫描期间它不涨 = 预算够; 每扫必涨 = 预算不够。
	Resets uint64
	// SearchFailures 是 DFA 放弃搜索的累计次数。单条 Regexp 撞上会退回 NFA (慢一个数量级,
	// 结果仍对); RegexpSet 不会走到这里 (RE2 对 kManyMatch 禁掉了 bail, 只 flush, 见 README)。
	SearchFailures uint64
	// LastStateBudget 是最近一次 Resets 发生时, 那个 DFA 的状态缓存预算 (字节)。
	// 它约等于 maxMem 扣掉编译期程序占用后剩下的那一半 —— 想知道"现在实际有多少额度给状态", 看它。
	LastStateBudget int64
	// LastCacheStates 是最近一次 Resets 发生时缓存里的状态数 (即这一次清掉了多少个状态)。
	// 它是 working set 的下界样本: 预算撑不住这么多状态, 才会有这次清空。
	LastCacheStates int64
}

DFAStats_t 是一份 DFA 状态缓存计数快照 (字段名与 C 侧 cre2_dfa_stats 逐字一致)。 四个字段各自无锁读取, 相互之间不保证同一瞬间 (并发下 Resets 与 Last* 可能差一拍)。

func DFAStats

func DFAStats() DFAStats_t

DFAStats 取一份当前快照。开销 = 四次原子读, 可以随便调。

type FindStringIndex_ctx_t

type FindStringIndex_ctx_t struct {
	// contains filtered or unexported fields
}

FindStringIndex_ctx_t 持有 FindStringIndex 复用所需的全部 scratch: cbuf 是喂给 cgo 回填 group0 [start,end) 的 C.int 缓冲 (len 2); ret 是返回给调用方的 [start,end]。 零值即可用 (首次调用惰性分配 cbuf); 也可用 NewFindStringIndex_ctx 预分配。

func NewFindStringIndex_ctx

func NewFindStringIndex_ctx() *FindStringIndex_ctx_t

NewFindStringIndex_ctx 预分配好 scratch, 返回一个可复用的 ctx。

func (*FindStringIndex_ctx_t) FindIndex

func (ctx *FindStringIndex_ctx_t) FindIndex(re *Regexp, b []byte) []int

FindIndex 同上面的 FindStringIndex, 但正文是 []byte (零拷贝喂给同一内核, 不做 string(b) 全量拷贝)。 方法名对齐 stdlib 的 FindIndex ↔ FindStringIndex 对应关系。返回切片同样切自 ctx.ret, 仅在下次用本 ctx 调用前有效。

func (*FindStringIndex_ctx_t) FindStringIndex

func (ctx *FindStringIndex_ctx_t) FindStringIndex(re *Regexp, s string) []int

FindStringIndex 同 (*Regexp).FindStringIndex, 但复用 ctx 的 scratch 缓冲, 单线程顺序反复调 稳态零分配. 返回最左匹配的 [start,end) (切自 ctx.ret · 仅在下次用本 ctx 调用前有效), 无匹配返回 nil。

func (*FindStringIndex_ctx_t) FindStringIndexAtWithin

func (ctx *FindStringIndex_ctx_t) FindStringIndexAtWithin(re *Regexp, s string, from, bound int) []int

FindStringIndexAtWithin 同上, 但复用 ctx 的 scratch (单线程顺序反复调稳态零分配)。 返回的切片切自 ctx.ret, 仅在【下次用同一 ctx 调用前】有效。

func (*FindStringIndex_ctx_t) FindStringIndexFrom

func (ctx *FindStringIndex_ctx_t) FindStringIndexFrom(re *Regexp, s string, pos int) []int

FindStringIndexFrom 同上, 但复用 ctx 的 scratch (单线程顺序反复调稳态零分配)。 返回的切片切自 ctx.ret, 仅在【下次用同一 ctx 调用前】有效。

与 ctx.FindStringIndex 的差别只有 startpos 一个参数 —— nmatch=1 同样只回填 group0, 不进 findWithin 那条"按子组个数现 make 两块"的路。

type MatchScanMode_t

type MatchScanMode_t string

MatchScanMode_t 是【每条 pattern 要什么】。两态, 零值 = 默认档 (要区间)。 全文见文件头"两态旋钮"那一节。

const MatchScanMode_boolOnly MatchScanMode_t = "boolOnly"

MatchScanMode_boolOnly 只要"命中没命中", 一处区间都不收口、一次端点都不补。 这几条照样进命中表 (Hit/HitIDs), 只是一处区间都不给。

🔴 这是最值钱的一档: 真表上 57% 的游程来自两条只当 bool 用的宽 pattern, 这一挡挡掉的是

它们的端点补全 (真花钱的那步), 不只是内存。

const MatchScanMode_span MatchScanMode_t = ""

MatchScanMode_span 要区间, 无条件 leftmost-longest (定长走减法, 变长走"回推候选 + 升序验", 两者必然同解)。零值就是它 —— mask 里没显式写的那几条都是这一档。

type MatchScanStats_t

type MatchScanStats_t struct {
	Walks int64 // 反向走了几趟 (= 处理了几个"没被游标盖住的"右端)
	Cands int64 // 这些趟一共给出多少候选起点
	Tries int64 // 一共拿正向锚定验了几次 (<= Cands; 命中即停)
	Emits int64 // 交出去几处区间 (含定长条)
}

MatchScanStats_t 是一遍 Scan 的账。加它是因为变长条的钱全在"验了几个假候选"上, 而那一笔从外面一个字都看不见 —— 没有这几个数就没法判断某张表的形状适不适合这条路。

🔴 前三个【只统计变长条】: 定长条走 e-minL 那句减法, 一次回看都不做, 不进这三个分母。

Emits 不一样, 它数的是【全部】吐出去的区间 (定长的也算) —— 所以要看"平均验了几次"
得用 Tries/Walks, 拿 Tries/Emits 会被定长条稀释成假象。

type MatchScanner

type MatchScanner struct {
	// contains filtered or unexported fields
}

MatchScanner 是可复用工作区。用 (*RegexpSet).NewMatchScanner 开, 不用了 Close。

func (*MatchScanner) Close

func (m *MatchScanner) Close()

Close 释放底层的 FindAllIndex 工作区。可重复调。

func (*MatchScanner) Hit

func (m *MatchScanner) Hit(i int) bool

Hit 报第 i 条上一次 Scan 有没有命中 (O(1) 查表)。

func (*MatchScanner) HitIDs

func (m *MatchScanner) HitIDs() []int32

HitIDs 返回上一次 Scan 命中过的 pattern 下标 (无序 · 不重复), 与 Set.Match 给的是同一张表。 切片下次 Scan 会被覆写。

func (*MatchScanner) Scan

func (m *MatchScanner) Scan(text string, batchFn func(ms []SetMatch)) error

Scan 扫 text 一遍 —— 这是【唯一】一遍全文。命中区间攒够一批 (matchScanBatch 处) 就调一次 batchFn; 扫完把不足一批的余数也交出去。全程没有任何命中就一次都不调。 返回之后 HitIDs/Hit 可用。

🔴 交给 batchFn 的切片是内部缓冲本身, 下一批原地覆写 —— 要留就 append 走。 🔴 各条 pattern 的结果是【交错】着来的 (同一条内部按 Lo 升序), 不按 pattern 分组。

🔴 【要么全给, 要么整遍报错】—— 没有"这几条没给全, 你自己补"这种中间态。返回 err 的时候

这一遍不算数 (交出去的批次也不算), 调用方就整篇走老路 FindAll。三种 err:
  ① 底下那一遍 FindAllIndex 自己失败 (native 侧 DFA 预算不够);
  ② 某条 pattern 补端点要的那个【单条对象】编不出来 —— 配置错 (maxMem 太小), 与正文
     无关, 把 maxMem 调大即可;
  ③ 某条 pattern 的锚定解析失败 (DFA 放弃) —— 同样是 maxMem 的事。
另有一种"游程乱序", 那是【库内不变量崩了】= 本库的 bug, 也从这里以 err 交出来。
能匹配空串的那几条不在这里出现: NewMatchScanner 建的时候就报过名单了。

batchFn 传 nil 合法: 只要命中表 (等价于 Set.Match), 一处区间都不收口。

func (*MatchScanner) SetModes

func (m *MatchScanner) SetModes(modes []MatchScanMode_t) error

SetModes 声明每条 pattern 要什么 (下标即 pattern 下标, 长度不足的按零值 = 默认档)。 传 nil = 全默认档。调用方那边这是【静态】信息, 建集的时候就知道, 热路径上不该每遍改。

🔴 NewMatchScanner 报出来的 unsupported 那几条只允许配 boolOnly, 否则这里【当场报错】——

那张名单就是给这一步对的。理由见 NewMatchScanner。

func (*MatchScanner) Stats

func (m *MatchScanner) Stats() MatchScanStats_t

Stats 返回上一次 Scan 的账 (见 MatchScanStats_t)。

type MatchScannerReverse

type MatchScannerReverse struct {
	// contains filtered or unexported fields
}

MatchScannerReverse 是反向那一侧的可复用工作区。用 (*RegexpSetReverse).NewMatchScanner 开, 不用了 Close。字段含义与 MatchScanner 一一对应, 差别只在 cur 的方向。

func (*MatchScannerReverse) Close

func (m *MatchScannerReverse) Close()

Close 释放底层的 FindAllIndex 工作区。可重复调。

func (*MatchScannerReverse) Hit

func (m *MatchScannerReverse) Hit(i int) bool

Hit 报第 i 条上一次 Scan 有没有命中 (O(1) 查表)。

func (*MatchScannerReverse) HitIDs

func (m *MatchScannerReverse) HitIDs() []int32

HitIDs 返回上一次 Scan 命中过的 pattern 下标 (无序 · 不重复), 与 RegexpSetReverse.Match 给的是同一张表。切片下次 Scan 会被覆写。

func (*MatchScannerReverse) Scan

func (m *MatchScannerReverse) Scan(text string, batchFn func(ms []SetMatch)) error

Scan 从末尾往前扫 text 一遍 —— 这是【唯一】一遍全文。命中区间攒够一批 (matchScanBatch 处) 就调一次 batchFn; 扫完把不足一批的余数也交出去。全程没有任何命中就一次都不调。 返回之后 HitIDs/Hit 可用。

🔴 交给 batchFn 的切片是内部缓冲本身, 下一批原地覆写 —— 要留就 append 走。 🔴 各条 pattern 的结果是【交错】着来的, 同一条 pattern 内部按 Lo 【降序】(不是升序)。

🔴 【要么全给, 要么整遍报错】, 与正向同解 —— 没有"这几条没给全, 你自己补"的中间态。

err 的三种来由 (底下那遍 FindAllIndex 失败 / 单条正向 set 编不出来 / 锚定解析放弃)
都是 maxMem 的事, 与正文无关; 另有"游程乱序"= 本库 bug, 也从这里交出来。
能匹配空串的那几条不在这里出现: NewMatchScanner 建的时候就报过名单了。

batchFn 传 nil 合法: 只要命中表 (等价于 RegexpSetReverse.Match), 一处区间都不收口。

func (*MatchScannerReverse) SetModes

func (m *MatchScannerReverse) SetModes(modes []MatchScanMode_t) error

SetModes 声明每条 pattern 要什么 (下标即 pattern 下标, 长度不足的按零值 = 默认档)。 传 nil = 全默认档。与正向那个逐字同解, 两档:

MatchScanMode_span      要区间 (零值 · 默认)。口径 rightmost-longest, 无条件。
MatchScanMode_boolOnly  只要"命中没命中", 一处区间都不收口、一次端点都不补。

🔴 2026-08-28 之前正向那侧多一个 MatchScanMode_spanFast, 这一侧【当场报错】把它挡掉。

那一档整个删了 (见 matchscan.go 头注), 于是这里也不再有那道闸 —— 两侧的档位从此
是同一套两态, 不必再解释"为什么这边少一档"。

🔴 能匹配空串的 pattern (PatternLenRange 的 min <= 0) 只允许配 boolOnly, 否则这里当场报错

而不是运行时静默退回老路 —— 理由同正向: 每个位置都是一处零长命中, 游标压不住。

type PatternCost

type PatternCost struct {
	// Index 是它在 Set 里的下标 (与 Match 返回的下标同一套)。
	Index int
	// States 是有多少个新建状态里出现了这条 pattern 【独占】的 NFA 指令。
	// 🔴 非锚定搜索下这个数会【饱和】: DFA 在每个位置都得考虑"新的匹配可能从这里开始",
	// 所以每条 pattern 的入口指令躺在几乎每个状态里 ⇒ 大半条 pattern 都是 100%。
	// 它只能用来看"这条 pattern 有没有参与", 不能用来排序。
	States int64
	// Insts 是那些独占指令一共出现了多少次 (按状态加权)。这才是排序该看的数:
	// 一条 pattern 只有在它的容差窗口【正在悬着】的时候, 才会往状态里塞多个零件。
	Insts int64
	// Excess = Insts - States, 即"扣掉每状态都躺着的那 1 个入口指令之后, 多出来的零件数"。
	// 这是最干净的病灶指标: 窄窗口/纯字面量的 pattern 常年 Excess≈0, 共现型才会飙上去。
	Excess int64
}

PatternCost 是一条 pattern 在"造状态"这件事上的账 (要 -DRE2_DFA_ATTRIB=1 编译)。

type Prefilter

type Prefilter struct {
	// contains filtered or unexported fields
}

Prefilter 持有一棵编译好的 AND-OR 原子树 + 每条 pattern 各一个 RE2 对象。

func NewPrefilter

func NewPrefilter(patterns []string, minAtomLen int, maxMem int64) (*Prefilter, error)

NewPrefilter 把 patterns 顺序喂进 FilteredRE2 并编译。

minAtomLen 是原子的最短长度 (<=0 用 RE2 默认): 调大 ⇒ 原子更少更长 (匹配器更快, 但更多 pattern 掉进不可过滤集); 调小 ⇒ 筛得更细但原子表膨胀、短原子在任何正文里都到处是, 筛不掉东西。 maxMem 是每条 pattern 各自那个 RE2 的预算 (<=0 = RE2 默认 8MB)。

任一条解析失败即返回 error —— 与 NewRegexpSet 一致: 静默丢掉一条会让 Potentials 的下标 与调用方的 patterns 下标错位, 那是最难查的一类错。

func (*Prefilter) Atoms

func (p *Prefilter) Atoms() []string

Atoms 返回原子表 (已小写化、去重)。下标就是 Potentials 要的那个 atom 下标。 返回的是内部切片, 【不要改写】。

func (*Prefilter) FreeC

func (p *Prefilter) FreeC()

FreeC 显式释放 native 资源 (否则靠 finalizer)。释放后不得再调本对象任何方法。

func (*Prefilter) GetPatternLen

func (p *Prefilter) GetPatternLen() int

GetPatternLen 返回 pattern 条数。

func (*Prefilter) Potentials

func (p *Prefilter) Potentials(atomIdx []int32) []int32

Potentials 给定"正文里找到的原子下标"(升序不升序都行, 重复也无所谓), 返回还可能命中的 pattern 下标 (升序)。没进名单的 pattern 保证不命中。

传 nil / 空切片 ⟹ 返回【不可过滤集】, 见 Unfiltered。

func (*Prefilter) Unfiltered

func (p *Prefilter) Unfiltered() []int32

Unfiltered 返回【一个原子都不需要】的那批 pattern 下标 —— 它们无论正文长什么样都得跑。

这个数是任何"前置字面量粗筛"方案的天花板: 筛得掉的那部分再便宜, 也省不掉这批的钱。 做粗筛之前先量它, 别先做完再发现天花板在 3%。

type Regexp

type Regexp struct {
	// contains filtered or unexported fields
}

Regexp 持有一个原生 RE2 句柄. 默认靠 finalizer 释放 (不强制 Close); 大量动态编译 pattern 想及时回收 native 内存时可显式调 FreeC.

func Compile

func Compile(pattern string) (*Regexp, error)

Compile 编译一个 RE2 正则. 编译错误返回 error (不 panic). 内存预算 = RE2 默认 DefaultMaxMem (8MB); 要自己定预算用 CompileMaxMem.

func CompileLongest

func CompileLongest(pattern string) (*Regexp, error)

CompileLongest 同 Compile, 但这条 pattern 的匹配口径是 leftmost-longest (POSIX), 而不是 RE2/PCRE 默认的 leftmost-first (贪心)。等价于 stdlib 的 re.Longest()。

func CompileLongestMaxMem

func CompileLongestMaxMem(pattern string, maxMem int64) (*Regexp, error)

CompileLongestMaxMem 同 CompileMaxMem, 但口径是 leftmost-longest。

── 两者差在哪 ─────────────────────────────────────────────────────────────── 起点【完全相同】(都是最靠左的那个能起头的位置), 只在终点上分歧: 贪心给的是 NFA 指令 优先序先撞上的那个终点, longest 给的是同一起点上最长的那个。

abc|b     撞 "abc"   贪心 [0,3)   longest [0,3)   ← 同一起点, 这里恰好同解
a|ab      撞 "ab"    贪心 [0,1)   longest [0,2)   ← 差在这儿

── 什么时候要它 ───────────────────────────────────────────────────────────── 调用方要的是"从某处起【最长】的那个匹配"时。没有它就得两趟: 先贪心搜一次定起点, 再另找 一条路 (锚定解析) 把终点重取成最长。有了它是一趟的事, 而且这一趟走的是 RE2::Match 那条 完整的路 —— DFA 放弃了还能退到 OnePass/BitState/NFA, 不像 set 那侧的锚定解析是 DFA 独一条。

🔴 【截断是有后果的】: 变长 pattern 取到短的那个终点 = 把命中截断, 下游拿 text[Lo:Hi] 去过

校验位 (身份证 · IBAN mod-97 · Luhn) 会失败, 整条真命中被自己毙掉 = 无声漏报。
所以"要位置再拿去判"的调用方一律该用这一个。

🔴 longest 是【编译期】的事 (它定的是 RE2 内部搜索的 MatchKind), 所以它是另一个对象,

不是某个方法上的开关。要两个口径就编两个对象。

func CompileMaxMem

func CompileMaxMem(pattern string, maxMem int64) (*Regexp, error)

CompileMaxMem 同 Compile, 但显式指定这一条 pattern 的内存预算 maxMem (字节; <=0 = 用默认 8MB)。

maxMem 就是 RE2::Options::max_mem, 一个旋钮同时抬两条天花板:

• 编译期: 这条 pattern 的程序指令条数上限。撞了 → 本函数返回 error ("pattern too large - compile failed"), 翻倍重试即可。
• 运行期: 剩下的额度给 DFA 状态缓存 (正向 prog 拿 2/3, 反向 prog 拿 1/3)。缓存装不下 当前语料走出来的状态集时 DFA 不是 LRU 淘汰而是【整表清空重建】—— 结果仍然正确, 所以调用方看不见任何信号, 但吞吐是几十倍的悬崖 (见 dfastats.go 的开头)。

什么时候需要动它: 单条 pattern 里有【起始类窄于重复类】的计数重复 (如 `[A-Za-z][A-Za-z0-9]{2,19}key`) 时, 正向 DFA 的状态数对界指数增长, 默认 8MB 装不下, 于是每份新正文都把缓存冲垮一次。两条出路二选一或都用:

• 把预算调大 (本函数), 用内存换掉 thrash;
• 反着扫 (CompileReverse 编一个 RegexpReverse), 让状态数从指数塌回线性 —— 这一条不花内存, 但只回答"命中没有"。

怎么标定: 拿一批【互不相同】的真语料单线程跑一遍, 看 RegexpReverse.MatchStats/ScanStats 的 Flushes 或进程级 DFAStats().Resets 增量; >0 就翻倍重来, 直到增量归零。

func MustCompile

func MustCompile(pattern string) *Regexp

MustCompile 同 Compile, 失败 panic. 对齐 go-re2/stdlib MustCompile.

func MustCompileLongest

func MustCompileLongest(pattern string) *Regexp

MustCompileLongest 同 CompileLongest, 失败 panic。

func (*Regexp) AppendAllStringIndexFlat

func (re *Regexp) AppendAllStringIndexFlat(dst []int, s string, n int) []int

🔴 待删除 (2026-08-26) —— 新代码一律改用 (*Regexp).StepAllStringIndex (match_step.go)。

理由: 本方法只干掉了「Go 侧那笔 flat + [][]int 外壳」, 干不掉 C 侧 cre2_match_all 的 std::vector 累积表 + malloc (峰值是整张命中表的两份, 纯 RSS, Go profile 上看不见); 而且它省下的累计分配是拿 live-max 换的 —— dst 这块复用缓冲会涨到历史最大命中数就再也不缩, 挂在 plan/pool 上 × 并发度常驻。step 形态两头都没有: C 直接写进 Go 缓冲, 缓冲固定一批大小。 StepAllStringIndex 与本方法语义逐处相同 (同一段 C 循环), 对拍见 match_step_test.go 的 TestStepAllStringIndex_VsFindAll —— 它每轮都拿本方法的结果再对一遍。

现在还留着只是让 7 个存量调用点能编过; 调用点全部换完、性能确认之后, 本文件连同 cre2_match_all / cre2_match_all_r 一起删。

AppendAllStringIndexFlat 把 re 在 s 上前 n 处匹配的 [start,end) 追加进 dst, 返回追加后的切片。 n < 0 = 全部。无匹配时原样返回 dst (一个元素都不追加, 同 FindAllStringIndex 返 nil 的语义)。

追加的元素成对出现: 第 k 处匹配是 dst[2k], dst[2k+1]。要复用缓冲就传 buf[:0]。 与 FindAllStringIndex 的差别只有"结果放在哪里": 匹配集合、顺序、空匹配处理逐处一致。 子组不回填 —— 要子组请用 FindAllStringSubmatchIndex。

func (*Regexp) AppendFindReplaceWithin

func (find *Regexp) AppendFindReplaceWithin(dst []byte, strip *Regexp, src, repl string) ([]byte, bool)

AppendFindReplaceWithin 把 find.FindReplaceWithin(strip, src, repl) 的结果追加进 dst, 返回 (追加后的切片, 结果与 src 相比是否真的变了)。变了的话 dst 末尾多出来的那一段就是 FindReplaceWithin 会返回的那个串; 没变的话 dst 一个字节都没多 —— 调用方该用原 src。

语义与 FindReplaceWithin 逐字节一致 (同一个 cre2_find_replace_within 内核, 同一份 changed 判据):

out, changed := find.AppendFindReplaceWithin(buf[:0], strip, src, repl)
// changed ⟺ find.FindReplaceWithin(strip, src, repl) != src
// changed ⟹ string(out) == find.FindReplaceWithin(strip, src, repl)

与 FindReplaceWithin 的唯一差别是结果落在哪: 那边每趟现开一个 Go string, 这边拷进调用方 已有的底 (cap 够就零 Go 堆分配)。C 侧那块 malloc 缓冲两边都要付, 拷完立刻 free。

🔴 返回的是调用方那块底上的视图: 再往同一块底上追加 (或把它切回 [:0]) 之后就失效, 要留存自己物化。 🔴 一律用返回值 —— cap 不够时里面换了底, 原来那个 dst 变量就落后了。

func (*Regexp) Find

func (re *Regexp) Find(b []byte) []byte

Find 返回最左匹配的字节 (b 的子切片, 零拷贝), 无匹配返回 nil。

func (*Regexp) FindAll

func (re *Regexp) FindAll(b []byte, n int) [][]byte

FindAll 返回前 n 个匹配的字节 (各为 b 的子切片) (n<0 = 全部), 无匹配返回 nil。

func (*Regexp) FindAllIndex

func (re *Regexp) FindAllIndex(b []byte, n int) [][]int

FindAllIndex 返回前 n 个匹配的 [start,end) (n<0 = 全部), 无匹配返回 nil。

func (*Regexp) FindAllString

func (re *Regexp) FindAllString(s string, n int) []string

FindAllString 返回前 n 个匹配文本 (n<0 = 全部), 无匹配返回 nil.

func (*Regexp) FindAllStringIndex

func (re *Regexp) FindAllStringIndex(s string, n int) [][]int

FindAllStringIndex 返回前 n 个匹配的 [start,end) (n<0 = 全部), 无匹配返回 nil.

func (*Regexp) FindAllStringSubmatch

func (re *Regexp) FindAllStringSubmatch(s string, n int) [][]string

FindAllStringSubmatch 返回前 n 个匹配的 (匹配+各子组文本) (n<0 = 全部), 无匹配返回 nil.

func (*Regexp) FindAllStringSubmatchIndex

func (re *Regexp) FindAllStringSubmatchIndex(s string, n int) [][]int

FindAllStringSubmatchIndex 返回前 n 个匹配的 index 区间 (n<0 = 全部), 无匹配返回 nil.

func (*Regexp) FindAllSubmatch

func (re *Regexp) FindAllSubmatch(b []byte, n int) [][][]byte

FindAllSubmatch 返回前 n 个匹配的 (匹配+各子组字节) (n<0 = 全部), 无匹配返回 nil。

func (*Regexp) FindAllSubmatchIndex

func (re *Regexp) FindAllSubmatchIndex(b []byte, n int) [][]int

FindAllSubmatchIndex 返回前 n 个匹配的 index 区间 (n<0 = 全部), 无匹配返回 nil。

func (*Regexp) FindIndex

func (re *Regexp) FindIndex(b []byte) []int

FindIndex 返回最左匹配的 [start,end), 无匹配返回 nil。

func (*Regexp) FindReplaceWithin

func (find *Regexp) FindReplaceWithin(strip *Regexp, src, repl string) string

FindReplaceWithin 等价于

find.ReplaceAllStringFunc(src, func(m string) string { return strip.ReplaceAllString(m, repl) })

但把【外层 find 逐处匹配循环 + 每处匹配内层 strip 替换】整体下沉到 C++ (cre2_find_replace_within), 全程只一次 cgo 跨界、Go 侧零 per-match 分配。算法与上式逐字一致: find 仍可零捕获组走最快 DFA, strip 仍只在【已命中段内】替换。典型用途: 去混淆还原 (find=被分隔符拆开的关键词骨架正则, strip=分隔符字符类, repl="")。

结果惰性物化: 若 src 经过替换后【逐字节没有任何变化】(最常见: 全程无匹配 / 命中但删 0 个字符), C 侧不分配也不拷贝, 本方法直接返回原 src (零分配)。仅在确有改动时才拷一次结果。

注意 repl 是 RE2 重写串 (交给 RE2 GlobalReplace), 捕获组引用用 \1..\9; 而 ReplaceAllString 的 repl 是纯字面 (不解释任何引用)。对常见的字面 repl (如 "") 二者无差别。

func (*Regexp) FindReplaceWithinBytes

func (find *Regexp) FindReplaceWithinBytes(strip *Regexp, src, repl []byte) []byte

FindReplaceWithinBytes 是 FindReplaceWithin 的 []byte 门面 (同一次 cgo 内核): 等价于 find.ReplaceAllFunc(src, func(m []byte) []byte { return strip.ReplaceAll(m, repl) }), 但外层逐处匹配循环 + 每处命中段内的 strip 替换整体下沉 C++, 全程一次 cgo、Go 侧零 per-match 分配。

repl 是 RE2 重写串 (捕获组引用用 \1..\9), 与 FindReplaceWithin 一致 —— 注意不同于 ReplaceAll 的纯字面 repl。 惰性物化同 ReplaceAll: 逐字节无改动时直接返回原 src 切片 (零分配, 不得改写), 结果为空返回 nil。

func (*Regexp) FindString

func (re *Regexp) FindString(s string) string

FindString 返回最左匹配的文本, 无匹配返回 "".

func (*Regexp) FindStringIndex

func (re *Regexp) FindStringIndex(s string) []int

FindStringIndex 返回最左匹配的 [start,end), 无匹配返回 nil.

func (*Regexp) FindStringIndexAtWithin

func (re *Regexp) FindStringIndexAtWithin(s string, from, bound int) []int

FindStringIndexAtWithin 返回【起点就是 from】的匹配 [from,end), 且整个匹配不越过 bound。 无匹配 (这条 pattern 在 from 这一点上起不了头, 或者伸不到 bound 以内) 返回 nil。 越界 (from<0 / bound>len(s) / from>bound) 当无匹配。

bound = 最远看到哪。判定用的上下文恒是【整篇 s】, 所以掐 bound 只会让答案变短, 不会让它 变错。不想掐就传 len(s)。

配 CompileLongest* 的对象用: 给的是"从 from 起、不越过 bound 的【最长】那个匹配"。

func (*Regexp) FindStringIndexFrom

func (re *Regexp) FindStringIndexFrom(s string, pos int) []int

FindStringIndexFrom 返回【起点 >= pos】的最左匹配 [start,end), 无匹配返回 nil。 pos 越界 (<0 或 >len(s)) 当无匹配。

非锚定。口径跟着 re 这个对象走: 默认编的是 leftmost-first (贪心), CompileLongest* 编的是 leftmost-longest。 🔴 两者【选同一个起点】, 只在终点上分歧 —— 所以要 leftmost-longest 的整段区间, 用一个

longest 对象调这一句就够了 (一趟)。拿贪心对象定起点、再另跑一趟锚定解析重取最长终点
也对, 但那是两趟; MatchScanner 的默认档 2026-08-27 就是这么从两趟压成一趟的。

func (*Regexp) FindStringSubmatch

func (re *Regexp) FindStringSubmatch(s string) []string

FindStringSubmatch 返回最左匹配 + 各子组文本, 无匹配返回 nil.

func (*Regexp) FindStringSubmatchIndex

func (re *Regexp) FindStringSubmatchIndex(s string) []int

FindStringSubmatchIndex 返回最左匹配 + 各子组的 index 区间, 无匹配返回 nil.

func (*Regexp) FindStringSubmatchIndexWithin

func (re *Regexp) FindStringSubmatchIndexWithin(s string, from, bound int) []int

FindStringSubmatchIndexWithin 在 s 的 [from,bound) 这一段里找最左匹配, 连子组偏移一起回填 (布局同 FindStringSubmatchIndex: 2*(numSubexp+1) 个, 没参与匹配的组是 -1)。无匹配返回 nil。 偏移都是【原串 s 上的】。越界 (from<0 / bound>len(s) / from>bound) 当无匹配。

🔴 它存在的理由与本文件头那段一字不差, 只是换个消费点: 调用方已经从别处 (比如

MatchScanner) 拿到了某一处匹配的【整段区间】 [from,bound), 现在还想要这一段里某个捕获组
的位置。没有这个入口的话只能 re.FindStringSubmatchIndex(s[from:bound]) —— 那两刀切完,
^ / $ / \b 在两端看到的是假邻居, 捕获组偏移会偏; 而捕获组偏移的下游往往是脱敏切片。

🔴 两端都收是因为调用方【本来就两端都知道】(区间是它给的), 白送的信息不用白不用:

RE2 只在这一段里搜, 越过 bound 的那部分正文一个字节都不碰。而且这道边界顺手把
"从 from 起头没匹配, 于是往后找到了另一处"这种答非所问挡在外面 —— 但挡不干净
(另一处也可能整个落在段内), 所以下面那句仍然作数。

非锚定: 调用方拿到 m 之后【仍须自己核对 m[0]/m[1] 就是那个 from/bound】—— 想问的是 "这一处的子组", 不是"这一段里随便哪一处的子组"。

func (*Regexp) FindSubmatch

func (re *Regexp) FindSubmatch(b []byte) [][]byte

FindSubmatch 返回最左匹配 + 各子组的字节 (都是 b 的子切片; 未参与的组为 nil), 无匹配返回 nil。

func (*Regexp) FindSubmatchIndex

func (re *Regexp) FindSubmatchIndex(b []byte) []int

FindSubmatchIndex 返回最左匹配 + 各子组的 index 区间, 无匹配返回 nil。

func (*Regexp) FreeC

func (re *Regexp) FreeC()

FreeC 立即释放内部的原生 RE2(C++)资源并清掉 finalizer. 用于大量动态编译 pattern、 想及时回收 native 内存而不等 GC 的场景. 释放后该 Regexp 的所有方法不可再用.

注意(故意不做防护, 由调用方保证): 非线程安全, 不可与其它方法/另一个 FreeC 并发调用; 释放后再调用任何方法是 use-after-free, 行为未定义. 不需要及时回收就别调, 交给 finalizer 兜底即可.

func (*Regexp) Match

func (re *Regexp) Match(b []byte) bool

Match 报告 b 是否含任意匹配 (非锚定)。同 MatchString, 走不取子组的快路径。

func (*Regexp) MatchString

func (re *Regexp) MatchString(s string) bool

MatchString 报告 s 是否含任意匹配 (非锚定). 走快路径, 不取子组.

func (*Regexp) MaxMem

func (re *Regexp) MaxMem() int64

MaxMem 返回这条 Regexp 编译时实际生效的内存预算 (字节)。Compile 出来的就是 DefaultMaxMem。

func (*Regexp) NumSubexp

func (re *Regexp) NumSubexp() int

NumSubexp 返回捕获组个数 (不含整体匹配).

func (*Regexp) ReplaceAll

func (re *Regexp) ReplaceAll(src, repl []byte) []byte

ReplaceAll 把每处匹配整体换成【字面】repl 并返回结果。repl 按原始字节插入, 不解释 $1/${name}/\1 —— 与 ReplaceAllString 同一内核、同一字面语义 (故同样不是 stdlib drop-in, 需捕获组展开用 ReplaceAllFunc)。

惰性物化: 全程无字节改动 (无匹配 / repl 与命中段逐字节相同) 时【直接返回原 src 切片, 零分配】, 此时返回值与 src 共享底层数组, 不得改写。确有改动时才拷一次新切片。结果为空返回 nil (同 stdlib)。

func (*Regexp) ReplaceAllFunc

func (re *Regexp) ReplaceAllFunc(src []byte, f func([]byte) []byte) []byte

ReplaceAllFunc 用 f(匹配字节) 的返回值替换所有匹配。与 ReplaceAllStringFunc 同一套匹配定位 (matchAllFlat 一次取齐所有位置, cgo 跨界只 1 次), 差别仅在这里按 []byte 拼接。

传给 f 的是 src 的子切片 (零拷贝, cap 已限到匹配末尾; stdlib 不限 cap, 本库限住以防 f 内 append 越写到 src 后续字节)。f 的返回值会被立即拷进结果缓冲, 可复用。 无匹配时【直接返回原 src 切片, 零分配】(不得改写); 结果为空返回 nil (同 stdlib)。

func (*Regexp) ReplaceAllString

func (re *Regexp) ReplaceAllString(src, repl string) string

ReplaceAllString 把每处匹配整体换成【字面】repl 并返回新串。repl 按原始字节插入, 不解释任何 转义/捕获组引用 —— 既不照搬 stdlib 的 $1/${name}/$$ 展开, 也不照搬 RE2 GlobalReplace 的 \1 重写串 (那两套都需各自的转义分析, 易错且本库无调用方需要; 见 README 的 Differences from stdlib 一节)。这意味着 ReplaceAllString 不是 stdlib *regexp.Regexp 的 drop-in —— 需要 $1 捕获展开请改用 ReplaceAllStringFunc 自行拼。

整循环 (逐处匹配 + 字面拼接) 下沉 C++ (cre2_replace_all_literal), 单次 cgo; 惰性物化: 全程无字节 改动 (无匹配 / repl 与命中段逐字节相同) 直接复用原 src, 零分配。

func (*Regexp) ReplaceAllStringFunc

func (re *Regexp) ReplaceAllStringFunc(src string, f func(string) string) string

ReplaceAllStringFunc 用 f(匹配文本) 的返回值替换所有匹配。f 是 Go 回调无法下沉 C++ (下沉需每处 匹配回调 Go, 反而增加跨界), 故拼接循环留在 Go; 但匹配位置一次取齐, cgo 调用数已从 O(匹配数) 压到 1。 取位置的那段 C 循环已按 stdlib allMatches 语义做了空匹配去重 + UTF-8 rune 推进, 故每处投递的匹配都 满足 stdlib replaceAll 的写入条件 (m1>lastMatchEnd || m0==0), 这里无条件写即与 stdlib 逐字一致。 惰性物化 (同 ReplaceAllString): 全程无字节改动 —— 无匹配, 或有匹配但每处 f 都把原文照样写回 —— 直接复用原 src 返回, 零分配。

结果底按 len(src) 【一次开够】(同 stdlib replaceAll 的 make([]byte,0,len(src)), 也同本库 []byte 门面 ReplaceAllFunc): 从 0 开始长的话累计分配收敛到 5×len(src) (Go 大切片 1.25 倍增长 ⇒ 1/(1-1/1.25)), 拷贝还白付 4 份 —— 实测 64MB 正文上是 329MB。要连这一块底都复用 (逐段反复调的热路径), 用 ReplaceAllStringFunc_ctx_t.AppendReplaceAllStringFunc 追加进自己的缓冲。

func (*Regexp) Split

func (re *Regexp) Split(s string, n int) []string

Split 把 s 按正则匹配处切开, 返回匹配之间的子串 (不含匹配本身)。n>0 最多返回 n 段 (最后一段是余下全部); n<0 返回全部。语义与 stdlib regexp.Split 一致 (含空匹配处理)。

func (*Regexp) StepAllStringIndex

func (re *Regexp) StepAllStringIndex(s string, n int, batchFn func(flat []int32) bool)

StepAllStringIndex 同 StepAllStringSubmatchIndex, 但只回填 group0 (per = 2, 本批第 k 处 = flat[2k], flat[2k+1])。

不是"取子组版的前两个"那么简单: nmatch=1 让 C 侧的 vector<StringPiece> 也从 numSubexp+1 缩到 1, 每处匹配少填 numSubexp 组区间。只要子组 (AppendAllStringIndexFlat 原来的场景) 就用这个。

func (*Regexp) StepAllStringIndexCAlloc

func (re *Regexp) StepAllStringIndexCAlloc(s string, n int, batchFn func(flat []int32) bool)

func (*Regexp) StepAllStringSubmatchIndex

func (re *Regexp) StepAllStringSubmatchIndex(s string, n int, batchFn func(flat []int32) bool)

StepAllStringSubmatchIndex 把 re 在 s 上前 n 处匹配 (n<0 = 全部) 分批交给 batchFn。

flat 的布局与 FindAllStringSubmatchIndex 的单行【逐字相同】:

per = 2*(re.NumSubexp()+1); 本批第 k 处 = flat[k*per : (k+1)*per]; 未参与的组是 -1,-1。

per 由调用方拿 re.NumSubexp()+1 现算 —— 不在回调里带 count/per (那就又是要传的结构), 调用方本来就知道自己那条正则。

🔴 flat 只在本次回调内有效: 下一批就地覆写同一块内存, 而且【本次调用一返回, 这块就还回池子了】, 别人下一次 step 会写它。要留存请自己 copy。 batchFn 返回 false = 提前停, 剩下的正文不再扫 (这是一次性 API 做不到的事)。 无匹配: batchFn 一次都不调, 且【全程零 Go 堆分配】(缓冲是借的, 不是现开的)。

匹配集合 / 顺序 / 空匹配去重推进与 FindAllStringSubmatchIndex 逐处相同 —— 同一段 C 循环, 差别只有结果落在哪、以及是不是一次吐完。对拍门见 match_step_test.go。

func (*Regexp) StepAllStringSubmatchIndexCAlloc

func (re *Regexp) StepAllStringSubmatchIndexCAlloc(s string, n int, batchFn func(flat []int32) bool)

func (*Regexp) StepAllStringSubmatchIndexGoLocal

func (re *Regexp) StepAllStringSubmatchIndexGoLocal(s string, n int, batchFn func(flat []int32) bool)

func (*Regexp) String

func (re *Regexp) String() string

String 返回编译时的源 pattern.

func (*Regexp) SubexpNames

func (re *Regexp) SubexpNames() []string

SubexpNames 返回各捕获组的名字 (下标 0 为整体匹配, 恒为 "").

type RegexpReverse

type RegexpReverse struct {
	// contains filtered or unexported fields
}

RegexpReverse 是【反着扫】的正则对象 —— 与正向的 Regexp 分开的一个类型, 各编各的。

【为什么是两个对象, 不是一个对象上的两个方法】 一条 pattern 的两个方向是【两套程序、两份 DFA 状态缓存】(缓存挂在 re2::Prog 上, 反向那份是 Regexp::CompileToReverseProg 单独编的)。方向是每条 pattern 各自的决定, 且一条 pattern 通常只 走一个方向 —— 拆成两个对象, 调用方一眼就知道自己手里这条走的是哪个方向, 内存也是一条一份, 不会出现"一个对象悄悄挂了两份缓存"。要两个方向就编两个对象。

【只回答"有没有匹配", 不回答"在哪"】 没有 Find 系列, 现在也不打算补: 反向扫天然只走到匹配的【左端】就停, 拿不到右端, 而且它先撞上的 是正文里【靠后】的那处匹配 —— 真要做 Find, 语义只能是 rightmost 一路, 与正向 Regexp 的 leftmost-first 不是一回事, 徒增两套语义。需要位置的调用方: 拿这个当便宜的门筛一道, 极少数命中的再走一次正向 Regexp 的 FindStringIndex, 位置语义仍然是正向那套。

func CompileReverse

func CompileReverse(pattern string) (*RegexpReverse, error)

CompileReverse 编译一条【反向】正则 (内存预算 = 默认 8MB; 要自己定预算用 CompileReverseMaxMem)。 编译错误返回 error (不 panic) —— 错误判定与 Compile 完全一样, 反向程序本身是首次扫描时才惰性编的。

func CompileReverseMaxMem

func CompileReverseMaxMem(pattern string, maxMem int64) (*RegexpReverse, error)

CompileReverseMaxMem 同 CompileReverse, 但显式指定内存预算 maxMem (字节; <=0 = 默认 8MB)。 含义同 CompileMaxMem: 编译期指令上限 + 运行期 DFA 状态缓存额度。反着扫的意义正是把状态数从 指数塌回线性, 所以这里通常【不需要】调大预算 —— 先按默认量一遍 MatchStats 的 Flushes 再说。

func MustCompileReverse

func MustCompileReverse(pattern string) *RegexpReverse

MustCompileReverse 同 CompileReverse, 失败 panic。

func (*RegexpReverse) FreeC

func (rr *RegexpReverse) FreeC()

FreeC 立即释放原生资源, 语义与 (*Regexp).FreeC 一致 (非线程安全, 释放后不可再用)。

func (*RegexpReverse) Match

func (rr *RegexpReverse) Match(b []byte) bool

Match 同 MatchString, 但正文是 []byte (零拷贝)。

func (*RegexpReverse) MatchStats

func (rr *RegexpReverse) MatchStats(s string, st *ScanStats) bool

MatchStats 同 MatchString, 外加把【这一次扫描】的 DFA 计数写进 st (st 可为 nil)。 st 不必预先清零。热路径上不想要这份开销就用 MatchString —— 不传 st 时 C 侧完全不统计。

这是标定"该正着扫还是反着扫"的量器: 同一条 pattern 编一个 Regexp 和一个 RegexpReverse, 拿同一批真语料各跑一遍, 比 Flushes (>0 = 在悬崖上) 与 StatesEnd (缓存里堆了多少状态), 小的那个方向就是这条 pattern 该留下的那个对象。

func (*RegexpReverse) MatchString

func (rr *RegexpReverse) MatchString(s string) bool

MatchString 报告 s 是否含任意匹配 (非锚定) —— 命中与否与正向 Regexp.MatchString 逐字相同, 只是 DFA 从正文末尾往前走【原始 buffer】(不反转正文, 不复制正文)。

反向程序在首次调用时惰性编出来 (线程安全, 预算用这条 pattern 的 MaxMem)。 万一反向程序编不出来 / 反向 DFA 中途放弃, 自动退回一次正向匹配: 答案永远正确, 只是那次没省到 状态 (退回走的是这个对象内部自己的正向程序, 也只有退回时才会建正向那份缓存)。 想知道有没有退回过, 用 MatchStats 看 FellBack。

func (*RegexpReverse) MaxMem

func (rr *RegexpReverse) MaxMem() int64

MaxMem 返回实际生效的内存预算 (字节)。

func (*RegexpReverse) MemInfo

func (rr *RegexpReverse) MemInfo() SetMemInfo

MemInfo 查这条 pattern 的【反向程序】当前那份 DFA 缓存的水位 (字段含义同 (*RegexpSet).MemInfo)。没走过反向 (程序还没惰性编出来 / DFA 还没建) 返回 Built=false —— 量具不制造被量的东西, 查询【不会】把 DFA 建出来。

用途与 (*RegexpSet).MemInfo 一样: 标定这条 pattern 反着走到底花了多少状态。 单条反向的意义正是把状态数从指数塌回线性, 所以这个数应该【很小】; 大了就是选错方向了。

func (*RegexpReverse) ResolveSpanWithin

func (rr *RegexpReverse) ResolveSpanWithin(text string, from, bound int32) (pos int32, ok bool, err error)

ResolveSpanWithin 求【左端】: 给一个匹配右端 from (不含), 返回最靠左的那个左端 pos (含), text[pos:from] 就是这条 pattern 的一个匹配。bound 是回看的左下界 (负数 = 不限)。 ok=false 表示这条 pattern 在这个右端上根本伸不出匹配 (或者伸不到 bound 以内)。

判定用的上下文恒是【整篇正文】, 所以 \b / ^ / $ 看到的永远是真实邻居; 掐 bound 只会让 答案变短, 不会让它变错。代价 = 实际回看了多远, 与正文长度无关。

🔴 给的是【最靠左】的那个左端, 不是碰到的第一个。反向走到死状态才知道还能不能更靠左;

"撞到第一个 match 状态就收工"给的是最短匹配 = 把命中截断。

── 它和 (*RegexpSetReverse).ResolveSpanWithin 的关系 ──────────────────────── 语义逐字相同, 差别只在【对象是谁】: 那个是一张表 (要一个 id 说是哪条), 这个是一条 pattern。 一条 pattern 就该用这一个 —— 套一条 pattern 的 set 去凑要多背一张 id 表 (kManyMatch 的 状态更大), 而且 set 与单条对 ^ / $ 的处理方式不同, 走的根本不是同一条代码路。

无状态、只读; 反向程序首次调用时惰性编出来 (线程安全)。编不出来 / DFA 放弃都返回 err —— 这一层不猜、不静默退回, 因为"没有匹配"和"算不出来"对调用方是两件完全不同的事。

func (*RegexpReverse) String

func (rr *RegexpReverse) String() string

String 返回编译时的源 pattern (不是反转后的文本 —— 反的是程序, 不是 pattern 文本)。

type RegexpSet

type RegexpSet struct {
	// contains filtered or unexported fields
}

RegexpSet 是多正则集合 (构建期一次编译 · 扫描期只读 · 并发安全)。

func NewRegexpSet

func NewRegexpSet(patterns []string) (*RegexpSet, error)

NewRegexpSet 把 patterns 顺序编进一个 RE2::Set (内存预算 = RE2 默认 8MB)。任一条解析失败 / 编译失败 → 返回 error (并释放已分配的 native 资源)。Match 输出的 index 即 patterns 的下标。

条数多了会撞上 8MB 这道【编译期】预算 (报 "set compile failed"), 此时别把表拆成两个 set, 用 NewRegexpSetMaxMem 调大预算即可。

func NewRegexpSetMaxMem

func NewRegexpSetMaxMem(patterns []string, maxMem int64) (*RegexpSet, error)

NewRegexpSetMaxMem 同 NewRegexpSet, 但显式指定 RE2 的内存预算 maxMem (字节; <=0 = 用默认 8MB)。

maxMem 是 RE2::Options::max_mem, 一个旋钮同时抬两条天花板 (Prog::CompileSet 拿它一次算完):

• 编译期: 整个 set 的程序指令条数上限 ((maxMem-sizeof(Prog))/4/sizeof(Inst), 封顶 2^24)。 pattern 多 / 每条复杂 → 撞的是这条; 撞了 Compile 直接失败, 即本函数返回 error。
• 运行期: 剩下的额度给 DFA 状态缓存。缓存满了 DFA 自己 flush 重来 (只是变慢, 结果仍正确), Set 不会退化成"没命中"。

怎么定: 从默认 8MB 起, 撞了就翻倍 (16/32/64MB) 直到 Compile 通过。构建期一次性开销, 之后常驻 只读; 内存换的是"一个 set 装下整表" —— 拆成两个 set 要扫两遍正文, 更贵。

func (*RegexpSet) Attrib

func (s *RegexpSet) Attrib() AttribInfo

Attrib 查建状态的归因。只读, 短暂拿一次 DFA 读锁, 不会因为查询而建状态。 没开 RE2_DFA_ATTRIB 编译时返回 AttribInfo{Enabled: false}。

典型用法 (找出该从表里拎走的那几条):

a := set.Attrib()
for _, p := range a.Pats[:10] {
    fmt.Printf("#%d 多塞了 %d 个零件 (平均每状态 %.2f 个)\n",
        p.Index, p.Excess, float64(p.Excess)/float64(a.StatesTotal))
}

func (*RegexpSet) FindAllIndex

func (s *RegexpSet) FindAllIndex(text string, alloc *RegexpSet_FindAllIndex_Alloc_t,
	batchFn func(runs []RegexpSet_FindAllIndex_Run_t)) error

FindAllIndex 扫 text 一遍, 每攒够一批端点游程就调一次 batchFn。 runs 里每一条的 Lo..Hi 是匹配【右端】(不含) 的取值范围, 两端都含。

🔴 runs 是工作区里那块缓冲本身, 下一批原地覆写 —— 要留就 append 走。

alloc 传 nil = 当场建一个用完就扔 (多一笔 native 分配); 热路径上传一个长期复用的。 没有任何命中时 batchFn 一次都不会被调 (不会拿空切片去骚扰调用方)。

返回 error 只有三种情况: alloc 已 Close / alloc 不是这个 set 的 / native 侧 DFA 中途放弃 (预算实在不够) —— 最后一种跟 Match 返回空是同一类事故, 用 NewRegexpSetMaxMem 调大即可。

func (*RegexpSet) FindAllIndexBytes

func (s *RegexpSet) FindAllIndexBytes(text []byte, alloc *RegexpSet_FindAllIndex_Alloc_t,
	batchFn func(runs []RegexpSet_FindAllIndex_Run_t)) error

FindAllIndexBytes 同 FindAllIndex, 但正文是 []byte (零拷贝)。

func (*RegexpSet) GetPatternLen

func (s *RegexpSet) GetPatternLen() int

GetPatternLen 返回集合里的 pattern 条数 (= Match 输出 index 的上界, 也是 buf 该开的长度)。

func (*RegexpSet) Match

func (s *RegexpSet) Match(text string, buf []int32) []int32

Match 扫 text 一遍, 把命中的 pattern index 写进 buf (传入复用切片避免每次分配) 并返回其前缀切片。 返回切片里每个元素是 patterns 的下标 (无序 · 不重复)。无命中返回长度 0 的切片。

buf 用 int32 (= C.int): 直接给 cgo 回填, 避免 Go int(64位) 与 C.int(32位) 尺寸不符的拷贝。

func (*RegexpSet) MatchAny

func (s *RegexpSet) MatchAny(text string) bool

MatchAny 报告 text 是否命中集合里【任一】正则 —— 【第一个命中位置就返回】, 不把正文扫完。

与 len(Match(...))>0 的差别不只是省一个切片: Match 要回答"哪几条", DFA 必须走到正文末尾才 知道命中集全不全; MatchAny 不取 index, 底下 RE2 的 SearchDFA 就打开 want_earliest_match, 扫到第一个命中位置立刻收工 —— 命中越早、正文越长, 省得越多 (不命中仍然是全扫一遍)。 因为不回填 index, 这里也不需要调用方传 buf。

走的是与 Match 同一份 DFA 状态缓存 (kManyMatch 那一份), 不会因为多这条快路径而多占一份。

func (*RegexpSet) MatchAnyBytes

func (s *RegexpSet) MatchAnyBytes(text []byte) bool

MatchAnyBytes 同 MatchAny, 但正文是 []byte (零拷贝)。

func (*RegexpSet) MatchBytes

func (s *RegexpSet) MatchBytes(text []byte, buf []int32) []int32

MatchBytes 同 Match, 但正文是 []byte (零拷贝喂给同一内核, 不做 string(text) 全量拷贝)。 供正文本来就是 []byte 的调用方直接用; 语义/返回值与 Match 完全一致。见 bytes.go 的说明。

func (*RegexpSet) MatchStats

func (s *RegexpSet) MatchStats(text string, buf []int32, st *ScanStats) []int32

MatchStats 同 Match, 外加把【这一次扫描】的 DFA 计数写进 st (st 可为 nil)。 st 不必预先清零。热路径上不想要这份开销就继续用 Match —— 不传 st 时 C 侧完全不统计。

func (*RegexpSet) MatchStatsBytes

func (s *RegexpSet) MatchStatsBytes(text []byte, buf []int32, st *ScanStats) []int32

MatchStatsBytes 同 MatchStats, 但正文是 []byte (零拷贝)。

func (*RegexpSet) MemInfo

func (s *RegexpSet) MemInfo() SetMemInfo

MemInfo 查这个 Set 当前的 DFA 缓存水位: 额度用掉多少、装了多少状态、生涯清空过几次。 只读, 内部短暂拿一次 DFA 读锁, 可以和扫描并发调 (会和正在 flush 的写锁互斥, 别在热路径上高频调)。

func (*RegexpSet) NewFindAllIndexAlloc

func (s *RegexpSet) NewFindAllIndexAlloc() (*RegexpSet_FindAllIndex_Alloc_t, error)

NewFindAllIndexAlloc 给这个正向 set 开一个 FindAllIndex 工作区。 热路径上建一次长期留着, 别每次扫描新建。

func (*RegexpSet) NewMatchScanner

func (s *RegexpSet) NewMatchScanner() (m *MatchScanner, unsupported []int32, err error)

NewMatchScanner 开一个工作区。热路径上建一次长期留着, 别每次扫描新建。

unsupported 是【走不了区间这条路】的那几条 pattern 的下标 —— 当下只有一个原因: 这条能 匹配空串 (PatternLenRange 的 min <= 0)。每个位置都是一处零长命中, 游标压不住, 吐出来的 text[Lo:Lo] 对下游也没有意义。

🔴 这张名单是【建工作区那一刻就定死】的: 它只看 pattern 本身, 与你之后喂什么正文无关,

所以它也是唯一一处"这条给不了你"的交代 —— Scan 那一遍要么全给, 要么整遍报错, 不存在
"扫到一半反悔"。名单上的那几条: 配 MatchScanMode_boolOnly (命中表照样有它们), 或者
自己走老路 FindAll。不配 boolOnly 而配了要区间的档, SetModes 当场报错; 压根不调
SetModes (全默认档) 的, 它们自动按 boolOnly 处理 —— 反正你在这里已经知道了。

名单可以直接写回归测试: 把 a* 之类扔进 set, 这里必然报出它的下标。

func (*RegexpSet) PatternLenRange

func (s *RegexpSet) PatternLenRange(i int) (min, max int)

PatternLenRange 返回集合里第 i 条的长度区间, 越界返回 (0, PatLenUnbounded)。 结果在建集期算好存着, 这里只是查表。

func (*RegexpSet) ResolveSpan

func (s *RegexpSet) ResolveSpan(text string, from, id int32) (pos int32, ok bool, err error)

ResolveSpan 求【另一端】: 给定 FindAllIndex 吐出来的一个端点, 返回第 id 条 pattern 在这个 端点上能达到的另一端。

from = 匹配左端(含), 返回右端(不含) —— text[from:pos] 就是这条 pattern 的匹配

ok=false 表示这条 pattern 在这个端点上根本不匹配 (调用方给错端点了, 或者给错 id)。

🔴 返回的是【最长】的那个匹配, 不是最短的。变长 pattern 在同一个端点上通常有一串长度都

成立 (`AAA-[A-Za-z0-9]{8,16}` 在同一个右端上有 9 个合法左端), "碰到第一个就收工"给的是
最短那个 = 把命中截断, 下游拿去做定长校验就会把真命中判成假命中。

无状态、只读, 可以和别的 goroutine 的 FindAllIndex 并发调 (与 Match 同一个口径)。

func (*RegexpSet) ResolveSpanBytes

func (s *RegexpSet) ResolveSpanBytes(text []byte, from, id int32) (pos int32, ok bool, err error)

ResolveSpanBytes 同 ResolveSpan, 但正文是 []byte (零拷贝)。

func (*RegexpSet) ResolveSpanWithin

func (s *RegexpSet) ResolveSpanWithin(text string, from, bound, id int32) (pos int32, ok bool, err error)

ResolveSpanWithin 同 ResolveSpan, 但限定【最远看到哪】: bound 是右上界, 负数 = 不限。

什么时候需要它: 走到死状态的成本 = 这条命中实际能延伸到多远 —— 除非 pattern 本身就能无限 延伸 ((?s).*KEY 那种), 那一条在整篇正文上解析一次就是 O(正文)。给这类 pattern 配一个 "看到哪为止"就把它钉回常数。

判定用的上下文恒是【整篇正文】, 所以 \b / ^ / $ 看到的永远是真实邻居字节, 而不是 bound 切出来的假边界 —— 掐 bound 只会让答案变短, 不会让它变错。

func (*RegexpSet) ViableOneStats

func (s *RegexpSet) ViableOneStats() (n int, states, arenaCap int64)

ViableOneStats 报【已经被建出来】的那些"反向单条 set"的账: 几条 · 状态数合计 · 状态区实际字节合计。与 MemInfo 同一个用途 (量内存去哪了), 不制造状态。 惰性建 ⟹ 没被 MatchScanner 问过位置的 pattern 一条都不占。

🔴 这是补起点这条路的【常驻】开销, 也是它相对 2026-08-28 之前那条老默认档 ("路 B",

只要正向单条, 一条反向都不建) 净增的那一笔 —— 挂新表之前先量这个数。
最大的那张 158 条生产表实测: 89 条被真问到位置, 合计 9.6MB。

type RegexpSetReverse

type RegexpSetReverse struct {
	// contains filtered or unexported fields
}

RegexpSetReverse 是反向编译的多正则集合 (构建期一次编译 · 扫描期只读 · 并发安全)。 用 NewRegexpSetReverseMaxMem 建。

func NewRegexpSetReverseMaxMem

func NewRegexpSetReverseMaxMem(patterns []string, maxMem int64) (*RegexpSetReverse, error)

NewRegexpSetReverseMaxMem 建一张【反向编译】的多正则表 (类型 RegexpSetReverse, 与正向的 RegexpSet 是两个类型 —— 为什么必须拆开见 regexpset_reverse.go 的文件头)。Match 从正文末尾 往前扫【原始 buffer】(不反转正文, 不复制正文), 命中集与正向逐条相同。 maxMem 的含义与 NewRegexpSetMaxMem 完全一样 (<=0 = RE2 默认 8MB) —— 反向 set 只有这一个 构造函数: 会想反着扫的表通常就是正向撞过预算的那张, 建它的时候顺手把预算定了。

⚠ 方向是【整张表一个】的选择: 一条 pattern 反着便宜不等于另一条也便宜。 实践做法是把表按方向拆成两张, 各扫一遍 —— 正文过两遍 DFA 仍然远比一张表在悬崖上跑便宜 (两份 Match 结果按下标并集即可; 注意 Match 返回的下标【无序】, 要比对得先排)。 拆之前先量: 每条 pattern 各建一个单条的正向 set 和反向 set, 用同一批真语料跑一遍比 MemInfo().States, 小的那边就是它该去的那一组。

func (*RegexpSetReverse) Attrib

func (r *RegexpSetReverse) Attrib() AttribInfo

Attrib 查建状态的归因 (要 -DRE2_DFA_ATTRIB=1 编译), 含义同 (*RegexpSet).Attrib。

func (*RegexpSetReverse) FindAllIndex

func (r *RegexpSetReverse) FindAllIndex(text string, alloc *RegexpSet_FindAllIndex_Alloc_t,
	batchFn func(runs []RegexpSet_FindAllIndex_Run_t)) error

FindAllIndex 扫 text 一遍 (从末尾往前走【原始 buffer】, 不反转正文, 不复制正文), 每攒够一批端点游程就调一次 batchFn。runs 里每一条的 Lo..Hi 是匹配【左端】(含) 的 取值范围, 两端都含。

🔴 与正向的差别不只是"端点换了一头": 反向 set 的状态数是每条 pattern 各自最坏情况

【相乘】出来的。真表实测 155 条反向扫 6.4MB = 65 秒 / arena 顶满 254MB 还在 flush,
正向同一张表 18ms / 零 flush。拿反向 set 扫全文之前先量一遍 MemInfo().FlushesTotal;
只是想补一处命中的左端, 用 ResolveSpanWithin (单点锚定, 代价与正文长度无关)。

其余 (alloc 语义 · 闭区间 · 顺序 · 缓冲原地复用 · error) 与正向那个完全一致。

func (*RegexpSetReverse) FindAllIndexBytes

func (r *RegexpSetReverse) FindAllIndexBytes(text []byte, alloc *RegexpSet_FindAllIndex_Alloc_t,
	batchFn func(runs []RegexpSet_FindAllIndex_Run_t)) error

FindAllIndexBytes 同 FindAllIndex, 但正文是 []byte (零拷贝)。

func (*RegexpSetReverse) GetPatternLen

func (r *RegexpSetReverse) GetPatternLen() int

GetPatternLen 返回集合里的 pattern 条数 (= Match 输出 index 的上界, 也是 buf 该开的长度)。

func (*RegexpSetReverse) Match

func (r *RegexpSetReverse) Match(text string, buf []int32) []int32

Match 从正文【末尾往前】扫一遍原始 buffer (不反转正文, 不复制正文), 把命中的 pattern index 写进 buf 并返回其前缀切片。命中集与正向 (*RegexpSet).Match 逐条相同, 只是扫的方向反了。

func (*RegexpSetReverse) MatchAny

func (r *RegexpSetReverse) MatchAny(text string) bool

MatchAny 报告 text 是否命中集合里【任一】正则 —— 第一个命中位置就返回, 不把正文扫完。

func (*RegexpSetReverse) MatchAnyBytes

func (r *RegexpSetReverse) MatchAnyBytes(text []byte) bool

MatchAnyBytes 同 MatchAny, 但正文是 []byte (零拷贝)。

func (*RegexpSetReverse) MatchBytes

func (r *RegexpSetReverse) MatchBytes(text []byte, buf []int32) []int32

MatchBytes 同 Match, 但正文是 []byte (零拷贝)。

func (*RegexpSetReverse) MatchStats

func (r *RegexpSetReverse) MatchStats(text string, buf []int32, st *ScanStats) []int32

MatchStats 同 Match, 外加把【这一次扫描】的 DFA 计数写进 st (st 可为 nil)。

这是标定"这张表该正着扫还是反着扫"的量器: 同一批 pattern 建一个 RegexpSet 和一个 RegexpSetReverse, 拿同一批真语料各跑一遍, 比 Flushes (>0 = 在悬崖上) 与 StatesEnd。

func (*RegexpSetReverse) MatchStatsBytes

func (r *RegexpSetReverse) MatchStatsBytes(text []byte, buf []int32, st *ScanStats) []int32

MatchStatsBytes 同 MatchStats, 但正文是 []byte (零拷贝)。

func (*RegexpSetReverse) MemInfo

func (r *RegexpSetReverse) MemInfo() SetMemInfo

MemInfo 查这个 set 当前的 DFA 缓存水位 (额度用掉多少 · 装了多少状态 · 生涯清空过几次)。 反向 set 上这个数尤其该看 —— 反向的状态数是每条各自最坏情况【相乘】出来的。

func (*RegexpSetReverse) NewFindAllIndexAlloc

func (r *RegexpSetReverse) NewFindAllIndexAlloc() (*RegexpSet_FindAllIndex_Alloc_t, error)

NewFindAllIndexAlloc 给这个反向 set 开一个 FindAllIndex 工作区。 正向 set 的 alloc 【不能】拿到这里来用, 反过来也不行。

func (*RegexpSetReverse) NewMatchScanner

func (r *RegexpSetReverse) NewMatchScanner() (m *MatchScannerReverse, unsupported []int32, err error)

NewMatchScanner 开一个反向工作区。热路径上建一次长期留着, 别每次扫描新建。

🔴 反向 set 本身仍然该是【一条一个】或者至少是很小的一张表: set 里的状态数是相乘的

(doc/状态数为什么会相乘.txt), 155 条的反向表在 6.4MB 正文上实测 65 秒 / arena 顶满
254MB 还在 flush。这一层不改变那件事 —— 它只是把"扫出来的左端"补成完整区间。

unsupported 与正向那个同解: 走不了区间这条路的那几条下标 (当下只有"能匹配空串"一个原因)。 建工作区那一刻就定死, 与正文无关 —— 这是"Scan 要么全给要么整遍报错"的前提。

func (*RegexpSetReverse) ResolveSpan

func (r *RegexpSetReverse) ResolveSpan(text string, from, id int32) (pos int32, ok bool, err error)

ResolveSpan 求【另一端】: 方向跟着反向 set 走, 与正向那个正好相反 ——

from = 匹配右端(不含), 返回左端(含) —— text[pos:from] 就是这条 pattern 的匹配

这就是"补左端"该走的那条路: 单点、锚定、代价与正文长度无关。上面那层 (matchscan.go) 给每条 pattern 惰性建一个【只有这一条】的反向 set, 就是为了在这里问这一句。

func (*RegexpSetReverse) ResolveSpanBytes

func (r *RegexpSetReverse) ResolveSpanBytes(text []byte, from, id int32) (pos int32, ok bool, err error)

ResolveSpanBytes 同 ResolveSpan, 但正文是 []byte (零拷贝)。

func (*RegexpSetReverse) ResolveSpanWithin

func (r *RegexpSetReverse) ResolveSpanWithin(text string, from, bound, id int32) (pos int32, ok bool, err error)

ResolveSpanWithin 同 ResolveSpan, 但限定【最远看到哪】: 反向的 bound 是左下界 (回看不越过 它), 负数 = 不限。上面那层把 bound 掐在游标上 —— 那是【正确性】不是省钱, 见 matchscan.go。

func (*RegexpSetReverse) ViableStarts

func (r *RegexpSetReverse) ViableStarts(text string, from, bound, id int32, out []int32) (n int, err error)

ViableStarts 把 [bound, from) 里全部候选起点写进 out, 返回【找到的总条数】n。

from  匹配右端 (不含) —— 就是正向 set 的 FindAllIndex 吐出来的那种端点;
bound 回看的左下界 (含), 负数 = 不限。判定用的上下文恒是【整篇正文】, 所以 \b / ^ / $
      看到的永远是真实邻居字节, 掐 bound 只会让候选变少, 不会让它变错。
id    第几条 pattern (与 Match 返回的下标同一套)。

🔴 out 里是【降序】的 (机器从右往左走, 先看见的位置更大)。要 leftmost 就【倒着遍历】。

🔴 n 可能【大于 len(out)】—— 那表示缓冲不够, 里面写下的是最大的那几个 (恰好最没用的

那几个)。调用方该按 n 换个更大的缓冲重来一次, 不要拿这批半成品往下走。

🔴 from 这个位置本身【不算】候选 (text[from:from) 是空的可行前缀, 对调用方没有意义)。

无状态、只读 (自己拿 DFA 的缓存读锁), 可以和别的 goroutine 的扫描并发调。

type RegexpSet_FindAllIndex_Alloc_t

type RegexpSet_FindAllIndex_Alloc_t struct {
	// contains filtered or unexported fields
}

RegexpSet_FindAllIndex_Alloc_t 是 FindAllIndex 的可复用工作区 (native 侧的游程表 + 挂起点 + 一批输出缓冲)。用 (*RegexpSet).NewFindAllIndexAlloc 或 (*RegexpSetReverse).NewFindAllIndexAlloc 开, 不用了调 Close (不调也有 finalizer 兜底)。

🔴 不是并发安全的: 一个 goroutine 一个。也不能跨 set 用 —— native 句柄是从那个 set 的程序 上开出来的, 串用会返回 error 而不是给错答案。

func (*RegexpSet_FindAllIndex_Alloc_t) Close

func (a *RegexpSet_FindAllIndex_Alloc_t) Close()

Close 释放 native 侧的工作区。可重复调; 调过之后拿它去 FindAllIndex 返回 error。

type RegexpSet_FindAllIndex_Run_t

type RegexpSet_FindAllIndex_Run_t struct {
	ReIndex int32
	Lo      int32
	Hi      int32
}

RegexpSet_FindAllIndex_Run_t 是一条【端点游程】: 第 ReIndex 条 pattern 的匹配端点落在 Lo..Hi 里的每一个值上 (两端都含)。正向 set 里是匹配【右端】(不含), 反向 set 里是匹配 【左端】(含) —— 见文件头。

🔴 这个布局是 native 直接写进来的 (紧挨着的 int32 三元组), 底下两条常量把它钉在编译期:

尺寸不是 12 字节就编不过。别在中间加字段、别改字段顺序、别换宽度。

type ReplaceAllStringFunc_ctx_t

type ReplaceAllStringFunc_ctx_t struct {
	// contains filtered or unexported fields
}

ReplaceAllStringFunc_ctx_t 持有 ReplaceAllStringFunc 复用所需的 scratch: idx 是匹配位置表 [s0,e0,s1,e1,…] (走 AppendAllStringIndexFlat 回填, 只要 group0 —— 拼接本来也只读 group0)。 零值即可用; 也可用 NewReplaceAllStringFunc_ctx 预分配。

func NewReplaceAllStringFunc_ctx

func NewReplaceAllStringFunc_ctx(nMatchHint int) *ReplaceAllStringFunc_ctx_t

NewReplaceAllStringFunc_ctx 预分配好 scratch (够放 nMatchHint 处匹配的位置表), 返回可复用的 ctx。 nMatchHint <= 0 就不预分配 (等首次调用自己长)。

func (*ReplaceAllStringFunc_ctx_t) AppendReplaceAllStringFunc

func (ctx *ReplaceAllStringFunc_ctx_t) AppendReplaceAllStringFunc(dst []byte, re *Regexp, src string, f func(string) string) ([]byte, bool)

AppendReplaceAllStringFunc 把「re 在 src 上每处匹配整体换成 f(匹配文本) 之后的完整结果」追加进 dst, 返回 (追加后的切片, 结果与 src 相比是否真的变了)。变了的话 dst 末尾多出来的那一段就是 re.ReplaceAllStringFunc(src, f); 没变的话 dst 一个字节都没多 (见下面那条回滚)。

🔴 第二个返回值是 changed 而【不是】matched —— 它的定义就是 `re.ReplaceAllStringFunc(src, f) != src`, 与那句常见的 `if out := re.ReplaceAllStringFunc(s, f); out != s` 取值一模一样。两种情况都报 false:

① 压根没匹配 —— 快返, 什么都不写;
② 有匹配, 但每一处 f 都把原文照样写了回去 ⇒ 逐字节没变 —— 收尾把 dst 截回调用前的长度。

②不是可有可无的保险。走这套 API 的替换绝大多数是"解码 / 去混淆", f 自己带着合法性判断, 判不过 就 `return m` 原样退回 —— HTML 数字实体 `&#…;` 里 ParseInt 失败或码点越界 · 十六进制串长度为奇数 或解出来不可打印, 都是【正则命中了但一个字节没改】。调用方问的是"到底变没变", 拿 matched 当 changed 用就会凭空多出一份与原文相同的产物: 多存一块底 · 多扫一遍 · 多一次去重, 甚至多一条告警。 代价接近零: 长度不等直接判定变了, 只有恰好等长才真跑一趟 memcmp; 而解码类替换的产物几乎恒比 输入短, 那一比通常在长度上就否掉了。

两处复用: ①dst 传 buf[:0] 跨调用复用结果底; ②同一个 ctx 复用匹配位置表。第一趟仍要把两块底长到位, 之后稳态零分配。首趟先按 len(src) 一次开够 —— 换字符串的结果长度事先不可知, 但绝大多数替换 (解码/去混淆/脱敏) 的产物与输入同量级或更短, 按 len(src) 开既躲开增长阶梯又不浪费; 真长出去了 后面的 append 照常接着长, 只是多付那一小截。

🔴 ②那条回滚只退 len 不退 cap: 报 false 时 dst 的【长度与内容】与传进来的完全一致, 但底可能已经 换成一块更大的了 (刚为它开的那 len(src) 字节)。这对调用方是好事 —— 下一趟不用再开; 唯一的要求 是别把传进去的那个切片变量当作"还指着老底"接着用, 一律拿返回值。

传给 f 的是 src 的子串 (零拷贝); f 的返回值会被立即拷进 dst, 可复用。

type ScanStats

type ScanStats struct {
	// Flushes 是本次扫描里状态缓存被【整表清空】的次数。
	// 0 = 这次调用全程吃缓存; >0 = 这次调用有一段在"每个字节都重新造状态"的速度上跑
	// (慢两个数量级), 而且清空要拿写锁, 同时扫这个 Set 的其它 goroutine 全停。
	Flushes int64
	// Grows 是本次扫描里 arena 扩容的次数。扩容【不丢任何状态】, 只是把状态区 realloc 到
	// 更大再重定位 —— 是"缓存在长大", 不是 thrash。单独列出来免得跟 Flushes 混。
	Grows int64
	// StatesBuilt 是本次扫描新建的状态数 (缓存未命中才建)。
	// 稳态下趋近 0。⚠ 口径是 DFA 上一个累计计数器的前后差, 并发扫同一个 Set 时会把
	// 别的 goroutine 建的算进来 —— 要精确归因请单线程量。
	StatesBuilt int64
	// Bytes 是本次扫描的正文字节数。Bytes/Flushes 就是 Rust 那边判"缓存还有没有用"的比值。
	Bytes int64
	// StatesEnd 是扫完时缓存里的状态数。
	StatesEnd int64
	// StateBudget 是这个 Set 的 DFA 状态缓存额度 (字节, 约等于 maxMem 扣掉程序占用),
	// MemLeft 是扫完时的剩余额度。已用 = StateBudget - MemLeft; MemLeft 见底就是下次 Flush 的前夜。
	StateBudget int64
	MemLeft     int64
	// FellBack 只有 (*RegexpReverse).MatchStats 会置 true: 反向 DFA 这次没跑成 (反向程序
	// 编译不出来 / DFA 中途放弃), 结果是退回正向 MatchString 得到的 —— 答案仍然正确, 只是
	// 这次没省到状态, 其余字段全 0。RegexpSet 的 MatchStats 恒为 false。
	FellBack bool
}

ScanStats 是【一次 Match 调用】的 DFA 计数 (字段名与 C 侧 cre2_scan_stats 逐字一致)。

跟 DFAStats() 那份【进程级】计数是两回事: 那份挂在 re2 的全局钩子上, 回调不带上下文, 只能回答"这个进程里有人在 thrash"; 这份是调用方自己在栈上开的对象, 沿调用链传下去, 能回答"这一次调用发生了什么"。没有全局状态, 没有 thread_local, 并发下各算各的。 做法照搬 Rust regex-automata 的 per-Cache 计数 (clear_count / 已扫字节数)。

type SetMatch

type SetMatch struct {
	Index int32
	Lo    int32
	Hi    int32
}

SetMatch 是一处命中: text[Lo:Hi] 是第 Index 条 pattern 的一个真匹配。

type SetMemInfo

type SetMemInfo struct {
	// Built=false 表示这个 Set 的 DFA 还没被建出来, 其余字段无意义。
	// 实测上 NewRegexpSet* 返回时它就已经是 true 了 —— RE2::Set::Compile 自己会跑一次冒烟
	// 搜索, 顺手把 DFA 建出来并留下 1 个状态。查询本身【不会】制造状态, 也不会替你建 DFA。
	Built bool
	// StateBudget 是状态缓存额度 (字节), MemLeft 是当前剩余。Used = StateBudget - MemLeft。
	StateBudget int64
	MemLeft     int64
	// States 是当前缓存里的状态数。
	States int64
	// ArenaCap 是实际向系统要到的状态区字节数。默认构建 (arena 按需翻倍) 下它才有意义:
	// 它 << StateBudget 说明"额度给多了也不占内存", 它逼近 StateBudget 说明快满了。
	ArenaCap int64
	// FlushesTotal / StatesBuiltTotal 是这个 Set 生涯里整表清空的次数 / 建过的状态数。
	// 前者就是 Rust regex-automata 那个 clear_count 的等价物, 只不过按 Set 归因。
	FlushesTotal     int64
	StatesBuiltTotal int64
}

SetMemInfo 是一个 RegexpSet 的 DFA 状态缓存水位 + 生涯累计。

func (SetMemInfo) Used

func (m SetMemInfo) Used() int64

Used 返回状态缓存已用额度 (字节)。Built=false 时返回 0。