onigmo

README

go-regexp/engine

A pure-Go (cgo-free) regular-expression engine compatible with Onigmo — the regular-expression library Ruby uses — with a public API shaped like the standard library's regexp package.

Why

The standard library regexp is built on RE2, which guarantees linear-time matching by forbidding features that require backtracking. go-regexp/engine supports the constructs RE2 rejects at compile time:

• Lookahead (?=…) / (?!…) and lookbehind (?<=…) / (?<!…)
• Backreferences \1, \k<name>
• Atomic groups (?>…) and possessive quantifiers
• Subexpression calls \g<name>, \g<0> (recursive patterns)
• Ruby/Onigmo character properties, \h/\H, \R, named groups, and more

It matches on the backtracking VM for those patterns and transparently falls back to a linear-time lazy-NFA/DFA accelerator for the RE2-compatible subset, so the common case stays fast while the extra features remain available.

The package is named onigmo (not regexp), so it can be imported alongside the standard library regexp without an alias.

Install

go get github.com/go-regexp/engine

Usage

package main

import (
	"fmt"

	onigmo "github.com/go-regexp/engine"
)

func main() {
	// A lookahead — (?=["<]) — that RE2/stdlib regexp cannot compile.
	re := onigmo.MustCompile(`v\d+\.\d+\.\d+(?=["<])`)

	fmt.Println(re.FindAllString(`grab v1.2.3" and v4.5.6< but not v9.9.9`, -1))
	// [v1.2.3 v4.5.6]

	fmt.Println(re.MatchString(`v2.0.0"`)) // true
	fmt.Println(re.FindStringIndex(`x v3.4.5<`)) // [2 8]
}

API

The *Regexp API mirrors the standard library regexp where it overlaps:

Method	Meaning
Compile, MustCompile	compile a pattern (panicking variant)
String	the source pattern
MatchString(s), Match(b)	does the input contain a match?
FindString, FindStringIndex	leftmost match text / [begin,end)
FindAllString, FindAllStringIndex	all non-overlapping matches (n < 0 = all)
FindStringSubmatch, FindStringSubmatchIndex	leftmost match with capture groups
NumSubexp, SubexpNames, SubexpIndex	capturing-group introspection

Extensions beyond the standard library:

Method	Meaning
CompileEnc, MustCompileEnc, Encoding	UTF-8 vs binary (ASCII8BIT) matching
MatchBounds, MatchBoundsAt	allocation-free whole-match [begin,end); …At anchors at a byte offset without scanning forward
FindStringSubmatchIndexAt	anchored submatch (cursor-style lexing)
WithTimeout, Timeout	per-match wall-clock limit for pathological patterns

FindAll* follow the standard library's non-overlapping, left-to-right semantics, including empty-match handling (an empty match advances by one rune and an empty match adjacent to a previous match is skipped).

A *Regexp is immutable once compiled and safe for concurrent use by multiple goroutines. The heavy matcher state is built lazily on the first match.

Status

Engine and public API are complete with 100% test coverage and CI on the six supported 64-bit architectures (amd64, arm64, riscv64, loong64, ppc64le, s390x). A full org landing page, logo and hosted documentation are a follow-up.

License

BSD-3-Clause. See LICENSE.

Documentation

Overview

Package onigmo is a pure-Go (cgo-free) regular-expression engine compatible with Onigmo — the regular-expression library Ruby uses — that exposes an API shaped like the standard library's regexp package.

Unlike the standard library regexp (and RE2, which it is built on), this engine supports lookahead, lookbehind, backreferences, atomic groups and subexpression calls. Patterns that require those features — which stdlib regexp rejects at Compile time — compile and match here.

The package name is onigmo (not regexp), so it can be imported alongside the standard library regexp without an alias:

import (
	"regexp"
	onigmo "github.com/go-regexp/engine"
)

re := onigmo.MustCompile(`v\d+\.\d+\.\d+(?=["<])`) // lookahead: RE2 cannot
re.FindAllString(`grab v1.2.3" and v4.5.6< but not v9.9.9`, -1)
// -> ["v1.2.3", "v4.5.6"]

A *Regexp is immutable once compiled and safe for concurrent use by multiple goroutines. The heavy matcher state is built lazily on the first match, so a compiled-but-unmatched Regexp pays only the parse cost.

Index

• Constants
• type Encoding
• type Regexp
  • func Compile(expr string) (*Regexp, error)
  • func CompileEnc(expr string, enc Encoding) (*Regexp, error)
  • func MustCompile(expr string) *Regexp
  • func MustCompileEnc(expr string, enc Encoding) *Regexp
  • func (re *Regexp) Encoding() Encoding
  • func (re *Regexp) FindAllString(s string, n int) []string
  • func (re *Regexp) FindAllStringIndex(s string, n int) [][]int
  • func (re *Regexp) FindString(s string) string
  • func (re *Regexp) FindStringIndex(s string) []int
  • func (re *Regexp) FindStringSubmatch(s string) []string
  • func (re *Regexp) FindStringSubmatchIndex(s string) []int
  • func (re *Regexp) FindStringSubmatchIndexAt(s string, pos int) []int
  • func (re *Regexp) Match(b []byte) bool
  • func (re *Regexp) MatchBounds(s string) (begin, end int, ok bool)
  • func (re *Regexp) MatchBoundsAt(s string, pos int) (begin, end int, ok bool)
  • func (re *Regexp) MatchString(s string) bool
  • func (re *Regexp) NumSubexp() int
  • func (re *Regexp) ReplaceAll(src, repl []byte) []byte
  • func (re *Regexp) ReplaceAllFunc(src []byte, repl func([]byte) []byte) []byte
  • func (re *Regexp) ReplaceAllLiteral(src, repl []byte) []byte
  • func (re *Regexp) ReplaceAllLiteralString(src, repl string) string
  • func (re *Regexp) ReplaceAllString(src, repl string) string
  • func (re *Regexp) ReplaceAllStringFunc(src string, repl func(string) string) string
  • func (re *Regexp) String() string
  • func (re *Regexp) SubexpIndex(name string) int
  • func (re *Regexp) SubexpNames() []string
  • func (re *Regexp) Timeout() time.Duration
  • func (re *Regexp) WithTimeout(d time.Duration) *Regexp

Constants

const (
	// UTF8 is the default encoding: the dot and byte-oriented classes advance by a
	// whole UTF-8 code point.
	UTF8 = compile.UTF8
	// ASCII8BIT is the binary encoding: every atom advances one byte.
	ASCII8BIT = compile.ASCII8BIT
)

Types

type Encoding

type Encoding = compile.Encoding

Encoding selects how the byte-oriented input-advancing atoms — the dot (`.`) and a byte-oriented character class — traverse the input.

In UTF8 (the default) the dot and byte-oriented classes advance by a whole UTF-8 code point, so `.` matches a complete multi-byte character. In ASCII8BIT (binary mode) every atom advances one byte, and Unicode case-folding (/i) and \p{…} properties operate per byte. Match offsets are byte offsets in both modes.

type Regexp

type Regexp struct {
	// contains filtered or unexported fields
}

Regexp is a compiled regular expression, safe for concurrent use by multiple goroutines. A Regexp is immutable once compiled; WithTimeout returns a copy carrying a wall-clock match limit rather than mutating the receiver, so a shared Regexp stays concurrency-safe.

func Compile

func Compile(expr string) (*Regexp, error)

Compile parses a pattern and returns a compiled Regexp in the default UTF-8 encoding, or an error if the pattern is malformed. Unlike the standard library regexp.Compile, a pattern using lookaround, backreferences, atomic groups or subexpression calls is accepted.

func CompileEnc

func CompileEnc(expr string, enc Encoding) (*Regexp, error)

CompileEnc is Compile with an explicit input encoding (see Encoding). UTF8 makes the dot and byte-oriented classes advance by a whole code point; ASCII8BIT makes every atom advance one byte.

func MustCompile

func MustCompile(expr string) *Regexp

MustCompile is like Compile but panics if the pattern cannot be compiled. It simplifies safe initialization of package-level compiled regular expressions.

func MustCompileEnc

func MustCompileEnc(expr string, enc Encoding) *Regexp

MustCompileEnc is like CompileEnc but panics if the pattern cannot be compiled.

func (*Regexp) Encoding

func (re *Regexp) Encoding() Encoding

Encoding returns the input encoding the Regexp matches under: UTF8 by default, ASCII8BIT for a binary pattern. It does not trigger the deferred machine build.

func (*Regexp) FindAllString

func (re *Regexp) FindAllString(s string, n int) []string

FindAllString returns a slice of all successive non-overlapping matches of the regular expression in s. A value of n >= 0 limits the result to at most n matches; n < 0 returns all of them. A return value of nil indicates no match. This mirrors regexp.Regexp.FindAllString.

func (*Regexp) FindAllStringIndex

func (re *Regexp) FindAllStringIndex(s string, n int) [][]int

FindAllStringIndex returns a slice of all successive non-overlapping matches of the regular expression in s, expressed as index pairs (see FindStringIndex). The matches are found left to right. A value of n >= 0 limits the result to at most n matches; n < 0 returns all of them. A return value of nil indicates no match. This mirrors regexp.Regexp.FindAllStringIndex.

func (*Regexp) FindString

func (re *Regexp) FindString(s string) string

FindString returns the text of the leftmost match in s of the regular expression. If there is no match, the return value is an empty string, but it will also be empty if the regular expression successfully matches an empty string. Use FindStringIndex if it is necessary to distinguish these cases.

func (*Regexp) FindStringIndex

func (re *Regexp) FindStringIndex(s string) []int

FindStringIndex returns a two-element slice of integers defining the location of the leftmost match in s of the regular expression. The match itself is at s[loc[0]:loc[1]]. A return value of nil indicates no match. This mirrors regexp.Regexp.FindStringIndex.

func (*Regexp) FindStringSubmatch

func (re *Regexp) FindStringSubmatch(s string) []string

FindStringSubmatch returns a slice of strings holding the text of the leftmost match of the regular expression in s and the matches, if any, of its subexpressions. A return value of nil indicates no match. An entry is the empty string when the corresponding subexpression did not participate in the match. This mirrors regexp.Regexp.FindStringSubmatch.

func (*Regexp) FindStringSubmatchIndex

func (re *Regexp) FindStringSubmatchIndex(s string) []int

FindStringSubmatchIndex returns a slice holding the index pairs identifying the leftmost match of the regular expression in s and the matches, if any, of its subexpressions, as defined by the 'Submatch' and 'Index' descriptions of regexp.Regexp. A return value of nil indicates no match. This mirrors regexp.Regexp.FindStringSubmatchIndex.

func (*Regexp) FindStringSubmatchIndexAt

func (re *Regexp) FindStringSubmatchIndexAt(s string, pos int) []int

FindStringSubmatchIndexAt is FindStringSubmatchIndex for a match anchored exactly at byte offset pos: the whole match must begin at pos (it does not scan forward), with the full string visible so ^, \A and lookbehind see the real prefix s[:pos]. It returns the capture index pairs, or nil if the pattern does not match anchored at pos. pos out of range yields nil.

func (*Regexp) Match

func (re *Regexp) Match(b []byte) bool

Match reports whether the byte slice b contains any match of the regular expression.

func (*Regexp) MatchBounds

func (re *Regexp) MatchBounds(s string) (begin, end int, ok bool)

MatchBounds scans s left to right for the leftmost match and returns its whole-match [begin, end) byte span, without extracting submatches. On the lazy-NFA subset the search runs on the linear-time NFA; otherwise it falls to the backtracking VM. The span is identical to FindStringIndex(s).

func (*Regexp) MatchBoundsAt

func (re *Regexp) MatchBoundsAt(s string, pos int) (begin, end int, ok bool)

MatchBoundsAt reports the whole match's [begin, end) byte span for a match anchored exactly at byte offset pos (begin == pos on success), without extracting submatches. Unlike MatchBounds it does not scan forward: it matches at pos or reports ok == false. The whole string stays visible, so ^, \A and lookbehind see the real prefix s[:pos] — the primitive a cursor-anchored tokenizer needs. pos out of range yields ok == false.

func (*Regexp) MatchString

func (re *Regexp) MatchString(s string) bool

MatchString reports whether the string s contains any match of the regular expression. When the program is in the lazy-NFA subset (no backreference, call, lookaround, atomic group, or over-large bounded loop) the question is answered by the linear-time NFA simulation rather than the backtracking VM.

func (*Regexp) NumSubexp

func (re *Regexp) NumSubexp() int

NumSubexp returns the number of parenthesized capturing subexpressions in the pattern, not counting the whole match (group 0). It matches the semantics of the standard library regexp.Regexp.NumSubexp.

func (*Regexp) ReplaceAll

func (re *Regexp) ReplaceAll(src, repl []byte) []byte

ReplaceAll is the []byte form of ReplaceAllString: it returns a copy of src with every non-overlapping match replaced by the $-expansion of repl.

func (*Regexp) ReplaceAllFunc

func (re *Regexp) ReplaceAllFunc(src []byte, repl func([]byte) []byte) []byte

ReplaceAllFunc is the []byte form of ReplaceAllStringFunc.

func (*Regexp) ReplaceAllLiteral

func (re *Regexp) ReplaceAllLiteral(src, repl []byte) []byte

ReplaceAllLiteral is the []byte form of ReplaceAllLiteralString: repl is used literally with no $ expansion.

func (*Regexp) ReplaceAllLiteralString

func (re *Regexp) ReplaceAllLiteralString(src, repl string) string

ReplaceAllLiteralString returns a copy of src, replacing every non-overlapping match of re with repl used literally — no $ expansion is performed. This mirrors regexp.Regexp.ReplaceAllLiteralString.

func (*Regexp) ReplaceAllString

func (re *Regexp) ReplaceAllString(src, repl string) string

ReplaceAllString returns a copy of src, replacing every non-overlapping match of re with the expansion of repl. Inside repl a $ introduces a submatch reference: $name or ${name} is replaced by the submatch named or numbered name, and $$ is a literal $. A reference to a group that did not participate expands to the empty string. This mirrors regexp.Regexp.ReplaceAllString.

func (*Regexp) ReplaceAllStringFunc

func (re *Regexp) ReplaceAllStringFunc(src string, repl func(string) string) string

ReplaceAllStringFunc returns a copy of src, replacing every non-overlapping match of re with the return value of repl applied to the matched substring. No $ expansion is performed on repl's result. This mirrors regexp.Regexp.ReplaceAllStringFunc.

func (*Regexp) String

func (re *Regexp) String() string

String returns the source text of the pattern the Regexp was compiled from.

func (*Regexp) SubexpIndex

func (re *Regexp) SubexpIndex(name string) int

SubexpIndex returns the index of the first subexpression with the given name, or -1 if there is no subexpression with that name. Group 0 (the whole match) has no name. This mirrors regexp.Regexp.SubexpIndex.

func (*Regexp) SubexpNames

func (re *Regexp) SubexpNames() []string

SubexpNames returns the names of the parenthesized capturing subexpressions. The name of the first sub-expression is names[1], so that the index into the slice matches the group number used by FindStringSubmatchIndex. Because the whole match has no name, names[0] is always "". A subexpression without a name has an empty string entry. The result is freshly allocated on each call and the caller may modify it. This mirrors regexp.Regexp.SubexpNames.

func (*Regexp) Timeout

func (re *Regexp) Timeout() time.Duration

Timeout returns the wall-clock limit applied to a single match, or zero if no limit is set.

func (*Regexp) WithTimeout

func (re *Regexp) WithTimeout(d time.Duration) *Regexp

WithTimeout returns a copy of the Regexp that aborts any single match taking longer than d of wall-clock time, reporting no match. A non-positive d clears the limit. The copy shares the compiled program with the receiver, which is left unchanged, so a Regexp can be shared across goroutines and given per-use timeouts without data races.