gex

README

gex

A generic framework for lexical analysis of UTF-8 text in Go, based on the implementation discussed in "Lexical Scanning in Go" by robpike.

Installation

$ go get -u github.com/bdreece/gex

Usage

import (
    "github.com/bdreece/gex"
)

Examples

Coming soon.

---

MIT License

Copyright (c) 2025 Brian Reece

Documentation

Overview

Package gex provides a generic framework for lexical analysis of UTF-8 text in Go, based on the implementation discussed in "Lexical Scanning in Go" by robpike.

Index

• func Start[T any](input string, init State[T], opts ...Option[T]) <-chan Token[T]
• func StartWithConfig[T any](config *Config[T]) <-chan Token[T]
• type Config
  • func DefaultConfig[T any]() *Config[T]
• type Lexer
  • func (l *Lexer[T]) Accept(valid string) bool
  • func (l *Lexer[T]) AcceptRun(valid string)
  • func (l *Lexer[T]) Back() (ok bool)
  • func (l Lexer[T]) Current() string
  • func (l Lexer[T]) EOF() rune
  • func (l *Lexer[T]) Emit(kind T)
  • func (l *Lexer[T]) Errorf(format string, args ...any) State[T]
  • func (l Lexer[T]) Input() string
  • func (l Lexer[T]) Name() string
  • func (l *Lexer[T]) Next() rune
  • func (l *Lexer[T]) Peek() rune
  • func (l Lexer[T]) Pos() int
  • func (l *Lexer[T]) Runes() iter.Seq[rune]
  • func (l *Lexer[T]) Skip()
  • func (l Lexer[T]) Start() int
  • func (l Lexer[T]) Width() int
• type Option
  • func WithCapacity[T any](capacity int) Option[T]
  • func WithEOF[T any](eof rune) Option[T]
  • func WithErrorType[T any](typ T) Option[T]
  • func WithName[T any](name string) Option[T]
• type State
• type Token
  • func (t Token[T]) String() string

Functions

func Start

func Start[T any](input string, init State[T], opts ...Option[T]) <-chan Token[T]

Starts analyzing the input text with the provided init state in a new goroutine, and returns a channel to receive the emitted Token objects.

Additional options may be provided as variadic arguments.

func StartWithConfig

func StartWithConfig[T any](config *Config[T]) <-chan Token[T]

Starts analyzing the input text using parameters provided through the Config object.

See Start for more details.

Types

type Config

type Config[T any] struct {
	// The input text (required).
	Input string
	// The initial state (required).
	Init State[T]

	// The name of the lexer (defaults to "gex").
	Name string
	// The buffer size of the lexer token stream (defaults to 2).
	Capacity int
	// The sentinel `rune` value used to demarcate the end of the
	// token stream (defaults to `*new(rune)`).
	EOF rune
	// The sentinel `T` value used when emitting a synthetic error
	// token (defaults to `*new(T)`).
	Error T
}

A Config provides additional configuration parameters for the lexer state machine in the form of a struct.

func DefaultConfig

func DefaultConfig[T any]() *Config[T]

DefaultConfig returns a Config object populated with the default values.

type Lexer

type Lexer[T any] struct {
	// contains filtered or unexported fields
}

Lexer provides utility methods for emitting tokens and directing the state of iteration.

func (*Lexer[T]) Accept

func (l *Lexer[T]) Accept(valid string) bool

Accept advances the iterator position if the next rune is in the valid set.

func (*Lexer[T]) AcceptRun

func (l *Lexer[T]) AcceptRun(valid string)

AcceptRun advances the iterator position until it encounters a rune not found in the valid set.

func (*Lexer[T]) Back

func (l *Lexer[T]) Back() (ok bool)

Back decrements the current iterator position. Can only be run once per call to Next.

func (Lexer[T]) Current

func (l Lexer[T]) Current() string

Current returns a string slice representing the current token.

func (Lexer[T]) EOF

func (l Lexer[T]) EOF() rune

EOF returns the sentinel EOF rune associated with the lexer.

func (*Lexer[T]) Emit

func (l *Lexer[T]) Emit(kind T)

Emit emits a token with the provided kind to the output stream.

func (*Lexer[T]) Errorf

func (l *Lexer[T]) Errorf(format string, args ...any) State[T]

Errorf formats and emits a synthetic error token using the configured error kind (defaults to the zero value).

func (Lexer[T]) Input

func (l Lexer[T]) Input() string

Input returns the original input text.

func (Lexer[T]) Name

func (l Lexer[T]) Name() string

Name returns the name of the lexer.

func (*Lexer[T]) Next

func (l *Lexer[T]) Next() rune

Next returns the next rune in the input text, advancing the iterator position.

func (*Lexer[T]) Peek

func (l *Lexer[T]) Peek() rune

Peek returns the next rune in the input text without advancing the iterator position.

func (Lexer[T]) Pos

func (l Lexer[T]) Pos() int

Pos returns the position of the next rune.

func (*Lexer[T]) Runes

func (l *Lexer[T]) Runes() iter.Seq[rune]

Runes returns an iterator over the runes in the sequence, until EOF is reached.

func (*Lexer[T]) Skip

func (l *Lexer[T]) Skip()

Skip advances the start cursor to the current iterator position.

func (Lexer[T]) Start

func (l Lexer[T]) Start() int

Start returns the start position of the current token.

func (Lexer[T]) Width

func (l Lexer[T]) Width() int

Width returns the width of the last scanned rune.

type Option

type Option[T any] interface {
	// contains filtered or unexported methods
}

An Option provides an internal mechanism for mutating a Config.

func WithCapacity

func WithCapacity[T any](capacity int) Option[T]

WithCapacity returns an option configuring the capacity of the Lexer token stream.

func WithEOF

func WithEOF[T any](eof rune) Option[T]

WithEOF returns an option configuring the sentinel EOF rune associated with the lexer. See Config for more details.

func WithErrorType

func WithErrorType[T any](typ T) Option[T]

WithErrorType returns an option configuring the sentinel error value associated with the lexer. See Config for more details.

func WithName

func WithName[T any](name string) Option[T]

WithName returns an option configuring the name of the Lexer.

type State

type State[T any] func(l *Lexer[T]) (next State[T])

A State is a node in the lexer state machine.

Implementors of this function contract can take advantage of utility functions provided through the Lexer object to emit tokens and direct iteration over the input text.

type Token

type Token[T any] struct {
	// The token type.
	Type T
	// The token value.
	Value string
}

A Token is a basic, abstract unit of lexical meaning.

Token is generic over type T (typically an enum), representing the token type.

func (Token[T]) String

func (t Token[T]) String() string

String implements fmt.Stringer