gbnf

README

github.com/tabnas/gbnf/go

Go port of @tabnas/gbnf — a llama.cpp GBNF front-end for the tabnas parsing engine.

Port status: front-end implemented. ParseGbnf reads .gbnf text into the shared grammar IR using a tabnas meta-grammar — the rule table in parser_gbnf.go is executed by the engine this front-end compiles for, the same design as ts/src/converter.ts — and Gbnf / ToSpec / Install (facade.go) emit and install a GrammarSpec with GBNF's exact lexing — empty ignore set, default matchers off, lex.empty computed from the IR. The same validation passes as the TypeScript front-end run here: mandatory root, defined references, tokenizer-token terminals rejected by policy. The suite mirrors ts/test/gbnf.test.js, compiles all eight llama.cpp corpus grammars in ../test/corpus/, and grades accept/reject on all of them.

import (
    tabnas "github.com/tabnas/parser/go"
    gbnf "github.com/tabnas/gbnf/go"
)

tn := tabnas.Make()
spec, err := gbnf.Install(tn, `root ::= "hi" | "hello"`, nil)

Parity. Two pieces made grading possible in both directions. Negotiated lexing landed in parser/go v0.8.5 and this front-end opts in; the shared compiler's contested-alternative guards — FOLLOW / FOLLOW₂ repetition exits, keyword-shadow guards, left factoring — landed in bnf/go v0.1.4 (tabnas/bnf#13). With both in place, all eight corpus grammars agree with TypeScript in both directions; bnf/go/doc/differences.md records how that was verified (65/65 accept, 30/30 reject against the tables ts/test/corpus.test.js pins, and rule-for-rule emitter comparison on c.gbnf).

corpusExpectedFailures in gbnf_test.go is empty and stays: a new gap goes in the table, and a closed one turns the suite red rather than passing unnoticed. (Two of its former entries — c's int x = 1; and english's trailing newline — turned out to be outside their grammars' languages, mislabelled as gaps; they moved to the reject table.)

Astral character classes additionally have no serialisable form the Go runtime can load (ts/doc/known-gaps.md §5).

The TypeScript implementation is canonical. The dialect, and the scannerless limitations recorded in ts/doc/known-gaps.md, are the contract this port is held to.

Documentation

Overview

facade.go — the package's public surface, mirroring ts/src/gbnf.ts.

Package gbnf is the Go port of @tabnas/gbnf: a llama.cpp GBNF front-end for the tabnas parsing engine. It parses GBNF text into the grammar IR that github.com/tabnas/bnf/go compiles.

PORT STATUS: front-end implemented. ParseGbnf reads GBNF text — character classes, escapes, postfix repetition, comments — and the same validation passes as ts/src/converter.ts run here: mandatory root, defined references, tokenizer-token terminals rejected by policy. Gbnf/ToSpec/Install emit a spec carrying GBNF's exact lexing, negotiated lexing included (Lex.Relex, parser/go v0.8.5) — with which, and with the contested-alternative guards from bnf/go v0.1.4, all eight llama.cpp corpus grammars grade accept/reject here exactly as they do in ts/. This front-end still has no markClassesEager port; it is inert where it does not apply.

The TypeScript implementation is canonical. The dialect, and the scannerless limitations documented in ts/doc/known-gaps.md, are the contract this port is held to.

parser_gbnf.go — the GBNF front-end: llama.cpp GBNF text in, grammar IR out.

Everything downstream of the IR belongs to github.com/tabnas/bnf/go and is shared with the ABNF and EBNF front-ends:

GBNF text --ParseGbnf--> bnf.Grammar --bnf.EmitGrammarSpec--> GrammarSpec

The grammar that reads GBNF is ITSELF a tabnas grammar — `defineGbnfRules` below is a table of open/close rule alternates executed by the very engine this front-end compiles for, mirroring `gbnfRules` in ts/src/converter.ts. That is the point of having a parser engine: the notation is declared once, in the engine's own terms, rather than reimplemented by hand per runtime. An earlier version of this file was hand-written recursive descent; it was replaced by this, and the whole suite — including all eight corpus grammars graded both ways and all 70 live-corpus grammars — passed unchanged, which is the evidence that the language did not move.

The dialect is llama.cpp's `grammars/README.md`; ts/doc/known-gaps.md records where this front-end and llama.cpp's own parser diverge, and applies to this runtime equally.

TWO GO-SPECIFIC DEPARTURES from the TS table, both forced by the language rather than chosen:

• Accumulating nodes are POINTERS to slices. TS pushes into a shared array because JS arrays are references; a Go `append` may reallocate, so a child appending to a value-copy of its parent's slice would silently drop elements. `*[]T` restores the semantics the rule table assumes.
• A terminal decoder cannot throw. Actions have no error return, so a decode failure is recorded on per-parse state reached through ctx.Meta and re-raised by ParseGbnf, which mirrors what the TS engine does by letting the exception propagate.

validate_gbnf.go — the semantic checks that run after the notation is read and BEFORE the IR reaches github.com/tabnas/bnf/go.

The ordering is load-bearing: the shared compiler maps an undefined TX/NR/ST/VL reference onto the engine's own lexer tokens — right for ABNF, wrong for GBNF, where those are ordinary rule names. Checking first means a typo is reported as a typo.

Index

• Constants
• func Gbnf(src string, opts *ConvertOptions) (*tabnas.GrammarSpec, error)
• func Install(j *tabnas.Tabnas, src string, opts *ConvertOptions) (*tabnas.GrammarSpec, error)
• func ParseGbnf(src string) (*bnf.Grammar, error)
• func ToSpec(src string, opts *ConvertOptions) (*tabnas.GrammarSpec, error)
• type CompileError
  • func (e *CompileError) Error() string
  • func (e *CompileError) Unwrap() error
• type ConvertOptions
• type GbnfElement
• type GbnfGrammar
  • func EliminateLeftRecursion(g *GbnfGrammar) *GbnfGrammar
• type GbnfProduction
• type GbnfSequence
• type ParseError
  • func (e *ParseError) Error() string
  • func (e *ParseError) Unwrap() error

Constants

const VERSION = "0.1.7"

VERSION is this module's version. It MUST equal ts/package.json "version": the release orchestrator rewrites both, and the version test fails the build if they drift.

Functions

func Gbnf

func Gbnf(src string, opts *ConvertOptions) (*tabnas.GrammarSpec, error)

Gbnf converts GBNF source into a tabnas grammar spec.

Installing the result is what switches an instance to GBNF's exact lexing: the spec carries an empty ignore set and no default matchers. Use a fresh instance per grammar — those settings are instance-wide.

func Install

func Install(
	j *tabnas.Tabnas, src string, opts *ConvertOptions,
) (*tabnas.GrammarSpec, error)

Install converts GBNF source and installs it on a tabnas instance.

func ParseGbnf

func ParseGbnf(src string) (*bnf.Grammar, error)

ParseGbnf parses GBNF source into the grammar IR. The returned Grammar is ready for bnf.EmitGrammarSpec: `root` is present, every reference resolves, and no tokenizer terminals remain.

func ToSpec

func ToSpec(src string, opts *ConvertOptions) (*tabnas.GrammarSpec, error)

ToSpec is Gbnf under the name the TS package uses.

Types

type CompileError

type CompileError struct {
	Message string
	Cause   error
}

CompileError is raised when the GBNF parsed cleanly but the grammar cannot be compiled — a tokenizer-token terminal, an unknown rule reference, a purely left-recursive rule.

func (*CompileError) Error

func (e *CompileError) Error() string

func (*CompileError) Unwrap

func (e *CompileError) Unwrap() error

type ConvertOptions

type ConvertOptions struct {
	Start        string
	Tag          string
	Builtins     bool
	Marks        bool
	WordKeywords bool
}

ConvertOptions is the shared compiler's options plus this front-end's own knob.

type GbnfElement

type GbnfElement = bnf.Element

The IR types under this package's own names. Aliases, not definitions: a value has to cross the package boundary.

type GbnfGrammar

type GbnfGrammar = bnf.Grammar

The IR types under this package's own names. Aliases, not definitions: a value has to cross the package boundary.

func EliminateLeftRecursion

func EliminateLeftRecursion(g *GbnfGrammar) *GbnfGrammar

EliminateLeftRecursion exposes that pass for a caller that wants to inspect the rewritten IR.

type GbnfProduction

type GbnfProduction = bnf.Production

The IR types under this package's own names. Aliases, not definitions: a value has to cross the package boundary.

type GbnfSequence

type GbnfSequence = bnf.Sequence

The IR types under this package's own names. Aliases, not definitions: a value has to cross the package boundary.

type ParseError

type ParseError struct {
	Message string
	Line    int
	Column  int
	Cause   error
}

ParseError is raised when the GBNF source itself cannot be read.

func (*ParseError) Error

func (e *ParseError) Error() string

func (*ParseError) Unwrap

func (e *ParseError) Unwrap() error