grammar

Documentation

Overview

Package grammar converts a subset of JSON Schema into a context-free grammar for constrained decoding. (Stability: beta)

Package grammar converts a subset of JSON Schema into a context-free grammar state machine that can constrain token-by-token generation to produce only valid JSON conforming to the schema.

Index

• func TokenMask(g *Grammar, vocab []string) []bool
• type Grammar
  • func Convert(schema *JSONSchema) (*Grammar, error)
  • func (g *Grammar) Advance(b byte) (next *Grammar, ok bool)
  • func (g *Grammar) IsComplete() bool
  • func (g *Grammar) ValidBytes() []byte
• type JSONSchema

Examples

• Convert

Functions

func TokenMask

func TokenMask(g *Grammar, vocab []string) []bool

TokenMask returns a boolean mask over vocab. mask[i] is true if vocab token i is a valid next token at the current grammar state.

For each token, every byte must advance the grammar successfully. If any byte is rejected by Grammar.Advance, the token is invalid.

Types

type Grammar

type Grammar struct {
	// contains filtered or unexported fields
}

Grammar is an immutable state machine that tracks the current parse position within a JSON value constrained by a JSON Schema. Each call to Advance consumes one byte and returns a new Grammar (or reports the byte as invalid).

func Convert

func Convert(schema *JSONSchema) (*Grammar, error)

Convert transforms a JSONSchema into a Grammar state machine. It returns an error if the schema uses unsupported features or contains unresolvable $refs.

Example

package main

import (
	"fmt"

	"github.com/zerfoo/zerfoo/generate/grammar"
)

func main() {
	schema := &grammar.JSONSchema{
		Type: "object",
		Properties: map[string]*grammar.JSONSchema{
			"name": {Type: "string"},
			"age":  {Type: "number"},
		},
		Required: []string{"name", "age"},
	}

	g, err := grammar.Convert(schema)
	if err != nil {
		fmt.Println("error:", err)
		return
	}

	// Advance through a valid JSON string character by character.
	input := `{"name":"Alice","age":30}`
	current := g
	ok := true
	for _, b := range []byte(input) {
		current, ok = current.Advance(b)
		if !ok {
			fmt.Printf("rejected at byte %q\n", string(b))
			return
		}
	}
	fmt.Println(current.IsComplete())
}

Output:
true

func (*Grammar) Advance

func (g *Grammar) Advance(b byte) (next *Grammar, ok bool)

Advance consumes byte b and returns the resulting grammar state. If b is not a valid next byte, ok is false.

func (*Grammar) IsComplete

func (g *Grammar) IsComplete() bool

IsComplete returns true when the grammar is in an accepting state — i.e. a complete, valid JSON value has been consumed.

func (*Grammar) ValidBytes

func (g *Grammar) ValidBytes() []byte

ValidBytes returns every byte that is a valid next character in the current state. This is used by constrained decoding to mask logits.

type JSONSchema

type JSONSchema struct {
	Type       string                 // "object", "array", "string", "number", "integer", "boolean", "null"
	Properties map[string]*JSONSchema // for "object"
	Required   []string               // for "object"
	Items      *JSONSchema            // for "array"
	Enum       []any                  // enum values
	Const      any                    // const value
	MinLength  int                    // for "string"
	MaxLength  int                    // for "string", 0 = unlimited

	// $ref support: Ref is resolved against Definitions or Defs before conversion.
	Ref         string                 `json:"$ref,omitempty"`
	Definitions map[string]*JSONSchema `json:"definitions,omitempty"`
	Defs        map[string]*JSONSchema `json:"$defs,omitempty"`

	// Unsupported fields - presence triggers an error from Convert.
	OneOf                []any  `json:"oneOf,omitempty"`
	AnyOf                []any  `json:"anyOf,omitempty"`
	AllOf                []any  `json:"allOf,omitempty"`
	Pattern              string `json:"pattern,omitempty"`
	AdditionalProperties *bool  `json:"additionalProperties,omitempty"`
}

JSONSchema represents the subset of JSON Schema supported by the converter. Unsupported features (oneOf, anyOf, allOf, pattern, additionalProperties) cause Convert to return an error. $ref is resolved before conversion using the Definitions or Defs maps.