engine

README

engine — go-tex

A pure-Go (no cgo) TeX engine, under construction — aimed at functional parity with a TeX distribution, not a subset. This module is the engine's mouth and gullet: a faithful re-implementation of TeX's category-code tokenizer, the equivalents table (eqtb) with grouping/scoping, macro definition with delimited parameters, and the full expansion machinery.

It is developed the way parity is actually reachable — reimplement the engine faithfully, gated by TeX's own conformance oracle — and later run the real LaTeX kernel and packages on it (they are TeX macros), diffing output against pdftex/xetex. The primary gate here is the conformance ratchet (TestConformance): TeX snippets checked byte-for-byte against real-TeX output.

Working today (verified, faithful)

• Category-code tokenizer — full catcode table, comments, control words/symbols.
• Macros — \def (undelimited, delimited, and grouped parameters, with backtracking on partial delimiter matches), \edef/\gdef/\xdef, \let (to macros, primitives, undefined, and character tokens), \global.
• Expansion — \expandafter, \csname/\endcsname, \noexpand, \string, \the, \number, \romannumeral, \meaning, \uppercase/\lowercase.
• Conditionals — \if, \ifnum, \ifx, \ifcat, \ifodd, \ifcase, \iftrue/\iffalse, with \else/\or/\fi and nesting.
• Registers & arithmetic — \count, \advance, \multiply, \chardef, \catcode, read via \the/\count.
• Grouping — {…}, \begingroup/\endgroup, save/restore of meanings, registers, and catcodes; \global escapes the current group.

Faithfulness is checked on subtleties only real TeX gets right (a control word absorbing its following space; significant spaces in conditional branches).

out, _ := engine.New().Run(`\def\twice#1{#1#1}\message{\twice{\twice A}}`)
// out == "AAAA"

Status & roadmap to parity

This is stage 1 (mouth + gullet). The remaining stages, each gated by an objective oracle:

1. ✅ Mouth + gullet — tokenizer, eqtb, macros, expansion (this module).
2. ⬜ Stomach — box/glue/penalty model, h/v lists, Knuth–Plass line breaking, page builder, \halign — gated by the TRIP test.
3. ⬜ Math (Appendix G) — go-tex/math is the starting point.
4. ⬜ Fonts — TFM + OpenType (via go-opentype).
5. ⬜ Output — DVI → PDF (via go-pdfkit), then run the real latex.ltx + packages, gated by PDF-diff vs pdftex/xetex.

Because it is under construction, line coverage (currently ~84%) rises as subsystems land; the meaningful gate is the growing conformance ratchet, not a fixed coverage figure. Pure Go, CGO=0, go vet clean, green across six 64-bit arches plus js/wasm and wasip1/wasm.

License

BSD-3-Clause — see LICENSE. Copyright the go-tex/engine authors.

Documentation

Overview

Package engine is the core of a pure-Go (CGO=0) TeX engine: a faithful re-implementation of TeX's mouth and gullet — category-code tokenization, the equivalents table (eqtb) with grouping/scoping, macro definition with delimited parameters, and the expansion machinery (\def, \edef, \let, \expandafter, \csname, \noexpand, \string, \the, \number, conditionals, integer registers). It is the foundation on which the real LaTeX kernel and packages will run, gated by TeX's own conformance suite (the TRIP test) — the path to functional parity with a TeX distribution, not a subset.

Index

• Constants
• func RenderPageSVG(page VBox, font *OpenTypeFont, pageW, pageH, marginX, marginY float64) string
• type Char
• type Engine
  • func New() *Engine
  • func (e *Engine) LoadFormat(src string) error
  • func (e *Engine) LoadPlain() error
  • func (e *Engine) Page() *boxNode
  • func (e *Engine) Pages() []*boxNode
  • func (e *Engine) RenderBox(i int, margin float64) string
  • func (e *Engine) RenderPDF(w io.Writer, margin float64) error
  • func (e *Engine) RenderPage(margin float64) string
  • func (e *Engine) RenderPages(margin float64) []string
  • func (e *Engine) Run(src string) (string, error)
  • func (e *Engine) SetFont(f fontFace)
  • func (e *Engine) Typeset(src string, m FontMetrics, ...) (Paragraph, bool)
• type FontMetrics
• type HBox
• type Item
  • func Box(w float64) Item
  • func Glue(w, stretch, shrink float64) Item
  • func Glyph(r rune, w, h, d float64) Item
  • func Penalty(w, p float64, flagged bool) Item
• type ItemKind
• type Line
  • func KnuthPlass(items []Item, lineWidth, tolerance, linePenalty float64) ([]Line, bool)
• type Node
• type OpenTypeFont
  • func NewOpenTypeFont(fontBytes []byte, sizePx int) (*OpenTypeFont, error)
  • func (o *OpenTypeFont) CharDims(r rune) (float64, float64, float64)
  • func (o *OpenTypeFont) Space() (float64, float64, float64)
• type Paragraph
  • func BuildParagraph(items []Item, lineWidth, tolerance, linePenalty, baselineskip float64) (Paragraph, bool)
• type Rule
• type SetGlue
• type VBox
  • func BuildPages(paras []VBox, vsize, parskip float64) []VBox

Constants

const InfPenalty = 10000.0

InfPenalty is TeX's "infinite" penalty (∞ = forbidden break, −∞ = forced).

const MiniLaTeX = `` /* 167-byte string literal not displayed */

MiniLaTeX is a small LaTeX-flavoured kernel written *in TeX* — the engine runs these macro definitions through its gullet exactly as a real format does; they are not reimplemented in Go. It is deliberately tiny (the road to parity is to grow this by loading the real latex.ltx, not to hand-code commands).

const Plain = `` /* 275-byte string literal not displayed */

Plain is a small set of plain-TeX structural macros, written *in TeX* on top of the box/glue primitives (\hbox to, \hfil, \vskip). Loaded with LoadPlain, they let a document use the familiar commands without any Go-side support — the same growth path as the kernel: add macros, do not hand-code commands.

Functions

func RenderPageSVG

func RenderPageSVG(page VBox, font *OpenTypeFont, pageW, pageH, marginX, marginY float64) string

RenderPageSVG paints a page box onto an SVG of the given dimensions, with the content offset by (marginX, marginY). Glyphs are drawn as vector paths from the font; rules as rectangles.

Types

type Char

type Char struct {
	R       rune
	W, H, D float64
}

Char is a set glyph (a leaf box with width/height/depth).

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine holds all TeX state: the input stack, the eqtb (control-sequence meanings), integer registers, category codes, the grouping save stack, and the \message output buffer.

func New

func New() *Engine

New builds an engine with TeX's default category codes and primitives loaded.

func (*Engine) LoadFormat

func (e *Engine) LoadFormat(src string) error

LoadFormat executes a string of TeX definitions (a format/preamble) through the gullet, defining its macros in the engine's eqtb without typesetting.

func (*Engine) LoadPlain

func (e *Engine) LoadPlain() error

LoadPlain defines the Plain structural macros in the engine.

func (*Engine) Page

func (e *Engine) Page() *boxNode

Page vpacks the main vertical list (everything contributed at top level) into a single vbox at natural height. Empty (nil) if nothing was contributed.

func (*Engine) Pages

func (e *Engine) Pages() []*boxNode

Pages splits the main vertical list into pages, each vpacked at natural height (≤ \vsize where the material allows a legal break). A single page taller than \vsize because one box exceeds it is kept whole (overfull) rather than lost.

func (*Engine) RenderBox

func (e *Engine) RenderBox(i int, margin float64) string

RenderBox renders box register i to an SVG string with a uniform margin (pt). Empty if the register is void.

func (*Engine) RenderPDF

func (e *Engine) RenderPDF(w io.Writer, margin float64) error

RenderPDF writes the main vertical list, split into \vsize pages, as a PDF to w. Each page is (content width + 2·margin) × (content height + 2·margin) points. A current OpenType font (with embeddable bytes) is required to draw text.

func (*Engine) RenderPage

func (e *Engine) RenderPage(margin float64) string

RenderPage renders the main vertical list to an SVG page with the given margin.

func (*Engine) RenderPages

func (e *Engine) RenderPages(margin float64) []string

RenderPages renders each page of the main vertical list to its own SVG string.

func (*Engine) Run

func (e *Engine) Run(src string) (string, error)

Run tokenizes src as the base input and processes it to completion, returning the accumulated \message output.

func (*Engine) SetFont

func (e *Engine) SetFont(f fontFace)

SetFont sets the current font used to measure and render characters in horizontal mode. Passing an *OpenTypeFont (or any fontFace) is the Go-level stand-in for TeX's \font primitive until font-file loading via \font lands.

func (*Engine) Typeset

func (e *Engine) Typeset(src string, m FontMetrics, lineWidth, tolerance, linePenalty, baselineskip float64) (Paragraph, bool)

Typeset runs the gullet over src (expanding macros), builds a horizontal list with the given font's metrics (glyph boxes and interword glue), and breaks it into a paragraph box with Knuth–Plass. It stops at \par, end of input, or an undefined control sequence.

type FontMetrics

type FontMetrics interface {
	CharDims(r rune) (w, h, d float64)
	Space() (w, stretch, shrink float64)
}

FontMetrics supplies box dimensions to the typesetter: the advance/height/ depth of a glyph, and the interword glue.

type HBox

type HBox struct {
	W, H, D float64
	GlueSet float64 // adjustment ratio applied to the glue (for reference)
	List    []Node
}

HBox is a horizontal box: its list is set left-to-right on a common baseline.

type Item

type Item struct {
	Kind            ItemKind
	Width           float64
	Height, Depth   float64 // box only (glyph metrics)
	R               rune    // box only (the glyph, 0 if none)
	Stretch, Shrink float64 // glue only
	Penalty         float64 // penalty only
	Flagged         bool    // penalty only (e.g. a hyphen) — consecutive flags are penalised
}

Item is one element of a horizontal list.

func Box

func Box(w float64) Item

Box, Glue and Penalty are constructors.

func Glue

func Glue(w, stretch, shrink float64) Item

func Glyph

func Glyph(r rune, w, h, d float64) Item

Glyph is a box carrying a rune and its height/depth (used by the typesetter).

func Penalty

func Penalty(w, p float64, flagged bool) Item

type ItemKind

type ItemKind uint8

ItemKind classifies a horizontal-list item.

const (
	KBox     ItemKind = iota // a box of fixed width
	KGlue                    // stretchable/shrinkable space (a legal breakpoint after a box)
	KPenalty                 // a penalty (a legal breakpoint; ±InfPenalty = forbidden/forced)
)

type Line

type Line struct {
	Start, End int     // item index range [Start, End) actually set on the line
	Ratio      float64 // glue adjustment ratio r (−1 fully shrunk … +tolerance stretched)
}

Line describes one output line of a broken paragraph.

func KnuthPlass

func KnuthPlass(items []Item, lineWidth, tolerance, linePenalty float64) ([]Line, bool)

KnuthPlass breaks items into lines of the given width, minimising total demerits (linePenalty is TeX's \linepenalty). It returns the chosen lines in order and ok=false if no sequence of feasible breaks exists within tolerance.

type Node

type Node interface {
	// contains filtered or unexported methods
}

Node is an element of a packed box tree.

type OpenTypeFont

type OpenTypeFont struct {
	// contains filtered or unexported fields
}

OpenTypeFont adapts a go-opentype face to FontMetrics (pdftex uses TFM; a future TFM backend will satisfy the same interface).

func NewOpenTypeFont

func NewOpenTypeFont(fontBytes []byte, sizePx int) (*OpenTypeFont, error)

NewOpenTypeFont builds a metrics source from a font and a pixel size.

func (*OpenTypeFont) CharDims

func (o *OpenTypeFont) CharDims(r rune) (float64, float64, float64)

CharDims returns a glyph's advance width and its ink height above and depth below the baseline, in pixels.

func (*OpenTypeFont) Space

func (o *OpenTypeFont) Space() (float64, float64, float64)

Space returns TeX-like interword glue derived from the space advance.

type Paragraph

type Paragraph struct {
	Box   VBox
	Lines []Line
}

Paragraph is a fully built paragraph: a vertical box of line boxes.

func BuildParagraph

func BuildParagraph(items []Item, lineWidth, tolerance, linePenalty, baselineskip float64) (Paragraph, bool)

BuildParagraph runs Knuth–Plass over items, packs each resulting line to lineWidth with hpack, and stacks the line boxes with vpack. lineHeight and lineDepth are used for boxes lacking explicit metrics; baselineskip sets the inter-line spacing.

type Rule

type Rule struct{ W, H, D float64 }

Rule is a filled rectangle (e.g. a fraction bar or \hrule).

type SetGlue

type SetGlue struct {
	W, Stretch, Shrink float64
	Set                float64
}

SetGlue is glue after packing: Set is the actual width it was stretched or shrunk to.

type VBox

type VBox struct {
	W, H, D float64
	List    []Node
}

VBox is a vertical box: its list is stacked; the reference point is the baseline of the last box (TeX's \vbox).

func BuildPages

func BuildPages(paras []VBox, vsize, parskip float64) []VBox

BuildPages stacks paragraph boxes into pages no taller than vsize, inserting parskip between paragraphs. A paragraph taller than vsize occupies its own (overfull) page rather than being split (block splitting comes later).