engine

README

engine — go-tex

A pure-Go (no cgo) TeX engine, under construction — aimed at functional parity with a TeX distribution, not a subset. This module is the engine's mouth and gullet: a faithful re-implementation of TeX's category-code tokenizer, the equivalents table (eqtb) with grouping/scoping, macro definition with delimited parameters, and the full expansion machinery.

It is developed the way parity is actually reachable — reimplement the engine faithfully, gated by TeX's own conformance oracle — and later run the real LaTeX kernel and packages on it (they are TeX macros), diffing output against pdftex/xetex. The primary gate here is the conformance ratchet (TestConformance): TeX snippets checked byte-for-byte against real-TeX output.

Working today (verified, faithful)

• Category-code tokenizer — full catcode table, comments, control words/symbols.
• Macros — \def (undelimited, delimited, and grouped parameters, with backtracking on partial delimiter matches), \edef/\gdef/\xdef, \let (to macros, primitives, undefined, and character tokens), \global.
• Expansion — \expandafter, \csname/\endcsname, \noexpand, \string, \the, \number, \romannumeral, \meaning, \uppercase/\lowercase.
• Conditionals — \if, \ifnum, \ifx, \ifcat, \ifodd, \ifcase, \iftrue/\iffalse, with \else/\or/\fi and nesting.
• Registers & arithmetic — \count, \advance, \multiply, \chardef, \catcode, read via \the/\count.
• Grouping — {…}, \begingroup/\endgroup, save/restore of meanings, registers, and catcodes; \global escapes the current group.

Faithfulness is checked on subtleties only real TeX gets right (a control word absorbing its following space; significant spaces in conditional branches).

out, _ := engine.New().Run(`\def\twice#1{#1#1}\message{\twice{\twice A}}`)
// out == "AAAA"

Status & roadmap to parity

This is stage 1 (mouth + gullet). The remaining stages, each gated by an objective oracle:

1. ✅ Mouth + gullet — tokenizer, eqtb, macros, expansion (this module).
2. ⬜ Stomach — box/glue/penalty model, h/v lists, Knuth–Plass line breaking, page builder, \halign — gated by the TRIP test.
3. ⬜ Math (Appendix G) — go-tex/math is the starting point.
4. ⬜ Fonts — TFM + OpenType (via go-opentype).
5. ⬜ Output — DVI → PDF (via go-pdfkit), then run the real latex.ltx + packages, gated by PDF-diff vs pdftex/xetex.

Because it is under construction, line coverage (currently ~84%) rises as subsystems land; the meaningful gate is the growing conformance ratchet, not a fixed coverage figure. Pure Go, CGO=0, go vet clean, green across six 64-bit arches plus js/wasm and wasip1/wasm.

License

BSD-3-Clause — see LICENSE. Copyright the go-tex/engine authors.

Documentation

Overview

Package engine is the core of a pure-Go (CGO=0) TeX engine: a faithful re-implementation of TeX's mouth and gullet — category-code tokenization, the equivalents table (eqtb) with grouping/scoping, macro definition with delimited parameters, and the expansion machinery (\def, \edef, \let, \expandafter, \csname, \noexpand, \string, \the, \number, conditionals, integer registers). It is the foundation on which the real LaTeX kernel and packages will run, gated by TeX's own conformance suite (the TRIP test) — the path to functional parity with a TeX distribution, not a subset.

Index

• Constants
• func CompileToPDF(src []byte, opt Options, w io.Writer) (int, error)
• func CompileToSVGPages(src []byte, opt Options) ([]string, error)
• func LineAt(spans []SourceSpan, x, y float64) int
• func RenderPageSVG(page VBox, font *OpenTypeFont, pageW, pageH, marginX, marginY float64) string
• type Char
• type Engine
  • func New() *Engine
  • func NewDocument(opt Options) (*Engine, error)
  • func (e *Engine) LoadFormat(src string) error
  • func (e *Engine) LoadLaTeX() error
  • func (e *Engine) LoadPlain() error
  • func (e *Engine) Page() *boxNode
  • func (e *Engine) Pages() []*boxNode
  • func (e *Engine) Position() (line, col int)
  • func (e *Engine) RenderBox(i int, margin float64) string
  • func (e *Engine) RenderPDF(w io.Writer, margin float64) error
  • func (e *Engine) RenderPage(margin float64) string
  • func (e *Engine) RenderPages(margin float64) []string
  • func (e *Engine) Run(src string) (string, error)
  • func (e *Engine) SetFont(f fontFace)
  • func (e *Engine) SourceSpans(margin float64) [][]SourceSpan
  • func (e *Engine) Typeset(src string, m FontMetrics, ...) (Paragraph, bool)
• type FontMetrics
• type HBox
• type Item
  • func Box(w float64) Item
  • func Glue(w, stretch, shrink float64) Item
  • func Glyph(r rune, w, h, d float64) Item
  • func Penalty(w, p float64, flagged bool) Item
• type ItemKind
• type Line
  • func KnuthPlass(items []Item, lineWidth, tolerance, linePenalty float64) ([]Line, bool)
• type Node
• type OpenTypeFont
  • func NewOpenTypeFont(fontBytes []byte, sizePx int) (*OpenTypeFont, error)
  • func (o *OpenTypeFont) CharDims(r rune) (float64, float64, float64)
  • func (o *OpenTypeFont) Space() (float64, float64, float64)
• type Options
• type Paragraph
  • func BuildParagraph(items []Item, lineWidth, tolerance, linePenalty, baselineskip float64) (Paragraph, bool)
• type Rule
• type SetGlue
• type SourceError
  • func (s SourceError) Error() string
• type SourceSpan
  • func RectsForLine(spans []SourceSpan, line int) []SourceSpan
• type VBox
  • func BuildPages(paras []VBox, vsize, parskip float64) []VBox

Constants

const InfPenalty = 10000.0

InfPenalty is TeX's "infinite" penalty (∞ = forbidden break, −∞ = forced).

const MiniLaTeX = `` /* 167-byte string literal not displayed */

MiniLaTeX is a small LaTeX-flavoured kernel written *in TeX* — the engine runs these macro definitions through its gullet exactly as a real format does; they are not reimplemented in Go. It is deliberately tiny (the road to parity is to grow this by loading the real latex.ltx, not to hand-code commands).

const MiniLaTeXKernel = `` /* 17816-byte string literal not displayed */

MiniLaTeXKernel is the LaTeX-flavoured macro layer loaded by LoadLaTeX (after the Plain macros).

const Plain = `` /* 1517-byte string literal not displayed */

Plain is a small set of plain-TeX structural macros, written *in TeX* on top of the box/glue primitives (\hbox to, \hfil, \vskip). Loaded with LoadPlain, they let a document use the familiar commands without any Go-side support — the same growth path as the kernel: add macros, do not hand-code commands.

Functions

func CompileToPDF

func CompileToPDF(src []byte, opt Options, w io.Writer) (int, error)

CompileToPDF processes TeX source and writes a PDF to w, returning the page count. A document's own \font/\hsize/… override the option defaults.

func CompileToSVGPages

func CompileToSVGPages(src []byte, opt Options) ([]string, error)

CompileToSVGPages processes TeX source and returns one SVG string per page — the form an editor preview pane consumes directly.

func LineAt

func LineAt(spans []SourceSpan, x, y float64) int

LineAt returns the source line of the last glyph span whose box contains (x, y), or 0 when the point is over no glyph. Last-wins so nested/overlapping content (a table cell over its row) resolves to the innermost glyph painted there.

func RenderPageSVG

func RenderPageSVG(page VBox, font *OpenTypeFont, pageW, pageH, marginX, marginY float64) string

RenderPageSVG paints a page box onto an SVG of the given dimensions, with the content offset by (marginX, marginY). Glyphs are drawn as vector paths from the font; rules as rectangles.

Types

type Char

type Char struct {
	R       rune
	W, H, D float64
}

Char is a set glyph (a leaf box with width/height/depth).

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine holds all TeX state: the input stack, the eqtb (control-sequence meanings), integer registers, category codes, the grouping save stack, and the \message output buffer.

func New

func New() *Engine

New builds an engine with TeX's default category codes and primitives loaded.

func NewDocument

func NewDocument(opt Options) (*Engine, error)

NewDocument builds an engine configured from opts: the Plain macros loaded (unless NoPlain) and the text font family set. It is the starting point for programmatic use when callers want to Run source incrementally before rendering.

func (*Engine) LoadFormat

func (e *Engine) LoadFormat(src string) error

LoadFormat executes a string of TeX definitions (a format/preamble) through the gullet, defining its macros in the engine's eqtb without typesetting.

func (*Engine) LoadLaTeX

func (e *Engine) LoadLaTeX() error

LoadLaTeX loads the Plain macros (if not already) and the minimal LaTeX kernel.

func (*Engine) LoadPlain

func (e *Engine) LoadPlain() error

LoadPlain defines the Plain structural macros in the engine.

func (*Engine) Page

func (e *Engine) Page() *boxNode

Page vpacks the main vertical list (everything contributed at top level) into a single vbox at natural height. Empty (nil) if nothing was contributed.

func (*Engine) Pages

func (e *Engine) Pages() []*boxNode

Pages splits the main vertical list into pages using TeX's cost-based page builder (§1005): at each legal breakpoint it forms a cost = page badness + penalty, breaks at the least-cost point once the page would overflow, honours a forced break (\penalty ≤ −10000) immediately, and never breaks at \penalty ≥ 10000. Each page is vpacked at natural height. (Insertions/\topskip are future.)

func (*Engine) Position

func (e *Engine) Position() (line, col int)

Position returns the 1-based line and 0-based column of the input the engine is currently reading — the location a diagnostic should point at.

func (*Engine) RenderBox

func (e *Engine) RenderBox(i int, margin float64) string

RenderBox renders box register i to an SVG string with a uniform margin (pt). Empty if the register is void.

func (*Engine) RenderPDF

func (e *Engine) RenderPDF(w io.Writer, margin float64) error

RenderPDF writes the main vertical list, split into \vsize pages, as a PDF to w. Each page is (content width + 2·margin) × (content height + 2·margin) points. A current OpenType font (with embeddable bytes) is required to draw text.

func (*Engine) RenderPage

func (e *Engine) RenderPage(margin float64) string

RenderPage renders the main vertical list to an SVG page with the given margin.

func (*Engine) RenderPages

func (e *Engine) RenderPages(margin float64) []string

RenderPages renders each page of the main vertical list to its own SVG string.

func (*Engine) Run

func (e *Engine) Run(src string) (string, error)

Run tokenizes src as the base input and processes it to completion, returning the accumulated \message output.

func (*Engine) SetFont

func (e *Engine) SetFont(f fontFace)

SetFont sets the current font used to measure and render characters in horizontal mode. Passing an *OpenTypeFont (or any fontFace) is the Go-level stand-in for TeX's \font primitive until font-file loading via \font lands.

func (*Engine) SourceSpans

func (e *Engine) SourceSpans(margin float64) [][]SourceSpan

SourceSpans returns, for each page of the built document, the glyph spans that tie output back to source — the data behind click-to-source and jump-to-line. margin matches the value passed to the SVG/PDF renderers so coordinates align.

func (*Engine) Typeset

func (e *Engine) Typeset(src string, m FontMetrics, lineWidth, tolerance, linePenalty, baselineskip float64) (Paragraph, bool)

Typeset runs the gullet over src (expanding macros), builds a horizontal list with the given font's metrics (glyph boxes and interword glue), and breaks it into a paragraph box with Knuth–Plass. It stops at \par, end of input, or an undefined control sequence.

type FontMetrics

type FontMetrics interface {
	CharDims(r rune) (w, h, d float64)
	Space() (w, stretch, shrink float64)
}

FontMetrics supplies box dimensions to the typesetter: the advance/height/ depth of a glyph, and the interword glue.

type HBox

type HBox struct {
	W, H, D float64
	GlueSet float64 // adjustment ratio applied to the glue (for reference)
	List    []Node
}

HBox is a horizontal box: its list is set left-to-right on a common baseline.

type Item

type Item struct {
	Kind            ItemKind
	Width           float64
	Height, Depth   float64 // box only (glyph metrics)
	R               rune    // box only (the glyph, 0 if none)
	Stretch, Shrink float64 // glue only
	Penalty         float64 // penalty only
	Flagged         bool    // penalty only (e.g. a hyphen) — consecutive flags are penalised
}

Item is one element of a horizontal list.

func Box

func Box(w float64) Item

Box, Glue and Penalty are constructors.

func Glue

func Glue(w, stretch, shrink float64) Item

func Glyph

func Glyph(r rune, w, h, d float64) Item

Glyph is a box carrying a rune and its height/depth (used by the typesetter).

func Penalty

func Penalty(w, p float64, flagged bool) Item

type ItemKind

type ItemKind uint8

ItemKind classifies a horizontal-list item.

const (
	KBox     ItemKind = iota // a box of fixed width
	KGlue                    // stretchable/shrinkable space (a legal breakpoint after a box)
	KPenalty                 // a penalty (a legal breakpoint; ±InfPenalty = forbidden/forced)
)

type Line

type Line struct {
	Start, End int     // item index range [Start, End) actually set on the line
	Ratio      float64 // glue adjustment ratio r (−1 fully shrunk … +tolerance stretched)
}

Line describes one output line of a broken paragraph.

func KnuthPlass

func KnuthPlass(items []Item, lineWidth, tolerance, linePenalty float64) ([]Line, bool)

KnuthPlass breaks items into lines of the given width, minimising total demerits (linePenalty is TeX's \linepenalty). It returns the chosen lines in order and ok=false if no sequence of feasible breaks exists within tolerance.

type Node

type Node interface {
	// contains filtered or unexported methods
}

Node is an element of a packed box tree.

type OpenTypeFont

type OpenTypeFont struct {
	// contains filtered or unexported fields
}

OpenTypeFont adapts a go-opentype face to FontMetrics (pdftex uses TFM; a future TFM backend will satisfy the same interface).

func NewOpenTypeFont

func NewOpenTypeFont(fontBytes []byte, sizePx int) (*OpenTypeFont, error)

NewOpenTypeFont builds a metrics source from a font and a pixel size.

func (*OpenTypeFont) CharDims

func (o *OpenTypeFont) CharDims(r rune) (float64, float64, float64)

CharDims returns a glyph's advance width and its ink height above and depth below the baseline, in pixels.

func (*OpenTypeFont) Space

func (o *OpenTypeFont) Space() (float64, float64, float64)

Space returns TeX-like interword glue derived from the space advance.

type Options

type Options struct {
	Font       []byte  // roman text font (.ttf/.otf); nil ⇒ the built-in default
	BoldFont   []byte  // optional bold face, bound to \bf (so \textbf really bolds)
	ItalicFont []byte  // optional italic face, bound to \it (so \textit/\emph slant)
	MonoFont   []byte  // optional monospace face, bound to \tt (so \texttt is fixed-width)
	SansFont   []byte  // optional sans-serif face, bound to \sf (so \textsf is sans)
	Size       int     // font size in points (0 ⇒ 10)
	Margin     float64 // page margin in points (0 ⇒ 72)
	NoPlain    bool    // set to omit the Plain macros
	Date       string  // \today text (a pure-Go wasm build has no clock; supply it here)
}

Options configures a compile. The zero value is valid: a built-in font at 10pt with a 72pt (1 inch) margin.

type Paragraph

type Paragraph struct {
	Box   VBox
	Lines []Line
}

Paragraph is a fully built paragraph: a vertical box of line boxes.

func BuildParagraph

func BuildParagraph(items []Item, lineWidth, tolerance, linePenalty, baselineskip float64) (Paragraph, bool)

BuildParagraph runs Knuth–Plass over items, packs each resulting line to lineWidth with hpack, and stacks the line boxes with vpack. lineHeight and lineDepth are used for boxes lacking explicit metrics; baselineskip sets the inter-line spacing.

type Rule

type Rule struct{ W, H, D float64 }

Rule is a filled rectangle (e.g. a fraction bar or \hrule).

type SetGlue

type SetGlue struct {
	W, Stretch, Shrink float64
	Set                float64
}

SetGlue is glue after packing: Set is the actual width it was stretched or shrunk to.

type SourceError

type SourceError struct {
	Line, Col int // 1-based line, 0-based column (0/0 = unknown)
	Msg       string
}

SourceError is an engine error carrying the source location it occurred at, so a caller (a CLI, loom's compile panel) can point the user at the exact line.

func (SourceError) Error

func (s SourceError) Error() string

type SourceSpan

type SourceSpan struct {
	Line       int
	X, Y, W, H float64
}

SourceSpan is one rendered glyph's bounding box on a page (points, SVG coordinates: origin top-left, Y is the box top) tagged with the source line it came from. It is the programmatic form of the SVG's data-l groups: a caller maps a click (x, y) → line, or a line → its output rectangles.

func RectsForLine

func RectsForLine(spans []SourceSpan, line int) []SourceSpan

RectsForLine returns every glyph span originating from the given source line — the boxes an editor highlights when the cursor sits on that line.

type VBox

type VBox struct {
	W, H, D float64
	List    []Node
}

VBox is a vertical box: its list is stacked; the reference point is the baseline of the last box (TeX's \vbox).

func BuildPages

func BuildPages(paras []VBox, vsize, parskip float64) []VBox

BuildPages stacks paragraph boxes into pages no taller than vsize, inserting parskip between paragraphs. A paragraph taller than vsize occupies its own (overfull) page rather than being split (block splitting comes later).