engine

README

engine — go-tex

A pure-Go (no cgo) TeX engine aimed at functional parity with a TeX distribution, not a subset. It is a faithful re-implementation of TeX — the category-code tokenizer, the equivalents table (eqtb) with grouping/scoping, macro definition with delimited parameters and the full expansion machinery (the mouth and gullet); a scaled-point box/glue/penalty stomach with Knuth–Plass line breaking and a cost-based page builder; math via go-tex/math; OpenType fonts; and PDF + SVG output — and on top of it, it loads and runs the genuine LaTeX classes: \documentclass{article}, {report} and {book} execute the real, embedded .cls files, in native builds and in the browser (js/wasm), with no TeXLive.

It is developed the way parity is actually reachable — reimplement the engine faithfully, gated by objective oracles — then run the real LaTeX classes and packages on it (they are TeX macros). Two gates hold the line: the conformance ratchet (TestConformance, TeX snippets checked byte-for-byte against real-TeX output) and a fidelity check that compares whole-document prose against a real LaTeX engine (tectonic).

Working today (verified, faithful)

• Category-code tokenizer — full catcode table, comments, control words/symbols.
• Macros — \def (undelimited, delimited, and grouped parameters, with backtracking on partial delimiter matches), \edef/\gdef/\xdef, \let (to macros, primitives, undefined, and character tokens), \global.
• Expansion — \expandafter, \csname/\endcsname, \noexpand, \string, \the, \number, \romannumeral, \meaning, \uppercase/\lowercase.
• Conditionals — \if, \ifnum, \ifx, \ifcat, \ifodd, \ifcase, \iftrue/\iffalse, with \else/\or/\fi and nesting.
• Registers & arithmetic — \count, \advance, \multiply, \chardef, \catcode, read via \the/\count.
• Grouping — {…}, \begingroup/\endgroup, save/restore of meanings, registers, and catcodes; \global escapes the current group.

Faithfulness is checked on subtleties only real TeX gets right (a control word absorbing its following space; significant spaces in conditional branches).

out, _ := engine.New().Run(`\def\twice#1{#1#1}\message{\twice{\twice A}}`)
// out == "AAAA"

Real-world documents (lenient mode)

A real third-party paper pulls in classes, packages, fonts, figures and .bib files that a from-scratch engine does not carry. Strict mode aborts on the first such gap (as TeX does). Lenient mode (gotex -lenient, or Options{Lenient: true}) turns those gaps into best-effort no-ops so an editor preview shows the typesettable content instead of one hard error:

• an undefined command is skipped, along with its likely [opt]{arg} block;
• an unloadable figure becomes a framed placeholder of the requested size;
• a math macro go-tex/math doesn't know drops that one equation;
• a \setlength on an unmodelled length, and a missing \input/\bibliography/ \font file, are ignored.

Every skipped construct is tallied ((*Engine).SkippedCommands) so a caller can report what was dropped. On a sample of 54 real arXiv sources, strict mode compiled 0 end-to-end (each hit a package command in the preamble); lenient mode produces a multi-page PDF for all 54, with real, selectable prose text. It is a preview aid, not a fidelity claim — the roadmap below is how the gaps close for real.

Loading real classes and packages

\documentclass and \usepackage (and \RequirePackage, \LoadClass, \LoadClassWithOptions) do more than emulate: they resolve and load the real .cls/.sty — from the document's own directory (an arXiv paper's bundled class/package), a TEXINPUTS/GOTEX_TEXMF search path, or an embedded base set — making @ a letter and running the file's own \newcommand/\def/… on the engine. The LaTeX2e option mechanism runs too: \DeclareOption, \DeclareOption*, \ProcessOptions, \ExecuteOptions, \CurrentOption, \PassOptionsToPackage/\PassOptionsToClass, plus \IfFileExists/ \InputIfFileExists. A file loads tolerantly — a command the engine lacks is skipped, so a real class contributes what it can — and a runaway-expansion guard bounds macro expansion so a pathological or partially-supported file can never hang (it stops with partial output in lenient mode, an error in strict). Distribution-heavy packages the engine emulates natively or better as stubs (geometry, tikz, hyperref, graphicx, encodings, …) are not loaded from disk.

The standard base classes run for real. \documentclass{article}, {report} and {book} load and execute the genuine, embedded LaTeX classes (article.cls/report.cls/book.cls + their size option files, LPPL, verbatim) — not an emulation. Everything they need is in place: the LaTeX2e kernel helpers, a class-kernel substrate (constants, registers, \if@ flags, NFSS font-switch aliases), \newcommand*/\DeclareOldFontCommand, the rubber-glue and <factor><internal-dimen> length scanner, numbered \@startsection with \@tocentry, \secdef via \@dblarg (so \chapter works), \@float figure/table captions, \@starttoc bridged to the engine's two-pass contents table, and — the keystone — stable source lines (loading a 644-line class no longer shifts the line numbers the editor maps glyphs back to). A real \documentclass{article} document typesets a numbered title, a dotted \tableofcontents, numbered sections, and numbered figure/table captions, and it reproduces the reference engine's prose on the fidelity gate. Because the class files are go:embeded and the resolver needs no filesystem, the real classes also run in the js/wasm build — genuine LaTeX class rendering in the browser, with no TeXLive and no server. amsart is embedded and its class loads (its \maketitle even drove real token-register support — \toks/\newtoks — into the engine), but \documentclass{amsart} is kept on the emulation for now: its own \newtheorem…[section] machinery loops on the engine, so it waits on that fix before it is routed to the real class.

Status & roadmap to parity

Each stage is gated by an objective oracle:

1. ✅ Mouth + gullet — tokenizer, eqtb, macros, expansion.
2. ✅ Stomach — box/glue/penalty model in scaled points, h/v lists, Knuth–Plass line breaking with an emergency pass, cost-based page builder, \halign.
3. ✅ Math — $…$ and the display environments delegated to go-tex/math (vector output).
4. ✅ Fonts — OpenType via go-opentype; a built-in font so it runs with no assets, with kerning and ligatures.
5. ✅ Output — PDF (via go-pdfkit, embedded subset fonts, selectable text) and self-contained SVG pages; the SVG carries a source map for click-to-line.
6. ✅ Real classes — \documentclass{article|report|book} loads and runs the genuine embedded LaTeX class (see above), reproducing the reference engine's prose on the fidelity gate — in native builds and in js/wasm.

Next: amsart's \newtheorem fix (then route it to the real class), more real packages (amsmath, hyperref, graphicx), a broader real-document conformance corpus (PDF-diff vs pdftex/xetex), and the TRIP test. Coverage ~91%; the meaningful gate is the conformance ratchet plus the fidelity check against a real LaTeX engine, not a fixed coverage figure. Pure Go, CGO=0, go vet clean, green across three 64-bit arches under qemu plus js/wasm and wasip1/wasm.

License

BSD-3-Clause — see LICENSE. Copyright the go-tex/engine authors.

Documentation

Overview

Package engine is the core of a pure-Go (CGO=0) TeX engine: a faithful re-implementation of TeX's mouth and gullet — category-code tokenization, the equivalents table (eqtb) with grouping/scoping, macro definition with delimited parameters, and the expansion machinery (\def, \edef, \let, \expandafter, \csname, \noexpand, \string, \the, \number, conditionals, integer registers). It is the foundation on which the real LaTeX kernel and packages will run, gated by TeX's own conformance suite (the TRIP test) — the path to functional parity with a TeX distribution, not a subset.

Index

• Constants
• func CompileToPDF(src []byte, opt Options, w io.Writer) (int, error)
• func CompileToSVGPages(src []byte, opt Options) ([]string, error)
• func LineAt(spans []SourceSpan, x, y float64) int
• type Engine
  • func New() *Engine
  • func NewDocument(opt Options) (*Engine, error)
  • func (e *Engine) LoadFormat(src string) error
  • func (e *Engine) LoadLaTeX() error
  • func (e *Engine) LoadPlain() error
  • func (e *Engine) Page() *boxNode
  • func (e *Engine) Pages() []*boxNode
  • func (e *Engine) Position() (line, col int)
  • func (e *Engine) RenderBox(i int, margin float64) string
  • func (e *Engine) RenderPDF(w io.Writer, margin float64) error
  • func (e *Engine) RenderPage(margin float64) string
  • func (e *Engine) RenderPages(margin float64) []string
  • func (e *Engine) Run(src string) (string, error)
  • func (e *Engine) SetFont(f fontFace)
  • func (e *Engine) SkippedCommands() map[string]int
  • func (e *Engine) SourceSpans(margin float64) [][]SourceSpan
• type Item
  • func Box(w float64) Item
  • func Glue(w, stretch, shrink float64) Item
  • func Glyph(r rune, w, h, d float64) Item
  • func Penalty(w, p float64, flagged bool) Item
• type ItemKind
• type Line
  • func KnuthPlass(items []Item, lineWidth, tolerance, linePenalty float64) ([]Line, bool)
• type OpenTypeFont
  • func NewOpenTypeFont(fontBytes []byte, sizePx int) (*OpenTypeFont, error)
  • func (o *OpenTypeFont) CharDims(r rune) (float64, float64, float64)
  • func (o *OpenTypeFont) Space() (float64, float64, float64)
• type Options
• type SourceError
  • func (s SourceError) Error() string
• type SourceSpan
  • func RectsForLine(spans []SourceSpan, line int) []SourceSpan

Constants

const AMSClassSubstrate = `` /* 5120-byte string literal not displayed */

AMSClassSubstrate is the register-and-macro layer amsart builds on beyond the article substrate: plain-TeX scratch registers, the penalty/spacing parameters a class assigns, and best-effort no-ops for NFSS font selection and a few text commands. Loaded last by LoadLaTeX.

const InfPenalty = 10000.0

InfPenalty is TeX's "infinite" penalty (∞ = forbidden break, −∞ = forced).

const LaTeX2eClassKernel = `` /* 11224-byte string literal not displayed */

LaTeX2eClassKernel is the LaTeX2e class-kernel substrate, loaded by LoadLaTeX right after LaTeX2eKernelHelpers.

const LaTeX2eClassLead = `` /* 3875-byte string literal not displayed */

LaTeX2eClassLead is the best-effort high-level class machinery a real class builds on top of the kernel: sectioning (\@startsection and its helpers), the generic \list, and small no-ops. It is intentionally simplified — headings are typeset in their requested style without the exact spacing/numbering of real LaTeX — so that loading a real class produces readable structured output rather than aborting. It is loaded after LaTeX2eClassKernel.

const LaTeX2eKernelHelpers = `` /* 11391-byte string literal not displayed */

LaTeX2eKernelHelpers is the LaTeX2e low-level kernel helper layer, loaded by LoadLaTeX right after MiniLaTeXKernel.

const MiniLaTeX = `` /* 167-byte string literal not displayed */

MiniLaTeX is a small LaTeX-flavoured kernel written *in TeX* — the engine runs these macro definitions through its gullet exactly as a real format does; they are not reimplemented in Go. It is deliberately tiny (the road to parity is to grow this by loading the real latex.ltx, not to hand-code commands).

const MiniLaTeXKernel = `` /* 26311-byte string literal not displayed */

MiniLaTeXKernel is the LaTeX-flavoured macro layer loaded by LoadLaTeX (after the Plain macros).

const Plain = `` /* 1517-byte string literal not displayed */

Plain is a small set of plain-TeX structural macros, written *in TeX* on top of the box/glue primitives (\hbox to, \hfil, \vskip). Loaded with LoadPlain, they let a document use the familiar commands without any Go-side support — the same growth path as the kernel: add macros, do not hand-code commands.

Functions

func CompileToPDF

func CompileToPDF(src []byte, opt Options, w io.Writer) (int, error)

CompileToPDF processes TeX source and writes a PDF to w, returning the page count. A document's own \font/\hsize/… override the option defaults.

func CompileToSVGPages

func CompileToSVGPages(src []byte, opt Options) ([]string, error)

CompileToSVGPages processes TeX source and returns one SVG string per page — the form an editor preview pane consumes directly.

func LineAt

func LineAt(spans []SourceSpan, x, y float64) int

LineAt returns the source line of the last glyph span whose box contains (x, y), or 0 when the point is over no glyph. Last-wins so nested/overlapping content (a table cell over its row) resolves to the innermost glyph painted there.

Types

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine holds all TeX state: the input stack, the eqtb (control-sequence meanings), integer registers, category codes, the grouping save stack, and the \message output buffer.

func New

func New() *Engine

New builds an engine with TeX's default category codes and primitives loaded.

func NewDocument

func NewDocument(opt Options) (*Engine, error)

NewDocument builds an engine configured from opts: the Plain macros loaded (unless NoPlain) and the text font family set. It is the starting point for programmatic use when callers want to Run source incrementally before rendering.

func (*Engine) LoadFormat

func (e *Engine) LoadFormat(src string) error

LoadFormat executes a string of TeX definitions (a format/preamble) through the gullet, defining its macros in the engine's eqtb without typesetting.

func (*Engine) LoadLaTeX

func (e *Engine) LoadLaTeX() error

LoadLaTeX loads the Plain macros (if not already) and the minimal LaTeX kernel.

func (*Engine) LoadPlain

func (e *Engine) LoadPlain() error

LoadPlain defines the Plain structural macros in the engine.

func (*Engine) Page

func (e *Engine) Page() *boxNode

Page vpacks the main vertical list (everything contributed at top level) into a single vbox at natural height. Empty (nil) if nothing was contributed.

func (*Engine) Pages

func (e *Engine) Pages() []*boxNode

Pages splits the main vertical list into pages using TeX's cost-based page builder (§1005): at each legal breakpoint it forms a cost = page badness + penalty, breaks at the least-cost point once the page would overflow, honours a forced break (\penalty ≤ −10000) immediately, and never breaks at \penalty ≥ 10000. Each page is vpacked at natural height. (Insertions/\topskip are future.)

func (*Engine) Position

func (e *Engine) Position() (line, col int)

Position returns the 1-based line and 0-based column of the input the engine is currently reading — the location a diagnostic should point at.

func (*Engine) RenderBox

func (e *Engine) RenderBox(i int, margin float64) string

RenderBox renders box register i to an SVG string with a uniform margin (pt). Empty if the register is void.

func (*Engine) RenderPDF

func (e *Engine) RenderPDF(w io.Writer, margin float64) error

RenderPDF writes the main vertical list, split into \vsize pages, as a PDF to w. Each page is (content width + 2·margin) × (content height + 2·margin) points. A current OpenType font (with embeddable bytes) is required to draw text.

func (*Engine) RenderPage

func (e *Engine) RenderPage(margin float64) string

RenderPage renders the main vertical list to an SVG page with the given margin.

func (*Engine) RenderPages

func (e *Engine) RenderPages(margin float64) []string

RenderPages renders each page of the main vertical list to its own SVG string.

func (*Engine) Run

func (e *Engine) Run(src string) (string, error)

Run tokenizes src as the base input and processes it to completion, returning the accumulated \message output.

func (*Engine) SetFont

func (e *Engine) SetFont(f fontFace)

SetFont sets the current font used to measure and render characters in horizontal mode. Passing an *OpenTypeFont (or any fontFace) is the Go-level stand-in for TeX's \font primitive until font-file loading via \font lands.

func (*Engine) SkippedCommands

func (e *Engine) SkippedCommands() map[string]int

SkippedCommands returns, for a lenient compile, how many times each undefined control sequence was skipped (empty when strict or when none were undefined). It lets a caller surface "these commands were dropped" after a preview compile.

func (*Engine) SourceSpans

func (e *Engine) SourceSpans(margin float64) [][]SourceSpan

SourceSpans returns, for each page of the built document, the glyph spans that tie output back to source — the data behind click-to-source and jump-to-line. margin matches the value passed to the SVG/PDF renderers so coordinates align.

type Item

type Item struct {
	Kind            ItemKind
	Width           float64
	Height, Depth   float64 // box only (glyph metrics)
	R               rune    // box only (the glyph, 0 if none)
	Stretch, Shrink float64 // glue only
	Penalty         float64 // penalty only
	Flagged         bool    // penalty only (e.g. a hyphen) — consecutive flags are penalised
}

Item is one element of a horizontal list.

func Box

func Box(w float64) Item

Box, Glue and Penalty are constructors.

func Glue

func Glue(w, stretch, shrink float64) Item

func Glyph

func Glyph(r rune, w, h, d float64) Item

Glyph is a box carrying a rune and its height/depth (used by the typesetter).

func Penalty

func Penalty(w, p float64, flagged bool) Item

type ItemKind

type ItemKind uint8

ItemKind classifies a horizontal-list item.

const (
	KBox     ItemKind = iota // a box of fixed width
	KGlue                    // stretchable/shrinkable space (a legal breakpoint after a box)
	KPenalty                 // a penalty (a legal breakpoint; ±InfPenalty = forbidden/forced)
)

type Line

type Line struct {
	Start, End int     // item index range [Start, End) actually set on the line
	Ratio      float64 // glue adjustment ratio r (−1 fully shrunk … +tolerance stretched)
}

Line describes one output line of a broken paragraph.

func KnuthPlass

func KnuthPlass(items []Item, lineWidth, tolerance, linePenalty float64) ([]Line, bool)

KnuthPlass breaks items into lines of the given width, minimising total demerits (linePenalty is TeX's \linepenalty). It returns the chosen lines in order and ok=false if no sequence of feasible breaks exists within tolerance.

type OpenTypeFont

type OpenTypeFont struct {
	// contains filtered or unexported fields
}

OpenTypeFont adapts a go-opentype face for the engine: it measures glyphs (in points via CharDims and in scaled points via charDimsSP), supplies the interword glue, and returns glyph outline paths for the SVG driver.

func NewOpenTypeFont

func NewOpenTypeFont(fontBytes []byte, sizePx int) (*OpenTypeFont, error)

NewOpenTypeFont builds a metrics source from a font and a pixel size.

func (*OpenTypeFont) CharDims

func (o *OpenTypeFont) CharDims(r rune) (float64, float64, float64)

CharDims returns a glyph's advance width and its ink height above and depth below the baseline, in pixels.

func (*OpenTypeFont) Space

func (o *OpenTypeFont) Space() (float64, float64, float64)

Space returns TeX-like interword glue derived from the space advance.

type Options

type Options struct {
	Font       []byte  // roman text font (.ttf/.otf); nil ⇒ the built-in default
	BoldFont   []byte  // optional bold face, bound to \bf (so \textbf really bolds)
	ItalicFont []byte  // optional italic face, bound to \it (so \textit/\emph slant)
	MonoFont   []byte  // optional monospace face, bound to \tt (so \texttt is fixed-width)
	SansFont   []byte  // optional sans-serif face, bound to \sf (so \textsf is sans)
	Size       int     // font size in points (0 ⇒ 10)
	Margin     float64 // page margin in points (0 ⇒ 72)
	NoPlain    bool    // set to omit the Plain macros
	Date       string  // \today text (a pure-Go wasm build has no clock; supply it here)

	// Lenient turns an undefined control sequence from a fatal error into a
	// skipped command: the engine drops the unknown \cs (and its likely
	// [optional]/{mandatory} argument block) and carries on, recording the name
	// for reporting. This is best-effort "produce something" behaviour for real
	// third-party documents that pull in packages/classes gotex does not load —
	// an editor preview shows the typesettable content instead of one hard error.
	// The default (false) is strict: an undefined cs aborts, as TeX does.
	Lenient bool
}

Options configures a compile. The zero value is valid: a built-in font at 10pt with a 72pt (1 inch) margin.

type SourceError

type SourceError struct {
	Line, Col int // 1-based line, 0-based column (0/0 = unknown)
	Msg       string
}

SourceError is an engine error carrying the source location it occurred at, so a caller (a CLI, loom's compile panel) can point the user at the exact line.

func (SourceError) Error

func (s SourceError) Error() string

type SourceSpan

type SourceSpan struct {
	Line       int
	X, Y, W, H float64
}

SourceSpan is one rendered glyph's bounding box on a page (points, SVG coordinates: origin top-left, Y is the box top) tagged with the source line it came from. It is the programmatic form of the SVG's data-l groups: a caller maps a click (x, y) → line, or a line → its output rectangles.

func RectsForLine

func RectsForLine(spans []SourceSpan, line int) []SourceSpan

RectsForLine returns every glyph span originating from the given source line — the boxes an editor highlights when the cursor sits on that line.