engine

README

engine — go-tex

A pure-Go (no cgo) TeX engine, under construction — aimed at functional parity with a TeX distribution, not a subset. This module is the engine's mouth and gullet: a faithful re-implementation of TeX's category-code tokenizer, the equivalents table (eqtb) with grouping/scoping, macro definition with delimited parameters, and the full expansion machinery.

It is developed the way parity is actually reachable — reimplement the engine faithfully, gated by TeX's own conformance oracle — and later run the real LaTeX kernel and packages on it (they are TeX macros), diffing output against pdftex/xetex. The primary gate here is the conformance ratchet (TestConformance): TeX snippets checked byte-for-byte against real-TeX output.

Working today (verified, faithful)

• Category-code tokenizer — full catcode table, comments, control words/symbols.
• Macros — \def (undelimited, delimited, and grouped parameters, with backtracking on partial delimiter matches), \edef/\gdef/\xdef, \let (to macros, primitives, undefined, and character tokens), \global.
• Expansion — \expandafter, \csname/\endcsname, \noexpand, \string, \the, \number, \romannumeral, \meaning, \uppercase/\lowercase.
• Conditionals — \if, \ifnum, \ifx, \ifcat, \ifodd, \ifcase, \iftrue/\iffalse, with \else/\or/\fi and nesting.
• Registers & arithmetic — \count, \advance, \multiply, \chardef, \catcode, read via \the/\count.
• Grouping — {…}, \begingroup/\endgroup, save/restore of meanings, registers, and catcodes; \global escapes the current group.

Faithfulness is checked on subtleties only real TeX gets right (a control word absorbing its following space; significant spaces in conditional branches).

out, _ := engine.New().Run(`\def\twice#1{#1#1}\message{\twice{\twice A}}`)
// out == "AAAA"

Real-world documents (lenient mode)

A real third-party paper pulls in classes, packages, fonts, figures and .bib files that a from-scratch engine does not carry. Strict mode aborts on the first such gap (as TeX does). Lenient mode (gotex -lenient, or Options{Lenient: true}) turns those gaps into best-effort no-ops so an editor preview shows the typesettable content instead of one hard error:

• an undefined command is skipped, along with its likely [opt]{arg} block;
• an unloadable figure becomes a framed placeholder of the requested size;
• a math macro go-tex/math doesn't know drops that one equation;
• a \setlength on an unmodelled length, and a missing \input/\bibliography/ \font file, are ignored.

Every skipped construct is tallied ((*Engine).SkippedCommands) so a caller can report what was dropped. On a sample of 54 real arXiv sources, strict mode compiled 0 end-to-end (each hit a package command in the preamble); lenient mode produces a multi-page PDF for all 54, with real, selectable prose text. It is a preview aid, not a fidelity claim — the roadmap below is how the gaps close for real.

Loading real classes and packages

\documentclass and \usepackage (and \RequirePackage, \LoadClass, \LoadClassWithOptions) do more than emulate: they resolve and load the real .cls/.sty — from the document's own directory (an arXiv paper's bundled class/package), a TEXINPUTS/GOTEX_TEXMF search path, or an embedded base set — making @ a letter and running the file's own \newcommand/\def/… on the engine. The LaTeX2e option mechanism runs too: \DeclareOption, \DeclareOption*, \ProcessOptions, \ExecuteOptions, \CurrentOption, \PassOptionsToPackage/\PassOptionsToClass, plus \IfFileExists/ \InputIfFileExists. A file loads tolerantly — a command the engine lacks is skipped, so a real class contributes what it can — and a runaway-expansion guard bounds macro expansion so a pathological or partially-supported file can never hang (it stops with partial output in lenient mode, an error in strict). Distribution-heavy packages the engine emulates natively or better as stubs (geometry, tikz, hyperref, graphicx, encodings, …) are not loaded from disk. Base classes (article, amsart) are the next embed — the kernel gap to run them is mapped.

Status & roadmap to parity

This is stage 1 (mouth + gullet). The remaining stages, each gated by an objective oracle:

1. ✅ Mouth + gullet — tokenizer, eqtb, macros, expansion (this module).
2. ⬜ Stomach — box/glue/penalty model, h/v lists, Knuth–Plass line breaking, page builder, \halign — gated by the TRIP test.
3. ⬜ Math (Appendix G) — go-tex/math is the starting point.
4. ⬜ Fonts — TFM + OpenType (via go-opentype).
5. ⬜ Output — DVI → PDF (via go-pdfkit), then run the real latex.ltx + packages, gated by PDF-diff vs pdftex/xetex.

Because it is under construction, line coverage (currently ~84%) rises as subsystems land; the meaningful gate is the growing conformance ratchet, not a fixed coverage figure. Pure Go, CGO=0, go vet clean, green across six 64-bit arches plus js/wasm and wasip1/wasm.

License

BSD-3-Clause — see LICENSE. Copyright the go-tex/engine authors.

Documentation

Overview

Package engine is the core of a pure-Go (CGO=0) TeX engine: a faithful re-implementation of TeX's mouth and gullet — category-code tokenization, the equivalents table (eqtb) with grouping/scoping, macro definition with delimited parameters, and the expansion machinery (\def, \edef, \let, \expandafter, \csname, \noexpand, \string, \the, \number, conditionals, integer registers). It is the foundation on which the real LaTeX kernel and packages will run, gated by TeX's own conformance suite (the TRIP test) — the path to functional parity with a TeX distribution, not a subset.

Index

• Constants
• func CompileToPDF(src []byte, opt Options, w io.Writer) (int, error)
• func CompileToSVGPages(src []byte, opt Options) ([]string, error)
• func LineAt(spans []SourceSpan, x, y float64) int
• type Engine
  • func New() *Engine
  • func NewDocument(opt Options) (*Engine, error)
  • func (e *Engine) LoadFormat(src string) error
  • func (e *Engine) LoadLaTeX() error
  • func (e *Engine) LoadPlain() error
  • func (e *Engine) Page() *boxNode
  • func (e *Engine) Pages() []*boxNode
  • func (e *Engine) Position() (line, col int)
  • func (e *Engine) RenderBox(i int, margin float64) string
  • func (e *Engine) RenderPDF(w io.Writer, margin float64) error
  • func (e *Engine) RenderPage(margin float64) string
  • func (e *Engine) RenderPages(margin float64) []string
  • func (e *Engine) Run(src string) (string, error)
  • func (e *Engine) SetFont(f fontFace)
  • func (e *Engine) SkippedCommands() map[string]int
  • func (e *Engine) SourceSpans(margin float64) [][]SourceSpan
• type Item
  • func Box(w float64) Item
  • func Glue(w, stretch, shrink float64) Item
  • func Glyph(r rune, w, h, d float64) Item
  • func Penalty(w, p float64, flagged bool) Item
• type ItemKind
• type Line
  • func KnuthPlass(items []Item, lineWidth, tolerance, linePenalty float64) ([]Line, bool)
• type OpenTypeFont
  • func NewOpenTypeFont(fontBytes []byte, sizePx int) (*OpenTypeFont, error)
  • func (o *OpenTypeFont) CharDims(r rune) (float64, float64, float64)
  • func (o *OpenTypeFont) Space() (float64, float64, float64)
• type Options
• type SourceError
  • func (s SourceError) Error() string
• type SourceSpan
  • func RectsForLine(spans []SourceSpan, line int) []SourceSpan

Constants

const InfPenalty = 10000.0

InfPenalty is TeX's "infinite" penalty (∞ = forbidden break, −∞ = forced).

const MiniLaTeX = `` /* 167-byte string literal not displayed */

MiniLaTeX is a small LaTeX-flavoured kernel written *in TeX* — the engine runs these macro definitions through its gullet exactly as a real format does; they are not reimplemented in Go. It is deliberately tiny (the road to parity is to grow this by loading the real latex.ltx, not to hand-code commands).

const MiniLaTeXKernel = `` /* 24583-byte string literal not displayed */

MiniLaTeXKernel is the LaTeX-flavoured macro layer loaded by LoadLaTeX (after the Plain macros).

const Plain = `` /* 1517-byte string literal not displayed */

Plain is a small set of plain-TeX structural macros, written *in TeX* on top of the box/glue primitives (\hbox to, \hfil, \vskip). Loaded with LoadPlain, they let a document use the familiar commands without any Go-side support — the same growth path as the kernel: add macros, do not hand-code commands.

Functions

func CompileToPDF

func CompileToPDF(src []byte, opt Options, w io.Writer) (int, error)

CompileToPDF processes TeX source and writes a PDF to w, returning the page count. A document's own \font/\hsize/… override the option defaults.

func CompileToSVGPages

func CompileToSVGPages(src []byte, opt Options) ([]string, error)

CompileToSVGPages processes TeX source and returns one SVG string per page — the form an editor preview pane consumes directly.

func LineAt

func LineAt(spans []SourceSpan, x, y float64) int

LineAt returns the source line of the last glyph span whose box contains (x, y), or 0 when the point is over no glyph. Last-wins so nested/overlapping content (a table cell over its row) resolves to the innermost glyph painted there.

Types

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine holds all TeX state: the input stack, the eqtb (control-sequence meanings), integer registers, category codes, the grouping save stack, and the \message output buffer.

func New

func New() *Engine

New builds an engine with TeX's default category codes and primitives loaded.

func NewDocument

func NewDocument(opt Options) (*Engine, error)

NewDocument builds an engine configured from opts: the Plain macros loaded (unless NoPlain) and the text font family set. It is the starting point for programmatic use when callers want to Run source incrementally before rendering.

func (*Engine) LoadFormat

func (e *Engine) LoadFormat(src string) error

LoadFormat executes a string of TeX definitions (a format/preamble) through the gullet, defining its macros in the engine's eqtb without typesetting.

func (*Engine) LoadLaTeX

func (e *Engine) LoadLaTeX() error

LoadLaTeX loads the Plain macros (if not already) and the minimal LaTeX kernel.

func (*Engine) LoadPlain

func (e *Engine) LoadPlain() error

LoadPlain defines the Plain structural macros in the engine.

func (*Engine) Page

func (e *Engine) Page() *boxNode

Page vpacks the main vertical list (everything contributed at top level) into a single vbox at natural height. Empty (nil) if nothing was contributed.

func (*Engine) Pages

func (e *Engine) Pages() []*boxNode

Pages splits the main vertical list into pages using TeX's cost-based page builder (§1005): at each legal breakpoint it forms a cost = page badness + penalty, breaks at the least-cost point once the page would overflow, honours a forced break (\penalty ≤ −10000) immediately, and never breaks at \penalty ≥ 10000. Each page is vpacked at natural height. (Insertions/\topskip are future.)

func (*Engine) Position

func (e *Engine) Position() (line, col int)

Position returns the 1-based line and 0-based column of the input the engine is currently reading — the location a diagnostic should point at.

func (*Engine) RenderBox

func (e *Engine) RenderBox(i int, margin float64) string

RenderBox renders box register i to an SVG string with a uniform margin (pt). Empty if the register is void.

func (*Engine) RenderPDF

func (e *Engine) RenderPDF(w io.Writer, margin float64) error

RenderPDF writes the main vertical list, split into \vsize pages, as a PDF to w. Each page is (content width + 2·margin) × (content height + 2·margin) points. A current OpenType font (with embeddable bytes) is required to draw text.

func (*Engine) RenderPage

func (e *Engine) RenderPage(margin float64) string

RenderPage renders the main vertical list to an SVG page with the given margin.

func (*Engine) RenderPages

func (e *Engine) RenderPages(margin float64) []string

RenderPages renders each page of the main vertical list to its own SVG string.

func (*Engine) Run

func (e *Engine) Run(src string) (string, error)

Run tokenizes src as the base input and processes it to completion, returning the accumulated \message output.

func (*Engine) SetFont

func (e *Engine) SetFont(f fontFace)

SetFont sets the current font used to measure and render characters in horizontal mode. Passing an *OpenTypeFont (or any fontFace) is the Go-level stand-in for TeX's \font primitive until font-file loading via \font lands.

func (*Engine) SkippedCommands

func (e *Engine) SkippedCommands() map[string]int

SkippedCommands returns, for a lenient compile, how many times each undefined control sequence was skipped (empty when strict or when none were undefined). It lets a caller surface "these commands were dropped" after a preview compile.

func (*Engine) SourceSpans

func (e *Engine) SourceSpans(margin float64) [][]SourceSpan

SourceSpans returns, for each page of the built document, the glyph spans that tie output back to source — the data behind click-to-source and jump-to-line. margin matches the value passed to the SVG/PDF renderers so coordinates align.

type Item

type Item struct {
	Kind            ItemKind
	Width           float64
	Height, Depth   float64 // box only (glyph metrics)
	R               rune    // box only (the glyph, 0 if none)
	Stretch, Shrink float64 // glue only
	Penalty         float64 // penalty only
	Flagged         bool    // penalty only (e.g. a hyphen) — consecutive flags are penalised
}

Item is one element of a horizontal list.

func Box

func Box(w float64) Item

Box, Glue and Penalty are constructors.

func Glue

func Glue(w, stretch, shrink float64) Item

func Glyph

func Glyph(r rune, w, h, d float64) Item

Glyph is a box carrying a rune and its height/depth (used by the typesetter).

func Penalty

func Penalty(w, p float64, flagged bool) Item

type ItemKind

type ItemKind uint8

ItemKind classifies a horizontal-list item.

const (
	KBox     ItemKind = iota // a box of fixed width
	KGlue                    // stretchable/shrinkable space (a legal breakpoint after a box)
	KPenalty                 // a penalty (a legal breakpoint; ±InfPenalty = forbidden/forced)
)

type Line

type Line struct {
	Start, End int     // item index range [Start, End) actually set on the line
	Ratio      float64 // glue adjustment ratio r (−1 fully shrunk … +tolerance stretched)
}

Line describes one output line of a broken paragraph.

func KnuthPlass

func KnuthPlass(items []Item, lineWidth, tolerance, linePenalty float64) ([]Line, bool)

KnuthPlass breaks items into lines of the given width, minimising total demerits (linePenalty is TeX's \linepenalty). It returns the chosen lines in order and ok=false if no sequence of feasible breaks exists within tolerance.

type OpenTypeFont

type OpenTypeFont struct {
	// contains filtered or unexported fields
}

OpenTypeFont adapts a go-opentype face for the engine: it measures glyphs (in points via CharDims and in scaled points via charDimsSP), supplies the interword glue, and returns glyph outline paths for the SVG driver.

func NewOpenTypeFont

func NewOpenTypeFont(fontBytes []byte, sizePx int) (*OpenTypeFont, error)

NewOpenTypeFont builds a metrics source from a font and a pixel size.

func (*OpenTypeFont) CharDims

func (o *OpenTypeFont) CharDims(r rune) (float64, float64, float64)

CharDims returns a glyph's advance width and its ink height above and depth below the baseline, in pixels.

func (*OpenTypeFont) Space

func (o *OpenTypeFont) Space() (float64, float64, float64)

Space returns TeX-like interword glue derived from the space advance.

type Options

type Options struct {
	Font       []byte  // roman text font (.ttf/.otf); nil ⇒ the built-in default
	BoldFont   []byte  // optional bold face, bound to \bf (so \textbf really bolds)
	ItalicFont []byte  // optional italic face, bound to \it (so \textit/\emph slant)
	MonoFont   []byte  // optional monospace face, bound to \tt (so \texttt is fixed-width)
	SansFont   []byte  // optional sans-serif face, bound to \sf (so \textsf is sans)
	Size       int     // font size in points (0 ⇒ 10)
	Margin     float64 // page margin in points (0 ⇒ 72)
	NoPlain    bool    // set to omit the Plain macros
	Date       string  // \today text (a pure-Go wasm build has no clock; supply it here)

	// Lenient turns an undefined control sequence from a fatal error into a
	// skipped command: the engine drops the unknown \cs (and its likely
	// [optional]/{mandatory} argument block) and carries on, recording the name
	// for reporting. This is best-effort "produce something" behaviour for real
	// third-party documents that pull in packages/classes gotex does not load —
	// an editor preview shows the typesettable content instead of one hard error.
	// The default (false) is strict: an undefined cs aborts, as TeX does.
	Lenient bool
}

Options configures a compile. The zero value is valid: a built-in font at 10pt with a 72pt (1 inch) margin.

type SourceError

type SourceError struct {
	Line, Col int // 1-based line, 0-based column (0/0 = unknown)
	Msg       string
}

SourceError is an engine error carrying the source location it occurred at, so a caller (a CLI, loom's compile panel) can point the user at the exact line.

func (SourceError) Error

func (s SourceError) Error() string

type SourceSpan

type SourceSpan struct {
	Line       int
	X, Y, W, H float64
}

SourceSpan is one rendered glyph's bounding box on a page (points, SVG coordinates: origin top-left, Y is the box top) tagged with the source line it came from. It is the programmatic form of the SVG's data-l groups: a caller maps a click (x, y) → line, or a line → its output rectangles.

func RectsForLine

func RectsForLine(spans []SourceSpan, line int) []SourceSpan

RectsForLine returns every glyph span originating from the given source line — the boxes an editor highlights when the cursor sits on that line.