bibtex

README

bibtex

A Go library for reading BibTeX that uses only the standard library. It parses .bib files such as CITATION.bib and BibTeX entries embedded in READMEs, splits author lists using BibTeX's name rules, and converts TeX markup to plain Unicode text. It is built for scanning untrusted input in bulk, such as repository histories or archive blobs: malformed entries produce diagnostics and parsing continues past them. Configurable limits bound the work, and the test suite checks time and memory on pathological inputs.

Installation

go get github.com/git-pkgs/bibtex

Usage

package main

import (
	"fmt"
	"log"

	"github.com/git-pkgs/bibtex"
)

func main() {
	f, err := bibtex.ReadFile("CITATION.bib", bibtex.ParseOptions{})
	if err != nil {
		log.Fatal(err)
	}
	for _, e := range f.Entries {
		fmt.Println(e.Type, e.Key, e.Text("title"), e.Get("doi"))
		for _, a := range e.Authors() {
			fmt.Printf("  %s | %s | %s | %s\n", a.First, a.Von, a.Last, a.Jr)
		}
	}
	for _, d := range f.Diagnostics {
		fmt.Println(d.Line, d.Code, d.Message)
	}
}

Use Parse(data) for bytes from a Git blob or another source and ParseWithOptions(data, opts) to set limits. Read(reader, opts) reads from an io.Reader and leaves it open. Results store copies of their strings, so callers may reuse input buffers once a call returns. go run ./examples/read path/to/CITATION.bib runs a version of this example; -extract reads a README and - reads a Git blob from standard input.

Get returns a field with macros expanded and # concatenation applied but TeX markup and inner braces intact ({GPU}, M{\"u}ller), while Text also runs it through PlainText to give GPU and Müller. For biblatex's verbatim fields (url, doi, eprint, file and pdf), Text removes braces, \url{} and escapes such as \_ and leaves ~ and -- as written. Field names and entry types are lowercased, and citation keys are returned with their original case. Each entry stores its line, column and byte range in the input.

READMEs

Extract(data, opts) returns entries inside another document, with positions relative to the whole document. An entry is recognised where a line starts with @type{key,, which covers Markdown fenced code blocks (with an optional language tag), blockquotes, list items, inline code and reStructuredText code directives. Decorators and macros such as @dataclass(frozen=True) or Julia's @variable(model, x >= 0) are skipped. An unclosed entry stops at the end of its fenced block or, for indented blocks, at the first line indented less than the @.

Names

ParseNames splits a value on and outside braces and applies BibTeX's rules for First von Last, von Last, First and von Last, Jr, First, including the brace and accent rules that assign a word to the von part. A fully braced name such as {World Health Organization} is returned whole in Last with Literal set, which is how BibTeX files mark organisations, and and others becomes a Name with Others set. On 5,248 names collected from CITATION.bib files and READMEs, the split matched pybtex except for comma-separated lists such as Jifeng Dai, Yi Li, Kaiming He, which this library reads as several names.

The comma-list heuristic also reads the ambiguous de la Cruz, Maria Elena, Jose Luis as three names, although BibTeX reads it as one name with a Jr part. Get("author") retains the name syntax, including braces and commas, after macro expansion and whitespace folding, so callers can use a different name interpretation without parsing the file again.

Errors

Missing commas, missing keys, unclosed values, undefined macros, duplicate keys and similar problems are reported in File.Diagnostics with a code and position. An unclosed value ends at the next line that starts like an entry, and parsing resumes from that entry. Invalid options, UTF-16 input and exceeded limits return an error, matched with errors.Is against ErrOptions, ErrUnsupported and ErrLimit respectively; errors.As extracts the *bibtex.Error diagnostic.

Duplicate citation keys are matched without regard to case and produce a duplicate_key diagnostic. All entries are retained in source order; callers choose which occurrence to use.

Compatibility

The parser follows BibTeX's grammar: entries in braces or parentheses, quoted and braced values with literally counted braces, numbers, @string macros with case-insensitive names, the predefined month macros, # concatenation, @preamble, and @comment in both its braced and line forms. Text outside entries is ignored, and whitespace runs inside values collapse to one space.

Some behaviour is more forgiving than BibTeX because CITATION.bib files and README entries are often hand-written. An undefined macro's name is used as its value (booktitle = CCS gives CCS) and reported as a diagnostic. A name with two or more commas whose parts are each at least two words, such as Jifeng Dai, Yi Li, Kaiming He, is read as a list of names, where BibTeX rejects it. Input with invalid UTF-8 is decoded as Windows-1252 and reported through File.Encoding. Non-breaking and ideographic spaces count as whitespace between tokens.

PlainText composes accent commands into precomposed characters using a table generated from Unicode 17.0.0, with a combining mark as the fallback where Unicode has no precomposed form. It replaces known symbols such as \o, \ss and \LaTeX and converts --, --- and TeX quotes. Other commands are dropped and their arguments retained; math between $ signs is returned unchanged.

Text("title") preserves math too: An $O(n^2)$ algorithm keeps its dollar signs and TeX expression. Callers matching titles against services such as OpenAlex need to normalise math themselves.

Out of scope: user-defined @preamble macros, output formatting, BibLaTeX data model checks and per-type required field validation.

Limits

Default limits are 16 MiB of input, 200,000 entries, 1 MiB per value, 64 MiB of values in total including macro expansion, and 100 diagnostics. ParseOptions accepts other finite limits; a zero value uses the default and a negative value returns ErrOptions. The expansion limit stops macros defined in terms of each other from doubling the value size on each expansion.

Testing

go test ./...
go test -race ./...
go test -fuzz=FuzzParse -fuzztime=30s
go test -bench=. -run=^$

Tests run offline against checked-in fixtures. testdata/real-world contains 24 CITATION.bib files and READMEs from GitHub repositories, with their sources and Git blob IDs in downloads.json and hand-reviewed expectations in manifest.json. go run ./internal/generate UnicodeData.txt > compose_generated.go regenerates the accent table from the pinned Unicode 17.0.0 data file.

License

MIT. See NOTICE for the Unicode data license and fixture sources.

Documentation

Index

• Variables
• func PlainText(s string) string
• type Diagnostic
• type Entry
  • func (e *Entry) Authors() []Name
  • func (e *Entry) Editors() []Name
  • func (e *Entry) Get(name string) string
  • func (e *Entry) Lookup(name string) (string, bool)
  • func (e *Entry) Text(name string) string
• type Error
  • func (e *Error) Error() string
  • func (e *Error) Unwrap() error
• type Field
• type File
  • func Extract(data []byte, opts ParseOptions) (*File, error)
  • func Parse(data []byte) (*File, error)
  • func ParseWithOptions(data []byte, opts ParseOptions) (*File, error)
  • func Read(r io.Reader, opts ParseOptions) (*File, error)
  • func ReadFile(path string, opts ParseOptions) (*File, error)
• type Name
  • func ParseNames(s string) []Name
  • func (n Name) String() string
• type ParseOptions
• type Position

Variables

var (
	ErrLimit       = errors.New("resource limit exceeded")
	ErrOptions     = errors.New("invalid parse options")
	ErrUnsupported = errors.New("unsupported input")
)

Functions

func PlainText

func PlainText(s string) string

PlainText converts a BibTeX field value to plain Unicode text. Accent commands become precomposed characters where Unicode has one, known symbols are replaced, other commands are dropped while their arguments are kept, braces are removed and whitespace is collapsed. Math between $ signs is kept verbatim.

Types

type Diagnostic

type Diagnostic struct {
	Code    string `json:"code"`
	Message string `json:"message"`
	Position
}

Diagnostic describes a recoverable problem without requiring consumers to parse its message.

type Entry

type Entry struct {
	// Type is lowercased, e.g. "article".
	Type string `json:"type"`
	// Key is the citation key with its original case.
	Key    string  `json:"key"`
	Fields []Field `json:"fields"`
	Position
	// Start and End are byte offsets of the entry in the input, from the @ to
	// just after the closing delimiter.
	Start int `json:"start"`
	End   int `json:"end"`
}

Entry is one BibTeX entry such as @article{key, ...}.

func (*Entry) Authors

func (e *Entry) Authors() []Name

Authors returns the parsed author field.

func (*Entry) Editors

func (e *Entry) Editors() []Name

Editors returns the parsed editor field.

func (*Entry) Get

func (e *Entry) Get(name string) string

Get returns the first value for a field name, or "" when it is absent.

func (*Entry) Lookup

func (e *Entry) Lookup(name string) (string, bool)

Lookup returns the first value for a field name, matched case-insensitively.

func (*Entry) Text

func (e *Entry) Text(name string) string

Text returns a field converted to plain Unicode text with PlainText. The url, doi, eprint, file and pdf fields are returned with braces, \url{} and escapes such as \_ removed and other characters unchanged.

type Error

type Error struct {
	Diagnostic
	Cause error
}

Error wraps a failure that stopped parsing, for errors.Is and errors.As.

func (*Error) Error

func (e *Error) Error() string

func (*Error) Unwrap

func (e *Error) Unwrap() error

type Field

type Field struct {
	Name  string `json:"name"`
	Value string `json:"value"`
	Position
}

Field is a field or @string definition. Name is lowercased. Value has macros expanded, # concatenation applied, outer delimiters removed and whitespace runs collapsed; TeX markup and inner braces are retained.

type File

type File struct {
	Entries []Entry `json:"entries"`
	// Strings holds @string definitions in source order, with values expanded.
	Strings   []Field  `json:"strings,omitempty"`
	Preambles []string `json:"preambles,omitempty"`
	// Encoding is "utf-8", or "windows-1252" when the input was not valid
	// UTF-8 and was decoded as Windows-1252.
	Encoding    string       `json:"encoding"`
	Diagnostics []Diagnostic `json:"diagnostics,omitempty"`
}

File is the result of parsing a BibTeX database or extracting entries from a document. It owns its strings; the input buffer may be reused after parsing.

func Extract

func Extract(data []byte, opts ParseOptions) (*File, error)

Extract finds BibTeX entries embedded in other text, such as a README. An entry is recognised where a line begins with "@type{key," or "@type(key," after optional indentation. Inside a Markdown fenced code block, an entry cannot extend past the closing fence.

func Parse

func Parse(data []byte) (*File, error)

Parse reads a BibTeX database with default limits.

func ParseWithOptions

func ParseWithOptions(data []byte, opts ParseOptions) (*File, error)

ParseWithOptions reads a BibTeX database. Text outside entries is ignored, as BibTeX does. Malformed entries produce diagnostics; an error is returned only for invalid options, unsupported encodings, and exceeded limits.

func Read

func Read(r io.Reader, opts ParseOptions) (*File, error)

Read parses one bounded database from r and leaves r open.

func ReadFile

func ReadFile(path string, opts ParseOptions) (*File, error)

ReadFile parses the database at path.

type Name

type Name struct {
	First string `json:"first,omitempty"`
	Von   string `json:"von,omitempty"`
	Last  string `json:"last,omitempty"`
	Jr    string `json:"jr,omitempty"`
	// Others marks the "and others" placeholder for omitted names.
	Others bool `json:"others,omitempty"`
	// Literal marks a name written entirely in braces, which is how BibTeX
	// files give organisations.
	Literal bool `json:"literal,omitempty"`
}

Name is one person or organisation from a BibTeX name list, split into BibTeX's four parts and converted to plain text. A name written in braces, such as {World Health Organization}, is returned whole in Last.

func ParseNames

func ParseNames(s string) []Name

ParseNames splits an author or editor value on "and" outside braces and parses each name with BibTeX's rules for the "First von Last", "von Last, First" and "von Last, Jr, First" forms.

Unlike BibTeX, a name with two or more commas whose parts are each at least two words, such as "Jifeng Dai, Yi Li, Kaiming He", is read as a comma-separated list of "First Last" names. READMEs often list authors this way, and BibTeX rejects it.

func (Name) String

func (n Name) String() string

String formats the name as "First von Last, Jr".

type ParseOptions

type ParseOptions struct {
	MaxBytes         int64
	MaxEntries       int
	MaxValueBytes    int
	MaxExpandedBytes int
	MaxDiagnostics   int
}

ParseOptions bounds work for untrusted inputs. Zero fields select defaults.

type Position

type Position struct {
	Line   int `json:"line,omitempty"`
	Column int `json:"column,omitempty"`
}

Position identifies a one-based source line and column. Columns count runes. Zero means unavailable.