Documentation
¶
Overview ¶
Package doc is the domain model every other package converges on: what a document looks like after it has been read and before it has been written out.
Nothing here parses, positions, classifies, or renders. That is deliberate and it is the reason the package exists. The extractor, the layout heuristics, the OCR engine, and the structure-tree walker all produce this model, and the Markdown and OKF sinks all consume it — so a page recovered from glyph positions and a page recovered from a vision model are interchangeable by the time a sink sees them. Without a shared model in the middle, each producer and each sink would need to know about the others, which is how the libraries in §1 of docs/DESIGN.md ended up as one package that does everything.
The only dependency is geom, which is itself stdlib-only. geom owns Rect, Matrix, and the tolerance policy; re-declaring them here would give the repo two rectangles that must be kept in agreement, so the layout in DESIGN.md §4 listing them under doc is satisfied by using geom's.
Section is in section.go, and its shape came from measuring ISO 32000-2 rather than from anticipating it — see that file for what the measurement changed.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ListMarker ¶ added in v0.2.0
ListMarker returns the marker rune txt opens with, or zero.
It trims the text itself rather than trusting a caller to, which is what makes the two conditions below sufficient: on trimmed text a marker followed by whitespace must have something after that whitespace, since the last rune is not one. A separate "and content follows" check reads like the third requirement and is unreachable — a mutation removing it survives every test, which is how it was found.
The separator is what distinguishes a marker from the same glyph used as a character: every one of the 1302 bullet-initial blocks in the corpus has it and none is glued to its text, while the excluded "-" is glued in 12 of its 13 occurrences. The length requirement is the other measured case — mupdf_explored.pdf has blocks that are a lone Wingdings square with no text, which are decoration and not items.
Exported because the decision to promote a block and the act of stripping it are separate steps in layout: a block that will not be promoted must not have its text edited, so the test has to be available without the mutation.
Types ¶
type Block ¶
type Block struct {
// Role is what this block is.
Role Role
// Level is the heading depth for RoleHeading (1-6) or the nesting depth for
// RoleListItem (1-based). Zero for everything else.
Level int
// Marker is a list item's label with the item's own text — the bullet glyph a
// page draws, or the number or letter an ordered list counts. Empty for
// everything that is not a list item, and for an item whose label is neither
// declared nor drawn.
//
// It is a field rather than a prefix on the text for the reason Docling's
// ListItem separates the two: a marker kept inside the text has to be re-found
// by every sink, using a glyph allowlist each one would have to re-derive, and
// on the one sink that exists it doubles — markdown writes its own "- ". Kept
// rather than dropped because a bullet is losable and a label is not: "[1]" and
// "a." are text the page says, and Markdown has no syntax that restates them, so
// a sink that wants them needs to know they existed. Enumerated is that
// distinction, derived from this.
Marker string
// Spans are the block's styled runs in reading order.
Spans []Span
// Box is the block's bounding rectangle in the page's coordinate space.
Box geom.Rect
// Lang is a language override from the structure tree, when the block declares
// one different from the document's.
Lang string
// Alt is /Alt or /ActualText: what the content means when the glyphs do not
// spell it. For a RoleFigure it is the only text available, and for a block of
// artwork-as-text it is more correct than the glyphs, so a sink prefers it —
// which is why it is on the block rather than dropped once the spans are read.
Alt string
// MCIDs are the marked-content identifiers this block was assembled from, on
// the block's page. Carried for diagnosis: when tagged text comes out in the
// wrong order, the question is always which MCIDs went where, and answering it
// without this means re-running the extraction.
//
// Not the join key for a structure element. A block is a *layout* unit and an
// element is a *logical* one, and the two disagree often enough to matter: on
// Well-Tagged-PDF-WTPDF-1.0.pdf, 12% of headings share a block with the body text
// that follows them, because the extractor's paragraph heuristic saw one paragraph
// where the tree declares a heading and a definition. Joining on this set resolves
// such a heading's title to the heading plus the definition. Span.MCID is the key
// that does not have that failure.
MCIDs []int
}
Block is a run of content that a sink emits as one unit.
A block holds Spans rather than a string because style changes inside a paragraph carry meaning that Markdown can express — an italic term, a code identifier in prose — and flattening to a string at extraction time discards it irreversibly. The spans are also where the space-inference decisions land, so keeping them lets a defect be traced to the glyph run that caused it.
func (Block) Enumerated ¶ added in v0.2.0
Enumerated reports whether the block's marker is an ordered label — a number or a letter — rather than a bullet.
Derived from Marker rather than stored beside it, because the two cannot disagree: the marker either is a single glyph from the allowlist or it is something a producer counted. Docling carries this as its own field; here a second field could be set inconsistently with the first, and there is nothing it could express that this does not.
It exists because a sink cannot render what it cannot distinguish. Markdown has no syntax for "[1]" as a list marker, so a sink emitting "- " has to put an ordered label back into the line, and dropping it instead would lose text the page draws.
func (Block) IsEmpty ¶
IsEmpty reports whether the block would emit nothing. A block with no text and no Alt is a positioned rectangle a producer left behind, and every stage from sectionize onward has to skip it.
func (*Block) SetMarker ¶ added in v0.2.0
SetMarker records a marker the producer declared, and closes the gap its removal leaves in the text.
For the producer that has a declaration to read: sectionize takes the label's spans out of the item before gathering it, so there is no marker left in the text to find — but there is usually whitespace where it was. Measured over every tagged list item on disk that declares a /Lbl, dropping the label's spans leaves the item's text opening with whitespace in 133 of 147 cases, which a sink writing its own "- " renders as two spaces.
The trim is the same one StripMarker does after the marker, and stops at the first span with content left for the same reason: it must not reach into the item's own prose.
func (*Block) StripMarker ¶ added in v0.2.0
StripMarker moves a leading marker glyph out of the block's spans into Marker, and reports whether it found one.
For the producer that has no declaration to read: the marker is only in the drawn text, so recognizing it and removing it are the same act. ListMarker gates it, so the separator and length requirements apply here too — a lone bullet with no text is page decoration and keeps its glyph.
The span walk is not simply "edit the first span". A block's text is its spans concatenated with no separator, so the marker and the whitespace after it can be split across spans in three ways, each of which occurs on disk: the marker with its separator in one span, the marker alone with the separator opening the next (a bold en dash followed by a roman body, which is testdata/reference/lists.pdf's nested item), and a leading span holding nothing but whitespace with the marker in the one after it. Stopping at the first non-empty span leaves "- • Item." in the output.
unicode.IsSpace rather than a byte cutset: producers separate a marker from its text with U+00A0 routinely, and ListMarker admits the block on that basis, so the strip must accept the same separators the gate did.
A span the strip empties stays in place rather than being removed, so the span indices a caller holds stay valid and Span.MCID survives for diagnosis; an empty span writes nothing.
func (Block) Text ¶
Text returns the block's text, spans concatenated with no separator.
No separator because a span boundary is a style change, not a word boundary: the space between "an" and "italic" belongs to one span or the other, and inserting one here would double it. Space inference happens once, in the extractor, against font metrics — doing any of it in this package would put half the policy in the model and half in the producer.
type Document ¶
type Document struct {
// Meta is what the file says about itself, for frontmatter and for OKF
// provenance.
Meta Metadata
// Pages are in document order, and every page the file declares is present
// even when nothing could be extracted from it. A missing page would shift
// every page number after it, and page numbers are what a reader checks a
// conversion against.
Pages []Page
}
Document is one PDF, read.
type Metadata ¶
type Metadata struct {
// Path is the source file as given on the command line. Kept because a
// converted document with no record of where it came from cannot be
// regenerated or checked.
Path string
// Title, Author, Subject, Keywords, Creator, and Producer come from the
// document information dictionary (ISO 32000-2 §14.3.3) or from XMP.
Title string
Author string
Subject string
Keywords string
Creator string
Producer string
// Created and Modified are as written in the file, not parsed into a time.
// PDF date strings are frequently malformed, and a sink that emits the string
// it found is more useful than one that drops a date it could not parse.
Created string
Modified string
// Lang is the document's /Lang, which OKF and Markdown frontmatter both want
// and which is also the only reliable hint for hyphenation and casing rules.
Lang string
// Version is the PDF version, as reported by objects.Store.
Version string
// Tagged reports whether the file has a structure tree. It decides which
// extraction path ran, so it belongs in the output: a conversion that silently
// fell back to layout heuristics should be legible as such.
Tagged bool
// Encrypted reports an /Encrypt dictionary. A file may be readable and still
// report true, since empty-password encryption is common.
Encrypted bool
}
Metadata is the document-level information a sink can put in frontmatter.
Every field is optional, because every one of them is optional in the file. The zero value is a document that said nothing about itself, which is common and is not an error.
type Outline ¶
type Outline struct {
// Meta is the source document's metadata, carried so a sink emitting one file
// per clause can attribute each of them without also being handed the Document.
Meta Metadata
// Sections are the top-level sections in document order.
Sections []*Section
// Preamble holds content that appeared before the first heading — a title page,
// a copyright notice. It is kept rather than dropped because dropping content is
// never the right default for an extractor, and it is separate from Sections
// because it belongs to no clause.
Preamble []Block
// Unplaced holds text the reconstruction could not attribute to any section,
// grouped by the page it was found on. It is Pages rather than Blocks because
// without the page number this content cannot be checked against the original,
// and Page is the type that already carries one.
//
// It exists because the alternative to keeping this is one of two worse things.
// ISO 32000-2 draws the whole of clause 1 outside any marked-content sequence, so
// no structure element names it: dropping it loses a normative clause, and
// attaching it to the nearest preceding section files the Scope under "0.4
// Changes introduced in ISO 32000-2:2020" — a wrong attribution in a bundle a
// model will later read as fact, which is worse than an unattributed one.
// Measured at 0.23% of the specification's text and 0% of ISO/TS 32005.
//
// Recovering such content as a *clause* needs a heading that the tree does not
// contain, which is the layout path's job and not this one's.
Unplaced []Page
}
Outline is a document's sections: the top-level ones, in document order.
A slice rather than a synthetic root, because a document does not have one heading at the top. ISO 32000-2 opens with Foreword and Introduction at the same level as clause 1, and inventing a root to hold them would put a section in the tree that no heading in the file corresponds to — which a clause-per-file sink would then have to emit or special-case.
type Page ¶
type Page struct {
// Number is the 1-based page number.
Number int
// Box is the page's visible area in user space — the crop box where one is
// present, otherwise the media box. Blocks are positioned in the same space.
Box geom.Rect
// Rotate is /Rotate in degrees, a multiple of 90. Carried rather than applied:
// a sink emitting text does not need it, and a rasterizer does, so folding it
// into coordinates here would force the sink to undo it.
Rotate int
// Blocks are the page's content in reading order. Order is the producer's
// responsibility — the structure tree's logical order when tagged, geometry
// when not — because it is the one thing a sink cannot recover.
Blocks []Block
// Rasterized reports that this page's content came from OCR rather than from
// the content stream. It travels with the page so a sink can mark inferred
// text as inferred, which matters for a knowledge bundle a model will later
// read as fact.
Rasterized bool
}
Page is one page's content in reading order.
func (Page) Coverage ¶
Coverage returns the fraction of the page box its text occupies, from 0 to 1.
Zero for a page with no text or no box. A zero-area box is a defective page dictionary rather than a full page, so reporting no coverage routes it to the rasterizer, which is the outcome that recovers something.
func (Page) Text ¶
Text returns the page's text with one newline between blocks.
This is for measurement, not for output: the §9 benchmark counts characters, space ratio, and word lengths, and those numbers must come from one agreed rendering or they are not comparable across runs. Markdown formatting — heading markers, list bullets, paragraph spacing — is the sink's job, and a sink that called this would be reformatting a string instead of walking the model.
func (Page) TextBounds ¶
TextBounds returns the union of every block's rectangle, or the zero Rect when the page has no text.
This is the numerator of the OCR router's coverage rule: a page whose text covers too little of its box is a scan with no text layer, or one with a text layer so sparse that rasterizing it will do better. Expressed as a bounding union rather than a sum of areas because overlapping blocks would double-count and push a scanned page's coverage above the threshold.
type Role ¶
type Role string
Role is what a block is, in the vocabulary a Markdown or OKF sink can act on.
It is a deliberately smaller set than tag.Role, and the reduction is the point. ISO 32000-2 §14.8.4 defines around fifty structure types; Markdown can express perhaps eight of them. Mapping down happens once, where the structure tree is read, rather than in each sink — otherwise every sink grows its own opinion about whether a TOCI is a list item, and two sinks disagree about the same document.
The set is also what layout/ and ocr/ have to produce. A vision model emits prose and headings, not TBody, so a vocabulary any narrower than this could not carry a tagged document's structure, and any wider could not be filled from an untagged one.
const ( // RoleParagraph is body text, the default and the overwhelming majority. RoleParagraph Role = "paragraph" // RoleHeading is a heading; its depth is in Block.Level. RoleHeading Role = "heading" // RoleListItem is one item of a list. Nesting is in Block.Level, so a sink can // indent without reconstructing a tree. RoleListItem Role = "list_item" // RoleTableCell is one cell. Table geometry is not modeled: reconstructing a // grid from a tagged table is sectionize's problem, and no sink needs it to // emit the cell's text. RoleTableCell Role = "table_cell" // RoleCode is preformatted text, where the extractor must not collapse // whitespace. The spec corpus is full of it. RoleCode Role = "code" // RoleQuote is a block quotation. RoleQuote Role = "quote" // RoleCaption is a figure or table caption. Separate from RoleParagraph // because a caption belongs with its figure when sections are stitched, and a // sink that cannot tell them apart puts it in the following paragraph. RoleCaption Role = "caption" // RoleFigure is a figure with no text of its own. Its Alt carries the // description, which for an accessible document is the only text there is. RoleFigure Role = "figure" // RoleArtifact is page furniture — a running header, a folio, a rule. Kept // rather than dropped at extraction, because "is this a header" is a judgement // and the evidence for it is here; a sink omits them, and probe can count them. RoleArtifact Role = "artifact" )
type Section ¶
type Section struct {
// Title is the heading text. It is resolved from page content joined on
// (page, MCID), not from the structure element's /T: every heading in ISO
// 32000-2 has an empty /T, so /T is an optimization when present and never the
// source of truth.
Title string
// Level is the heading depth, 1-based. It comes from the declared heading role
// — H1..H6, or a bare H's nesting depth per ISO 32000-2 §14.8.4.4 — on the
// tagged path, and from font-size clustering on the untagged one. It is the
// hierarchy: a section's parent is the nearest preceding section of lower level.
Level int
// Number is the clause number parsed off the front of the title, "7.5.8" for
// "7.5.8 Filters", or empty when the title does not begin with one. It is kept
// separate because it is what a cross-reference names and what a stable URI is
// built from, and recovering it from the title at every use would mean parsing
// it in several places.
Number string
// Blocks are the section's own content in reading order, excluding the heading
// itself and excluding everything belonging to a subsection.
Blocks []Block
// Kids are subsections in document order.
Kids []*Section
// Parent is the enclosing section, nil at the top level.
Parent *Section
// Pages is the 1-based page range the section's content was found on, inclusive.
// Both are 0 for a section whose content could not be anchored to any page. This
// is a range and not a number on purpose: the median clause fits on one page and
// the ones that matter do not.
FirstPage, LastPage int
}
Section is one clause of a document: a heading, the content under it, and the subsections nested beneath.
The shape here follows a measurement rather than an expectation. An earlier draft of docs/DESIGN.md had a clause as "one contiguous subtree", which would have made Section a thin view over the structure tree. ISO 32000-2 has 7 Sect elements against 981 headings, with a single Part holding 13,442 direct children in a flat H1 P P P … stream, so 966 of those headings have no element children at all: a clause's body is its heading's *following siblings*, not its descendants. Section is therefore a real tree that sectionize builds, not a projection of one that already exists.
Two consequences visible in the fields. Blocks holds the content directly rather than referencing pages, because a clause has no page — clause 7.5.8 running from page 412 to 414 is one section whose blocks came from three pages, and there is nothing to stitch since the structure tree was never page-scoped to begin with. And Pages records where the content was found rather than where the section "is", because that is a range and a reader checking a conversion needs it.
func (*Section) Path ¶
Path returns the section's ancestry titles from the top level down to and including its own, which is what OKF tags and a breadcrumb both need.
type Span ¶
type Span struct {
// Text is the run's characters. Several characters may come from one glyph — a
// ligature — and one character may come from several glyphs.
Text string
// Style is how the run was drawn.
Style Style
// Box is the run's bounding rectangle in page coordinates.
Box geom.Rect
// MCID is the marked-content identifier this run was drawn inside, or -1 when it
// was drawn outside any marked-content sequence. Combined with the page number it
// is the join key between page text and a structure element.
//
// It is on the span rather than only on the block because a span never crosses a
// marked-content boundary while a block routinely does — the extractor starts a
// new run at every MCID change, so this is exact where Block.MCIDs is a union. On
// Well-Tagged-PDF-WTPDF-1.0.pdf that distinction is 12% of the headings.
//
// Zero is a valid MCID, so the absent value is -1 and cannot be the zero value of
// the field. Constructing a Span by hand therefore leaves it claiming MCID 0; the
// extractor always sets it, and a consumer joining on it should treat a page with
// no marked content as having no join at all rather than trusting a 0.
MCID int
}
Span is a run of text sharing one style.
type Style ¶
type Style struct {
// Font is the /BaseFont name with any subset prefix stripped.
Font string
// Size is the effective on-page size in user-space units, after the text
// matrix and CTM. Not the Tf operand: a Tf of 1 with a matrix scaling by 12 is
// how many producers set 12-point type, and reporting 1 makes every size-based
// heading heuristic useless.
Size float64
// Bold and Italic are as declared by the font descriptor's flags and weight.
Bold bool
Italic bool
// Mono reports a fixed-pitch font, from the descriptor's flag. It is the signal
// that distinguishes a code block from a paragraph in an untagged document,
// where nothing else does.
Mono bool
// Hidden reports rendering mode 3 or 7 — invisible, or clip-only. This is the
// text layer under a scanned page, which is exactly the text an extractor
// wants, so it is reported rather than filtered. A sink emits it; a coverage
// calculation counts it; only a renderer skips it.
Hidden bool
}
Style is the typographic identity of a span.
Font stays alongside the Bold and Italic flags rather than being replaced by them, because the two disagree and each has a consumer. The flags are what the descriptor asserts, and a font named "Arial-BoldMT" whose descriptor forgot the flag is ordinary — so the extractor sets the flags from both sources. A consumer grouping runs into blocks still needs the exact font name, which no pair of booleans can stand in for.
func (Style) SameRun ¶
SameRun reports whether two styles are close enough that their spans can be merged into one.
Size is compared with a tolerance because it is computed from a matrix, so two glyphs set in the same type can differ in the last bits. Without the tolerance a paragraph becomes one span per glyph, which is correct and useless.
Source Files