Documentation
¶
Overview ¶
Package layout infers roles from geometry and typography, for the documents that declare none.
Most PDFs are untagged, so for most PDFs there is no structure tree to read a heading rank out of. extract deliberately stops short of guessing: it marks every non-artifact block RoleParagraph and leaves rank to whichever package has the most evidence. Where a tree exists that is sectionize. Where none does it is here.
Why not "bold and larger means heading" ¶
The obvious rule is falsified by the corpus, and this package is shaped by how it fails. Measured over the untagged fixtures:
- pymupdf/v110-changes.pdf sets 8.04pt *bold* as 48.8% of its characters, against 49.4% at 9.96pt — so the body wins on size by half a point of margin while bold is not emphasis in that document at all, and a weight-implies-heading rule marks half of it. (Measured on the character tally rather than reached through this package: the file's headings fuse into the paragraphs after them, so nothing there is a candidate — see uniformStyle.)
- pymupdf/2201.00069.pdf (arXiv) and adobe-samples/autotagPDFInput.pdf set their headings *plain* — 11.96pt and 28pt respectively — and use bold only as inline emphasis, 0.3% and 0.5% of characters. A rule that requires bold finds nothing.
- pymupdf/dotted-gridlines.pdf has a table row at body size in bold, 41 characters of it. It is typographically indistinguishable from a real heading.
- testdata/reference/headings.pdf sets its third level at *body size* in bold, so a rule that requires a larger size loses the deepest level of the one fixture written to test heading depth.
So size and weight together cannot separate a heading from a bold table row, and neither can the space above it: on dotted-gridlines the gap above that row is 1.68 times the body size, inside the 1.60–1.96 range the reference headings occupy.
What the document states rather than implies ¶
A numbered heading carries its own rank. "4.2.1 Nested subclause" is level three because the document says so, not because 9.96pt happens to sit third in a size ladder. That inverts the usual arrangement: typography is the *gate* that admits a block as a candidate, and the section number is what assigns the level.
Ranking by position in the size ladder was measured and rejected. pymupdf's mupdf_explored.pdf has five distinct above-body sizes — 24.79 chapter titles, 20.66 "Chapter 1" labels, 17.22 the book title, 14.35 sections, 11.96 the author line and part entries — of which only some are heading levels at all. Ranking by ladder position disagreed with the document's own numbering on 296 of 296 numbered headings, because it counts rungs that are not levels. Numbering is not a heuristic about the document; it is the document's statement, so it wins.
The cost is that unnumbered headings stay paragraphs. That is deliberate for now: the fixtures show no signal that admits "Preface" without also admitting a bold table row, and a rule that promotes one and not the other would be tuned to a document rather than derived from one. Recording the limit is better than guessing past it — DESIGN.md §10 carries it as measured debt.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var DefaultOptions = Options{MaxHeading: 80, MaxLevel: 6, ListStep: 1.0}
DefaultOptions is inference as the CLI runs it.
ListStep of 1.0 sits in the middle of an empty band rather than at a tuned point. Measured over every PDF on disk, a marker run contains only eight distinct left-edge gaps in total: six at 0.011 type sizes, one at 0.241, and one at 2.403. The first seven are the same tier — float noise in a shared margin, and ISO 32000-2's PDFDocEncoding table where two adjacent rows open with an em dash and an en dash of different widths — while the last is reference/lists.pdf's genuine `itemize` nesting. Anything from roughly 0.3 to 2.4 yields the same result, so 1.0 is a statement that nesting indents by about a character, not a threshold fitted to a trough.
Functions ¶
This section is empty.
Types ¶
type ListStats ¶
type ListStats struct {
// Runs counts maximal sequences of consecutive list items.
Runs int
// Items counts blocks promoted to RoleListItem.
Items int
// MaxLevel is the deepest nesting assigned. One on every corpus fixture except
// reference/lists.pdf, which is the only document on disk with a nested list.
MaxLevel int
}
ListStats reports what Lists did. Quiet failure is the same risk as for Headings: a run that promotes nothing has not errored, and neither has one that promotes a table.
func Lists ¶
Lists promotes untagged blocks to RoleListItem in place, and reports what it did.
A block is a list item when it opens with a marker glyph as its own token. That is a weaker signal than the section number Headings runs on, and the reason it is usable anyway is that a bullet is not a character anyone sets in prose: measured over the corpus, of 1442 blocks opening with one of doc's marker glyphs, reading every ambiguous case leaves 5 that are not list items — all of them rows of ISO 32000-2's glyph tables in Annex A and D, where an em dash or en dash *is* the row's subject ("— 132 0x84 0204 U+2014 EM DASH"). At 5 in 1442 the population is 288:1 in favour of promoting, which is the inverse of the ratio that made DESIGN.md's lost-space defect not worth a rule.
The marker is removed, here and not in the sink ¶
Promoting a block to RoleListItem is the statement that its marker is structure rather than text, so the marker leaves the spans with it — into Block.Marker, which is where a sink that can render a label finds it. Leaving it in the text would mean every sink re-deriving the glyph allowlist to know which leading rune was the marker — the same split doc.Block warns against for space inference, half the policy in the model and half in the producer. It would also double the marker on the one sink that exists: markdown writes its own "- ".
Block.StripMarker does it, in doc rather than here, because this is no longer the only producer of a list item's marker: sectionize reads one the structure tree declares, and two copies of the allowlist is the same split one package further out. This is still the only place in *this* package that edits a block's text rather than its role.
Levels, and what was rejected ¶
Depth comes from the left edge, ranked within a maximal run of consecutive marker blocks. Ranking within the run rather than the page is what keeps two unrelated lists in different columns from being read as one nested list.
A minimum run length was measured and rejected. Requiring two consecutive marker blocks looks like the obvious guard against a stray table row, and it costs far too much: it drops 136 promotions across the corpus, and reading them shows they are overwhelmingly genuine — single-item lists, and multi-item lists that extract fused into one block ("■ machine-readable text presented in a declared language; ■ appropriate…"). It would have rejected 136 to catch about 3.
The fusion in that second group is a segmentation defect in extract rather than anything a role rule should paper over, and it was investigated rather than left as a suspicion: 98 line pairs across 6 files, of which exactly one reaches the emitted output, because every other affected file is tagged and sectionize splits its items from the structure tree before a sink sees them. Both fixes that suggest themselves are dead — the vertical step before a bullet line (1.220 to 1.486 line heights) overlaps ordinary wraps (1.100 to 1.500) completely, the bullet sits flush with the block margin rather than outdented from it at the 25th through 90th percentile, and breaking a block where the marked-content identifier changes would cost 6911 splits to buy 8. So the run minimum stays rejected on the 136-to-3 arithmetic and not pending a segmentation fix. DESIGN.md §10 carries the numbers and the larger defect the measurement did find, which is on the tagged path.
type Options ¶
type Options struct {
// ListStep is how far a list item's left edge must sit right of the tier above it
// to count as nested, as a multiple of the run's type size. Zero means the default;
// a negative value disables nesting, leaving every item at level 1.
ListStep float64
// MaxHeading bounds a heading's length in runes. A heading is short; a numbered
// paragraph is not, and specifications are full of them ("4.2.1 The value shall
// be…"). Without a bound, every numbered clause body in a standard becomes a
// heading. Measured against the fixtures: the longest true heading found is 44
// runes, so 80 leaves room without admitting prose. Zero means the default.
MaxHeading int
// MaxLevel bounds the depth a section number may assign. Markdown has six
// heading levels and deeper numbering exists — ISO clause 7.11.4.2.1 is five —
// so the cap is where the dialect's, not the document's. Zero means the default.
MaxLevel int
}
Options configures inference.
type Stats ¶
type Stats struct {
// BodySize is the size, in points, of the dominant text cluster.
BodySize float64
// BodyBold reports whether that dominant cluster is bold — that is, whether
// weight carries any heading signal in this document. False on every fixture in
// the corpus; v110-changes.pdf comes closest and still misses, its 8.04pt bold
// holding 48.8% of characters against a 9.96pt plain body that wins on size.
BodyBold bool
// Candidates counts blocks that passed the typographic gate.
Candidates int
// Headings counts blocks promoted to RoleHeading.
Headings int
}
Stats reports what inference did. The failure modes here are quiet — a run that promotes half a document has not errored, and neither has one that promotes nothing — so the numbers are the only way to see it.
func Headings ¶
Headings promotes untagged blocks to RoleHeading in place, and reports what it did.
In place rather than returning a new document because the caller already owns one and the change is per-block: a Role and a Level, on blocks that extract left as paragraphs. Building a doc.Outline here would duplicate sectionize, whose level stack already turns a linear (level, title, content) sequence into a tree — that is the next step on this path, not this one.
A document with no text, or one whose blocks are all artifacts, is left alone and reports zero. That is not an error: a page holding one image has no headings.
Source Files