Documentation
¶
Overview ¶
Package corpus builds the Byblos test corpus in memory.
Every document here is produced by this code, deterministically. No binary PDF fixture of uncertain provenance enters the repository, and `make corpus` reproduces the exact bytes the committed poppler goldens were made from.
The PDF writer below is deliberately minimal and hand-rolled rather than built on pdfcpu: the corpus must be able to express structures pdfcpu would never emit, including a truncated file.
Index ¶
- Constants
- Variables
- func ByName(name string) ([]byte, bool)
- func CorruptContentStream() []byte
- func CorruptContentStreamInArray() []byte
- func DupRasterWithInfo(key, value string) []byte
- func FirstGray(seed int) uint8
- func Gradient() image.Image
- func MixedPageTwoUnreadable() []byte
- func NoMediaBox() []byte
- func PageTwoStopsMidStream() []byte
- func Photo() image.Image
- func RotateInheritance() []byte
- func ScanFractionalCropBox(frac float64) []byte
- func ScanImageMask() []byte
- func ScanJPEG() []byte
- func ScanPlacedAt(m [6]float64) []byte
- func ScanSMaskJPEG() []byte
- func Scanjpeg() image.Image
- func Scanpage() image.Image
- type Doc
Constants ¶
const ( Count = 36 ReadableCount = 35 )
Count is how many documents All() returns; ReadableCount is how many of them a PDF reader can open, which is every one except "malformed", the scan truncated mid-body.
THESE ARE THE SINGLE SOURCE OF THE TWO NUMBERS. Both are quoted in prose all over the tree -- the design spec's section 8 acceptance row, and doc comments in optimize.go, optimize_test.go, stamp_test.go and linearize_test.go -- and every one of those said 27 through three successive documentation reconciliations while the corpus grew, because nothing connected the prose to the code (byb-a20).
Adding a document is therefore two edits and no more: put it in All(), and bump the constant here. TestAllMatchesTheDeclaredCount (this package) and TestCorpusReadableCountIsWhatTheCorpusDeclares plus TestCorpusCountClaimsMatchTheCorpus (root package, designspec_pin_test.go) then fail until every quoted figure in the tree agrees.
const ( PageWidthPt = 612 PageHeightPt = 792 ScanImageW, ScanImageH = 306, 396 // the full-page raster, 36 DPI TileImageW, TileImageH = 153, 396 // each half of the tiled page )
Geometry shared by every generated document. US Letter at 72 points/inch.
const ( StackedBaseSeed = 4 StackedTopSeed = 5 )
Seeds for the grey patterns of the stacked documents. The two layers must be distinguishable pixel by pixel, because "which one came back?" is the whole question those documents ask.
const ( // BornDigitalTextChars is len("Byblos born-digital page one.") + // len("Second") + len("line") + len("here.") = 29 + 6 + 4 + 5. BornDigitalTextChars = 44 // OverlayTextChars is len("Scanned 2026-07-27"). OverlayTextChars = 18 // InvisibleTextChars is len("References"). TextChars counts an invisible // layer like any other text; only classification treats it differently. InvisibleTextChars = 10 )
Text content, kept as constants so tests assert against a named value rather than a magic number.
const ( ImageW = 600 ImageH = 800 )
ImageW, ImageH are the pixel dimensions every generator in this file uses. byb-b3's design settled on 600x800: large enough that the median-cut histogram and Lloyd refinement see realistic costs, small enough that the oracle-backed tests stay fast.
const BookletPages = 8
BookletPages is how many pages booklet() carries. Six to ten was the range byb-woy asked for: enough that the page-offset hint table's per-page loop runs a meaningful number of times, and few enough that the generated document stays in the same size class as the rest of the corpus.
const TruncatedContentStreamChars = 11
TruncatedContentStreamChars is what a walk of TruncatedContentStreamText counts before the unterminated string: len("hello") + len("world!").
const TruncatedContentStreamText = "BT /F1 12 Tf 72 720 Td (hello) Tj 0 -14 Td (world!) Tj ET\n" +
"BT /F1 12 Tf 72 690 Td (never closed"
TruncatedContentStreamText is page 2's content in PageTwoStopsMidStream: two legal text-showing operators, then a literal string that is never closed.
The two Tj operands are 5 and 6 bytes, so a walk that gets as far as the damage has counted exactly TruncatedContentStreamChars, and one that throws its work away on the error counts zero. That difference is the whole point of the fixture.
Variables ¶
var ( // DeskewPlacement rotates the raster by 0.13 degrees clockwise — the median // deskew of those 147 pages, in the same sign convention as the measured // stream (b negative, c positive). govdocs1/005393.pdf p91 carries the whole // content stream `q 560.65283 -0.56462 0.76572 760.3374 14.97417 16.36581 cm // /Im0 Do Q`; the numbers here are that rotation applied to a raster // oversized by six points, so the placement still covers the page and the // document exercises the rotation rather than the coverage rule (byb-b1.3). DeskewPlacement = [6]float64{617.99841, -1.40220, 1.81061, 797.99795, -3.9, -3} // MirrorPlacement flips the raster vertically: b and c are zero, so no // rotation test can see it, and UnitSquareBox still reports a page-covering // box. Only the sign of d gives it away. MirrorPlacement = [6]float64{PageWidthPt, 0, 0, -PageHeightPt, 0, PageHeightPt} // QuarterTurnPlacement is an exact 90-degree rotation: a and d are zero. QuarterTurnPlacement = [6]float64{0, PageHeightPt, -PageWidthPt, 0, PageWidthPt, 0} // NaturalDPIPlacement is ia-DTIC_ADA383635.pdf p40's, the shape byb-b1.3 // measured on 132 pages across 17 files. The raster is placed at its own // resolution rather than stretched to the page: 2384x3321 pixels at 302 DPI // is 568.37 x 791.76 points on a nominal 612x792 MediaBox, which covers // 91.74% of it and leaves a 43.6 point blank strip down the right-hand side. // The page's entire content stream is `/GS1 gs q 568.3708 0 0 791.7616 0 0 // /Im40 Do Q` — nothing else can mark that strip. // // Measured placement DPI was a round number in every dominant file: DTIC // 302/303, CIA 299.3, dc-1238360 400.0, govdocs1 200/300/300.3. NaturalDPIPlacement = [6]float64{568.3708, 0, 0, 791.7616, 0, 0} // SubPointPlacement squeezes the raster into 0.4 point on BOTH axes. Every // edge of the placed box rounds to the same integer as its opposite edge, so // a projection that rounds to nearest reports an empty rectangle for a // placement that paints (byb-62t). // // It is content, not a rounding ghost. poppler 26.06.0 renders the page: a // 4x4 grey image under this matrix inks 2 pixels at 72 DPI and 9 at 300, and // pdfimages -list reports the image present at 720 ppi, which is what 4 // pixels across 0.4 point is. SubPointPlacement = [6]float64{0.4, 0, 0, 0.4, 10, 10} // SubPointStripePlacement is the same squeeze on ONE axis only: 0.4 point // wide and the full height of the page. It separates a per-axis projection // from a whole-box one, and it is the harder case to call invisible -- // poppler inks 792 pixels of it at 72 DPI and 9,900 at 300, a black stripe // running the length of the page. SubPointStripePlacement = [6]float64{0.4, 0, 0, PageHeightPt, 10, 0} )
The placement matrices the three off-axis scan documents use, in PDF matrix order [a b c d e f]. They are exported so a test asserts against a named value rather than repeating six magic numbers.
The shapes come from a measurement over govdocs1 (byb-b1.2): of 159 pages whose page-covering raster was not placed axis-aligned, 147 were sub-degree scanner deskew, 8 were a vertical mirror, and 4 were a true quarter turn.
var CorruptContentStreamPayload = []byte{0x78, 0x9c, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01}
CorruptContentStreamPayload is the nine bytes blankPage's empty content stream would carry if it had been damaged in transit: a valid zlib header (0x78 0x9c, exactly what compress/zlib writes) followed by a deflate block whose type bits are the reserved value 11. zlib.NewReader accepts the header, and the inflate then fails with "flate: corrupt input before offset N".
That prefix is load-bearing. pdfcpu's decodeContentStream swallows precisely that error under the relaxed validation mode pdfdoc.Open uses -- it logs "skipped" and returns nil -- which leaves the stream's content empty, and PageContent then reports the page with the SAME model.ErrNoContent it uses for a legitimately blank one. A shredded page and an empty page arrive at pdfdoc as the identical error.
Functions ¶
func CorruptContentStream ¶
func CorruptContentStream() []byte
CorruptContentStream is blankPage with page 2's zero-byte content stream replaced by CorruptContentStreamPayload: the same /FlateDecode dictionary, the same zero bytes of usable content, and a stream that did not decode.
CorruptContentStreamInArray is the same damage with page 2's /Contents written as the array form ISO 32000-1 table 30 also permits -- an intact empty stream first, the corrupt one second. Real producers write /Contents as an array routinely (every incremental append does), and a reader that checks only the first entry, or only handles the single-stream form, gets the identical zero bytes and the identical model.ErrNoContent from pdfcpu while being wrong about the page.
Neither is in All(). poppler reads both files as two pages and simply renders nothing for page 2, so a document that must make Byblos report a damaged page cannot also satisfy TestInspectAgreesWithPoppler -- the divergence is the point of the fixture. Same reasoning, same treatment as MixedPageTwoUnreadable.
func CorruptContentStreamInArray ¶
func CorruptContentStreamInArray() []byte
CorruptContentStreamInArray is CorruptContentStream with page 2's /Contents written as an array. See CorruptContentStream.
func DupRasterWithInfo ¶
DupRasterWithInfo is dupRaster with a single Info-dictionary entry set directly in the trailer, as a PDF literal string. It exists so a test can build a document that already carries an Info entry (e.g. a byblos-provenance record) WITHOUT that entry ever having passed through a pdfcpu optimize pass: writing it via the library's own WriteProvenance always runs one (it is itself a full pdfcpu read-validate-optimize-write pass), which leaves no headroom to observe a SECOND pdfcpu rewrite actually shrinking the document any further. Hand-rolling the Info entry here is what makes that second rewrite observable.
func FirstGray ¶
FirstGray is the first sample of the pattern grayPixels builds for a seed. A test that decodes a stacked document reads pixel (0,0) and compares it to this, which is cheaper than a second copy of the pattern and just as decisive.
func Gradient ¶
Gradient is a smooth RGB sweep -- the pathological case for a palette quantizer. Every pixel is a distinct or near-distinct colour, so there is no way to reproduce it losslessly at any small N.
func MixedPageTwoUnreadable ¶
func MixedPageTwoUnreadable() []byte
MixedPageTwoUnreadable exposes mixedPageTwoUnreadable directly rather than through ByName/All: poppler defaults a missing /MediaBox rather than erroring, so this document would fail TestInspectAgreesWithPoppler and TestExtractedRasterMatchesPdfimages for disagreeing with poppler on exactly the page it exists to make pdfdoc.Doc.Page reject -- that divergence is the point of the fixture, not a bug to reconcile against the oracle.
func NoMediaBox ¶ added in v0.3.0
func NoMediaBox() []byte
noMediaBox is a one-page document that declares no /MediaBox ANYWHERE: not on the page, not on the /Pages node, nowhere in the inheritance chain.
ISO 32000-1 7.7.3.3 makes /MediaBox a required inheritable page attribute, so this document is malformed. It is also real: byb-8ly measured 9 of 4,840 govdocs1 files exactly like it -- all PDF 1.0, none using object streams, with the string "MediaBox" absent from the file's bytes altogether. Byblos refused all 9 while poppler read every one and reported 612x792, which is not a measurement of the document but the universal reader default.
The raster is page-covering at that default size, so a reader that defaults as poppler does sees a page-covering scan. NoMediaBox is exported and deliberately NOT in All(): every write-path test iterates the corpus, and pdfcpu refuses to WRITE a page dict with no /MediaBox, so registering it would fail 12 optimize and linearize tests that have nothing to do with byb-8ly. Read-path tests reach for it by name.
func PageTwoStopsMidStream ¶ added in v0.3.0
func PageTwoStopsMidStream() []byte
PageTwoStopsMidStream is a two-page document whose second page decodes perfectly and then fails to LEX partway through, which is a different failure from mixedPageTwoUnreadable's: there the stream never decodes at all and the page yields nothing, and here the page yields real content up to the damage.
That is the case poppler handles by painting what it reached -- on 050734.pdf page 8 it renders 182 characters out of a content stream that stops after 1,156 bytes -- and the case byblos discarded entirely before byb-3jq.
Exported directly rather than through All(), like MixedPageTwoUnreadable: a document that exists to be half-broken has no business in the sweeps that compare every corpus document against poppler page for page.
func Photo ¶
Photo is a sinusoidal colour field: many distinct colours (451,597 at ImageW x ImageH, measured -- most of the image's 480,000 pixels), none of them dominant, deterministic.
func RotateInheritance ¶ added in v0.3.0
func RotateInheritance() []byte
RotateInheritance exposes rotateInheritance directly rather than through ByName/All, for the reason NoMediaBox gives: All() is iterated by every write-path test and its length is a measured figure in five files (byb-3y8), so a document added there costs edits (Count, wantNames) that have nothing to do with the bead adding it. This fixture exists for one test, byb-yul.2's proof that PageInfo.Rotate resolves /Rotate through page-tree inheritance rather than reading only the leaf dict.
func ScanFractionalCropBox ¶ added in v0.3.0
ScanFractionalCropBox is scan(0) with an explicit /CropBox offset by frac points from the MediaBox on every edge -- byb-2mt's review found PageGeometry.CoversPage and PageRaster.CoversPage can disagree on a fractional CropBox once RasterQuad is in play (PageRaster rounds the page box to integers before testing the quad against it; PageGeometry does not), and this is what reproduces that.
func ScanImageMask ¶ added in v0.3.0
func ScanImageMask() []byte
ScanImageMask is a one-page PDF whose only image XObject is an /ImageMask stencil: 1 bit per sample, no /ColorSpace, painted in the current fill colour (ISO 32000-1 section 8.9.6.2).
It is a standalone fixture rather than a member of All(), for the reason the "Known gap" note on jbig2() gives: a stencil is not extractable, so putting it in the corpus every extraction test walks would add a document that can only ever produce a failure. Nothing here extracts it.
It exists for byb-js5.2. ImageRef.Bitonal is "1 bit per component OR an image mask", so a caller picking bilevel candidates for a JBIG2 substitution by Bitonal alone selects exactly the images ReplaceImages refuses. This is the document that shows the two fields answering different questions.
func ScanJPEG ¶
func ScanJPEG() []byte
ScanJPEG is a two-page PDF whose single shared image XObject is a real DCTDecode (JPEG) stream rather than a raw DeviceGray sample array -- the shape byb-b3's JPEG recompression pass targets. Both pages reference the SAME object (unlike dupRaster, which duplicates the content into two objects), so a correct recompression pass touches it once and both pages' PageProvenance.Applied must record the substitution.
func ScanPlacedAt ¶ added in v0.3.0
ScanPlacedAt is the one-page scan document under a caller-chosen placement matrix, for a geometry no named corpus document carries.
Exported and deliberately NOT in All(), for the reason NoMediaBox gives: All() is iterated by every write-path test and its length is a measured figure in five files (byb-3y8), so a document added there costs nine edits that have nothing to do with the bead adding it. A test that needs one placement reaches for this by name.
func ScanSMaskJPEG ¶
func ScanSMaskJPEG() []byte
ScanSMaskJPEG is a one-page PDF whose image XObject is a DCTDecode stream carrying an /SMask -- the case a recompression pass must skip, since ReplaceImage refuses /SMask outright (write.go).
func Scanjpeg ¶
Scanjpeg is Scanpage round-tripped through JPEG at quality 75, 4:2:0 -- what a scanner's real /DCTDecode output actually looks like, chroma subsampling and DCT block artifacts included. This is the realistic B3 input, not the idealized flat-colour Scanpage.
Source Files