byblos

package module
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 19, 2026 License: Apache-2.0 Imports: 24 Imported by: 0

README

Byblos

A pure-Go PDF pipeline for scanned documents. No cgo, no shared libraries, no subprocesses.

Byblos is replacing the PDF-side tools a scan pipeline normally shells out to — ghostscript, jbig2enc, pngquant, poppler, img2pdf — with a Go library that compiles into your binary. None of the five is fully replaced yet: each is partially covered, and the residue is tracked per binary rather than estimated.

Byblos was the port that traded papyrus to Greece; "biblion", and eventually "book", comes from its name. This library handles the paper.

Status: under active implementation. Inspection, raster extraction, JBIG2 generic-region encoding and symbol-mode decoding, the invisible text layer, PDF assembly from images, structural optimization (including our own Annex F linearizer), and image recompression (quantization, downsampling, JPEG recompression) all exist and are exported. See the design spec and FUTURE.md for what remains.

What makes it tractable

Byblos does not render PDFs. Rendering arbitrary PDF means a content-stream interpreter and full font rasterization — plausibly more work than an OCR engine.

Scanned pages don't need it. They are, overwhelmingly, one page-covering image per page: that requires extraction, not rendering. Born-digital PDFs shouldn't be rasterized at all. Pages that are neither are detected and reported (ErrNotSingleRaster) rather than guessed at.

Design commitments

  • Lossless bitonal compression only. JBIG2 generic-region coding, not lossy symbol matching. Lossy JBIG2 can silently substitute characters in scanned documents — the 2013 Xerox scanner defect — and produces output that looks cleaner while being wrong. See FUTURE.md for the full reasoning. That is a commitment about what Byblos writes. It reads the symbol dictionaries and text regions other encoders wrote, including the shared dictionary a PDF keeps in /DecodeParms /JBIG2Globals, because a decoder places the symbols a stream names and never judges two glyphs alike — the substitution risk belongs entirely to the encoder.
  • Capability-based provenance. Every output Optimize rewrites records what each page actually received, and claims only the capabilities that call exercised — never the whole build's, which would suppress a later version's upgrade check. So a later version can identify exactly which stored documents would benefit from re-processing, and skip the ones that wouldn't.
  • Policy stays with the caller. Byblos exposes primitives — Inspect, ExtractPageRaster, Optimize, StampTextLayer, BuildPDF, and the image codecs. Preset ladders and validation rules belong to the application.

Relationship to other projects

  • cadmus — the OCR engine. Neither library imports the other; the application converts between them.
  • OCRmyPDF — a design reference, not a port target. It is MPL-2.0, and translating it would make those files MPL-2.0. It is also orchestration over the very binaries this project removes, so porting it would not have helped.

License

Apache-2.0. Reimplemented from format specifications, not ported.

Documentation

Overview

Package byblos is a pure-Go PDF pipeline for scanned documents: no cgo, no shared libraries, no subprocesses.

Byblos does not render PDFs. Scanned pages are overwhelmingly one page-covering image per page, which requires extraction rather than rendering; pages that are not are detected and reported with ErrNotSingleRaster rather than guessed at. See docs/superpowers/specs.

Licensing: Byblos is Apache-2.0 and is reimplemented from format specifications — principally ISO 32000-1:2008 — and from the documented behaviour of the tools it replaces. It is NOT a port of OCRmyPDF, which is MPL-2.0 file-level copyleft; no OCRmyPDF source is consulted or translated. See NOTICE.

Index

Constants

View Source 
const CapabilityJBIG2Generic = "jbig2-generic"

CapabilityJBIG2Generic is the provenance capability string recorded for a page compressed with lossless JBIG2 generic region coding. A document whose provenance carries it is exactly the upgrade set for a future jbig2-symbol capability (see FUTURE.md).

View Source 
const MaxSkewDeg = 2.0

MaxSkewDeg is how far a placement's axes may lie from the page's before the image is treated as rotated or sheared.

It is an angle rather than a matrix entry for a reason recorded in byb-b1.2: the tolerance used to be 1e-6 compared against the off-diagonal terms, which are in points. At a page scale of 560 that is an exact-zero test — about 1e-7 degrees — so it rejected all 147 sub-degree scanner deskews in the measurement. Two degrees clears the widest of them (1.09) with room to spare and stays nowhere near a quarter turn.

IT IS EXPORTED BECAUSE IT IS ALSO THE STRAIGHTEN ENVELOPE, and a consumer that hardcodes it drifts (gap G3). A StraightenSpec.Deg wider than this leaves a placement byblos then diverts as rotated, so an editor offering a straighten slider has to clamp to the same number this file diverts on. Kleio hardcoded 2.0 in its slider while this was unexported; read it from here instead.

View Source 
const Version = "0.2.0"

Version is the Byblos semver recorded in every Provenance. It exists for humans and bug reports; upgrade decisions are driven by Capabilities, not by comparing versions (design spec section 6).

Variables

View Source 
var ErrNotImplemented = errors.New("byblos: not implemented")

ErrNotImplemented reports that a caller asked for something Byblos does not do YET, as opposed to something that failed or something about this particular document.

The distinction is the whole point, and it is a distinction a caller has to be able to make in code rather than by reading a message. Three things can go wrong in a call like Optimize and they want three different responses: 

this document is broken        -> park it, review it
this document is not eligible  -> divert it, record why (ErrNotSingleRaster)
Byblos cannot do this at all   -> fall back to the old tool, for EVERY document

Only the third is a property of the build rather than of the input, so only the third should make a caller change its pipeline instead of its handling of one file. Retrying, laddering or quarantining a document because the library lacks a feature is wasted work at best; at worst it looks like a corpus of bad documents and hides the real cause.

Test with errors.Is. To find out WHICH capability is missing, so a caller can keep using Byblos for everything else, use errors.As with *NotImplemented.

View Source 
var ErrNotSingleRaster = errors.New("byblos: page is not a single raster")

ErrNotSingleRaster reports a page that is not one visible image: tiled rasters, visible vector content, or image-plus-overlay. The wrapped message names the specific reason.

How much of the page that one image covers is not part of the test, because nothing in the content stream can mark the rest of it (byb-b1.3). PageRaster.CoversPage is what tells the caller. Callers divert such documents for review; design spec section 2 explains why detecting rather than rendering is the whole reason this project is tractable.

"Visible" is doing work in that sentence. A path painted before the raster and inside its placement box marks nothing anyone can see, and byb-b1.5 measured 126 scan-shaped pages diverting on exactly that. See paintsHidden.

View Source 
var ErrUnstampableRune = errors.New("byblos: rune is outside the glyphless font's coverage")

ErrUnstampableRune reports a rune outside the glyphless font's coverage (internal/glyphless.FirstRune..LastRune, printable ASCII). Substituting a different glyph would mangle recognized text invisibly, so StampTextLayer errors instead of guessing; a Type0/Identity-H font is the follow-up for non-ASCII OCR text.

View Source 
var ErrUnsupportedImageCodec = errors.New("byblos: page raster uses an image codec byblos cannot decode")

ErrUnsupportedImageCodec reports a page raster stored in a codec Byblos cannot decode: JBIG2, JPEG 2000, or CMYK images pdfcpu re-renders as TIFF.

This error exists because of a specific correctness trap: pdfcpu does not error on JBIG2Decode or JPXDecode, it returns the raw opaque bytes. Handing those to an image decoder would either fail obscurely or, worse, appear to work. Byblos names the case instead.

View Source 
var ErrUnsupportedJBIG2Feature = errors.New("byblos: JBIG2 stream uses a feature byblos does not decode")

ErrUnsupportedJBIG2Feature reports a JBIG2 stream that parsed correctly and uses a coding feature byblos does not implement: a symbol dictionary or text region, refinement, halftones, MMR, or a generic region coded with anything other than GBTEMPLATE 0 and the nominal AT pixels.

It is worth distinguishing from an ordinary decode failure. This error means the bytes are fine and byblos is not enough; a plain error means the bytes are not fine. An archive deciding what to re-process later acts on that difference -- the first is a page a future decoder recovers, the second is damage.

Functions

func BuildFromPages added in v0.3.0 

func BuildFromPages(w io.Writer, pages []PageSource) error

BuildFromPages writes a document whose page i is pages[i], and a provenance record that describes it.

WHAT AN EXPORT KEEPS, AND WHAT IT DROPS. Each page arrives with its content stream, its resources, its annotations, its inherited attributes pushed down, and its provenance record moved to its new index. The document's catalog does NOT come with it: outlines, page labels, the structure tree, form fields and named destinations are dropped, because they describe the page SET or the page ORDER and an edit makes them silently wrong. Measured over the pinned sample, 61.9% of multi-page documents carry at least one such entry. See the design spec's 2026-08-13 amendment.

OUTPUT IS NOT BYTE-STABLE and must not be treated as though it were. Two builds of one sequence differ in object numbering and in length. Content- address an export -- write it once, under a key derived from its bytes -- and never rewrite a key in place: a second write hands a client mid-download a torn object and invalidates any checksum taken over the first.

OUTPUT IS NEVER LINEARIZED. The record says "rewritten-delinearized", which is what UpgradeCandidates reads to nominate the document for re-linearization. Compose with Optimize{Linearize: true} when a linearized export is wanted; asking for it here would mean re-opening the document from scratch.

It cannot be cancelled. Use BuildFromPagesContext when the caller has a deadline.

func BuildFromPagesContext added in v0.3.0 

func BuildFromPagesContext(ctx context.Context, w io.Writer, pages []PageSource) error

BuildFromPagesContext is BuildFromPages, cancellable at each source and at each page of the record (byb-xyn).

CANCELLATION LATENCY: EFFECTIVELY THE WHOLE CALL, and for the same reason BuildPDFContext gives. The checked boundary brackets the migration walk and its own write, which is now the ONE pdfcpu WRITE pass this makes -- pdfdoc's BuildFromPagesWithProperties folds the provenance record into the same build instead of a second read-validate-optimize-write pass afterwards (byb-yul.6, Correction 5; see its own doc comment for why that second pass is not just redundant but actively wrong). A second, READ-ONLY pass still runs below: pdfdoc.Validate, over the buffered output, restoring the gate the old second WRITE pass used to provide as a side effect of piping the bytes through api.AddProperties' own ReadValidateAndOptimize. Without it, a source pdfcpu's validator refuses -- an unsupported /PresSteps, a malformed date, a bad /ShowBookmarks -- built and wrote silently, with a nil error, once Correction 5 took the old validating pass away (found in review). Validate only reads; it does not reintroduce the dangling- reference bug a second WRITE pass caused. Budget for the whole build. A cancelled call writes nothing to w. See context.go.

func BuildPDF 

func BuildPDF(w io.Writer, pages []BuildPage) error

BuildPDF writes a PDF whose page i paints pages[i].Image and nothing else.

It supports FlateDecode, DCTDecode (DeviceGray/DeviceRGB only) and JBIG2Decode (carried verbatim); any other filter, or an unsupported BPC/colour-space combination for one of those, is rejected rather than written as a file no reader can open.

It cannot be cancelled. Use BuildPDFContext when the caller has a deadline.

func BuildPDFContext added in v0.3.0 

func BuildPDFContext(ctx context.Context, w io.Writer, pages []BuildPage) error

BuildPDFContext is BuildPDF, cancellable at each page boundary (byb-xyn).

CANCELLATION LATENCY: EFFECTIVELY THE WHOLE CALL. The page loop is checked, but it only resolves each page's box -- arithmetic -- while the pdfbuild.Write that follows is a single uninterruptible pass over every page's encoded bytes, and that is where all the time goes. Measured over 120 pages, the longest stretch between two context checks was 94% of the call. The per-page boundary here is real but nearly worthless; budget for the whole write. A cancelled call writes nothing to w. See context.go.

func Capabilities 

func Capabilities() []string

Capabilities returns, sorted, what this build can do.

func Downsample 

func Downsample(img image.Image, srcDPI, dstDPI float64) (image.Image, error)

Downsample resamples img from srcDPI to dstDPI with a Catmull-Rom kernel (golang.org/x/image/draw), the x/image scaler that agrees with ghostscript's /Bicubic downsampler (design spec byb-b3 section 1, 4).

srcDPI <= dstDPI, or a ratio that rounds to identical output dimensions, is a no-op: img is returned unchanged, not resampled. Kleio's compression ladder must be able to ask for 300 DPI on a 150 DPI scan and get the scan back, not a failure, and upsampling a scan invents detail byblos will not produce.

Downsample is DownsampleDeclaredBPC(img, 8, srcDPI, dstDPI) and nothing else: 8 is the depth it has silently assumed since it was written. That assumption is wrong for a source the PDF declares /BitsPerComponent 1, which Catmull-Rom blends into grey levels a bilevel image cannot hold (byb-plj; measured, a 600x800 bitonal text scan comes back with 99 distinct levels). An image.Image cannot say what depth it was declared at, so Downsample cannot tell that case apart and this signature cannot be made to -- see DownsampleDeclaredBPC, which takes the depth as an argument.

func DownsampleDeclaredBPC added in v0.2.0 

func DownsampleDeclaredBPC(img image.Image, declaredBPC int, srcDPI, dstDPI float64) (image.Image, error)

DownsampleDeclaredBPC is Downsample plus the one input an image.Image cannot carry: declaredBPC, the source's /BitsPerComponent as its PDF declares it. Pass 1 for an /ImageMask stencil, which is bilevel by definition and has no /BitsPerComponent entry. Every other rule -- the validation, the no-op cases, the output dimensions, the returned concrete type -- is Downsample's, because this is the function Downsample calls.

declaredBPC 1 resamples by point subsampling (draw.NearestNeighbor), so the output can only ever contain values the source already had and a bilevel scan cannot gain grey levels it never had. Every other declared depth interpolates with Catmull-Rom, exactly as before.

THE DEPTH IS DECLARED, NEVER DETECTED, and that distinction is the whole reason this function exists rather than a fix inside Downsample. A bitonal scan widened to 8 bpc -- a bitonal TIFF imported as 8-bpc DeviceGray or DeviceRGB, which is legitimate and common -- has nothing but pure black and white pixels and is still NOT a mono image to any PDF tool. Ghostscript draws the line at the declaration: /MonoImageDownsampleType governs images declared 1 bpc, everything else goes through /ColorImageDownsampleType. An earlier attempt at byb-plj inferred bilevel-ness by scanning the pixels; measured against the Ghostscript oracle it dropped such an 8-bpc page from 41.66 dB to 21.24 dB, 13 dB under the gate in downsample_oracle_test.go, because it fired on a source Ghostscript downsamples bicubicly.

Point subsampling is not an approximation of what Ghostscript does to a mono image, it is the same thing. Measured against gs pdfwrite on a 1-bpc DeviceGray page, draw.NearestNeighbor is PIXEL-IDENTICAL to its output for /Subsample, /Average AND /Bicubic alike: gs subsamples a mono image whatever /MonoImageDownsampleType asks for, because an interpolating filter produces values 1 bpc cannot store. Kleio's ladder pins /Subsample (compress.go) and gets what it asked for.

Callers already hold the declaration, by either route into this library. Inspect surfaces it as ImageRef.Bitonal, "1 bit per component, or an image mask", keyed by the same ImageRef.ObjNr that ReplaceImages substitutes on; ExtractPageRaster surfaces it as PageRaster.Bitonal, where the pixels are. Internally both are pdfdoc.ImageInfo.BPC, the same predicate extract.go's mrcLayers branches on, and TestBitonalAgreesBetweenInspectAndExtract pins that the two routes never disagree — a caller that extracts a page, resamples it, and substitutes the result by object number crosses from one to the other.

The PIXELS still cannot say, and that has not changed: pdfcpu renders a 1-bpc DeviceGray image to PNG and image.Decode returns *image.RGBA, so the raster is 8 bits per channel with two distinct values in it. PageRaster.Bitonal is the only surviving record of what those samples were widened from, which is why it is a field beside the image rather than something recoverable from it.

func EncodeJBIG2Generic 

func EncodeJBIG2Generic(b *Bitmap) ([]byte, error)

EncodeJBIG2Generic compresses a bitonal bitmap with lossless JBIG2 generic region coding and returns a JBIG2 bitstream in the embedded file organization required by the PDF JBIG2Decode filter.

The coding is lossless: a decoder reconstructs b exactly, so no character can be substituted for another. Byblos does not implement lossy symbol matching and will not -- see FUTURE.md.

To embed the result as a PDF image XObject, use these dictionary entries and nothing else: 

/Type             /XObject
/Subtype          /Image
/Width            b.Width
/Height           b.Height
/ColorSpace       /DeviceGray
/BitsPerComponent 1
/Filter           /JBIG2Decode

No /Decode array: the JBIG2Decode filter already presents a JBIG2 black pixel as the DeviceGray sample that renders black, so adding /Decode [1 0] inverts the page. No /DecodeParms and no /JBIG2Globals stream either: generic region coding produces no page-0 segments. The filter must not be used with inline images (ISO 32000-1:2008 7.4.7).

EncodeJBIG2Generic does not copy b.Pix. It zeroes everything in each row past pixel b.Width-1: the padding bits inside that pixel's byte, and any whole bytes between there and b.Stride when the stride is larger than the minimal (b.Width+7)/8. On a well-formed bitmap that is a no-op, since all of it is required to be zero already. Pass a copy if that matters.

func Optimize 

func Optimize(w io.Writer, r io.ReadSeeker, opts OptimizeOptions) error

Optimize writes a structurally-optimized copy of r's PDF to w.

The size and linearization tradeoff (recorded on the bead byb-b5, not reopened here): Optimize returns min(input, pdfcpu-rewritten-output). Priority is size, and choosing the smaller of the two costs nothing in quality -- both candidates are lossless structural rewrites of the same document. The real tradeoff is size versus LINEARIZATION: on the documents where pdfcpu's rewrite is larger than the input, returning the input means a caller who asked to linearize did not get it, SILENTLY. That is why the branch taken is recorded on the result's Provenance.Optimized field rather than merely logged -- a log line is not visible to a caller inspecting the PDF later, and a field on the provenance record is.

Because the pass-through branch returns the input's bytes byte-for-byte verbatim (required to satisfy "never larger than input" literally, since any in-band write would grow it), it cannot itself record which branch ran. Provenance.Optimized's zero value covers that: it means "not known to have been rewritten by Optimize", which is what both an unprocessed document and a pass-through result share.

A document that reaches the rewritten branch with no prior provenance gets a fresh record with an EMPTY Capabilities: Optimize did not extract or inspect anything, so it must not claim those capabilities are done (see the note further down, and upgrade.go's UpgradeCandidates). It cannot be cancelled. Use OptimizeContext when the caller has a deadline.

func OptimizeContext added in v0.3.0 

func OptimizeContext(ctx context.Context, w io.Writer, r io.ReadSeeker, opts OptimizeOptions) error

OptimizeContext is Optimize, cancellable between its stages and inside the JPEG recompression pass (byb-xyn).

CANCELLATION LATENCY: A WHOLE PDFCPU ROUND TRIP, and on the default options that is essentially the whole call. Optimize's work is four pdfcpu passes -- pdfdoc.Optimize, ReadProvenance, WriteProvenance, and optionally Linearize -- none of which is interruptible. What makes the default path cancellable at all is that the checks sit BETWEEN those passes, so the latency is one pass and not the whole call; with RecompressJPEG false there is no finer boundary than that. With RecompressJPEG true the recompression pass adds per-page and per-image boundaries that ARE checked, so a context cancelled during recompression is honoured within one image's re-encode.

A caller that needs Optimize to stop promptly does not have that option today; it must budget for the document's full rewrite. A cancelled call writes nothing to w. See context.go.

func QuantizePNG 

func QuantizePNG(img image.Image, colors int) ([]byte, error)

QuantizePNG reduces img to at most colors distinct colours by median cut (with Lloyd/k-means refinement) and returns a complete, palette-indexed PNG file. See design spec byb-b3 section 1-2 for the accepted rationale.

img must be opaque: image.Image.At().RGBA() returns alpha-premultiplied values, and quantizing those directly would be wrong, while an RGBA palette is a shape byblos has no use for (ReplaceImage refuses /SMask and /Mask outright). colors must be 2..256, exactly the range pngquant itself enforces -- clamping would silently substitute a different request than the one asked for.

func ReplaceImages 

func ReplaceImages(w io.Writer, r io.ReadSeeker, subs map[int]EncodedImage) error

ReplaceImages copies r's PDF to w with each named image XObject's stream and dictionary replaced by the encoded image given for it. Keys are ImageRef.ObjNr, from Inspect; an object number this document has no image for is an error, not a silently skipped substitution.

It is per-OBJECT, which is what ImageRef.ObjNr's doc comment warns about from the other side: one XObject painted on several pages is substituted once and changes all of them.

Placement is untouched. A page positions an image with a CTM on the unit square, so a raster with different pixel dimensions lands in exactly the same rectangle at a different resolution; that is what makes downsampling a substitution rather than a re-layout.

What it refuses, inherited from the seam and not re-derived here (internal/pdfdoc/write.go): an image carrying /SMask or /Mask, whose transparency is keyed to the samples being replaced; an /ImageMask stencil, whose dictionary is a different shape; an image stream that is a direct object, which has no cross-reference entry to write back to. A stale /Decode array is dropped, because a leftover [1 0] would silently invert the new samples. The first failure aborts the whole call and nothing is written -- a partly substituted document would open cleanly and be wrong.

What it does NOT do, all of it deliberate:

  • It does not check that Data decodes to Width x Height samples. That is the encoder's contract (EncodedImage), and this seam carries JBIG2 and JPX bytes Byblos itself cannot decode.
  • It does not compare sizes. A substitution that grows the document is written, because whether that trade is worth making depends on what the caller was buying -- Optimize's "never larger than input" rule is Optimize's, and applying it here would silently discard, for instance, a bitonal re-encode a caller asked for on purpose.
  • It does not resample, and that is where byb-05w's obligation lands. Under option P this seam takes bytes an encoder already produced, so Byblos never chooses between the contone and the bilevel kernel -- whoever called Downsample did. Reaching for Downsample on a source the PDF declares /BitsPerComponent 1 silently reintroduces byb-plj, because Downsample is DownsampleDeclaredBPC(img, 8, ...) and Catmull-Rom blends a bilevel scan into grey levels it cannot hold. Pass the declaration instead: 1 for a bitonal source or an /ImageMask. Both routes into this library carry it -- ImageRef.Bitonal from Inspect, keyed by the same ObjNr substituted on here, and PageRaster.Bitonal from ExtractPageRaster.
  • It writes no provenance, the same as StampTextLayer and BuildPDF. Under option P it CANNOT: the Applied vocabulary names the capability that ran ("downsample-150", "jbig2-generic"), and the same call substitutes bytes from any encoder or none. The caller records what it applied, through RecordExtraction and WriteProvenance.

That last point has a sharp edge worth stating: a provenance record the document already carried survives this call unchanged, and a substitution can make it stale. Replacing a raster with JBIG2 turns a page that used to extract into one ExtractPageRaster reports ErrUnsupportedImageCodec for, while the old record still says "extract-raster". Re-run RecordExtraction after substituting if the record has to stay true. Page GEOMETRY does not go stale that way -- it is measured in points, and the placement did not move. It cannot be cancelled. Use ReplaceImagesContext when the caller has a deadline.

func ReplaceImagesContext added in v0.3.0 

func ReplaceImagesContext(ctx context.Context, w io.Writer, r io.ReadSeeker, subs map[int]EncodedImage) error

ReplaceImagesContext is ReplaceImages, cancellable at each page boundary of the resolving walk and at each substitution (byb-xyn).

CANCELLATION LATENCY: ONE PDFCPU PASS, not one page. The walk and the substitution loop are both checked per item, but they are bracketed by two uninterruptible pdfcpu passes -- the Open before and the d.Write after -- and those dominate. Measured over 120 pages, the longest stretch between two context checks was 46% of the call. A cancelled call writes nothing to w. See context.go.

func ResetExtractStats 

func ResetExtractStats()

ResetExtractStats zeroes every counter. Intended for tests and for a long-lived process that reports per-batch rather than cumulative rates.

func StampTextLayer 

func StampTextLayer(w io.Writer, r io.ReadSeeker, tl TextLayer) error

StampTextLayer writes tl into r's pages as an invisible text-render-mode-3 layer, and copies the result to w.

A page whose tl.Pages[i] is empty (including one beyond the end of a shorter tl.Pages) is not touched at all: no font resource is added and no content is appended. len(tl.Pages) greater than r's page count is an error.

It cannot be cancelled. Use StampTextLayerContext when the caller has a deadline.

func StampTextLayerContext added in v0.3.0 

func StampTextLayerContext(ctx context.Context, w io.Writer, r io.ReadSeeker, tl TextLayer) error

StampTextLayerContext is StampTextLayer, cancellable at each page boundary and at each word within a page (byb-xyn).

CANCELLATION LATENCY: ONE PDFCPU PASS. The checks sit at the top of the page loop and the top of the word loop, so the stamping itself is interrupted between words -- but the per-page tail after that loop (AddFontResource and AppendContent) runs unchecked, and the Open before and the d.Write after are whole uninterruptible pdfcpu passes. Measured over 120 pages carrying one word each, the longest stretch between two context checks was 22% of the call; on a document with fewer, longer pages the write dominates further. A cancelled call writes nothing to w. See context.go.

func UpgradeCandidates 

func UpgradeCandidates(p *Provenance, current []string) []string

UpgradeCandidates returns, sorted, the capabilities in current that the document described by p does not already have AND that would actually change its output. An empty result means re-processing the document is wasted work.

A capability with no rule is reported: missing a real upgrade is worse than one wasted re-run, and TestEveryCapabilityHasARule keeps the gap from persisting. A nil p means nothing is known about the document, so every capability is a candidate.

func ValidatePages added in v0.4.0 

func ValidatePages(pages []PageSource) error

ValidatePages reports whether BuildFromPages would refuse pages before it opened a single source (gap G2).

IT IS THE CHECK A REQUEST HANDLER CAN AFFORD. Kleio accepts an edit list at PUT time and materialises it later, in a worker, off a queue — so a list BuildFromPages refuses surfaces as a failed export minutes later rather than as a 422 the editor can show. This is the half of that refusal that needs no document, no download and no build: the sequence is non-empty, every page names a Source, every Rotate is one of 0/90/180/270, every Straighten.Deg is finite, and no Straighten carries a Crop (still not implemented; gap G1).

WHAT IT CANNOT CHECK, AND THIS IS THE HALF THAT MATTERS MOST IN PRACTICE: whether PageSource.Page exists in its source. That needs the source's page count, which needs the document open — Inspect answers it, and a consumer storing a page count of its own can answer it without byblos at all. A nil return here is therefore "nothing structural is wrong", not "this will build". An encrypted source is likewise only refused at build time.

It never reads from any Source, so a PageSource whose reader is not yet positioned, or not yet fetched, still validates.

A CALLER THAT HAS NO SOURCES YET STILL HAS TO SUPPLY NON-NIL ONES. The nil-Source refusal is a real build precondition and stays, so a handler validating an edit list it has not fetched the documents for passes any non-nil reader -- one shared empty bytes.Reader for the whole list is enough, because nothing here touches it: 

var unfetched = bytes.NewReader(nil)
// ... p.Source = unfetched for every page ...
if err := byblos.ValidatePages(pages); err != nil { /* 422 */ }

That is deliberate rather than an oversight. Dropping the check would make this function's answer differ from the build's, and the ONE property worth having here is that the two cannot disagree.

func WriteProvenance 

func WriteProvenance(r io.ReadSeeker, w io.Writer, p Provenance) error

WriteProvenance marshals p to JSON and stores it under provenanceKey in r's Info dictionary, via pdfdoc.WriteProperties (pdfcpu's api.AddProperties underneath). It needs only an Info dictionary to exist, not an extraction outcome or a text layer -- see byb-0dz, which split this half of B5 off byb-b1/byb-b4 for exactly that reason.

It cannot be cancelled. Use WriteProvenanceContext when the caller has a deadline.

func WriteProvenanceContext added in v0.3.0 

func WriteProvenanceContext(ctx context.Context, r io.ReadSeeker, w io.Writer, p Provenance) error

WriteProvenanceContext is WriteProvenance, cancellable only before its work begins (byb-xyn).

CANCELLATION LATENCY: A WHOLE PDFCPU ROUND TRIP. Writing provenance is one indivisible read-validate-optimize-write pass through pdfcpu, which is not context-aware; there is no loop boundary byblos owns inside it, so the context is consulted once on entry and not again. The parameter exists so a caller can decline to START the work, and so this primitive composes with the other eight rather than being the one that silently takes no context. A cancelled call writes nothing to w. See context.go.

Types

type Bitmap 

type Bitmap struct {
	Width, Height int
	Stride        int // bytes per row
	Pix           []byte
}

Bitmap is a 1-bit-per-pixel bilevel image owned by Byblos.

Byblos deliberately does not share this type with Cadmus: neither library imports the other (design spec section 3), so each owns its own substrate.

A set bit (1) is BLACK ink. That matches JBIG2, where 1 is black, and is the inverse of PDF /DeviceGray, where 1 is white; any conversion across that boundary inverts.

The origin is top-left and y increases downward, matching image.Rectangle. Note that PageInfo.Bounds uses the opposite (PDF) convention.

Rows are packed MSB-first: pixel (x, y) is bit 7-(x%8) of Pix[y*Stride + x/8]. Bits past Width in the final byte of a row are always zero, and Set preserves that, so Equal may compare Pix directly.

func DecodeJBIG2Generic added in v0.2.0 

func DecodeJBIG2Generic(data []byte) (*Bitmap, error)

DecodeJBIG2Generic decodes a JBIG2 bitstream in the embedded file organization -- the form the PDF JBIG2Decode filter carries, and the form EncodeJBIG2Generic produces -- and returns the page bitmap. A set bit is ink, matching Bitmap's convention throughout.

It inverts EncodeJBIG2Generic UP TO A SIZE, bit-identically, and nothing wider. Below that size every stream EncodeJBIG2Generic produces decodes back exactly, which is what makes a byblos archive re-openable by byblos. Above it, byblos can write a page it will not read back -- and that asymmetry is deliberate, not a gap. EncodeJBIG2Generic shipped in v0.1.0 with no size budget at all and callers pin that tag, so the write side is fixed; the read side is where an untrusted stream arrives, and the resource budgets that make it safe (internal/jbig2, MaxPagePixels and maxStreamBitmapBytes) necessarily bound what it will accept.

The boundary, in terms a caller can act on:

  • It decodes any page of at most 67,108,864 pixels whose bitmaps pack into 16 MiB. That covers every 600-dpi preservation master byblos is handed -- A4 (4961x7016), US Letter (5100x6600), US Legal (5100x8400) -- and 800-dpi A4 (6614x9354, 61,867,356 pixels), which is the largest sheet size that round-trips.
  • It does not decode 600-dpi A3 (7016x9921, 69,605,736 pixels), anything at 1200 dpi, or a bitmap so narrow that row padding blows the 16 MiB -- a 1x8388609 column is an eighth of the pixel budget and two bytes past the memory one. EncodeJBIG2Generic writes every one of them, and cheaply: measured on a blank page with two corner pixels, 81 bytes for A3, for 1200-dpi Letter and for A2, and 109 for the 1x8388609 column, whose 8,388,609 single-pixel rows cost more coded bits than a page-shaped bitmap of the same area.
  • It does not decode a stream of more than 65,536 segments, whatever sizes those segments declare. That bound is one region per row of the tallest page the pixel budget admits, and what it limits is the cost of reading the headers, which the size budgets cannot charge for because they are evaluated from the headers (internal/jbig2, rule 5). Nothing EncodeJBIG2Generic writes comes near it: it emits two segments per page.

TestEncodeDecodeSizeBoundary pins that list. A caller holding a page larger than the envelope and needing to read it back must tile it; nothing here will silently produce a partial raster instead.

WHAT IT DECODES, and the name is now narrower than the function (byb-9v0): immediate generic regions coded with GBTEMPLATE 0 and the nominal AT pixels of T.88 Table 5, AND arithmetically coded symbol dictionaries with the immediate text regions that place them. The name is kept because callers pin it; what changed is the capability, not the contract.

Every other JBIG2 coding mode returns ErrUnsupportedJBIG2Feature and no bitmap: Huffman symbol coding, refinement, halftones, MMR, the other three generic templates and non-nominal AT pixels.

The refusal is the point, not a gap left to fill in later. The MQ arithmetic decoder returns a decision for any input whatsoever, so running it over a Huffman or MMR stream does not fail -- it yields a full-size bitmap of noise. An error a caller can route around is strictly better than a raster that is wrong without looking wrong.

PAGE-0 (GLOBAL) SEGMENTS FROM A PDF /JBIG2Globals STREAM ARE NOT VISIBLE ON THIS ENTRY POINT, because it is handed one stream and that object is another. A symbol dictionary very often lives there -- a bulk scanner writes one per document and every page points at it -- and a text region without its dictionary is refused here rather than decoded to a blank page. Extraction does not have that limitation: ExtractPageRaster reads /DecodeParms and hands both streams to the decoder.

func NewBitmap 

func NewBitmap(w, h int) *Bitmap

NewBitmap returns a w x h bitmap with every pixel clear. It panics on a negative dimension.

func Sauvola 

func Sauvola(img image.Image) (*Bitmap, error)

Sauvola binarizes img by local adaptive thresholding (J. Sauvola & M. Pietikäinen, "Adaptive document image binarization", Pattern Recognition 33(2), 2000), returning a Bitmap ready for EncodeJBIG2Generic.

Adaptive, not global: a scan with uneven illumination -- a shadowed gutter, a page corner lifted off the platen -- has no single cutoff that both keeps the shadow white and keeps the text in the bright half black, because the ink and paper intensity bands overlap. Sauvola computes, for every pixel, a threshold from the mean m and standard deviation s of an odd windowSize x windowSize neighbourhood centred on it: 

T(x,y) = m * (1 + k * (s/r - 1))

r is the dynamic range of s, fixed at 128 for 8-bit greyscale, and k controls how strongly local contrast lowers the threshold in text-bearing regions. k=0.5 is Sauvola's own paper's value; windowSize=31 suits the stroke widths in this corpus. Both were swept (windowSize 15..41, k 0.2..0.5) and the output moves smoothly and monotonically across that range -- more ink for a wider window, less for a larger k -- so these are a conservative point on a continuum, not a cliff edge.

A pixel with value strictly less than its local threshold is ink (bit 1); this matches Bitmap's convention (set bit = black), the inverse of /DeviceGray.

Known and intrinsic: a uniform dark region LARGER than the window comes out hollow. Inside such a region s is ~0 and m is the fill's own value, so T -> m*(1-k) and no pixel clears it. Sauvola marks edges of solid fills, not their interiors; that is the algorithm, not a defect in this code, and it costs nothing for text (strokes are thinner than the window) while visibly hollowing out stamps and logos.

The mean and variance are both obtained from a single pass over two integral images (sum and sum-of-squares) built from img, giving O(1) work per pixel regardless of window size after that O(width*height) setup.

func (*Bitmap) At 

func (b *Bitmap) At(x, y int) uint8

At returns 1 or 0. Pixels outside the bitmap read as 0, as required by ITU-T T.88 section 6.2.5.2 for JBIG2 template gathering.

func (*Bitmap) Bounds 

func (b *Bitmap) Bounds() image.Rectangle

Bounds returns the bitmap's extent with the origin at (0, 0).

func (*Bitmap) Clone 

func (b *Bitmap) Clone() *Bitmap

Clone returns a deep copy.

func (*Bitmap) Equal 

func (b *Bitmap) Equal(o *Bitmap) bool

Equal reports whether o has the same dimensions and the same pixels. It is the lossless check the JBIG2 round-trip test in B2 is built on.

func (*Bitmap) Set 

func (b *Bitmap) Set(x, y int, v uint8)

Set writes 1 when v is non-zero and 0 otherwise. Coordinates outside the bitmap are ignored.

type BuildPage 

type BuildPage struct {
	Image             EncodedImage
	WidthPt, HeightPt float64
	DPI               float64
}

BuildPage is one page of a document built from images: one encoded raster and the page box to paint it on.

WidthPt/HeightPt are the MediaBox in points, PDF default user space (origin lower-left, y increasing upward — the same convention as PageInfo.Bounds). Both zero means derive the box from the image's pixel dimensions and DPI, which is what a scan wants.

DPI is the raster's resolution, used only when the box is not given. Zero DPI with no box is an error, not a default: guessing 300 on a 600 DPI plate makes a page twice its true size and nothing downstream can tell.

Placement within the box is fit-centered (contain): the image is scaled uniformly to fit inside the box and centred there. When the box's aspect ratio does not match the image's, PageRaster.CoversPage will report false for that page — correct and expected, not a defect.

type ColorSpace 

type ColorSpace = pdfdoc.ColorSpace

type DecodeParms 

type DecodeParms = pdfdoc.DecodeParms

type Diagnostic added in v0.3.0 

type Diagnostic struct {
	Severity Severity
	Message  string
}

Diagnostic is one problem byblos worked around while reading a page.

byb-3jq: byblos used to refuse the WHOLE document for any of these. Measured over govdocs1 that cost 176 readable pages to 7 bad ones, 135 of them in one document over three pages. Poppler reads all four of those documents and reports the problem on stderr rather than withholding the file, which is the behaviour this mirrors.

Message is the underlying error's text. Poppler's callback also carries a machine-readable byte offset (Goffset pos); byblos's offsets are inside the message text for now, because the lexer formats them rather than returning a typed error.

type EncodedImage 

type EncodedImage = pdfdoc.EncodedImage

EncodedImage, ColorSpace and DecodeParms are the vocabulary the write seam already defines (internal/pdfdoc/write.go). Aliased here because internal/ packages are unreachable from Kleio, and a parallel type would be a second thing to keep in step with ReplaceImage's.

func QuantizeIndexed 

func QuantizeIndexed(img image.Image, colors int) (EncodedImage, error)

QuantizeIndexed reduces img to at most colors distinct colours by the same median-cut/Lloyd/population-order core QuantizePNG uses (quantizeCore, quantize.go), and packages the result as a PDF /Indexed, /FlateDecode image: EncodedImage.Data is the raw PNG-predicted, deflate-compressed scanline stream (not a PNG file -- see EncodedImage.Data's doc comment in internal/pdfdoc/write.go for why a PNG file is the wrong shape here), ready for pdfdoc.ReplaceImage.

Container decision (byb-96p, lead 5): QuantizeIndexed shares quantizeCore with QuantizePNG to get the identical *image.Paletted -- crucially including byb-20b's population-order permutation, which must not drift between the two entry points -- then runs that SAME image through the standard library's PNG encoder and pulls the /Indexed colour table (PLTE) and the concatenation of every IDAT chunk's payload straight out of the resulting chunk framing. Per the PNG spec the IDAT payloads concatenate into one continuous zlib stream regardless of how the encoder split them across chunks, and that stream IS the predictor-prefixed scanline data a PDF /FlateDecode filter with /DecodeParms /Predictor 15 expects -- so nothing needs decoding or re-encoding, only chunk parsing. This is deliberately option (a) from the bead (parse chunks out of a PNG), applied to quantizeCore's private *image.Paletted rather than to QuantizePNG's public []byte, so QuantizePNG's own bytes (and its calibrated oracle test) are untouched by this file.

type ExtractCounters 

type ExtractCounters struct {
	Attempted uint64
	Extracted uint64
	// Partial counts the extracted pages whose raster does not fill the page
	// box. It is not a fourth outcome — it is a subset of Extracted, and does
	// not disturb the sum above.
	//
	// It exists because byb-b1.3 retired the "not-page-covering" divert reason.
	// Those 132 measured pages are ordinary scans placed at their natural
	// resolution and they now extract, which is correct, but without this they
	// would vanish from the instrumentation the whole design is checked by.
	Partial  uint64
	Diverted uint64            // page understood, but not a single raster
	Failed   uint64            // could not be read at all: damaged file, missing page
	Reasons  map[string]uint64 // divert reason to count; see classify in extract.go
}

ExtractCounters is a snapshot of ExtractPageRaster/RecordExtraction outcomes since process start, or since the last ResetExtractStats.

The design of Byblos rests on the premise that a page which is not a single raster is rare (design spec section 2). These counters are how that premise is checked against reality rather than assumed. Export UnhandledRate from your application; if it is not small, the premise is wrong and the design needs revisiting.

Every attempt increments exactly one of Extracted, Diverted and Failed, so those three always sum to Attempted.

func ExtractStats 

func ExtractStats() ExtractCounters

ExtractStats returns a snapshot. The returned Reasons map is a copy, so the caller may keep or mutate it freely.

func (ExtractCounters) DivertRate 

func (c ExtractCounters) DivertRate() float64

DivertRate is the fraction of attempted pages that diverted. Failures are excluded from the numerator but not the denominator: a document Byblos could not read is a different problem from one it read and declined.

That makes DivertRate the wrong number to watch on its own, and byb-5kk is the demonstration: a page-tree bug made 1,266 pages of a real archive sample unreadable, and because every one of them landed in Failed, the defect pushed the measured divert rate *down*. A whole document class became unprocessable while the metric meant to detect exactly that moved in the reassuring direction. Watch UnhandledRate; reach for this one to split the total.

func (ExtractCounters) UnhandledRate 

func (c ExtractCounters) UnhandledRate() float64

UnhandledRate is the fraction of attempted pages that produced no raster, whether Byblos declined them or could not read them at all.

This is the premise check. It has no blind spot by construction — it is 1 - Extracted/Attempted — so no outcome can grow without moving it, and no change to the divert vocabulary can quietly stop counting a class of page. Diverted and Failed remain readable side by side when the number is bad and the question becomes which problem it is.

type ImageRef 

type ImageRef struct {
	Bounds    image.Rectangle
	Placement [6]float64
	// PlacementDeg is Placement's rotation about the page's axes, in degrees,
	// positive COUNTER-CLOCKWISE -- atan2(b, a). It is the SIGNED angle, where
	// skewDegrees (extract.go:71-75) is unsigned and cannot express a
	// direction.
	//
	// It exists so a caller can see a straighten's consequence before asking
	// for it: a correction of Deg leaves the placement at PlacementDeg + Deg,
	// and placementReason (extract.go:800) diverts a page past MaxSkewDeg =
	// 2.0. The precedent is Substitutable -- a caller that cannot see a
	// refusal coming cannot drive the primitive (byb-js5.2).
	PlacementDeg  float64
	Width, Height int    // pixel dimensions of the stored raster
	Bitonal       bool   // 1 bit per component, or an image mask
	Filter        string // the stored raster's declared codec; "" when it declares none
	ObjNr         int    // the image XObject's PDF object number
	Substitutable bool   // ReplaceImages will accept this image; see below
}

ImageRef is one painting of an image on a page.

Bounds is where the image is actually VISIBLE, in PDF default user space: points, origin lower-left, y increasing upward. image.Rectangle is used only as a convenient integer rectangle — do not read it as screen coordinates.

Placement is the matrix the image was painted with, in PDF matrix order [a b c d e f] (ISO 32000-1 section 8.3.3), mapping the image's unit square into user space. Before byb-b1.12, Bounds was exactly that mapped unit square's axis-aligned bounding box -- the same rectangle for a clean placement and for one a scanner deskewed by a fraction of a degree, with the rotation visible only in Placement. Since byb-b1.12, Bounds is that same bounding box intersected with any W/W* clip path or form /BBox in effect at the Do: a placement clipped to a corner of the page reports the corner, not the raster's own oversized extent, while Placement keeps describing the full unclipped matrix regardless. A caller deriving a scale or DPI from Bounds against the stored raster's pixel dimensions (Width, Height) must account for this — a clipped Bounds does not mean the raster was stored at a different resolution.

Bounds is never EMPTY for a placement that can paint (byb-62t). A placement narrower than a point on an axis — `q .4 0 0 .4 10 10 cm /Im0 Do Q`, which poppler renders — is reported as the smallest integer rectangle containing it, so it overstates the extent by up to a point per edge rather than vanishing. That is a further reason to read a scale off Placement and not off Bounds.

Filter is the codec the stored raster declares: "JBIG2Decode", "CCITTFaxDecode", "DCTDecode", "FlateDecode", and so on; "" when the stream declares no /Filter or declares one this does not recognize as a name. It is the codec, not the transport wrapper — a /Filter [/ASCII85Decode /JBIG2Decode] chain reports JBIG2Decode. Read it beside Bitonal to triage work without decoding anything: Bitonal with Filter "JBIG2Decode" is already what Byblos would produce, Bitonal with any other filter needs only a re-encode, and not Bitonal needs a binarizer first.

ObjNr identifies the image XObject itself, and is the handle ReplaceImages takes. It is per-OBJECT, not per-painting: one XObject can be painted on several pages, or twice on one page, and every such ImageRef reports the same ObjNr — which is exactly the signal a caller needs to re-encode a shared raster once rather than once per page. It is negative for an image stream that is a direct object, which ISO 32000-1 section 7.3.8.1 forbids and which ReplaceImages therefore refuses; such a stream has no cross-reference entry to write a substitution back to.

Substitutable reports whether ReplaceImages will accept this ObjNr. It is false for the four cases the write seam refuses (internal/pdfdoc/write.go: 205-212): an /SMask or a /Mask, whose transparency is keyed to the samples being replaced; an /ImageMask stencil, whose dictionary is a different shape; and a direct object, which is the negative-ObjNr case above.

IT EXISTS BECAUSE ReplaceImages IS ALL-OR-NOTHING. The first refusal aborts the whole call and nothing is written (substitute.go), so a caller building a substitution map has to be able to see a refusal coming rather than discover it as an error string after the batch is discarded (byb-js5.2).

READ IT BESIDE Bitonal, NOT INSTEAD OF IT. Bitonal is "1 bit per component OR an image mask", and an image mask is exactly what the seam refuses — so a caller selecting bilevel candidates for a JBIG2 substitution by Bitonal alone picks the images that will abort its call. The two fields answer different questions: Bitonal is about the samples, this is about the write path.

type NotImplemented 

type NotImplemented struct {
	Capability string // e.g. "linearize"
	Why        string // one sentence, in terms of what is missing, not what to do
	Issue      string // e.g. "byb-k48"
}

NotImplemented names a capability Byblos does not have yet, why not, and where the work is tracked.

Capability is a capability string from the same vocabulary provenance and UpgradeCandidates use (see buildCapabilities in provenance.go and capabilityRules in upgrade.go), NOT free text. That is deliberate: it means a caller that catches one of these can hand the string straight to UpgradeCandidates later to ask whether a newer build would now handle the documents it fell back on, instead of maintaining its own parallel list of what this version could not do. TestEveryNotImplementedSiteNamesTheRegister keeps the two vocabularies from drifting apart.

THAT SENTENCE NAMED A TEST THAT DID NOT EXIST, from byb-b3 until byb-bjh. It said TestEveryNotImplementedNamesAKnownCapability enforced this; byb-b3's GREEN stage had deleted that test, because its table lost its only row when RecompressJPEG stopped returning a *NotImplemented and it would have passed vacuously forever after (the tombstone is in optimize_test.go). A doc comment promising a guard that was removed is the same defect this whole type exists to fix -- a claim with nothing behind it -- so it is recorded here rather than quietly corrected.

Issue is the tracking id, so the error itself says where the answer lives rather than requiring a search. It is in the message for the same reason. It comes from capabilityIssue (upgrade.go) rather than being written at each construction site, so the bead an error reports and the bead the register tracks cannot be two different answers.

func (*NotImplemented) Error 

func (e *NotImplemented) Error() string

func (*NotImplemented) Unwrap 

func (e *NotImplemented) Unwrap() error

Unwrap makes errors.Is(err, ErrNotImplemented) true for every NotImplemented, so a caller that only wants "can this build do it at all?" does not have to know the concrete type.

type OptimizeOptions 

type OptimizeOptions struct {
	// Linearize requests a linearized ("fast web view") output.
	//
	// pdfcpu v0.13.0 cannot produce this and never will be asked to:
	// OptimizeContext frees linearization hint-table objects on write whenever
	// IsLinearizationObject is true (pdfcpu write.go, deleteRedundantObject),
	// and nothing in its write path ever emits a /Linearized dictionary.
	// Measured directly against pdfcpu's own linearized fixtures: a round trip
	// through pdfcpu's optimize pass STRIPS linearization rather than adding it
	// (bookletTest.pdf 50308 B Linearized=true -> 34531 B Linearized=false;
	// WaldenFull.pdf 3482146 B Linearized=true -> 1942597 B Linearized=false).
	// Byblos therefore owns the Annex F write path itself; see
	// internal/linearize and internal/pdfdoc/linearize.go (byb-1y7).
	//
	// Two consequences a caller has to know about:
	//
	//   - The output is larger than it would have been WITHOUT linearizing:
	//     measured over the 35 readable corpus documents, +649 to +2071 bytes
	//     against the same document rewritten and not linearized, with no
	//     exceptions. Linearization adds a second cross-reference section, a
	//     parameter dictionary and a hint stream, and it forbids the object
	//     streams pdfcpu's rewrite would otherwise use.
	//
	//     Against the INPUT it is usually but not always larger, because
	//     pdfcpu's rewrite can save more than linearization costs: the corpus
	//     spans +7 (dup-raster) to +1007, and 4 of the 49 PDFs in pdfcpu's own
	//     testdata come out smaller than they went in -- WaldenFull.pdf
	//     3482146 -> 2152320, VectorApple.pdf 1062861 -> 812959,
	//     testWithText.pdf 30008 -> 20859, read.go.pdf 254303 -> 254110.
	//     So "never larger than input" would not discard every linearized
	//     output; it would discard MOST of them, unpredictably, which is worse.
	//     The rule is therefore suspended for this branch and for this branch
	//     only -- a pass-through here would be a caller asking to linearize and
	//     silently not getting it, which is the exact failure
	//     Provenance.Optimized exists to make visible.
	//   - Linearization runs LAST. Writing provenance goes through pdfcpu's
	//     writer, which strips linearization, so nothing may re-serialize the
	//     document after this point.
	Linearize bool

	// RecompressJPEG re-encodes every DCTDecode image XObject at JPEGQuality
	// and substitutes it through pdfdoc.ReplaceImage. It is lossy and it is
	// the only lossy thing Optimize does.
	//
	// It touches only images pdfcpu renders as file type "jpg" -- DCTDecode
	// with at most three components -- whose /ColorSpace is the name
	// /DeviceGray or /DeviceRGB and which carry no /SMask, /Mask or
	// /ImageMask. Everything else is left byte-for-byte alone, with no error:
	// a page with no JPEG on it is not an ineligible document, it is a page
	// with no JPEG on it. Which pages were actually re-encoded is recorded as
	// "jpeg-recompress-<quality>" in each page's PageProvenance.Applied.
	//
	// A re-encoded stream is substituted only when it is strictly smaller than
	// the one it replaces, so this pass cannot grow a document and Optimize's
	// "never larger than input" rule needs no suspension for it (unlike
	// Linearize above).
	//
	// JPEGQuality must be 1..100 and Optimize validates it. image/jpeg would
	// silently clamp instead, and a caller who passed 150 would get 100 and no
	// way to know. Zero is an error rather than a default, for the same reason
	// BuildPage.DPI's zero is: guessing how much of someone's image to throw
	// away is not a default anyone can audit.
	RecompressJPEG bool
	JPEGQuality    int
}

OptimizeOptions controls Optimize's behaviour (design spec section 4).

type PageGeometry 

type PageGeometry struct {
	RasterBox  [4]float64  `json:"raster_box"`
	PageBox    [4]float64  `json:"page_box"`
	ClipBox    *[4]float64 `json:"clip_box,omitempty"`
	RasterQuad *[8]float64 `json:"raster_quad,omitempty"`
}

PageGeometry is a page's raster and page boxes as measured at write time, each [llx lly urx ury] in PDF default user space: points, origin at the lower-left corner, y increasing upward (ISO 32000-1 8.3). This is NOT the [a b c d e f] affine matrix order PageProvenance.Placement uses -- a box is two corners, a placement is a transform, and conflating their orderings would silently misread one as the other.

Geometry is a pointer, not a value, and must stay one. nil means "this build recorded no geometry" -- what every pre-byb-b5.1 record deserializes to -- and must never be read as "the raster covers the page". A non-nil Geometry with a zero box is a real, if degenerate, measurement, and JSON can tell the two apart because a pointer either marshals as the object or is omitted entirely.

Go 1.24's `omitzero` is deliberately refused here even though go.mod (go 1.26.4) has it available: omitzero on a value-typed PageGeometry would make a measured all-zero box and a never-measured field serialize identically, destroying exactly the distinction the paragraph above depends on. The pointer is the backward-compatibility story; do not "tidy" it away. TestPageProvenanceGeometryZeroValueIsNotOmitted is the tripwire.

This exists for the case byb-b1.3 measured: a raster placed at its own resolution on a nominal page box, axis-aligned (so Placement is empty for it), that does not fill the box. 132 such pages were measured; one of them is 2384x3321 px at 302 DPI, which is 568.37 x 791.76 pt on a 612x792 MediaBox -- 92.84% of the box (92.87% by width), a 43.6 pt blank strip the raster does not cover. Without Geometry, a later re-processing run has no way to tell that page apart from a full-page scan.

Sourcing note for whoever writes this record: PageRaster.Bounds and PageRaster.Page are image.Rectangle, built through round() (inspect.go:92- 100) -- i.e. points ROUNDED TO INTEGERS. A writer populating PageGeometry must source the unrounded floats -- content.Placement.Box and pdfdoc Page.CropBox -- not PageRaster. 568.3708 is not 568.

CoversPage below applies coverTolerancePt (extract.go:49, == 1.0) over these exact floats. The live PageRaster.CoversPage applies NO tolerance of its own -- it is a plain image.Rectangle.In test on rounded integer rectangles; the 1.0pt allowance belongs to contains() (extract.go), a different function. That means the two CAN disagree, for a shortfall in the (0.5, 1.0] pt band: PageRaster.CoversPage sees no cover (rounding already ate the sub-pixel slack, containment fails outright), while this one's tolerance still calls it covered. That fork is deliberate -- documented here, not left to be discovered later.

RasterBox and PageBox are both mandatory whenever Geometry is non-nil: a writer that measures one must measure the other, and a decoder that gets a short JSON array for either silently zero-fills the missing elements rather than erroring (a wrongly-typed value, e.g. a string, does still error), which would misrepresent a partial record as a real (if degenerate) measurement. Byblos's own writer always sets both together. The same short-array hazard applies to ClipBox, once its presence bit says it is there at all: a short clip_box array still zero-fills rather than erroring, reading as a real (if degenerate) measured clip.

ClipBox is the third box, added by byb-b1.12 once content.Walk started honouring form /BBox and clip paths. It carries its OWN presence bit -- *[4]float64, not a bare [4]float64 -- for exactly the reason the paragraph above warns about: a value-typed box would zero-fill on every record written before it existed, and a zero box inside a non-nil Geometry already means "measured, and the measurement was degenerate" (the paragraph above this one), so it would make every byb-b5.1-era record lie that it measured a [0 0 0 0] clip.

It is populated only when a clip actually narrowed the placement below its unclipped raster box (extract.go compares Placement.Box against Placement.CTM.UnitSquareBox(), the placement's own extent before any clip) -- not whenever a clip merely happened to be in effect. A clip landing exactly on the raster's own edge (a form BBox sized to match its image, say) narrows nothing, so it records no ClipBox, the same as a page with no clip at all: this field answers "did a clip change what shows", not "was a clip present". A page with no narrowing clip in effect leaves ClipBox nil, the same way Geometry itself is nil for a page a build never measured.

There is no honest value for "unbounded, nothing clipped this" that fits inside a plain [4]float64 the way RasterBox and PageBox do -- any box-shaped value collides with a real, if degenerate, measurement (the paragraph above). That is the reason ClipBox needs its own presence bit rather than being inferred by comparing RasterBox to something. RasterQuad is the placement's true quadrilateral -- the unit square mapped through its CTM, in ring order (0,0) (1,0) (1,1) (0,1) -- carrying its own presence bit for the same reason ClipBox does: nil for a placement with no rotation or shear (axisAligned), where the quad is the same shape as RasterBox and not worth a second field for, and nil for every record written before byb-2mt. CoversPage below uses it to tell a rotation-inflated RasterBox from the page it only appears to cover.

The same short-array hazard the paragraph above documents for ClipBox applies here too: a short raster_quad array zero-fills the missing elements rather than erroring. A zero-filled tail leaves at least one pair of coincident vertices, tripping containsPoints' zero-length-edge guard (internal/content) rather than its zero-area one -- the truncated quad is not generally zero-area -- so CoversPage answers false on it rather than a wrong true; a partial record reads as "does not cover" rather than a fabricated measurement.

func (PageGeometry) CoversPage 

func (g PageGeometry) CoversPage() bool

CoversPage reports whether RasterBox fills PageBox, within coverTolerancePt, and -- when RasterQuad is present -- whether the placement's true quadrilateral does too.

It mirrors PageRaster.CoversPage's guard against a degenerate box: a PageBox with zero or negative width or height -- including one with its corners swapped -- is covered by nothing, because a plain containment check would otherwise answer true for an empty or inverted box (image.Rectangle.In does, for an empty receiver), which would make an unmeasured or degenerate record report itself as a full-page scan.

type PageInfo 

type PageInfo struct {
	Index       int
	Bounds      image.Rectangle
	Rotate      int
	Images      []ImageRef
	TextChars   int
	Diagnostics []Diagnostic
}

PageInfo describes one page.

Bounds is the page's CropBox, or its MediaBox when it declares no CropBox, in the same user-space convention as ImageRef.Bounds.

Rotate is the page's effective /Rotate (ISO 32000-1 7.7.3.3), resolved through page-tree inheritance rather than read off the leaf page dict alone -- pdfdoc.Page.Rotate already does that resolution (internal/pdfdoc/pdfdoc.go:402), this is one more field carrying it out. It exists for byb-yul: an edit list storing an ABSOLUTE target rotation needs the page's current effective rotation to render a starting state or compute that target, and neither is derivable without it.

Rotate is normalized into [0,360) with Go's sign-preserving %, not copied verbatim from the file: a declared /Rotate -90 reads back as 270, not -90. It is also not guaranteed to be a multiple of 90 -- a declared /Rotate 45 reads back as 45 unchanged, even though pdfcpu refuses to WRITE anything that is not 0, 90, 180 or 270 (measured for byb-yul.2; see validateIntegerEntry). A caller comparing Rotate against a file's declared value, or treating it as one of {0,90,180,270}, must account for both.

/Rotate is NOT applied to Bounds, and must not be assumed to be: it is a display attribute (see PositionedWord's identical note, stamp.go), and byb-wtp measured that both poppler and byblos report a /Rotate 90 Letter page as 612 x 792 -- neither rotates the box. A consumer that needs the page as actually displayed must apply Rotate to Bounds itself.

TextChars counts the bytes shown by the page's text operators, including text reached through Form XObjects. It is a born-digital signal, not a text extractor: it counts stored code units, not Unicode code points, and it does not decode fonts. Byblos never recognizes text (design spec section 3).

Diagnostics holds what byblos had to work around on this page, and is empty for the overwhelming majority of pages. A page carrying one was still read.

func Inspect 

func Inspect(r io.ReadSeeker) ([]PageInfo, error)

Inspect reports what every page of r contains. It does not render anything.

It cannot be cancelled. Use InspectContext when the caller has a deadline.

func InspectContext added in v0.3.0 

func InspectContext(ctx context.Context, r io.ReadSeeker) ([]PageInfo, error)

InspectContext is Inspect, cancellable at each page boundary (byb-xyn).

CANCELLATION LATENCY: MOST OF THE CALL, despite the per-page check, and the per-page check is not the reason. pdfdoc.Open runs before the loop and is a single uninterruptible pdfcpu parse of the whole document; the walk that follows is comparatively cheap. Measured over 120 pages of 300-dpi scans, the longest stretch between two context checks was 69% of the call, and all of it was the open. So the loop check bounds the WALK by one page, which is worth having, but it does not bound the CALL: a caller must still budget for a full pdfcpu parse. See context.go.

type PageProvenance 

type PageProvenance struct {
	Applied       []string        `json:"applied,omitempty"`
	Diverted      string          `json:"diverted,omitempty"`
	Placement     []float64       `json:"placement,omitempty"`
	DroppedAnnots int             `json:"dropped_annots,omitempty"`
	Geometry      *PageGeometry   `json:"geometry,omitempty"`
	Straightened  *PageStraighten `json:"straightened,omitempty"`
}

PageProvenance records what one page actually received.

Applied entries are capability names, optionally carrying a numeric parameter as a "-N" suffix, e.g. "downsample-150".

Diverted is the reason a page was not processed, e.g. "not-single-raster", and is empty when the page was handled normally. It is coarser than classify's full reason vocabulary (extract.go) — the record only needs to be precise enough to answer "would re-processing help, and with what capability?" — but not maximally coarse: byb-z8j split the codec case into "unsupported-codec-jbig2"/"-jpx"/"-tiff" so a decoder capability can be nominated for only the pages that want it, rather than every codec-diverted page. The full classify vocabulary still goes to the divert counters, not here.

A document written before byb-z8j, or one whose codec this build could not name, carries the older coarse "unsupported-codec" instead of one of the three above. That string must keep matching every decode-* rule indefinitely, because nothing can tell after the fact which codec it held.

divertClass in extract.go is the one place that maps the fine reason to the value stored here, and capabilityRules in upgrade.go is what matches on it. The two must agree; TestDivertClassCoversEveryReason is the tripwire.

Placement, when present, is the affine the page's raster was painted with, in PDF matrix order [a b c d e f] — the same six numbers as ImageRef.Placement. It is recorded, not applied: a scanner's deskew rotation stays in the placement matrix and the raster is kept as stored, because resampling a bilevel raster to straighten it would break the lossless guarantee this library exists for (byb-b1.2). It is empty for an axis-aligned placement, which is almost every page.

DroppedAnnots is how many annotations painted on this page and are not in the raster that was stored. Like Placement it is recorded, not applied: the appearance streams are still in the PDF, and drawing them into the raster would be a renderer (design spec section 2). What the record carries is the fact that the stored image is not the whole of what a reader would see, so a later pass can find those pages without re-measuring the archive. It is zero for almost every page — byb-b1.11 measured 6 in 18,610 extracted — and omitted when zero, so the ordinary record does not grow.

Geometry, when present, is the raster and page boxes measured at write time (byb-b5.1). Placement deliberately stays a separate top-level field rather than moving inside PageGeometry: lifting an old record's top-level placement into the container would have to invent boxes for it, and a Geometry carrying a [0 0 0 0] page box is a wrong geometry stated confidently. The two fields have two separate presence bits because they have two separate histories. It is nil for every record this build did not measure and for every record written before byb-b5.1 -- see PageGeometry's doc comment for why that nilness must survive. Straightened, when present, is the absolute rotation byblos has applied to this page's content (byb-16j.4). See PageStraighten's own doc comment, beside PageGeometry below, for why it is a pointer and why it is REPLACED rather than unioned into Applied.

type PageRaster 

type PageRaster struct {
	Image  image.Image
	Bounds image.Rectangle // where the raster lands
	Page   image.Rectangle // the page's CropBox

	// RasterQuad is the placement's true quadrilateral -- the unit square
	// mapped through its CTM, in ring order (0,0) (1,0) (1,1) (0,1), in the
	// same unrounded PDF-space points Bounds's corners were rounded from --
	// or nil when the placement carries no rotation or shear (axisAligned),
	// which makes it the same shape as Bounds and not worth a second field
	// for. CoversPage uses it to tell a rotation-inflated bounding box from
	// the page it only appears to cover (byb-2mt).
	RasterQuad *[8]float64

	// ObjNr is the PDF object number of the image XObject this raster came
	// from -- the same handle ImageRef.ObjNr carries from Inspect, and exactly
	// what ReplaceImages keys its substitution map on (substitute.go:28-35,
	// :127-131). Without it, a caller wanting to write back an encoding of
	// this raster had no exported way to name which object to replace and had
	// to guess.
	//
	// The guess is wrong on real documents. classify picks the LAST placement
	// in paint order as the page's one visible raster (extract.go: top :=
	// len(s.Images) - 1) because a stacked page can hide an earlier image behind
	// an opaque one painted over it -- 16,241 measured Internet Archive pages
	// do exactly this, two page-covering images at the identical CTM, the
	// second occluding the first. On those pages the first placement's object
	// number is the HIDDEN under-layer: substituting on it writes a document
	// that opens cleanly, looks unchanged, and is wrong, with no error raised
	// anywhere. ObjNr is the object classify actually chose, which fixes that
	// case; it is negative for an image XObject that is a direct object rather
	// than an indirect reference, exactly as ImageRef.ObjNr is (inspect.go:55-58),
	// since ReplaceImages has no cross-reference entry to write such a
	// substitution back to.
	//
	// ObjNr identifies the XObject, not the placement: one image can be painted
	// on several pages, and ReplaceImages substitutes it once, changing every
	// page that shares it (optimize.go:317 memoizes per object id for exactly
	// this reason).
	ObjNr int
	// DroppedAnnots counts the annotations on this page that paint and are not
	// in Image.
	//
	// Annotations live beside the content stream, not in it, so classify never
	// sees them and no raster ever contains them: a stamp, a signature or a
	// form field on a scanned page is shown by a viewer and absent here. That
	// is true of every extracted page and always has been — byb-b1.3 did not
	// introduce it, it only removed the coverage gate that had been catching
	// some of it by accident.
	//
	// Non-zero means the caller is holding an image that is missing ink a
	// reader would see. What to do about it is the caller's policy, which is
	// why this is reported rather than diverted: byb-b1.11 measured 6 such
	// pages in 18,610 extracted, and refusing 6 real pages to avoid 6
	// incomplete ones is the worse trade for an archive. Run byblos-annots for
	// the breakdown by subtype.
	//
	// Rendering the appearance streams into the raster would be a renderer,
	// which design spec section 2 puts out of scope.
	DroppedAnnots int

	// Bitonal reports that the source XObject DECLARED one bit per component,
	// or was an image mask. It is the same predicate Inspect surfaces as
	// ImageRef.Bitonal, carried to where the pixels are.
	//
	// Image is an *image.RGBA either way: decoding widens the samples, and
	// Bitonal is the only surviving record of what they were widened from.
	// Callers feeding DownsampleDeclaredBPC map true to declaredBPC 1.
	//
	// It is a DECLARATION, never a measurement. Do not replace it with a pixel
	// test: an 8-bpc source whose pixels happen to be pure black and white --
	// a bitonal TIFF widened to 8 bpc, common -- is indistinguishable from a
	// genuine 1-bpc source by pixel data alone, but Ghostscript keys
	// /MonoImageDownsampleType off the declared depth and downsamples it
	// bicubically. byb-plj measured a first attempt at that sniff 13 dB under
	// byblos's own oracle.
	//
	// Bitonal is chosen over an int BPC because an /ImageMask carries no
	// /BitsPerComponent at all and would have to report 0; see byb-xcx.
	Bitonal bool
}

PageRaster is a page's raster and where it sits on the page.

It exists because the raster is not always the whole page. byb-b1.3 measured 132 pages across 17 files whose raster is placed at its own resolution on a nominal Letter box — 2384x3321 pixels at 302 DPI is 568.37 x 791.76 points on a 612x792 MediaBox — and those pages extract. Returning a bare image.Image for them would quietly hand a caller 91.74% of a page as if it were the page.

Bounds and Page are both in PDF default user space: points, origin lower-left, y increasing upward, the same convention as PageInfo.Bounds. image.Rectangle is used only as a convenient integer rectangle — do not read it as screen coordinates.

The residual affine of a deskewed placement (byb-b1.2) is not here; it is on ImageRef.Placement, reached through Inspect. byb-b5.1 designs the stored form of both.

Since byb-b1.12, Bounds is where the raster is VISIBLE — the placement's own extent intersected with any W/W* clip path or form /BBox in effect — and can be smaller than the placement's own unclipped extent. Image is always the full, uncropped raster as stored (Byblos never crops pixel data): a clip narrows where Bounds says the raster's visible mark falls, not what Image contains. A caller that assumed Bounds was simply Image's own placement box must not, now that a clipped page reports the narrower, honest rectangle instead. ImageRef.Bounds carries the identical relationship to Placement.

Bounds is never EMPTY beside a returned Image (byb-62t). A placement narrower than a point on an axis is reported as the smallest integer rectangle that contains it rather than collapsing to nothing, so an empty Bounds and returned bytes cannot occur together. The cost is that such a Bounds overstates the extent by up to a point per edge, which is one more reason not to derive a scale or DPI from it.

func ExtractPageRaster 

func ExtractPageRaster(r io.ReadSeeker, page int) (*PageRaster, error)

ExtractPageRaster returns the single raster of the given 1-based page.

A page may reach that through more than one placement. Scanner pipelines routinely paint two page-covering images at the same matrix, the second hiding the first; that page has one visible raster and this returns it. See classify for the conditions, which are stricter than they look — a layer that is not provably an opaque cover diverts.

It returns an error wrapping ErrNotSingleRaster when the page is anything else, and ErrUnsupportedImageCodec when the raster's codec is not decodable.

Note on rotation: a page's /Rotate is a display attribute and does not affect content space, so a rotated page still extracts cleanly. The returned image is the raster as stored; applying /Rotate is the caller's business.

Orientation within content space is a narrower promise: the returned raster is the page as it reads, to within MaxSkewDeg. Scanner deskew lives inside that tolerance — a bulk scanner writes a fraction of a degree into the placement matrix and leaves the pixels raw — so those pages extract as stored, with the residual affine recorded in ImageRef.Placement and, at write time, in PageProvenance.Placement. Byblos does not straighten them: resampling a bilevel raster to take out a tenth of a degree would break the lossless promise on exactly the pages this library exists for.

Vector paint is judged the same way: by what the content stream proves, not by rendering it. A path painted before the raster and landing inside its placement box is behind an opaque image and cannot be seen, so the page is still one page-covering raster and still extracts. Paint the raster does not hide diverts. Nothing is filled, scan-converted or composited to decide this — only painting order and a bounding box.

Coverage is not a condition at all when the page has one raster and nothing else. Every other arm of classify has already established that no text, path, shading, inline image or unresolved XObject marks the page, so nothing can put ink outside the placement and that raster IS the page, at any coverage. byb-b1.3 measured 132 such pages; on every one of them the region outside the placement held zero content operators. What the caller gets told is the geometry — PageRaster.CoversPage — not a divert.

Past the tolerance the page diverts, and byb-b1.2 settled that this includes the two cases a caller could correct exactly, a quarter turn and a mirror. Recording the affine does weaken the older argument that the caller has no signal to correct them by. What it does not weaken is the reason: the returned image.Image is consumed — OCR, thumbnails, human review — by code that never reads provenance, and a sideways or mirrored raster is wrong there in a way a fraction of a degree is not. It cannot be cancelled. Use ExtractPageRasterContext when the caller has a deadline.

func ExtractPageRasterContext added in v0.3.0 

func ExtractPageRasterContext(ctx context.Context, r io.ReadSeeker, page int) (*PageRaster, error)

ExtractPageRasterContext is ExtractPageRaster, cancellable at the two boundaries this primitive has (byb-xyn).

CANCELLATION LATENCY: THE WHOLE OF ONE PAGE'S EXTRACTION. This is the weakest guarantee of the nine and the honest statement of it matters more than the check does. Extracting one page is open, walk, classify, decode, and none of those is interrupted: pdfcpu is not context-aware, and byblos' own content walk (internal/content.Walk) DOES check now, since byb-fem: it consults ctx at every token, so a content stream with millions of operators costs one token rather than the whole walk. Before that bead the walk was 95.4% of this call on such a stream and would not stop at all.

What stays uninterruptible is pdfcpu. A caller needing a tighter bound than "one page" must still budget for one page's worst case in the DECODE, which is bounded by byb-riy's resource budget rather than by this context. Measured over a 120-page document of ordinary scans, the longest stretch between two context checks was 55% of the call; that figure predates byb-fem and is unchanged by it, because on those documents the walk was never the dominant unit.

The checks are placed to keep a cancelled call OUT of the extraction telemetry: a call abandoned because the caller's deadline expired is not a failed extraction, and counting it as one would pollute the divert rate that design spec section 2's premise rests on with what are really worker timeouts. That is why the pdfdoc.Open error branch re-checks before counting -- a caller that closes the reader on cancel makes Open fail, and without the re-check every timed-out document would land in the counters as Attempted+Failed and be reported to the caller as a pdfcpu error rather than a cancellation.

func (PageRaster) CoversPage 

func (p PageRaster) CoversPage() bool

CoversPage reports whether the raster fills the page box. When it is false the caller is holding the scanned area, not the MediaBox, and the difference between Bounds and Page is the part of the page the scan does not cover.

Byblos does not pad the raster out to the page. Synthesising pixels that were never scanned is the same kind of lie as resampling a deskewed raster to straighten it, and on a bilevel JBIG2 scan it would mean decoding and re-encoding the very pages the lossless promise exists for. A zero Page is not covered by anything. image.Rectangle.In answers true for an empty receiver, so without this the zero PageRaster — what every error return hands back — would report itself as a full-page scan, and a caller that read CoversPage before err would never notice. The RasterQuad conjunct, when present, is checked against the rounded integer Page rectangle rather than an unrounded float -- PageRaster carries no unrounded page box to check it against instead. Rounding widens the (0.5, 1.0] pt fork PageGeometry.CoversPage's doc comment already records between this method and that one: on a fractional CropBox the effective slack here can reach 1.5pt rather than coverTolerancePt's 1.0. This is not hypothetical: corpus.ScanFractionalCropBox at a 0.4pt offset, straightened 0.2deg, is a reproduced instance (TestRecordExtractionCoversPageForkIsSafeDirectionOnly) -- false here, true on the equivalent PageGeometry. The direction stays the one worth having: this method's rounding only ever makes it MORE conservative, never less, so a caller trusting a live "false" here is not later contradicted by a "true" on the record.

type PageSource added in v0.3.0 

type PageSource = pdfdoc.PageSource

PageSource names one page of one document, and the rotation to give it. Aliased from the write seam for the reason EncodedImage is (build.go): internal packages are unreachable from Kleio, and a parallel type would be a second thing to keep in step.

type PageStraighten added in v0.3.0 

type PageStraighten struct {
	Deg float64 `json:"deg"`
}

PageStraighten is the correction byblos applied to this page's content, in StraightenSpec.Deg's signed convention (byb-16j.4): degrees, positive counter-clockwise in PDF default user space.

It is ABSOLUTE and is REPLACED, never accumulated -- which is why it cannot live in Applied. unionSorted UNIONS, so two corrections of one page would leave two contradictory angles in the record, and a union cannot express an absolute value.

It is a pointer for the reason PageGeometry is one (see that type's doc comment above): a 0.0 degree correction is a real measurement, and `omitzero` on a value type would make it serialize identically to "never straightened".

type PositionedWord 

type PositionedWord struct {
	Text   string
	Bounds image.Rectangle
}

PositionedWord is one recognized word and where it sits on the page.

Bounds is in POINTS, in PDF default user space (origin lower-left, y increasing upward -- the same convention as PageInfo.Bounds and PageRaster.Bounds), with the byb-b1.2 residual affine ALREADY APPLIED and /Rotate NOT applied. StampTextLayer places text at Bounds directly; it applies no transform of its own. Two consequences worth stating up front, because both are easy to get backwards:

  • Points, not the raster's pixel space, so that a caller reading PageRaster.Bounds and a caller writing PositionedWord.Bounds speak the same unit without a conversion at the call site -- two units in one API is a caller trap.
  • The affine must already be applied. At the measured max 1.09 degree scanner deskew that is ~11.6 pt of drift across a 612 pt page -- two line heights -- and text stamped without it drifts progressively relative to the scan, which is exactly what makes an invisible layer mis-select. StampTextLayer cannot check this; a caller that forgets produces silently drifting text.
  • /Rotate is NOT applied, and must not be: it is a display attribute that already rotates stamped text along with the page's own content (see extract.go's identical note on PageRaster), so applying it here would rotate it twice.

type Provenance 

type Provenance struct {
	Version      string           `json:"version"`
	Capabilities []string         `json:"capabilities"`
	ProcessedAt  time.Time        `json:"processed_at"`
	Pages        []PageProvenance `json:"pages"`

	// Optimized records which branch Optimize (byb-b5) took, because that
	// choice is a whole-document property, not a per-page one. Absent a
	// linearization request, Optimize returns min(input,
	// pdfcpu-rewritten-output): both are lossless structural rewrites of the
	// same document, so there is no quality tradeoff to record, only a
	// size-vs-linearization one. OptimizeOptions.Linearize:true suspends that
	// rule outright and always returns the linearized bytes, which are always
	// larger; see "rewritten-linearized" below. pdfcpu's rewrite
	// pass strips linearization rather than adding it (see
	// OptimizeOptions.Linearize's measurement), so the rewritten branch loses
	// whatever linearization the INPUT already had, silently -- the field
	// records that the rewrite ran, not that linearization was lost, but it is
	// the caller's only signal that this document did not simply pass
	// through untouched. The pass-through branch, by contrast, preserves
	// whatever linearization the input had, verbatim, because it does not
	// touch the bytes at all.
	//
	// "" is the zero value written by every build before byb-b5, AND what a
	// pass-through emits -- a pass-through returns the input's bytes
	// verbatim, so it cannot write anything into it without growing it past
	// that input and breaking Optimize's "never larger" guarantee. "" must
	// therefore be read as "not known to have been rewritten by Optimize",
	// never as "confirmed pass-through".
	//
	// "rewritten" means pdfcpu's optimized bytes were kept and the input was
	// not linearized, so nothing was lost to get them.
	//
	// "rewritten-delinearized" means the same, EXCEPT that the input carried a
	// linearization parameter dictionary and the output does not. That is a
	// real property traded for bytes, and it is the one case where the "no
	// quality tradeoff" claim above does not hold. It is recorded separately
	// rather than folded into "rewritten" because the two need different
	// answers: a caller that re-linearizes downstream can ignore the first and
	// must act on the second. Kleio's born-digital path exists precisely to
	// produce linearized files (its compress stage runs ocrmypdf in
	// linearize-only mode and does nothing else), so a later Optimize pass
	// quietly undoing that is a regression byblos must not hide.
	//
	// "rewritten-linearized" means Optimize was asked to linearize and did:
	// the output carries Annex F structure byblos itself wrote (byb-1y7,
	// internal/linearize). It is the only value that asserts a POSITIVE
	// property of the output rather than merely naming a branch, which is why
	// capabilityRules keys its upgrade rule on it (upgrade.go).
	//
	// "rewritten-delinearized" SURVIVES byb-1y7 and stays reachable; it is not
	// superseded by "rewritten-linearized". The two are mutually exclusive by
	// construction and describe different calls, not different eras: they are
	// the Linearize:false and Linearize:true branches over the same linearized
	// input. Optimize cannot infer that a caller who did not ask to linearize
	// wanted it, so the delinearizing branch still exists and must still say so.
	// Deleting the value would silence the exact regression it was added to
	// make visible, on the exact documents Kleio cares about.
	//
	// Reserve, do not yet emit, "passed-through".
	//
	// The converse also holds and is just as important: a pass-through run
	// carries whatever record the input already had forward untouched, so
	// "rewritten" on a document that has since had a pass-through Optimize
	// run on it is stale -- it describes the most recent REWRITE, not the
	// most recent call to Optimize. Nothing clears it, for the same reason
	// "" cannot be written by a pass-through: doing so would require an
	// in-band write to bytes that must stay byte-identical to the input.
	Optimized string `json:"optimized,omitempty"`
}

Provenance is the record Byblos writes into a processed PDF, as JSON under a custom Info-dictionary key. The PDF is authoritative; any mirror of these fields in a caller's database is a cache.

func ReadProvenance 

func ReadProvenance(r io.ReadSeeker) (*Provenance, error)

ReadProvenance reads back the record WriteProvenance stored under provenanceKey, via pdfdoc.ReadProperties (pdfcpu's api.Properties underneath). It returns (nil, nil) for a document no Byblos build has processed -- absence is not an error, and UpgradeCandidates already treats a nil *Provenance as "every capability is a candidate", so callers need no special case for a never-seen file.

It cannot be cancelled. Use ReadProvenanceContext when the caller has a deadline.

func ReadProvenanceContext added in v0.3.0 

func ReadProvenanceContext(ctx context.Context, r io.ReadSeeker) (*Provenance, error)

ReadProvenanceContext is ReadProvenance, cancellable only before its work begins (byb-xyn).

CANCELLATION LATENCY: A WHOLE PDFCPU READ. Reading the Info dictionary means pdfcpu parsing the document, which is not interruptible and has no byblos-owned loop inside it, so the context is consulted once on entry and not again. It is the cheapest of the nine, but "cheapest" is a property of the document, not a bound this context provides. See context.go.

func RecordExtraction 

func RecordExtraction(r io.ReadSeeker) (Provenance, error)

RecordExtraction runs extraction over every page of r and returns the record of what happened, ready for WriteProvenance. Pages is exactly one entry per page, in page order, so index i describes page i+1 -- PageProvenance carries no page number of its own.

It is the only producer of PageGeometry: the unrounded placement and CropBox floats live inside the extraction path and nothing else can see them.

A page Byblos could not READ at all aborts the whole call. There is no "failed" value in PageProvenance's vocabulary, and the two ways to carry on are both worse: skipping the page shifts every later page's index, and appending a zero PageProvenance is indistinguishable to any reader from a page that was handled and had nothing applied. A caller that needs per-page resilience has ExtractPageRaster.

Capabilities claims only "extract-raster", for the reason Optimize's fresh record claims nothing (optimize.go): this call did not build, encode, stamp or linearize anything, and a record saying otherwise would suppress those capabilities in UpgradeCandidates. A caller that goes on to do more appends to both Capabilities and the pages' Applied.

It reads back whatever record r already carries, the same way Optimize does (optimize.go), and merges into it rather than overwriting it: Optimized is preserved verbatim (extraction is not a rewrite and has no opinion on it), Capabilities is the union of the old record's and this call's, and each page's Applied is the union of the old page's and this call's. Without this, running RecordExtraction over a document Optimize (or an earlier RecordExtraction) already processed would silently erase real capabilities -- including "rewritten-linearized", which a later re-run without Linearize:true would then falsely nominate for reprocessing. A record under provenanceKey that fails to parse as JSON is treated as no record at all, matching ReadProvenance/Optimize's errCorruptProvenance handling.

It cannot be cancelled. Use RecordExtractionContext when the caller has a deadline.

func RecordExtractionContext added in v0.3.0 

func RecordExtractionContext(ctx context.Context, r io.ReadSeeker) (Provenance, error)

RecordExtractionContext is RecordExtraction, cancellable at each page boundary (byb-xyn).

This is the primitive the context convention is really for. It runs extraction over EVERY page, so it is both the most expensive entry point in the package and the one where a per-page check buys the most: on a long document it is the difference between a worker returning in one page's time and a worker held until the SQS visibility timeout redelivers the same file.

CANCELLATION LATENCY: one page's extraction -- open, walk, classify, decode -- which is the same indivisible unit ExtractPageRasterContext documents, bounded by byb-riy's decoder budget and not by this context. The initial ReadProvenance and pdfdoc.Open are single uninterruptible pdfcpu passes; a context cancelled during those is not noticed until the page loop is reached.

WHAT ONE PAGE COSTS, measured, because this is the number a caller has to budget for and the reassuring one is misleading: 

ordinary 300-dpi JPEG scans, 120 pages:  12 ms   (3.2% of the call)
hostile JBIG2 admitted by byb-riy:        seconds per page

The second number is the contract. byb-riy's budget admits a page of 67,092,481 pixels, and decoding one costs seconds, so "cancellation stops this within a page" is only a useful promise to a caller who has budgeted SECONDS for that page -- not the milliseconds an ordinary document suggests. A kleio worker whose SQS visibility timeout is shorter than that will still be redelivered onto the same document, which is the exact failure this bead exists to prevent, so the timeout has to be set against the hostile number. See TestCancellationLatencyOnAHostilePage and context.go.

type RasterRefusal added in v0.4.0 

type RasterRefusal struct {
	// Page is the 1-based page number that was refused.
	Page int
	// Reason is the fine-grained refusal reason.
	Reason string
	// Class is the coarse divert class recorded in PageProvenance.Diverted.
	Class string
	// contains filtered or unexported fields
}

RasterRefusal names, in a type rather than in prose, why ExtractPageRaster refused a page (gap G4).

IT CHANGES NOTHING ABOUT THE ERROR'S TEXT OR ITS CHAIN. Error returns the message the wrapped chain always produced, and Unwrap keeps ErrNotSingleRaster, ErrUnsupportedImageCodec and any *NotImplemented reachable by errors.Is and errors.As exactly as before. It only adds a way to read the reason WITHOUT parsing that message: 

var ref *byblos.RasterRefusal
if errors.As(err, &ref) { ... ref.Reason ... }

Kleio's preview handler forwarded err.Error() verbatim into a 422 body because the fine reason existed only as a text suffix, which made every wording change here a silent break of the tile a user sees.

REASON IS FINE-GRAINED AND CLASS IS COARSE, and they are not the same vocabulary. Reason is what classify (or the codec switch) actually decided: "vector-paint", "multiple-images", "mrc-layers", "unsupported-codec-jbig2", and the rest. Class is what divertClass maps that to, which is what PageProvenance.Diverted records and what UpgradeCandidates matches on — it collapses every unrecognised reason to "not-single-raster". Branch on Class to decide whether a future byblos could recover the page; show Reason to a human. Neither is a closed set: byb-z8j owns the vocabulary and adds to it, so treat an unknown value as "some other refusal" rather than failing.

func (*RasterRefusal) Error added in v0.4.0 

func (e *RasterRefusal) Error() string

Error returns the wrapped chain's message, unchanged from what this call returned before RasterRefusal existed.

A HAND-BUILT RasterRefusal HAS NO CHAIN AND MUST NOT PANIC. The wrapped error is unexported, so a consumer writing a test for its own refusal handling can only construct the exported fields; that is a legitimate use and it would otherwise nil-panic here. Such a value reports its reason and nothing else, and errors.Is finds no sentinel on it -- which is honest, because there is none.

func (*RasterRefusal) Unwrap added in v0.4.0 

func (e *RasterRefusal) Unwrap() error

Unwrap exposes the sentinel and any cause beneath it, so errors.Is on ErrNotSingleRaster and ErrUnsupportedImageCodec still answers. It is nil for a hand-built value; see Error.

type Severity added in v0.3.0 

type Severity uint8

Severity says how much of a PageInfo to believe when byblos had to work around something on the page.

It is poppler's distinction, taken from its Error.h: errSyntaxWarning is "PDF syntax error which can be worked around; output will probably be correct", and errSyntaxError is the same sentence ending "output will probably be incorrect". Note what BOTH halves say -- a syntax problem is always worked around. Neither category removes a page, and neither ends a document. 

const (
	// SeverityWarning: byblos worked around the problem and the page's numbers
	// are probably right.
	SeverityWarning Severity = iota
	// SeverityError: byblos worked around the problem and the page's numbers
	// are probably WRONG, and wrong LOW. A content stream that stops early
	// paints fewer images and shows less text than the page really holds, so a
	// scanned page can look like an empty born-digital one -- which is the
	// exact classification byblos exists to get right. A caller must not read
	// TextChars or Images off such a page without accounting for this.
	SeverityError
)

func (Severity) String added in v0.3.0 

func (s Severity) String() string

type StraightenSpec added in v0.3.0 

type StraightenSpec = pdfdoc.StraightenSpec

StraightenSpec is a lossless rotation of one page's content. Aliased from the write seam for the same reason PageSource is (byb-16j.4); see its own doc comment in internal/pdfdoc for the sign convention and the absolute-not-delta contract BuildFromPagesContext enforces below.

type TextLayer 

type TextLayer struct {
	Pages [][]PositionedWord
}

TextLayer is the recognized text for one document, one page at a time. Pages[i] is page i+1's words; a page with no recognized text is an empty slice, and StampTextLayer does not touch such a page at all.

Source Files

View all Source files

Directories

Path Synopsis
cmd
byblos-annots command
Command byblos-annots measures how often an annotation that paints is lost because ExtractPageRaster returned the page's raster without it.
 Command byblos-annots measures how often an annotation that paints is lost because ExtractPageRaster returned the page's raster without it.
byblos-corpus command
Command byblos-corpus writes the generated test corpus to a directory.
 Command byblos-corpus writes the generated test corpus to a directory.
byblos-ctm-census command
This file holds the CTM census's own arithmetic (shearDegrees, skewDegrees, placementDeg) plus a byte-for-byte reimplementation of classify (extract.go) and the private helpers it calls, so this tool can compute classify's divert reason for every page without decoding a single image -- byb-06n's recon (part C) established that every gate up to and including placementReason is decidable from one content.Walk, so this is legitimate and not a guess.
 This file holds the CTM census's own arithmetic (shearDegrees, skewDegrees, placementDeg) plus a byte-for-byte reimplementation of classify (extract.go) and the private helpers it calls, so this tool can compute classify's divert reason for every page without decoding a single image -- byb-06n's recon (part C) established that every gate up to and including placementReason is decidable from one content.Walk, so this is legitimate and not a guess.
byblos-divert command
Command byblos-divert measures how often ExtractPageRaster declines a page.
 Command byblos-divert measures how often ExtractPageRaster declines a page.
byblos-encrypt-census command
Command byblos-encrypt-census measures the population that a refusal at pdfdoc.Open (option (a)) would break, not merely the population that carries an /Encrypt dictionary.
 Command byblos-encrypt-census measures the population that a refusal at pdfdoc.Open (option (a)) would break, not merely the population that carries an /Encrypt dictionary.
byblos-fonts command
Command byblos-fonts classifies every /Type /Font dict in a corpus by what a code-to-Unicode decoder would have to do with it.
 Command byblos-fonts classifies every /Type /Font dict in a corpus by what a code-to-Unicode decoder would have to do with it.
internal
content
Package content lexes and walks PDF content streams.
 Package content lexes and walks PDF content streams.
corpus
Package corpus builds the Byblos test corpus in memory.
 Package corpus builds the Byblos test corpus in memory.
glyphless
Package glyphless is a minimal TrueType font whose glyphs carry no outlines: every character prints nothing, but still occupies exactly as much horizontal space as a normal proportional glyph would.
 Package glyphless is a minimal TrueType font whose glyphs carry no outlines: every character prints nothing, but still occupies exactly as much horizontal space as a normal proportional glyph would.
jbig2
Package jbig2 implements a lossless JBIG2 codec: generic regions in both directions, and symbol mode for reading.
 Package jbig2 implements a lossless JBIG2 codec: generic regions in both directions, and symbol mode for reading.
linearize
Package linearize implements ISO 32000-1:2008 Annex F linearization: the object partition, the two cross-reference sections, the linearization parameter dictionary and the primary hint stream.
 Package linearize implements ISO 32000-1:2008 Annex F linearization: the object partition, the two cross-reference sections, the linearization parameter dictionary and the primary hint stream.
pdfbuild
Package pdfbuild constructs a PDF from already-encoded page images.
 Package pdfbuild constructs a PDF from already-encoded page images.
pdfdoc
Package pdfdoc is the only package in Byblos that imports pdfcpu.
 Package pdfdoc is the only package in Byblos that imports pdfcpu.
sample
Package sample walks a directory of PDF documents and reports the POPULATION that a rate measured over that directory is a rate of.
 Package sample walks a directory of PDF documents and reports the POPULATION that a rate measured over that directory is a rate of.
skew
Package skew estimates the angle of the text lines INSIDE a raster.
 Package skew estimates the angle of the text lines INSIDE a raster.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.