zrecipe

README

zrecipe

zrecipe is a Go library that makes compressed files reproducible from their uncompressed content: given a gzip or zstd file it decompresses once, searches a set of real-world compression engines and parameter grids for the combination that recreates the exact same compressed bytes, and records that combination as a small JSON document alongside blake3 digests of both streams; given the uncompressed content and that document it rebuilds the original compressed file bit for bit. The intended use is storage systems that want to keep only the uncompressed content — which deduplicates and chunks well — while still being able to hand back the original compressed artifact on demand.

Install and build

zrecipe ships as a Nix flake that provides the Go toolchain and the C libraries the cgo engines link against (system zlib and zstd, plus gzip and pigz for the wild-fixture tests). Enter the shell and build from there:

nix develop
go build ./cmd/zrecipe

Everything below assumes commands run inside nix develop (or nix develop --command ... from outside it). The library also builds without cgo, in which case only the five pure-Go engines, gnu-gzip, go-flate, klauspost-flate, pgzip and klauspost-zstd, are available and zlib, pigz and libzstd are absent from DefaultEngines().

Library usage

The root package is github.com/draganm/zrecipe. Analyze takes an io.ReadSeeker over a compressed file, decompresses it once while hashing both the compressed and uncompressed bytes, and returns *Params, which records the engine, its version, and the parameters that reproduce the file. Recompress takes those *Params and the uncompressed content and streams the compressed file back out, verifying both the input and the output against the digests in Params.

ctx := context.Background()

in, err := os.Open("archive.tar.gz")
// ...
defer in.Close()

uncompressed, err := os.Create("archive.tar")
// ...
defer uncompressed.Close()

// Analyze decompresses once, hashes both streams, and finds an engine and
// parameter set that reproduces archive.tar.gz byte for byte from
// archive.tar. Store archive.tar and the params JSON; the .gz can now be
// discarded.
params, err := zrecipe.Analyze(ctx, in, &zrecipe.Options{Uncompressed: uncompressed})
// ...

paramsFile, err := os.Create("params.json")
// ...
err = params.Write(paramsFile)
paramsFile.Close()

// Later, rebuild the original .gz from archive.tar and params.json.
src, err := os.Open("archive.tar")
// ...
defer src.Close()

out, err := os.Create("rebuilt.tar.gz")
// ...
defer out.Close()

err = zrecipe.Recompress(ctx, params, src, out, nil)

For an uncompressed input Analyze returns Params with Format set to none, both digests equal, and no engine, so callers have one code path for every input. Options.Parallelism (default runtime.NumCPU()) evaluates candidates concurrently, but only takes effect when the reader also implements io.ReaderAt (as *os.File does); otherwise the search runs sequentially over the single io.ReadSeeker. Large inputs spool the decompressed content to a temp file once they exceed Options.MaxInMemory (default 64 MiB), so memory stays bounded regardless of input size — TestLargeInput in this package exercises a 2 GiB input and stays well under that bound (see Testing below).

Analyze recompresses the input once. It is Start, Confirm and Close, which a caller that also wants the decompressed content can drive itself:

a, err := zrecipe.Start(ctx, in, nil) // pass one, then the engine search
// ...
defer a.Close()

// Confirm reads the content once, writes it to the tee (nil to skip), and
// rebuilds the input from it to prove the parameters through the same code
// Recompress uses. It returns the same Params Analyze would.
params, err := a.Confirm(ctx, uncompressed)

Start decompresses the input once and narrows the candidates to one by feeding them a growing prefix and dropping each at its first divergent byte; Confirm then reproduces the input from the content in a single pass while handing that content to the tee, so a caller that decomposes or stores the content pays for one decompression and one recompression, not two of each. Options.Uncompressed is the tee Analyze passes to Confirm, so it receives the content during the confirming pass, not the first pass: an input that is not reproducible writes nothing to it, and one whose confirmation fails writes a prefix.

Options.VerifyLimit bounds the recompression on large inputs: once a candidate has reproduced that many bytes of the compressed input, the search and the confirming pass accept it without running it to the end (the confirming pass still streams the whole content to its tee). A candidate that matches that far and diverges later is rare, and Recompress checks the output digest, so such a divergence still surfaces at rebuild time rather than silently. The default, zero, verifies every byte.

The search advances every candidate in lockstep over growing windows of the content and drops each one at its first divergent byte; a lone survivor is settled on once it has matched a margin past its last competitor's death, and the confirming pass then checks it over the whole input. Every engine's candidates take part at once, ordered by tier (the likely ones first) but not separated by it, since a likely candidate may agree with an unlikely one over a long prefix and only the lockstep can tell them apart. A candidate that has shown no output by search.UntestedLimit (33 MiB of content) is dropped there, and one whose engine says it buffers more than that before its first write (engine.ZstdBuffering) is not started on a longer input: without that bound the lockstep would run every such candidate over the whole content, at its own level, before it could settle or give up. The candidates this leaves out are libzstd's job-based path at the ultra levels 20 to 22 and in long distance matching mode, whose first job is 64 MiB to 512 MiB, on inputs longer than the limit, and klauspost's one-shot paths on such inputs.

ReadParams decodes and validates a Params document read back from JSON, rejecting an unknown schema version with ErrParamsVersion and an internally inconsistent document — malformed digest hex, or a gzip/zstd section that does not match format — with ErrInvalidParams.

CLI usage

cmd/zrecipe wraps the library in four subcommands:

# Print gzip, zstd or none for a file.
zrecipe detect archive.tar.gz

# Find parameters that reproduce a compressed file. Params JSON goes to
# stdout by default, or to --params; --uncompressed additionally writes the
# decompressed content. Flags must come before the positional <file>: this
# is a urfave/cli v2 limitation (it stops parsing flags at the first
# positional argument), not a choice made by this tool.
zrecipe analyze --params params.json --uncompressed archive.tar archive.tar.gz

# Rebuild the compressed file from params and the uncompressed content.
# Writes to a temp file next to <out> and renames on success, so a failed
# run never leaves a partial file at the destination. Same flags-first rule.
zrecipe recompress --params params.json archive.tar rebuilt.tar.gz

# List the engines compiled into this binary, with their format and version.
zrecipe engines

Exit status is 0 on success and 1 on any error, with the error printed to stderr. recompress also accepts --allow-version-mismatch to try an engine whose recorded version differs from the one compiled into the binary; analyze accepts --parallelism and --temp-dir to override the defaults described above.

Engines

Name	Format	Binding	Version source
gnu-gzip	gzip	pure-Go port of GNU gzip's compressor	the ported release, 1.14
zlib	gzip	cgo, system zlib	zlibVersion()
pigz	gzip	cgo, system zlib driven the way pigz does	2.8+zlib plus zlibVersion()
libzstd	zstd	cgo, system libzstd	ZSTD_versionString()
go-flate	gzip	stdlib compress/flate	runtime.Version()
klauspost-flate	gzip	github.com/klauspost/compress/flate	module version from debug.ReadBuildInfo()
pgzip	gzip	pure-Go port of klauspost/pgzip's writer over a copy of klauspost/compress/flate v1.11.3	the ported releases, 1.2.6+klauspost-compress1.11.3
klauspost-zstd	zstd	github.com/klauspost/compress/zstd	module version from debug.ReadBuildInfo()

gnu-gzip is a line-by-line port of the compressor in GNU gzip 1.14 (deflate.c, trees.c and bits.c) and produces exactly the bytes the gzip program writes when it compresses a regular file, at every level and with --rsyncable. It is listed first in DefaultEngines() because GNU gzip is the most common producer of gzip files in the wild, and because a pure-Go engine keeps the resulting Params usable from a binary built without cgo. The ported files are GPL-licensed; see License below.

pigz reproduces pigz, the parallel gzip. pigz compresses with zlib but cuts the input into blocks (128 KiB by default), restarts the compressor on every block primed with the previous 32 KiB, and byte-aligns each block with empty deflate blocks, so a plain zlib stream matches its output only for input that fits in one block. The engine drives the system zlib exactly as pigz 2.8 does, covering both of pigz's code paths (-p 1 keeps one stream and flushes at the same boundaries; more threads reset per block), --independent, --rsyncable, -b block sizes, and the -H/-U strategies. zopfli (-11) is not covered. Like pigz, the engine compresses the blocks of the parallel path concurrently, on Engine.Workers zlib streams (GOMAXPROCS by default); the output does not depend on the count. Its version string names both the pigz release ported and the zlib linked, since the output depends on both.

pgzip reproduces klauspost/pgzip, the parallel gzip written in Go that umoci compresses layers with, and through umoci rockcraft: every Canonical rock on Docker Hub, ubuntu included. pgzip cuts the input into 1 MiB blocks, compresses each with klauspost/compress/flate primed with the last 16 KiB of the block before it, sync-flushes after every block, and closes the stream after the last block, which it compresses even when empty. Those bytes depend on the klauspost/compress generation as much as on the scheme, and umoci pins v1.11.3, whose encoder differs from the v1.20.0 the klauspost-flate engine links. Go cannot load two versions of one module, so the engine drives a verbatim copy of that release's flate package (engine/pgzip/flate, BSD-licensed) and its version string names both the pgzip release ported and the klauspost/compress release copied. block_size covers callers of SetConcurrency. pgzip files are easy to recognise: OS byte 255 and, unless the producer set a modification time, an mtime of 0x886e0900 (the zero time.Time truncated to 32 bits), which the header captured in Params carries verbatim.

klauspost-zstd reproduces github.com/klauspost/compress/zstd, which compresses the zstd layers that skopeo, podman, buildah (all through containers/image), BuildKit and nix2container push. Its head parameter covers a shape containers/image leaves on every layer it compresses from an uncompressed source: it writes the 8 bytes it peeked at to detect the source compression, then streams the rest through Encoder.ReadFrom, which first flushes what Write buffered as a block of its own. The frame therefore opens with a raw 8-byte block, and the engine writes head bytes, flushes once and streams on. Such a block is recognised from the frame itself (a raw or RLE first block, not the last one, shorter than the producer's block), and since no libzstd path emits one, the libzstd engine offers no candidates for those frames at all. A head that compresses into a compressed block, or one longer than a block, is not recognised.

libzstd covers the zstd CLI and everything else linked against libzstd. Its candidates are the levels whose window, for the pledged size or for an unknown one, is the one the frame header declares: libzstd writes the window descriptor from its parameters alone, so any other level would fail at byte six, and the engine does not offer it. The first tier is the single-thread path at those levels; the second adds the job-based path (workers 1: libzstd's multithreaded compressor, which the zstd CLI uses even with one thread, and whose output is the same for any thread count), the ultra and negative levels, long distance matching for windows of 128 MiB and up, and the other levels with the header's window set explicitly. libzstd's job-based compressor shows nothing until its first job is full, four times its window (32 MiB at level 19), and stopping it before then still costs that job, since freeing it waits for the job in flight. The engine therefore compresses the first job itself with libzstd's buffer-less API, chunk by chunk exactly as libzstd's job does, so a wrong candidate is dropped after one 512 KiB chunk; if the input outlasts the job, the rest goes to libzstd's own compressor, the first job replayed into it and its bytes for that job checked against the ones already written. Jobs over 32 MiB (the ultra levels, wide windows) and long distance matching go to libzstd's compressor from the start, and the engine reports their job through engine.ZstdBuffering so the search leaves them out when the job would not fit its limit. A first block flushed before it was full rules libzstd out altogether (see klauspost-zstd).

The three cgo engines are present only in binaries built with cgo enabled. Against the versions pinned by this repository's flake, zlib reports 1.3.2 and libzstd reports 1.5.7; klauspost-flate and klauspost-zstd both report the github.com/klauspost/compress module version, v1.20.0 at the time of writing. Build info carries no module version inside a go test binary, so the klauspost engines report (devel) there; Recompress treats that like any other version string. go-flate's version is whatever Go toolchain built the binary. gnu-gzip reports the GNU gzip release it ports; that string only changes if a change to the port alters its output.

Version granularity. go-flate is versioned by the full Go release (runtime.Version(), e.g. go1.22.3), not just the major/minor line, so even a Go patch release changes engine_version and makes Recompress refuse a Params document made with a different patch release unless AllowVersionMismatch (--allow-version-mismatch on the CLI) is set — the digest check still decides whether the attempt actually reproduces the file. The klauspost engines report (devel) not only from a go test binary but from any build where debug.ReadBuildInfo() cannot resolve a concrete version for the module, such as a Go workspace (go.work) build that replaces it with a local checkout.

Different implementations produce different bytes at the same nominal level, and libzstd's output changes between releases while zlib's has been stable for years, so a binary reproduces zstd files only against the libzstd version it is linked against — see Limitations below.

Write-size independence. An engine's output must depend only on the content and the parameters, never on how the content was split across Write calls: Analyze verifies a candidate from its spool while Recompress rebuilds from whatever reader the caller passes. Most engines have this property by construction. zlib at level 0 does not — zlib sizes each stored block by the input one deflate() call can see, so 32 KiB writes gave 32768-byte blocks where one large write gave maximal 65535-byte ones — and neither does libzstd with end_with_data, which hands whatever it holds back to ZSTD_e_end. Both engines therefore batch their input internally (64 KiB for zlib, libzstd's own stream input size for libzstd), and on top of that the search and Recompress both feed every engine through engine.Feed, in fixed 32 KiB writes, so a candidate is always verified under exactly the write shape later used to rebuild the file.

Params JSON

Params is versioned (currently 1), carries blake3 digests and sizes of both the compressed and uncompressed content, the winning engine's name and version, and either a gzip or a zstd section with that engine's parameters. Here is what zrecipe analyze prints for a small file gzipped by the system's gzip -6, reproduced by the zlib engine:

{
  "version": 1,
  "format": "gzip",
  "compressed": {
    "blake3": "7ecf3d688fe675af5b076c00c4dd7b63a4a3af45df12bc3232d35ba842eb84e2",
    "size": 114
  },
  "uncompressed": {
    "blake3": "56f278db158c5413742c7209762b6dd4c0b776a6246cd29397f54e18f277b81c",
    "size": 8800
  },
  "engine": "zlib",
  "engine_version": "1.3.2",
  "gzip": {
    "header_b64": "H4sICNlmmWoAA3NhbXBsZS50eHQA",
    "level": 6,
    "strategy": "default",
    "window_bits": 15,
    "mem_level": 8
  }
}

header_b64 is the gzip header captured verbatim, byte 0 up to the first byte of the deflate stream, so mtime, filename, OS byte, XFL and every optional field come back exact without needing to be parsed individually. strategy is meaningful only for the zlib engine (default, filtered, huffman_only, rle or fixed); the pure-Go engines omit it because they have no equivalent knob. rsyncable records --rsyncable for gnu-gzip and pigz. block_size (in KiB) is the block the input is cut into for pigz (-b, default 128) and pgzip (SetConcurrency, default 1024); independent and single_thread are pigz only: its -i and the -p 1 code path.

The same file compressed with the zstd CLI instead produces a zstd section:

{
  "version": 1,
  "format": "zstd",
  "compressed": {
    "blake3": "45cd33d739d17e4190228332f616794eca0fb92be4445f8eeb7851ac3694a9d4",
    "size": 67
  },
  "uncompressed": {
    "blake3": "56f278db158c5413742c7209762b6dd4c0b776a6246cd29397f54e18f277b81c",
    "size": 8800
  },
  "engine": "libzstd",
  "engine_version": "1.5.7",
  "zstd": {
    "level": 3,
    "checksum": true,
    "content_size": true,
    "pledged_size": true,
    "single_segment": true,
    "workers": 0,
    "end_with_data": true
  }
}

window_log, long, encode_all and head are omitted here because they are zero-valued or false; they, along with the always-present workers, cover long-distance matching mode, an explicit window size, klauspost's one-shot EncodeAll encoding path, and the prefix klauspost's streaming writer flushes before it streams (see the klauspost-zstd engine above). end_with_data is an addition beyond the original design. It does not record what the producer did — zrecipe has no way to observe that — but what the frame itself shows: whether the last block is non-empty (true) or an explicit empty block trails the data (false). A known-size producer such as the zstd CLI reading a file typically yields the non-empty shape by passing its final chunk of input together with the end directive; a producer that only learns end-of-input later must call end separately once there is nothing left to flush, leaving that trailing empty block. For input that is not aligned to zstd's block size, though, there is always unflushed data buffered when end is signalled, so the last block comes out non-empty regardless of which way the producer called it — the two directives converge on the same frame, and end_with_data reads true either way.

Limitations

A binary reproduces zstd files made only by the exact libzstd version it is linked against; that version is recorded in engine_version and, absent AllowVersionMismatch (--allow-version-mismatch on the CLI), Recompress refuses to even try an engine whose current version differs from the recorded one. Passing that flag makes it try anyway — the digest check still decides whether the attempt actually succeeded.

Analyze recognises but does not handle four kinds of input, all returned as ErrUnsupported: multi-member gzip files, multi-frame zstd files, zstd skippable frames, and zstd frames built against a dictionary.

Analyze's zstd decoder is bounded only by the window size declared in the frame header, up to zstd's own maximum of 2 GiB (WithDecoderMaxWindow). An untrusted zstd input can therefore make decompression demand up to that much memory for its window alone; callers decompressing files from untrusted sources should account for this.

When the klauspost-zstd engine produces a single-segment frame (one without a window descriptor) or uses its one-shot EncodeAll path, it must buffer the entire uncompressed content in memory first; both are capped at 1 GiB of uncompressed size, above which those candidates are skipped rather than exhausting memory.

Recompress does not roll back bytes already written to its output writer on error, since arbitrary io.Writers are not generally seekable or truncatable; the CLI works around this itself by writing to a temporary file next to the destination and renaming it into place only once Recompress succeeds, and library callers that need the same safety should do likewise.

Finally, not every real-world compressor is reproducible in this version. A spike against this repository's flake-pinned tools (zstd CLI 1.5.7, GNU gzip 1.14, pigz 2.8) found that the zstd CLI reproduces in all 96 tested variants — levels 1 through 22, --fast, -T0/-T1/-T4, --single-thread, --no-check, --long, from both stdin and a file. GNU gzip reproduces at every level, with and without --rsyncable, through the gnu-gzip engine; zlib alone matched it only at levels 8 and 9 on plain text, because gzip ends deflate blocks with its own heuristic. One caveat: gnu-gzip models gzip reading a regular file, where every read(2) returns the full amount asked for. gzip reading from a pipe can get short reads, which shift the point where its window slides; that changes the output only when the last few hundred bytes of input fall in the region where gzip stops matching, or when a final match runs past the end of input into stale window bytes, and in those cases gzip's own pipe output is timing-dependent. pigz is reproduced through the pigz engine at every level, on both of its code paths, with -i, -R and -b; zlib alone matched only input that fits in a single pigz block. Files produced by zlib itself, Go's compress/gzip, and both klauspost engines are reproduced. klauspost/pgzip over klauspost/compress v1.11.3, the pair umoci and rockcraft ship, is reproduced through the pgzip engine at every level, block size and thread count; pgzip over other klauspost generations is not, since the encoder changed in v1.11.13 and again later, and those generations would each need their own copy of the flate package.

Testing

Besides the unit, round-trip and wild-fixture tests that run on every go test, TestLargeInput in this package streams a synthetic 2 GiB gzip file through Analyze and Recompress to confirm the spool and the search keep memory bounded on inputs far larger than MaxInMemory. It is gated behind an environment variable because it takes minutes and several gigabytes of scratch disk:

ZRECIPE_LARGE=1 go test . -run LargeInput -v -timeout 30m

License

zrecipe is licensed under the GNU Affero General Public License, version 3 or later; see LICENSE. The engine/gnugzip package contains code ported from GNU gzip, which is licensed under the GNU General Public License, version 3 or later; those files keep their upstream copyright notices and engine/gnugzip/COPYING holds that license. Section 13 of each license permits combining the two: the ported files remain under the GPL, the rest of the project is under the AGPL, and the AGPL's network-interaction terms apply to the AGPL-covered parts. The engine/pgzip/flate package is a copy of github.com/klauspost/compress's flate package at v1.11.3 under its BSD-3-Clause license, kept in engine/pgzip/flate/LICENSE. Every other dependency is under a permissive license (BSD-3-Clause, MIT or the zlib license) that is compatible with both.

In practice this means a program that imports zrecipe must itself be distributed under AGPL-compatible terms, and one that offers it as a network service must offer its source to the users of that service.

Design and plan

The full design, including the search algorithm, the candidate grids for each engine, and the spike results that shaped the limitations above, lives at docs/superpowers/specs/2026-09-03-zrecipe-design.md, with the GNU gzip engine and the AGPL relicensing in docs/superpowers/specs/2026-09-03-gnu-gzip-engine-design.md, the pigz engine in docs/superpowers/specs/2026-09-03-pigz-engine-design.md and the pgzip engine in docs/superpowers/specs/2026-09-04-pgzip-engine-design.md. The implementation plan that built this library task by task lives at docs/superpowers/plans/2026-09-03-zrecipe.md.

Documentation

Overview

Package zrecipe makes compressed files reproducible from their uncompressed content: Analyze finds the engine and parameters that re-create a gzip or zstd file exactly, and Recompress rebuilds it.

Index

• Constants
• Variables
• func DefaultEngines() []engine.Engine
• func Recompress(ctx context.Context, p *Params, uncompressed io.Reader, w io.Writer, ...) error
• type Analysis
  • func Start(ctx context.Context, r io.ReadSeeker, opts *Options) (*Analysis, error)
  • func (a *Analysis) Close() error
  • func (a *Analysis) Compressed() Digest
  • func (a *Analysis) Confirm(ctx context.Context, tee io.Writer) (*Params, error)
  • func (a *Analysis) Format() Format
  • func (a *Analysis) Uncompressed() Digest
  • func (a *Analysis) Verified() bool
• type DeflateParams
• type Digest
• type Format
  • func Detect(r io.ReadSeeker) (Format, error)
• type GzipParams
• type Options
• type Params
  • func Analyze(ctx context.Context, r io.ReadSeeker, opts *Options) (*Params, error)
  • func ReadParams(r io.Reader) (*Params, error)
  • func (p *Params) Write(w io.Writer) error
• type RecompressOptions
• type ZstdParams

Constants

const (
	FormatNone = format.None
	FormatGzip = format.Gzip
	FormatZstd = format.Zstd
)

const DefaultMaxInMemory = 64 << 20

DefaultMaxInMemory is the spool size above which content goes to a temp file.

const ParamsVersion = 1

ParamsVersion is the schema version written by this package.

Variables

var (
	// ErrUnsupported reports an input the library recognises but does not
	// handle: multi-member gzip, multi-frame zstd, skippable frames,
	// dictionaries.
	ErrUnsupported = errors.New("zrecipe: unsupported input")
	// ErrCorrupt reports an input that failed to decompress or verify.
	ErrCorrupt = errors.New("zrecipe: corrupt input")
	// ErrNotReproducible reports that no candidate reproduced the input.
	ErrNotReproducible = errors.New("zrecipe: not reproducible")
	// ErrInputMismatch reports uncompressed input that does not match Params.
	ErrInputMismatch = errors.New("zrecipe: uncompressed input does not match params")
	// ErrDigestMismatch reports recompressed output that does not match Params.
	ErrDigestMismatch = errors.New("zrecipe: recompressed output does not match params")
	// ErrEngineUnavailable reports an engine name that is not in the set.
	ErrEngineUnavailable = errors.New("zrecipe: engine unavailable")
	// ErrEngineVersionMismatch reports an engine version different from Params.
	ErrEngineVersionMismatch = errors.New("zrecipe: engine version mismatch")
	// ErrParamsVersion reports an unknown Params schema version.
	ErrParamsVersion = errors.New("zrecipe: unsupported params version")
	// ErrInvalidParams reports Params that are internally inconsistent.
	ErrInvalidParams = errors.New("zrecipe: invalid params")
)

Functions

func DefaultEngines

func DefaultEngines() []engine.Engine

DefaultEngines returns every engine compiled into the binary, most likely producers of files from the wild first: the GNU gzip port, then the cgo engines zlib, pigz and libzstd (present only when built with cgo), then the remaining pure-Go engines, with the klauspost/pgzip port after the klauspost/compress engine whose current version it does not share.

func Recompress

func Recompress(ctx context.Context, p *Params, uncompressed io.Reader, w io.Writer, opts *RecompressOptions) error

Recompress rebuilds the compressed file described by p from uncompressed and streams it to w. It verifies the input against p.Uncompressed and the output against p.Compressed. Bytes already written to w are not rolled back on error; write to a temporary file and rename on success.

Types

type Analysis

type Analysis struct {
	// contains filtered or unexported fields
}

Analysis is a decompressed input together with the candidate the elimination settled on, waiting for Confirm to reproduce the input from the content while streaming that content to the caller. Close releases the spool holding the content.

func Start

func Start(ctx context.Context, r io.ReadSeeker, opts *Options) (*Analysis, error)

Start detects the format of r, decompresses it once into a spool while hashing both streams, validates the container, and eliminates the candidates down to one. It returns ErrNotReproducible when every candidate diverged from the input inside the elimination, and ErrUnsupported, ErrCorrupt or the context's error as Analyze does. For an uncompressed input it hashes r and there is nothing to eliminate. The caller must Close the returned Analysis.

func (*Analysis) Close

func (a *Analysis) Close() error

Close releases the spool. It is idempotent; Confirm fails after it.

func (*Analysis) Compressed

func (a *Analysis) Compressed() Digest

Compressed is the BLAKE3 digest and size of the input.

func (*Analysis) Confirm

func (a *Analysis) Confirm(ctx context.Context, tee io.Writer) (*Params, error)

Confirm reads the content once, writes every block to tee (nil to skip) and rebuilds the input from it through Recompress's own code, comparing the output with the input byte for byte. It returns the Params on success; ErrNotReproducible, naming the offset, when the output diverges; tee's own error, wrapped, when a tee write fails; the context's error; or an engine error. After a failure tee has received a prefix of the content. A block reaches tee before the rebuilder sees it, so tee may be a few blocks ahead of the comparison when the pass stops. With Options.VerifyLimit set the rebuild stops once that many bytes matched and the rest of the content goes to tee alone. Confirm may be called once.

func (*Analysis) Format

func (a *Analysis) Format() Format

Format is the container format of the input.

func (*Analysis) Uncompressed

func (a *Analysis) Uncompressed() Digest

Uncompressed is the BLAKE3 digest and size of the decompressed content.

func (*Analysis) Verified

func (a *Analysis) Verified() bool

Verified reports that the elimination already reproduced the whole input (a small input, one the fallback search ran to the end, or an uncompressed one). Confirm runs the rebuild regardless, so callers need not care; it is exposed for logs and tests.

type DeflateParams

type DeflateParams = engine.DeflateParams

Parameter types are defined in package engine and re-exported here.

type Digest

type Digest struct {
	Blake3 string `json:"blake3"` // 64 hex characters
	Size   int64  `json:"size"`
}

Digest identifies content by blake3 hash and size.

type Format

type Format = format.Format

Format identifies a compression container.

func Detect

func Detect(r io.ReadSeeker) (Format, error)

Detect reads the magic bytes and seeks r back to its start. An input shorter than any magic sequence is FormatNone.

type GzipParams

type GzipParams = engine.GzipParams

Parameter types are defined in package engine and re-exported here.

type Options

type Options struct {
	// TempDir holds the spool for large inputs. Default os.TempDir().
	TempDir string
	// MaxInMemory is the spool size kept in memory. Default DefaultMaxInMemory.
	MaxInMemory int64
	// Parallelism is the number of candidates evaluated at once. Default
	// runtime.NumCPU(). Takes effect only when r implements io.ReaderAt.
	Parallelism int
	// Uncompressed, if set, receives the decompressed content while
	// Analyze confirms the parameters it found (the confirming pass, see
	// Analysis.Confirm), in order and in engine.FeedSize writes. An input
	// that is not reproducible writes nothing to it; one whose confirmation
	// fails writes a prefix.
	Uncompressed io.Writer
	// Engines to search. Default DefaultEngines().
	Engines []engine.Engine
	// VerifyLimit, when positive, accepts a candidate once it has
	// reproduced this many bytes of the compressed input instead of
	// running it to the end, both in the search and in the confirming
	// pass: a candidate that matches that far and diverges later is rare
	// enough that recompressing the rest of a large input is not worth
	// its time. The confirming pass still streams the whole content to
	// its tee. Zero, the default, verifies the whole input. Recompress
	// checks the output digest, so a divergence past the limit surfaces
	// there.
	VerifyLimit int64
}

Options configures Analyze. The zero value uses the defaults.

type Params

type Params struct {
	Version       int         `json:"version"`
	Format        Format      `json:"format"`
	Compressed    Digest      `json:"compressed"`
	Uncompressed  Digest      `json:"uncompressed"`
	Engine        string      `json:"engine,omitempty"`
	EngineVersion string      `json:"engine_version,omitempty"`
	Gzip          *GzipParams `json:"gzip,omitempty"`
	Zstd          *ZstdParams `json:"zstd,omitempty"`
}

Params records how to rebuild a compressed file from its content.

func Analyze

func Analyze(ctx context.Context, r io.ReadSeeker, opts *Options) (*Params, error)

Analyze detects the format of r, decompresses it once while hashing both streams, finds an engine and parameters that reproduce r exactly and confirms them through the pull path: it is Start, Confirm with Options.Uncompressed as the tee, and Close. For an uncompressed input it returns Params with FormatNone.

func ReadParams

func ReadParams(r io.Reader) (*Params, error)

ReadParams decodes and validates a Params document.

func (*Params) Write

func (p *Params) Write(w io.Writer) error

Write encodes p as indented JSON.

type RecompressOptions

type RecompressOptions struct {
	// Engines to look the recorded engine up in. Default DefaultEngines().
	Engines []engine.Engine
	// AllowVersionMismatch tries the engine even when its version differs
	// from the recorded one. The digest check still decides the outcome.
	AllowVersionMismatch bool
}

RecompressOptions configures Recompress. The zero value uses the defaults.

type ZstdParams

type ZstdParams = engine.ZstdParams

Parameter types are defined in package engine and re-exported here.