xz

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 1, 2026 License: BSD-3-Clause Imports: 11 Imported by: 0

README

go-compressions/xz

CI coverage

A pure-Go (CGO_ENABLED=0) decoder for .xz, raw LZMA2 and .lzma ("LZMA alone") whose memory follows the decoded data, not the sizes a header declares.

import "github.com/go-compressions/xz"

r, err := xz.NewReader(file)          // one or several concatenated streams
if err != nil { /* ... */ }
_, err = io.Copy(dst, r)              // streaming: output is never buffered whole

It replaces the decode side of github.com/ulikunitz/xz with the same shape of API (xz.NewReader, xz.ReaderConfig, lzma.NewReader, lzma.NewReader2), so a consumer swaps the import path.

Why

ulikunitz/xz v0.5.17 allocates every LZMA2 dictionary whole, from the block header: lzmafilter.go:75-77 takes max(ReaderConfig.DictCap, header) (so DictCap is a minimum) and lzma/buffer.go:23 does make([]byte, size+1). A header of about a hundred bytes can ask for up to 4 GiB, and no option bounds it. Measured with go run ./internal/bombdemo (44 bytes of data in one block, the header declaring a large dictionary):

declared dictionary this library ulikunitz/xz v0.5.17
1 GiB 22 672 bytes allocated 1 073 793 080 bytes
1.5 GiB 22 672 bytes allocated 1 610 658 656 bytes
4 GiB - 1 22 672 bytes allocated not run (allocates 4 GiB by the code above)

Design

Bibliography (opened before any code was written)
Source Used for Licence
xz-file-format.txt (Lasse Collin) the container: stream header/footer, block header, index, padding, checks, filter IDs, VLI, Delta, BCJ properties public domain, may also be distributed under 0BSD (XZ Utils COPYING)
doc/lzma-file-format.txt the .lzma header public domain / 0BSD (same)
LZMA SDK DOC/lzma-specification.txt and its reference decoder LzmaSpec.cpp (Igor Pavlov, 2015-06-14) the LZMA decoder: ported exactly, it is the equality oracle public domain (header of the file)
XZ Utils liblzma/lzma/lzma2_decoder.c LZMA2 chunk control bytes: dictionary / state / properties resets, uncompressed chunks, chunk size checks 0BSD (SPDX header)
XZ Utils liblzma/simple/*.c, liblzma/delta/delta_decoder.c the Delta and BCJ filters (x86, PowerPC, IA-64, ARM, ARM Thumb, SPARC, ARM64, RISC-V) 0BSD
XZ Utils tests/files and its README 80 conformance files (25 good, 50 bad, 5 unsupported), embedded under internal/testfiles/data/xz-utils 0BSD (README of that directory)
Linux lib/xz/xz_dec_bcj.c (0BSD) and fs/squashfs/xz_wrapper.c (GPL-2.0-or-later, read only, nothing copied) which filters squashfs images can use (below) —

All of it is public domain or 0BSD except the squashfs wrapper, which was only read for the list of filters. The repository is BSD-3-Clause.

The lazy window

The reference decoder (COutWindow::Create(dictSize)) allocates the whole dictionary up front. That is the one place this port departs from it:

  • the window starts at 16 KiB (or the dictionary size if smaller) and doubles as Pos advances, up to the dictionary size; it wraps only once it has reached it;
  • growth happens only when the write position reaches the end of the buffer, so there is no per-byte check in the hot loop;
  • a growth that would pass MaxDictionary fails with ErrTooLarge before anything is allocated, and the bytes decoded up to that point have already been delivered.

Why the output is unchanged: the reference's own check is CheckDistance(dist) = dist <= Pos || IsFull. While the window is not full, every distance the reference accepts points into bytes already written, at Pos - dist; a buffer that holds exactly [0, Pos) contains all of them and returns the same bytes. Once Pos reaches the dictionary size the window is the dictionary size, wraps as in the reference, and from then on the two are identical. A header declaring a 4 GiB dictionary over a few bytes of data therefore allocates kilobytes. TestRandomScripts (symbol-level streams, dictionaries 4 KiB to 1 MiB, wrapping several times) and the 36 + 80 fixture files check this.

A second, smaller departure: the reference's CheckDistance(rep0) takes rep0 <= Pos, which for rep0 == Pos reads one byte before the data (Buf[Size-1], uninitialised in the reference when the window is not full). LzmaDec.c and liblzma refuse rep0 >= Pos; so does this decoder. (The xz-utils test file bad-1-lzma2-2.xz exercises this.)

Everything else follows the reference: range decoder, probability tables, literal / match / rep / short-rep coding, end marker, and the three unpackSize modes (unknown size with mandatory marker, known size with an optional marker, known size without marker, the LZMA2 chunk case).

Other allocations follow the data too: the literal probability tables of a .lzma header with lc+lp up to 12 (6 MiB if fully used) grow with the highest literal context actually seen; an LZMA2 chunk is read in 16 KiB steps whatever its header claims (the format caps it at 64 KiB); block sizes and Index counts are validated against what arrives and never used to size a buffer (the Index is checked with a running SHA-256 of the records, in constant memory).

Bounds

ReaderConfig{MaxDictionary int; SingleStream bool}. A zero MaxDictionary is DefaultMaxDictionary, 256 MiB (above the 64 MiB of xz -9). The ceiling is exact: data that fits in it decodes, whatever dictionary the header declares; one byte more is ErrTooLarge (TestCeilingExactness, byte for byte, both for .xz and .lzma). DictCap is accepted and ignored so that code written for ulikunitz/xz compiles; it was a minimum there.

Errors are typed and work with errors.Is: ErrCorrupt (every malformed input), ErrChecksum and ErrTruncated (both are also ErrCorrupt; ErrTruncated is also io.ErrUnexpectedEOF), ErrTooLarge, ErrUnsupported. Nothing panics on input; a failure is sticky. Data of a block is delivered before its check is verified; a failed check follows the data.

Container

Stream header/footer with CRC32, block headers (up to 4 filters, CRC32, padding), Index validation (record count, unpadded and uncompressed sizes, padding, CRC32, Backward Size), Stream Padding and concatenated streams (or SingleStream), checks None / CRC32 / CRC64 / SHA-256 (verified). Reserved check types, unknown filter IDs, reserved flag bits and a filter chain the format does not allow are ErrUnsupported, as liblzma reports them.

Filters
ID Filter Status
0x21 LZMA2 yes
0x03 Delta yes
0x04 BCJ x86 yes
0x05 BCJ PowerPC yes
0x06 BCJ IA-64 yes
0x07 BCJ ARM yes
0x08 BCJ ARM Thumb yes
0x09 BCJ SPARC yes
0x0A BCJ ARM64 yes (with start offsets)
0x0B BCJ RISC-V yes

All eight BCJ filters and Delta are checked against fixtures written by the xz command line tool (go run ./internal/mkfixtures), including chains of three and four filters, start offsets, and the filter test files of XZ Utils.

squashfs. The kernel decodes squashfs XZ images with its XZ Embedded copy (fs/squashfs/xz_wrapper.c calling lib/xz), whose BCJ filters in lib/xz/xz_dec_bcj.c are x86, PowerPC, ARM, ARM Thumb, SPARC, ARM64 and RISC-V (IA-64 was dropped with the architecture); mksquashfs -Xbcj writes x86, arm, armthumb, powerpc, sparc and ia64. This decoder covers every one of them, so any squashfs XZ image decodes, whichever filter its blocks use.

API

package xz
func NewReader(r io.Reader) (*Reader, error)
func (c ReaderConfig) NewReader(r io.Reader) (*Reader, error)
func (c *ReaderConfig) Verify() error
type ReaderConfig struct { MaxDictionary int; DictCap int /* ignored */; SingleStream bool }
const DefaultMaxDictionary = 256 << 20
var ErrCorrupt, ErrChecksum, ErrTruncated, ErrTooLarge, ErrUnsupported error

package lzma // .lzma and raw LZMA2 (7z, zip method 14)
func NewReader(r io.Reader) (*Reader, error)                    // .lzma
func (c ReaderConfig) NewReader(r io.Reader) (*Reader, error)
func (r *Reader) Header() Header
func NewReader2(r io.Reader) (*Reader2, error)                  // raw LZMA2
func (c Reader2Config) NewReader2(r io.Reader) (*Reader2, error)
type Reader2Config struct { DictCap int; MaxDictionary int }    // DictCap: the container's dictionary size
func DictCapFromProp(prop byte) (uint32, error)                  // the 7z / xz property byte
func (c Reader1Config) NewReader1(r io.Reader) (*Reader1, error) // headerless LZMA1 with end marker (lzip)
type Reader1Config struct { LC, LP, PB uint; DictCap, MaxDictionary int }
func (r *Reader1) Unconsumed() []byte                            // read-ahead left over after io.EOF

Reader1 decodes the headerless LZMA1 coding inside an lzip member (lc, lp, pb and the dictionary come from the container, the size is unknown, an end marker ends the stream). The decoder reads ahead from its source, so the bytes that follow the stream (the lzip trailer, the next member) are returned by Unconsumed once Read has reported io.EOF; the caller pushes them back.

Both packages re-export the same error variables. All readers are streaming io.Readers. The .xz reader reads from the source exactly the bytes it uses (no read-ahead); the .lzma reader buffers 32 KiB.

Tests

  • Fixtures from the C reference (xz 5.8.4, internal/mkfixtures): 36 files (all checks, presets 0 and 9e, 4 KiB dictionary so the window wraps, lc/lp/pb variants, multi-block, stored chunks, empty, all filters, .lzma).
  • XZ Utils test files (tests/files): every good file decodes (and equals xz -dc where xz is installed), every bad file is refused with a typed error, every unsupported file is ErrUnsupported.
  • Differential against ulikunitz/xz's own writer (10 configurations × 4 kinds of data: checks, block sizes, properties, match algorithms) and, as decoder oracle, against its reader.
  • Symbol-level LZMA encoder (internal/lzmacore/enc_test.go) reaches every path of the decoder, valid or not: all unpackSize modes, every corruption branch, the end marker with trailing state, repeat/short-repeat in an empty window, distances beyond the data or the dictionary, matches crossing the declared size, and LZMA2 chunk sequences with every reset level.
  • Bombs (TestBomb*, run in CI as compiled binaries under ulimit -v and GOMEMLIMIT): headers declaring 1, 1.5, 2, 3 and 4 GiB - 1 over tiny data (stored and LZMA-compressed) allocate < 1 MiB; ceilings refuse before allocating; block sizes and index counts of 2^62 are refused; a chunk that claims 64 KiB and delivers 10 bytes allocates nothing near 64 KiB.
  • Truncation and bit flips: every proper prefix of a stream is an ErrCorrupt; flipping any byte never panics and never yields wrong data without an error under SHA-256.
  • Fuzzing: FuzzXZ, FuzzLZMA, FuzzLZMA2 (differential against ulikunitz/xz wherever both decode, output bounded, errors must be typed), seeds from every fixture; see the results below. Crashers are committed under testdata/fuzz/ and written out as named unit tests, because the emulated CI lanes run away from testdata/.
  • 100 % statement coverage, gated in CI across the library packages, on amd64, arm64, windows and — through cross-compiled test binaries under QEMU — riscv64, ppc64le, s390x (big-endian) and loong64; plus 386 for 32-bit.
Fuzzing results

100 s per target, 3 workers, on an Apple silicon laptop (GOMEMLIMIT=3GiB):

target executions corpus after the run crashers
FuzzXZ 6.83 M 232 0
FuzzLZMA 3.76 M 306 0
FuzzLZMA2 5.96 M 168 0

The only disagreements the seeds produced are on the oracle's side: ulikunitz/xz v0.5.17 accepts at least two files that XZ Utils lists as bad- (one with a non-minimal variable-length integer, one with an LZMA2 chunk whose data exceeds its declared size) and this decoder refuses them, so the differential only demands equal bytes where both decoders succeed. A weekly workflow (fuzz.yml) keeps fuzzing; a crasher it finds is committed under testdata/fuzz/ and written out as a named unit test.

Performance

go run ./internal/benchxz -work <durable dir> decodes a 64 MiB corpus (the Go source tree and toolchain binaries of the installed Go, compressed once with xz -6 -T1) with this library, with ulikunitz/xz v0.5.17 and with xz -dc (C), single core, in interleaved rounds, and prints MB/s and ratios. It refuses to run when the 1-minute load average is above 8, because the figure would measure the machine.

Measurement deferred. The 1-minute load average of the development machine was 32 to 47 throughout this work (about 25 sessions share it), so the tool refused to measure (exit status 3) and no throughput figure is claimed here. The decode loop is the reference's straight port with no per-byte check besides the range decoder's input refill; optimisation, algorithm first, waits for the table.

Layout

.                      package xz: container, checks, filter chain
lzma/                  package lzma: .lzma and raw LZMA2
internal/lzmacore/     LZMA decoder (port of LzmaSpec.cpp), lazy window, LZMA2 chunks
internal/filter/       Delta and BCJ filters
internal/xerr/         shared error values
internal/testfiles/    embedded fixtures (xz CLI + XZ Utils tests/files)
internal/mkfixtures/   regenerates the fixtures with the xz command line tool
internal/bombdemo/     the table above
internal/benchxz/      the benchmark below

Licence

BSD-3-Clause. The code ported from the LZMA SDK reference decoder (public domain) and from XZ Utils (0BSD) is credited in the sources.

Documentation

Overview

Package xz is a pure-Go decoder for the .xz container (xz-file-format.txt, tukaani.org) whose memory follows the decoded data and not the sizes a header declares.

Its API mirrors the decode side of github.com/ulikunitz/xz so that a consumer can swap the import. The dictionary window of the LZMA2 filter is allocated lazily and never beyond ReaderConfig.MaxDictionary; a stream that needs more fails with ErrTooLarge before the allocation is made.

Example
package main

import (
	"bytes"
	"encoding/hex"
	"fmt"
	"io"

	"github.com/go-compressions/xz"
)

// A complete .xz file as written by `xz` (CRC64 check, one LZMA2 block).
const helloXZ = "fd377a585a000004e6d6b4460200210116000000742fe5a301000f636875" +
	"6e6b20636f6d7072657373696f00ab31da83b7defdd400012810e50b6c60" +
	"1fb6f37d010000000004595a"

func main() {
	data, _ := hex.DecodeString(helloXZ)
	r, err := xz.NewReader(bytes.NewReader(data))
	if err != nil {
		fmt.Println(err)
		return
	}
	out, err := io.ReadAll(r)
	fmt.Printf("%q %v\n", out, err)
}
Output:
"chunk compressio" <nil>

Index

Examples

Constants

View Source
const DefaultMaxDictionary = 256 << 20

DefaultMaxDictionary is the window ceiling used when ReaderConfig.MaxDictionary is zero: 256 MiB, four times the largest dictionary of xz -9 (64 MiB).

View Source
const HeaderLen = 12

HeaderLen is the length of a stream header and of a stream footer.

Variables

View Source
var (
	// ErrCorrupt is wrapped by every error caused by malformed input.
	ErrCorrupt = xerr.ErrCorrupt
	// ErrChecksum is a mismatch of a block check (CRC32, CRC64, SHA-256).
	ErrChecksum = xerr.ErrChecksum
	// ErrTruncated is input that ends inside a structure; it is also an
	// ErrCorrupt and an io.ErrUnexpectedEOF.
	ErrTruncated = xerr.ErrTruncated
	// ErrTooLarge means the stream needs a dictionary window larger than
	// MaxDictionary. No memory was allocated for it.
	ErrTooLarge = xerr.ErrTooLarge
	// ErrUnsupported is a well-formed feature that is not implemented: an
	// unknown filter ID, a reserved check type, a new stream version.
	ErrUnsupported = xerr.ErrUnsupported
)

Errors. They are shared with package lzma and work with errors.Is.

Functions

This section is empty.

Types

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader decodes one or several concatenated xz streams.

func NewReader

func NewReader(xz io.Reader) (*Reader, error)

NewReader creates a reader with the default parameters. It reads and checks the header of the first stream.

func (*Reader) Read

func (r *Reader) Read(p []byte) (int, error)

Read reads uncompressed data. Data of a block is delivered before the block's check is verified; an error from the check follows the data.

type ReaderConfig

type ReaderConfig struct {
	// MaxDictionary is the ceiling in bytes on the dictionary window the
	// reader will allocate for a block; zero means DefaultMaxDictionary.
	// It bounds memory, not the format: a block that declares a 4 GiB
	// dictionary decodes fine as long as the data it produces fits.
	MaxDictionary int

	// DictCap is accepted so that code written for ulikunitz/xz compiles.
	// There it is a minimum allocation; here the window follows the data,
	// so the field has no effect.
	//
	// Deprecated: use MaxDictionary.
	DictCap int

	// SingleStream makes the reader expect exactly one stream, without
	// padding, and report anything after it as an error.
	SingleStream bool
}

ReaderConfig defines the parameters of the xz reader.

Example

MaxDictionary bounds the memory a hostile file can make the reader use: the window grows with the decoded data and stops, with ErrTooLarge, at the ceiling. A header that merely declares a huge dictionary costs nothing.

package main

import (
	"bytes"
	"encoding/hex"
	"errors"
	"fmt"
	"io"

	"github.com/go-compressions/xz"
)

// A complete .xz file as written by `xz` (CRC64 check, one LZMA2 block).
const helloXZ = "fd377a585a000004e6d6b4460200210116000000742fe5a301000f636875" +
	"6e6b20636f6d7072657373696f00ab31da83b7defdd400012810e50b6c60" +
	"1fb6f37d010000000004595a"

func main() {
	data, _ := hex.DecodeString(helloXZ)
	cfg := xz.ReaderConfig{MaxDictionary: 64 << 20, SingleStream: true}
	r, err := cfg.NewReader(bytes.NewReader(data))
	if err != nil {
		fmt.Println(err)
		return
	}
	_, err = io.Copy(io.Discard, r)
	fmt.Println(err, errors.Is(err, xz.ErrTooLarge))
}
Output:
<nil> false

func (ReaderConfig) NewReader

func (c ReaderConfig) NewReader(xz io.Reader) (*Reader, error)

NewReader creates a reader using the configuration. Unless SingleStream is set it processes concatenated streams and stream padding.

func (*ReaderConfig) Verify

func (c *ReaderConfig) Verify() error

Verify checks the parameters and replaces zero values by defaults.

Directories

Path Synopsis
internal
benchxz command
Command benchxz measures single-core decode throughput of this library, of github.com/ulikunitz/xz and of the C reference (xz -dc) on a 64 MiB realistic corpus: the Go source tree and the Go toolchain binaries of the running Go installation, concatenated, then compressed once by xz.
Command benchxz measures single-core decode throughput of this library, of github.com/ulikunitz/xz and of the C reference (xz -dc) on a 64 MiB realistic corpus: the Go source tree and the Go toolchain binaries of the running Go installation, concatenated, then compressed once by xz.
bombdemo command
Command bombdemo prints how much memory this library and github.com/ulikunitz/xz allocate to decode a few bytes of data stored in a block whose header declares a large LZMA2 dictionary.
Command bombdemo prints how much memory this library and github.com/ulikunitz/xz allocate to decode a few bytes of data stored in a block whose header declares a large LZMA2 dictionary.
corpus
Package corpus generates the deterministic inputs used by the tests and by the fixture generator (internal/mkfixtures).
Package corpus generates the deterministic inputs used by the tests and by the fixture generator (internal/mkfixtures).
covgaps command
Command covgaps prints the blocks of a Go coverage profile that were never executed (summing the counts a block has in every test binary), and the overall statement coverage.
Command covgaps prints the blocks of a Go coverage profile that were never executed (summing the counts a block has in every test binary), and the overall statement coverage.
filter
Package filter implements the non-last filters of the xz format in the decoding direction: the Delta filter and the Branch/Call/Jump (BCJ) filters for executables, ported from liblzma/simple/*.c and liblzma/delta/delta_decoder.c (0BSD) of XZ Utils.
Package filter implements the non-last filters of the xz format in the decoding direction: the Delta filter and the Branch/Call/Jump (BCJ) filters for executables, ported from liblzma/simple/*.c and liblzma/delta/delta_decoder.c (0BSD) of XZ Utils.
lzmacore
Package lzmacore is the LZMA decoder shared by the xz and lzma packages.
Package lzmacore is the LZMA decoder shared by the xz and lzma packages.
mkfixtures command
Command mkfixtures regenerates the .xz and .lzma files under internal/testfiles/data with the xz command line tool (the C reference, xz-utils).
Command mkfixtures regenerates the .xz and .lzma files under internal/testfiles/data with the xz command line tool (the C reference, xz-utils).
testfiles
Package testfiles embeds the fixtures made by the xz command line tool (see internal/mkfixtures).
Package testfiles embeds the fixtures made by the xz command line tool (see internal/mkfixtures).
xerr
Package xerr holds the error values shared by the xz and lzma packages, so that both re-export the very same variables and errors.Is works across them.
Package xerr holds the error values shared by the xz and lzma packages, so that both re-export the very same variables and errors.Is works across them.
Package lzma decodes the .lzma ("LZMA alone") format, raw LZMA2 streams and headerless LZMA1 streams with an end marker (lzip members).
Package lzma decodes the .lzma ("LZMA alone") format, raw LZMA2 streams and headerless LZMA1 streams with an end marker (lzip members).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL