lzip

README

lzip

A pure-Go reader for the lzip (.lz) compressed file format.

func NewReader(r io.Reader) (io.Reader, error)
func NewReaderLimit(r io.Reader, maxDictSize int) (io.Reader, error) // ceiling on the dictionary a member may declare

f, err := os.Open("archive.tar.lz")
// ...
r, err := lzip.NewReader(f)
// ...
_, err = io.Copy(dst, r)

CGO_ENABLED=0. The LZMA coding itself comes from github.com/go-compressions/xz/lzma; this package is the framing around it, which is why it is small.

There is no writer, on purpose

xz and zstd are read everywhere a .lz file might go and compress as well or better. A new .lz written today would be a file fewer tools can open for no gain in ratio, so it serves nobody. What a Go program does need is to read the .lz files that already exist — GNU project tarballs, distribution archives, anything a plzip pipeline produced. That is what this package does.

What the format is

A .lz file is one or more members concatenated. Each member is:

magic	LZIP, 4 bytes
version	1. Version 0 is obsolete and refused by name: it has no end-of-stream marker, so a version-0 member's LZMA stream cannot be found the way this reader finds a version-1 one.
coded dictionary size	1 byte, not a size — see below
LZMA stream	raw, with no 13-byte .lzma header of its own
trailer	little-endian: CRC32 of the uncompressed data (u32), uncompressed size (u64), size of the whole member (u64)

Because the member carries no .lzma header, the LZMA decoder is started headerless with the standard properties lc=3 lp=0 pb=2 (property byte 0x5d) and the decoded dictionary size; the uncompressed size is unknown, so the decoder stops at the end-of-stream marker rather than at a byte count. The decoder reads ahead of that marker; the surplus (the trailer, the next member) is handed back to the member reader before the trailer is read.

All three trailer fields are checked when the member ends. A decompressor that does not check its own trailer is how a corrupt file becomes a silently short one.

The coded dictionary size

The byte at offset 5 is not a size. Its low five bits are a base-2 exponent, and its top three bits subtract that many sixteenths of the resulting power of two — which is how lzip expresses the sizes that fall between two powers of two (224 KiB, 1.5 MiB, 3 MiB …):

base = 1 << (b & 0x1f)
size = base - (base / 16) * ((b >> 5) & 7)

lzip -9 on the 229 336-byte testdata/big.txt writes 0x52: exponent 18 with fraction 2, so 262144 - 16384*2 = 229376, the 224 KiB dictionary. Read as a size that byte is 82; read as an exponent alone it is 262144. Neither is the number in the file.

Test corpus

Every .lz file in testdata/ was written by the real GNU lzip 1.26 and plzip 1.13, not hand-assembled from the specification. testdata/generate.sh regenerates them and asserts its own premise: lzip -dc must read each file back before it is allowed to become a fixture. Judging a new decoder against a generator nobody checked is how hours disappear.

The fixtures for the malformed cases — bad magic, version 0, a wrong trailer field, a truncated member, a damaged LZMA body — are derived in the tests by mutating those real files, so even the negative cases start from bytes a real compressor wrote. Tests assert bytes, not lengths.

fixture	producer	members	what it covers
single.lz	lzip -9	1	one member
empty.lz	lzip -9	1	empty input still has a member
onebyte.lz	lzip -9	1	one byte
concat3.lz	cat of three lzip runs at -9, -6, -0	3	separately-compressed members concatenated
big-split.lz	plzip -9 -B 64Ki	4	one payload split into members by the producer
big-0.lz	lzip -0	1	229 336 bytes through a 64 KiB dictionary — data larger than the dictionary
big-3.lz, big-6.lz, big-9.lz	lzip -3/-6/-9	1	compression levels; a 224 KiB (non-power-of-two) dictionary

The corpus is embedded with //go:embed: the emulated CI lanes ship a go test -c binary into a container with no testdata/ directory, and a test that opened a file there would be vacuous on six of the eight lanes.

An outside judge

witness_test.go uses the real lzip and plzip as a judge that shares no code with this package. When they are on PATH it has them compress inputs that are not in the repository -- incompressible data, 400 KB of zeros, a length that lands one byte past the 64 KiB dictionary of lzip -0 -- at all ten levels, and splits a payload at four member sizes with plzip -B, asserting the premise each time (lzip -dc must return its own input) before judging us. It also checks that we refuse the mutations lzip -t refuses.

Those tests skip on all eight CI lanes, which have no lzip. So what CI checks is the committed corpus, and the judge is what a developer's machine adds on top. Both matter: testdata/generate.sh guards against a mis-generated fixture at generation time, and these tests guard against one that slipped through.

Untrusted input

The dictionary size is the only statement a member makes of its memory, and nothing in the framing contradicts it until the trailer, which is at the far end. ulikunitz/xz's LZMA allocated the whole dictionary from that size before decoding (newBuffer is make([]byte, size+1)), so an eleven-byte file asked for a 512 MiB allocation (536 913 640 bytes, measured with TotalAlloc). The LZMA layer is now go-compressions/xz (lzma.Reader1Config.NewReader1): its window starts small and grows with the decoded data, up to the dictionary, so the same eleven bytes cost kilobytes (TestPlantedBombCostsKilobytes: a member coding 512 MiB, the most lzip permits, allocates < 1 MiB; CI runs it as a compiled binary under ulimit -v and GOMEMLIMIT).

r, err := lzip.NewReader(f)                  // dictionary ceiling: 64 MiB
r, err := lzip.NewReaderLimit(f, 512<<20)    // read everything lzip can write
r, err := lzip.NewReaderLimit(f, 1<<20)      // files known to be small

The ceiling stays: a member whose coded dictionary is above it is refused with ErrTooLarge, before anything is allocated, and it holds for every member of a multi-member stream. The default, 64 MiB, is twice the 32 MiB of lzip -9 and plzip -9 (GNU lzip manual, "Invoking lzip", level table); a file written with a larger -s needs NewReaderLimit. The ceiling bounds memory per member, not output: a streaming reader leaves output to its consumer (io.LimitReader). FuzzNewReader runs the reader under a 1 MiB ceiling on arbitrary bytes, with a bounded read.

Memory now follows the data: a member allocates a window only as large as the output it has produced (up to its dictionary and the ceiling).

Ablations

Each row is the whole test suite run against a deliberately broken copy of the decoder. A row that passes is a claim the tests do not actually support.

#	ablation	whole suite	corpus tests alone
1a	coded dictionary byte read as a literal size	FAILS	FAILS
1b	coded dictionary byte read as an exponent only, top three bits dropped	FAILS	⚠ PASSES
1c	dictionary fraction added instead of subtracted	FAILS	⚠ PASSES
2	stop after the first member	FAILS	FAILS
3	trailer CRC32 ignored	FAILS	FAILS
4	trailer uncompressed size ignored	FAILS	FAILS
4b	trailer member size ignored	FAILS	FAILS
5	trailer fields read big-endian	FAILS	FAILS
6	version 0 accepted	FAILS	FAILS

Ablations 1b and 1c pass against the corpus, and that is worth stating plainly rather than hiding behind the green column. An LZMA dictionary larger than the one the encoder used is harmless: the distances in the stream never reach past the real size, so the decoder never notices the slack. And the range check cannot catch an oversized reading either — TestExponentOnlyReadingIsInvisibleToDecoding walks all 256 byte values and shows that for each of the 137 legal ones, the exponent-only reading is also within 4 KiB…512 MiB and never smaller than the true size. No .lz file can witness that mistake. The only thing that can is TestDictSizeFromCode, which reads the formula directly against values taken from the fixtures. That is why the formula has a unit test of its own and not merely a round trip.

It still matters: the size in the file is what a caller is told about the memory the decode will need, and 1b is a 14% over-allocation on big-9.lz alone.

CI

Eight lanes, all gated on 100% statement coverage: linux/amd64, linux/arm64, darwin/arm64, windows/amd64 native, and riscv64, loong64, ppc64le, s390x under QEMU. s390x is big-endian, which is the lane that would notice if the little-endian trailer reads were ever written with the host's byte order.

Licence

BSD-3-Clause. See LICENSE.

Documentation

Overview

Package lzip decodes the lzip (.lz) compressed file format.

lzip is LZMA with its own framing. A .lz file is a sequence of one or more members concatenated; each member is a six-byte header, a raw LZMA stream with no .lzma header of its own, and a twenty-byte little-endian trailer carrying the CRC32 of the uncompressed data, the uncompressed size and the size of the whole member. The format is specified in the GNU lzip manual, "File format".

NewReader returns a reader over the concatenation of every member's uncompressed data. The CRC32 and both sizes in each member's trailer are verified as the member is finished, so a truncated or corrupted file is reported as an error rather than returned as short data.

There is no writer

This package decodes and does not encode, on purpose. xz and zstd are read everywhere a .lz file might go and compress as well or better, so a new .lz file written today serves nobody: it would be a file fewer tools can open, for no gain in ratio. What a program does need is the ability to read the .lz files that already exist -- Linux distribution archives, GNU project tarballs, and anything a plzip pipeline produced. That is what this package is for. The LZMA coding itself comes from github.com/go-compressions/xz/lzma.

Memory

The size coded in a member header is the only statement a member makes of its memory, and nothing in the framing contradicts it until the trailer, which is at the far end. lzip permits up to 512 MiB. The LZMA layer allocates its window lazily, growing with the decoded data, so an eleven-byte file that codes 512 MiB costs kilobytes (an earlier LZMA layer allocated the whole dictionary: 536 913 640 bytes).

NewReader still refuses, as a ceiling, with ErrTooLarge and before allocating, a member whose coded dictionary is above DefaultMaxDictSize (64 MiB, twice the 32 MiB of lzip -9). NewReaderLimit takes the ceiling from the caller: pass 512 MiB to read everything lzip can write, or something small for files that are known to be small. The limit applies to every member of the stream. It bounds memory per member; it does not bound output, which a streaming reader leaves to its consumer (use io.LimitReader).

Test corpus

Every .lz fixture in this repository was produced by the real GNU lzip 1.26 and plzip 1.13, not hand-assembled from the specification, and testdata/generate.sh asserts that premise: lzip -dc must read each file back before it is allowed to be a fixture. Fixtures for the error paths -- a bad magic, version 0, a wrong trailer, a truncated member -- are derived in the tests by mutating those real files, so even the negative cases start from bytes a real compressor wrote.

witness_test.go goes further where it can: when lzip(1) and plzip(1) are on PATH it has them compress inputs that are not in the repository, at every level and at several member sizes, and checks this package against them. Those tests skip on CI, which has no lzip, so what CI checks is the committed corpus and what a developer's machine can add is a judge that shares no code with us.

Index

• Constants
• Variables
• func NewReader(r io.Reader) (io.Reader, error)
• func NewReaderLimit(r io.Reader, maxDictSize int) (io.Reader, error)

Examples

• NewReader

Constants

const DefaultMaxDictSize = 64 << 20

DefaultMaxDictSize is the dictionary size NewReader accepts: 64 MiB, twice what lzip -9 and plzip -9 choose (32 MiB; GNU lzip manual, "Invoking lzip", level table). A file written with a larger -s needs NewReaderLimit.

Variables

var (
	// ErrMagic reports that the four bytes where a member header was
	// expected are not "LZIP".
	ErrMagic = errors.New("lzip: not an lzip stream")
	// ErrVersion0 reports the obsolete version 0 of the format by name.
	ErrVersion0 = errors.New("lzip: format version 0 is obsolete and not supported")
	// ErrVersion reports a format version this package does not know.
	ErrVersion = errors.New("lzip: unsupported format version")
	// ErrDictSize reports a coded dictionary size outside the range the
	// format permits.
	ErrDictSize = errors.New("lzip: dictionary size out of range")
	// ErrCRC reports that a member's data does not match the CRC32 in its
	// trailer.
	ErrCRC = errors.New("lzip: CRC mismatch")
	// ErrSize reports that a member's uncompressed size does not match its
	// trailer.
	ErrSize = errors.New("lzip: uncompressed size mismatch")
	// ErrMemberSize reports that the bytes consumed by a member do not
	// match the member size in its trailer.
	ErrMemberSize = errors.New("lzip: member size mismatch")
	// ErrTooLarge reports a member whose coded dictionary size is valid but
	// above the limit the reader was given. It is a ceiling on what a member
	// may declare, checked before the LZMA layer starts; nothing is allocated.
	ErrTooLarge = errors.New("lzip: dictionary size exceeds the limit")
)

Errors reported for a malformed or corrupt stream. Each is wrapped with detail, so test with errors.Is.

Functions

func NewReader

func NewReader(r io.Reader) (io.Reader, error)

NewReader returns a reader over the uncompressed data of every member of the lzip stream r, in order. The first member's header is read and checked immediately, so a stream that is not lzip at all is reported here.

Each member's trailer is verified when that member ends: the CRC32 and the uncompressed size must match the data produced, and the member size must match the bytes consumed. Read reports the first disagreement as an error and every later call returns the same one.

A member whose header codes a dictionary above DefaultMaxDictSize is refused with ErrTooLarge before the dictionary is allocated; use NewReaderLimit to choose another ceiling.

Example

r, err := NewReader(bytes.NewReader([]byte{
	// `printf 'hi' | lzip -9`, byte for byte.
	0x4c, 0x5a, 0x49, 0x50, 0x01, 0x0c, 0x00, 0x34, 0x1a, 0x5c, 0xff, 0xff, 0xff,
	0xff, 0xf0, 0x00, 0x00, 0x00, 0xac, 0x2a, 0x93, 0xd8, 0x02, 0x00, 0x00, 0x00,
	0x00, 0x00, 0x00, 0x00, 0x26, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
}))
if err != nil {
	fmt.Println(err)
	return
}
out, err := io.ReadAll(r)
fmt.Printf("%q %v\n", out, err)

Output:
"hi" <nil>

func NewReaderLimit

func NewReaderLimit(r io.Reader, maxDictSize int) (io.Reader, error)

NewReaderLimit is NewReader with a caller-chosen ceiling on the dictionary size a member may declare. The LZMA layer allocates the whole dictionary when it starts a member, so this is the memory a stream can make the reader spend per member at most, and only for a member whose data really reaches back that far: the window grows with the decoded data, so an eleven-byte file that codes 512 MiB costs kilobytes. lzip's own range is 4 KiB to 512 MiB (ErrDictSize outside it); a limit above 512 MiB is the same as 512 MiB.