compress

README

compress

Reads the .Z format of compress(1) — pure Go, CGO_ENABLED=0, builds for every target Go builds for.

r, err := compress.NewReader(f)
if err != nil { return err }
io.Copy(dst, r)

Why the standard library cannot

compress/lzw says what it is for: "LZW as used by the GIF and PDF file formats, which means variable-width codes up to 12 bits". .Z needs three things it does not have:

codes up to 16 bits	compress/lzw stops at 12
a clear code	the writer resets the table mid-stream when compression stops paying
the group alignment	codes are written in groups of eight, and the stream is padded to the end of a group before the width grows or the table clears

The last one is what makes .Z hard. It is invisible in small files: a reader that ignores it decodes everything up to the first widening and then produces plausible rubbish, which is worse than failing.

There is no writer, on purpose

Reading .Z opens archives that already exist. Writing one makes a file nobody should be making in 2026 — gzip, xz and zstd all compress better and are read everywhere. The format is here to be understood, not perpetuated.

Untrusted input

The header can claim at most 16 bits, so the three tables are under 256 KiB, and the reader decodes one code at a time, only once the caller has taken the previous string (at most 64 KiB). What it holds does not depend on the stream.

What does depend on the stream is how much it expands, and LZW has no ceiling to find: a code that names a 65 536-byte string costs sixteen bits. compress(1) turns 64 MiB of zeros into 18 KB (3 640:1), and a hand-built stream of 123 KB (lzwSquare(16) in the tests) reaches 2.1 GB. That expansion is paid in time by whoever reads; a caller that collects everything (io.ReadAll) must cap it:

out, err := io.ReadAll(io.LimitReader(r, 64<<20))

One defect was found and fixed: every code allocated its string afresh, because the output slice was resliced past what had been copied and left no room at its front — 71.5 MB of garbage for 64 MiB of zeros (TotalAlloc, measured on a compress(1) stream), now 0.3 MB, the tables and one string. FuzzReader found no panic.

⛔ The fixture generator was broken before the reader was

macOS's compress(1) writes streams its own -d refuses, at -b 9, -b 10 and -b 11. It exits 0 doing it, and gzip -d refuses them too:

$ printf 'x' | compress -b 9 -c > t.Z ; compress -dc < t.Z
compress: Inappropriate file type or format

Three of those were used as fixtures at first, and every failure they caused was reported against this reader, which was right to refuse them. Hours went into chasing a defect that was in the generator.

So squeeze now asks the reference to read back what it just wrote, and skips a width it cannot. The fixture is checked before the subject.

The other half of the same trap: measuring this from a shell gave "every width fails", because compress -d refuses a file whose name does not end in .Z — a failure for a reason that had nothing to do with the bytes.

What the tests assert beyond "it decodes"

• that the width actually grew — a corpus that never widens tests nothing while looking thorough (it ends at 12 bits with 3550 table entries);
• that block mode was exercised, which only data defeating LZW provokes;
• that a code past the table is ErrCorrupt and not io.EOF, because a .Z ends with whatever bits fill its last byte, so running out mid-code is the normal end and calling damage "the end" hands back a silently short file.

The suite does not need the binary

Eight .Z streams are committed, each one verified to round-trip through compress -d before being stored, and each one's plaintext is generated rather than stored — so the testdata is 84 KB and there is nothing to keep in step by hand.

That was not the first design. The suite made every fixture with compress(1), which is absent on three of this project's eight CI lanes: every test there skipped, and the coverage gate fell with them while the tests themselves reported success. The live binary is now an extra judge rather than the only one, and coverage is 100% with it removed from PATH.

The corpus asserts, on the committed files, that some stream widened and some stream cleared — the two moments the group alignment runs. Only one does clear, and it had to be built for it: text first so the table fills, then noise so the ratio collapses.

100% statement coverage, race clean, 8 CI lanes including 4 emulated architectures.

Licence

BSD-3-Clause.

Documentation

Overview

Package compress reads the .Z format of compress(1) in pure Go, with no cgo.

Why the standard library cannot

Go has compress/lzw, and it says what it is for: "LZW as used by the GIF and PDF file formats, which means variable-width codes up to 12 bits". The .Z format needs three things it does not have:

• codes up to SIXTEEN bits;
• a clear code that resets the table mid-stream, which the writer sends when compression stops paying;
• and the group alignment — compress(1) writes codes in groups of eight, so a group of nine-bit codes is nine whole bytes, and when the width grows or the table is cleared it PADS to the end of the current group before changing anything.

The last is the one that makes .Z hard. It is invisible in small files: a reader that ignores it decodes everything until the first widening and then produces plausible rubbish, which is worse than failing.

r, err := compress.NewReader(f)
if err != nil { return err }
io.Copy(dst, r)

There is no writer, on purpose

Reading .Z opens archives that already exist, which is a real need. Writing one makes a file nobody should be making in 2026: gzip, xz and zstd all compress better and are read everywhere. The format is here to be understood, not perpetuated.

Verification, and what it cost

Every fixture is made by the system's compress(1) and compared byte for byte. The corpus is built to reach the parts that matter: data that fills the table so the width grows, and data that defeats LZW so the writer clears — and the tests ASSERT that both happened, because a corpus that never widens tests nothing while looking thorough.

⛔ macOS's compress(1) writes streams its own -d refuses, at -b 9, -b 10 and -b 11. It exits 0 doing it. Three of these were used as fixtures at first and every failure they caused was reported against this reader, which was right to refuse them. So squeeze asks the reference to read back what it just wrote, and skips a width it cannot — the fixture is checked before the subject.

Index

• Variables
• func NewReader(r io.Reader) (io.Reader, error)

Variables

var ErrCorrupt = errors.New("compress: corrupt .Z stream")

ErrCorrupt is returned when the code stream cannot be decoded.

It is distinct from io.ErrUnexpectedEOF on purpose: a .Z stream ENDS with whatever bits fill the last byte, so running out mid-code is the normal end and not damage. Only a code that cannot exist yet is damage.

var ErrHeader = errors.New("compress: not a .Z stream")

ErrHeader is returned when the first three bytes are not a .Z header.

Functions

func NewReader

func NewReader(r io.Reader) (io.Reader, error)

NewReader returns a reader over the .Z stream in r.

The header is read immediately, so a stream that is not .Z is refused here rather than at the first Read.