README
¶
Package xz
This Go language package supports the reading and writing of xz compressed streams. It includes also a gxz command for compressing and decompressing data. The package is completely written in Go and doesn't have any dependency on any C code.
APIs are not considered stable. Compression speed and ratio do not match
the xz tool, whose algorithms have been tuned over a long time.
Decompression is a different story: see the numbers below, and
ParallelReader for block-parallel decoding of multi-block archives.
About this fork
This here is a friendly fork of https://github.com/ulikunitz/xz. Upstream seems inactive. However, if you have time and interest in doing that, feel free to carry these changes over there.
The fork diverges from upstream in three areas:
Performance. Serial decoding is a bit over twice as fast as
upstream, mostly by buffering each LZMA2 chunk in memory so the range
decoder reads bytes by index instead of through a per-byte interface
call, and by keeping the hot bit-decoding loops free of calls and error
branches. Decoder state, probability models, dictionary and read
buffers are reused across chunks and across the blocks of a file, which
takes allocations down from about 1.2 million to 132 per 10 MB decode
and from 1.2 million to 306 per encode. ParallelReader decodes the
blocks of multi-block archives concurrently on top of that. The decoder
dictionary grows on demand instead of being allocated at its declared
size — a stream that produces little never pays for the 4 GiB its
header may declare, at the cost of roughly twice the final size in
allocations for streams that fill it.
Robustness. Every number in an xz index is attacker controlled, so
the parallel reader binds its memory use to what a block actually
decodes to rather than what the index declares, and rejects record
counts, sizes and overflows that upstream fed into allocations or loop
bounds. A ParallelReader that is dropped without Close winds down
its goroutines instead of leaking them. Decoding errors are classified:
everything that means "this input is not valid xz" matches
ErrCorrupt, unsupported-but-valid features match ErrUnsupported,
and I/O errors from the underlying reader pass through untouched, so
callers can tell a corrupt file from a failed transport. A truncated
file is reported as such instead of decoding as a shorter one, and a
writer flush failure surfaces instead of silently producing a short
stream, as upstream v0.5.16 does for small-dictionary configurations —
configurations this fork briefly rejected and now encodes correctly.
Verification. The decoder is differentially tested against
upstream and against the xz tool across encoder configurations,
payload shapes, multi-stream files and dictionary-growth boundaries,
and fuzzed both in-repo (serial and parallel readers must agree) and
against upstream. Malformed-input tests cover truncation and bit flips
at every offset and a corpus of hostile index constructions with an
allocation budget. AUDIT.md records a full audit of the tree,
including the findings that led to the fixes above and the negative
results that were measured and rejected.
Benchmarks
Measured on testdata/enwik7 (10 MB of Wikipedia text), Apple M5 Pro,
Go 1.26, 2026-08-02. Upstream is github.com/ulikunitz/xz v0.5.16 on
the same benchmark bodies. Reproduce with:
go test -run '^$' -bench 'Reader|Writer' -benchmem -benchtime=5x -count=6 .
| Benchmark | Upstream v0.5.16 | This fork | Change |
|---|---|---|---|
| Reader (decompress) | 48 MB/s | 100 MB/s | +110% |
| Reader allocs/op | 1,213,039 | 132 | −99.99% |
| Writer (compress) | 14.8 MB/s | 16 MB/s | +8% |
| Writer allocs/op | 1,217,296 | 306 | −99.97% |
Multi-block files (the shape xz -T produces, and the one
ParallelReader exists for), same corpus:
| Benchmark | Throughput | Allocs/op |
|---|---|---|
| Reader, 153 × 64 KiB blocks | 73 MB/s | 2,576 |
| ParallelReader, 10 × 1 MiB blocks, 18 workers | 700 MB/s | ~1,265 |
| ParallelReader, 153 × 64 KiB blocks, 18 workers | ~980 MB/s | — |
Compression ratio is identical to upstream in the default configuration. Multi-block serial decoding is slower than single-block per byte because each block restarts the dictionary; that cost is intrinsic to the format, not to this implementation.
Using the API
The following example program shows how to use the API.
package main
import (
"bytes"
"io"
"log"
"os"
"github.com/forkcloser/xz"
)
func main() {
const text = "The quick brown fox jumps over the lazy dog.\n"
var buf bytes.Buffer
// compress text
w, err := xz.NewWriter(&buf)
if err != nil {
log.Fatalf("xz.NewWriter error %s", err)
}
if _, err := io.WriteString(w, text); err != nil {
log.Fatalf("WriteString error %s", err)
}
if err := w.Close(); err != nil {
log.Fatalf("w.Close error %s", err)
}
// decompress buffer and write output to stdout
r, err := xz.NewReader(&buf)
if err != nil {
log.Fatalf("NewReader error %s", err)
}
if _, err = io.Copy(os.Stdout, r); err != nil {
log.Fatalf("io.Copy error %s", err)
}
}
Documentation
You can find the full documentation at pkg.go.dev.
Using the gxz compression tool
The package includes a gxz command line utility for compression and decompression.
Use following command for installation:
$ go install github.com/forkcloser/xz/cmd/gxz@latest
To test it call the following command.
$ gxz bigfile
After some time a much smaller file bigfile.xz will replace bigfile. To decompress it use the following command.
$ gxz -d bigfile.xz
Security & Vulnerabilities
The security policy is documented in SECURITY.md.
The software is not affected by the supply chain attack on the original xz implementation, CVE-2024-3094. This implementation doesn't share any files with the original xz implementation and no patches or pull requests are accepted without a review.
All security advisories for this project are published under github.com/forkcloser/xz/security/advisories.
Documentation
¶
Overview ¶
Package xz supports the compression and decompression of xz files. It supports version 1.0.4 of the specification without the non-LZMA2 filters. See http://tukaani.org/xz/xz-file-format-1.0.4.txt
Example ¶
const text = "The quick brown fox jumps over the lazy dog."
var buf bytes.Buffer
// compress text
w, err := NewWriter(&buf)
if err != nil {
log.Fatalf("NewWriter error %s", err)
}
if _, err := io.WriteString(w, text); err != nil {
log.Fatalf("WriteString error %s", err)
}
if err := w.Close(); err != nil {
log.Fatalf("w.Close error %s", err)
}
// decompress buffer and write result to stdout
r, err := NewReader(&buf)
if err != nil {
log.Fatalf("NewReader error %s", err)
}
if _, err = io.Copy(os.Stdout, r); err != nil {
log.Fatalf("io.Copy error %s", err)
}
Output: The quick brown fox jumps over the lazy dog.
Index ¶
Examples ¶
Constants ¶
const ( None byte = 0x0 CRC32 byte = 0x1 CRC64 byte = 0x4 SHA256 byte = 0xa )
Constants for the checksum methods supported by xz.
const HeaderLen = 12
HeaderLen provides the length of the xz file header.
Variables ¶
var ( // ErrCorrupt reports that the data being read is not a valid xz stream: // bad magic, a failed checksum, sizes that disagree, a reserved field // that is set. Every such error from this package matches it. An I/O // error from the underlying reader is passed through untouched, so it // does not match. ErrCorrupt = errors.New("xz: corrupt input") // ErrClosed reports that a reader was used after Close. ErrClosed = errors.New("xz: already closed") // ErrUnsupported reports a stream this package cannot decode even though // it may be well formed, such as a filter other than LZMA2. ErrUnsupported = errors.New("xz: unsupported feature") )
Sentinel errors that callers can test for with errors.Is.
The distinction that matters in practice is between a file that is not a valid xz stream and a transport that failed underneath us: the first means reject the input, the second means the read may be worth retrying. Telling them apart used to require matching on message text.
Functions ¶
func ValidHeader ¶
ValidHeader checks whether data is a correct xz file header. The length of data must be HeaderLen.
Types ¶
type ParallelReader ¶
type ParallelReader struct {
ParallelReaderConfig
// contains filtered or unexported fields
}
ParallelReader decodes the blocks of an xz file concurrently. It requires random access to the input (io.ReaderAt) and its total size, because the block locations are read from the stream indexes at the end of each stream before any data is decoded. The decoded stream is presented in order through the io.Reader (or io.WriterTo) interface.
Only files consisting of multiple blocks — as produced for example by xz with a block size limit or in multi-threaded mode, or by this package's writer with WriterConfig.BlockSize — decode with real concurrency; a single-block file decodes on one worker. Memory usage is proportional to Workers times the uncompressed block size.
The ParallelReader verifies the block checks, the block sizes against the index, and the header, footer and index checksums of every stream.
Read and WriteTo must be called from one goroutine at a time. Close is the exception: it may be called from another goroutine to cancel a Read that is waiting on a block, which is what makes it usable for abandoning a reader whose input has gone slow.
func NewParallelReader ¶
func NewParallelReader(xz io.ReaderAt, size int64) (r *ParallelReader, err error)
NewParallelReader creates a reader that decodes the blocks of an xz file concurrently using the default parameters. See ParallelReader for the conditions under which this actually parallelizes.
func (*ParallelReader) Close ¶
func (r *ParallelReader) Close() error
Close stops the background workers. It must be called when the reader is abandoned before io.EOF was reached; it is a no-op otherwise. The error is always nil.
Unlike Read and WriteTo, Close may be called from another goroutine, and doing so cancels a read that is waiting on a block. A reader that already reached io.EOF keeps reporting io.EOF rather than the close.
func (*ParallelReader) Read ¶
func (r *ParallelReader) Read(p []byte) (n int, err error)
Read reads the uncompressed data stream. The blocks are decoded concurrently but delivered in order.
func (*ParallelReader) Size ¶
func (r *ParallelReader) Size() int64
Size returns the total number of uncompressed bytes in the file, as recorded in the stream indexes.
type ParallelReaderConfig ¶
ParallelReaderConfig defines the parameters for the parallel xz reader. Workers is the number of blocks decoded concurrently; values below 1 select runtime.GOMAXPROCS(0).
func (ParallelReaderConfig) NewParallelReader ¶
func (c ParallelReaderConfig) NewParallelReader(xz io.ReaderAt, size int64) (r *ParallelReader, err error)
NewParallelReader creates a new parallel reader using the given configuration. It reads and verifies the stream headers, footers and indexes, but does not decode any block data yet.
func (*ParallelReaderConfig) Verify ¶
func (c *ParallelReaderConfig) Verify() error
Verify checks the configuration for errors and replaces zero values with their defaults, so afterwards DictCap and Workers both hold the values that will actually be used.
type Reader ¶
type Reader struct {
ReaderConfig
// contains filtered or unexported fields
}
Reader supports the reading of one or multiple xz streams.
Example ¶
package main
import (
"bufio"
"io"
"log"
"os"
"github.com/forkcloser/xz"
)
func main() {
f, err := os.Open("fox.xz")
if err != nil {
log.Fatalf("os.Open(%q) error %s", "fox.xz", err)
}
defer func() {
if err := f.Close(); err != nil {
log.Printf("f.Close() error %s", err)
}
}()
r, err := xz.NewReader(bufio.NewReader(f))
if err != nil {
log.Printf("xz.NewReader(f) error %s", err)
return
}
if _, err = io.Copy(os.Stdout, r); err != nil {
log.Printf("io.Copy error %s", err)
return
}
}
Output: The quick brown fox jumps over the lazy dog.
func NewReader ¶
NewReader creates a new xz reader using the default parameters. The function reads and checks the header of the first XZ stream. The reader will process multiple streams including padding.
func (*Reader) Read ¶
Read reads uncompressed data from the stream.
As the io.Reader contract permits, Read can return decoded data together with an error. When the error reports corrupt or truncated input, the trailing bytes of that data may stem from decoder state that was fed input past the point of corruption; discard data received alongside such an error rather than treating it as a correct prefix of the stream.
type ReaderConfig ¶
ReaderConfig defines the parameters for the xz reader. The SingleStream parameter requests the reader to assume that the underlying stream contains only a single stream.
DictCap is the smallest dictionary the reader will use. A block whose header asks for more gets what it asks for, so this raises the floor rather than capping memory; the dictionary itself is grown on demand and costs only what the stream actually decodes.
func (ReaderConfig) NewReader ¶
func (c ReaderConfig) NewReader(xz io.Reader) (r *Reader, err error)
NewReader creates an xz stream reader. The created reader will be able to process multiple streams and padding unless a SingleStream has been set in the reader configuration c.
func (*ReaderConfig) Verify ¶
func (c *ReaderConfig) Verify() error
Verify checks the reader parameters for validity and replaces zero values with their defaults, so afterwards DictCap holds the value that will actually be used.
type Writer ¶
type Writer struct {
WriterConfig
// contains filtered or unexported fields
}
Writer compresses data written to it. It is an io.WriteCloser.
Example ¶
package main
import (
"fmt"
"log"
"os"
"path/filepath"
"github.com/forkcloser/xz"
)
func main() {
// A temporary path keeps the example from dropping a file into whatever
// directory it is run from — which, when it runs as a test, is the
// package source directory.
name := filepath.Join(os.TempDir(), "example.xz")
f, err := os.Create(name)
if err != nil {
log.Fatalf("os.Create(%q) error %s", name, err)
}
defer func() { _ = os.Remove(name) }()
defer func() {
if err := f.Close(); err != nil {
log.Printf("f.Close() error %s", err)
}
}()
w, err := xz.NewWriter(f)
if err != nil {
log.Printf("xz.NewWriter(f) error %s", err)
return
}
if _, err = fmt.Fprintln(w, "The brown fox jumps over the lazy dog."); err != nil {
log.Printf("fmt.Fprintln error %s", err)
return
}
// Close finishes the compressed stream. Skipping it, or ignoring what it
// returns, is how a truncated archive gets written without anyone
// noticing.
if err = w.Close(); err != nil {
log.Printf("w.Close() error %s", err)
return
}
}
Output:
type WriterConfig ¶
type WriterConfig struct {
Properties *lzma.Properties
DictCap int
BufSize int
BlockSize int64
// CheckSum selects the check method: CRC32, CRC64 or SHA256 (default:
// CRC64). It cannot select None: None is zero, which is indistinguishable
// from the field being unset, so a zero CheckSum means the default. Use
// NoCheckSum to write a stream with no check.
CheckSum byte
// NoCheckSum writes a stream with no integrity check, overriding
// CheckSum (default: false).
NoCheckSum bool
// match algorithm
Matcher lzma.MatchAlgorithm
}
WriterConfig describe the parameters for an xz writer.
func (WriterConfig) NewWriter ¶
func (c WriterConfig) NewWriter(xz io.Writer) (w *Writer, err error)
NewWriter creates a new Writer using the given configuration parameters.
func (*WriterConfig) Verify ¶
func (c *WriterConfig) Verify() error
Verify checks the configuration for errors. Zero values will be replaced by default values.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
gxz
command
Command gxz supports the compression and decompression of LZMA files.
|
Command gxz supports the compression and decompression of LZMA files. |
|
xb
command
Command xb supports the xz for Go project builds.
|
Command xb supports the xz for Go project builds. |
|
internal
|
|
|
gflag
Package gflag implements GNU-style command line flag parsing.
|
Package gflag implements GNU-style command line flag parsing. |
|
hash
Package hash provides rolling hashes.
|
Package hash provides rolling hashes. |
|
randtxt
Package randtxt supports the generation of random text using a trigram model for the English language.
|
Package randtxt supports the generation of random text using a trigram model for the English language. |
|
term
Package term provides the IsTerminal function.
|
Package term provides the IsTerminal function. |
|
xlog
Package xlog provides a simple logging package that allows to disable certain message categories.
|
Package xlog provides a simple logging package that allows to disable certain message categories. |
|
Package lzma supports the decoding and encoding of LZMA streams.
|
Package lzma supports the decoding and encoding of LZMA streams. |
|
Package xio provides tools to handle I/O operations.
|
Package xio provides tools to handle I/O operations. |