xz

README

This here is a friendly fork of https://github.com/ulikunitz/xz with several performance improvements, mostly focused on decoding, roughly doubling decoding speed for serial encoding, and about 20x with parallel block decoding for multiblock archives.

Upstream seems inactive. However, if you have time and interest in doing that, feel free to carry these changes over there.

Upstream as of 2026-07-16

Benchmark	Throughput	Time/op	Bytes/op	Allocs/op
Reader (decompress)	48.63 MiB/s	196.1 ms	31.94 MiB	1,213,036
Writer (compress)	14.09 MiB/s	678.3 ms	69.35 MiB	1,217,288

This fork

Benchmark	Throughput	Time/op	Bytes/op	Allocs/op
Reader (decompress)	106.32 MiB/s (+118.6%)	89.7 ms (-54.3%)	13.56 MiB (-57.6%)	284 (-99.98%)
Writer (compress)	18.27 MiB/s (+29.7%)	522.1 ms (-23.0%)	50.85 MiB (-26.7%)	4,622 (-99.6%)

LLVM xz release tarball decoding (1.6G, multiblock)

Decoder	Time	Throughput
Reader (serial, Patch 9)	79.7 s	141 MiB/s
ParallelReader (18 workers)	7.4 s	1,514 MiB/s

---

Package xz

This Go language package supports the reading and writing of xz compressed streams. It includes also a gxz command for compressing and decompressing data. The package is completely written in Go and doesn't have any dependency on any C code.

The package is currently under development. There might be bugs and APIs are not considered stable. At this time the package cannot compete with the xz tool regarding compression speed and size. The algorithms there have been developed over a long time and are highly optimized. However there are a number of improvements planned and I'm very optimistic about parallel compression and decompression. Stay tuned!

Using the API

The following example program shows how to use the API.

package main

import (
    "bytes"
    "io"
    "log"
    "os"

    "github.com/forkcloser/xz"
)

func main() {
    const text = "The quick brown fox jumps over the lazy dog.\n"
    var buf bytes.Buffer
    // compress text
    w, err := xz.NewWriter(&buf)
    if err != nil {
        log.Fatalf("xz.NewWriter error %s", err)
    }
    if _, err := io.WriteString(w, text); err != nil {
        log.Fatalf("WriteString error %s", err)
    }
    if err := w.Close(); err != nil {
        log.Fatalf("w.Close error %s", err)
    }
    // decompress buffer and write output to stdout
    r, err := xz.NewReader(&buf)
    if err != nil {
        log.Fatalf("NewReader error %s", err)
    }
    if _, err = io.Copy(os.Stdout, r); err != nil {
        log.Fatalf("io.Copy error %s", err)
    }
}

Documentation

You can find the full documentation at pkg.go.dev.

Using the gxz compression tool

The package includes a gxz command line utility for compression and decompression.

Use following command for installation:

$ go get github.com/forkcloser/xz/cmd/gxz

To test it call the following command.

$ gxz bigfile

After some time a much smaller file bigfile.xz will replace bigfile. To decompress it use the following command.

$ gxz -d bigfile.xz

Security & Vulnerabilities

The security policy is documented in SECURITY.md.

The software is not affected by the supply chain attack on the original xz implementation, CVE-2024-3094. This implementation doesn't share any files with the original xz implementation and no patches or pull requests are accepted without a review.

All security advisories for this project are published under github.com/forkcloser/xz/security/advisories.

Documentation

Overview

Package xz supports the compression and decompression of xz files. It supports version 1.0.4 of the specification without the non-LZMA2 filters. See http://tukaani.org/xz/xz-file-format-1.0.4.txt

Example

const text = "The quick brown fox jumps over the lazy dog."
var buf bytes.Buffer

// compress text
w, err := NewWriter(&buf)
if err != nil {
	log.Fatalf("NewWriter error %s", err)
}
if _, err := io.WriteString(w, text); err != nil {
	log.Fatalf("WriteString error %s", err)
}
if err := w.Close(); err != nil {
	log.Fatalf("w.Close error %s", err)
}

// decompress buffer and write result to stdout
r, err := NewReader(&buf)
if err != nil {
	log.Fatalf("NewReader error %s", err)
}
if _, err = io.Copy(os.Stdout, r); err != nil {
	log.Fatalf("io.Copy error %s", err)
}

Output:
The quick brown fox jumps over the lazy dog.

Index

• Constants
• func ValidHeader(data []byte) bool
• type ParallelReader
  • func NewParallelReader(xz io.ReaderAt, size int64) (r *ParallelReader, err error)
  • func (r *ParallelReader) Close() error
  • func (r *ParallelReader) Read(p []byte) (n int, err error)
  • func (r *ParallelReader) Size() int64
  • func (r *ParallelReader) WriteTo(w io.Writer) (n int64, err error)
• type ParallelReaderConfig
  • func (c ParallelReaderConfig) NewParallelReader(xz io.ReaderAt, size int64) (r *ParallelReader, err error)
  • func (c *ParallelReaderConfig) Verify() error
• type Reader
  • func NewReader(xz io.Reader) (r *Reader, err error)
  • func (r *Reader) Read(p []byte) (n int, err error)
• type ReaderConfig
  • func (c ReaderConfig) NewReader(xz io.Reader) (r *Reader, err error)
  • func (c *ReaderConfig) Verify() error
• type Writer
  • func NewWriter(xz io.Writer) (w *Writer, err error)
  • func (w *Writer) Close() error
  • func (w *Writer) Write(p []byte) (n int, err error)
• type WriterConfig
  • func (c WriterConfig) NewWriter(xz io.Writer) (w *Writer, err error)
  • func (c *WriterConfig) Verify() error

Examples

• Package
• Reader
• Writer

Constants

const (
	None   byte = 0x0
	CRC32  byte = 0x1
	CRC64  byte = 0x4
	SHA256 byte = 0xa
)

Constants for the checksum methods supported by xz.

const HeaderLen = 12

HeaderLen provides the length of the xz file header.

Functions

func ValidHeader

func ValidHeader(data []byte) bool

ValidHeader checks whether data is a correct xz file header. The length of data must be HeaderLen.

Types

type ParallelReader

type ParallelReader struct {
	ParallelReaderConfig
	// contains filtered or unexported fields
}

ParallelReader decodes the blocks of an xz file concurrently. It requires random access to the input (io.ReaderAt) and its total size, because the block locations are read from the stream indexes at the end of each stream before any data is decoded. The decoded stream is presented in order through the io.Reader (or io.WriterTo) interface.

Only files consisting of multiple blocks — as produced for example by xz with a block size limit or in multi-threaded mode, or by this package's writer with WriterConfig.BlockSize — decode with real concurrency; a single-block file decodes on one worker. Memory usage is proportional to Workers times the uncompressed block size.

The ParallelReader verifies the block checks, the block sizes against the index, and the header, footer and index checksums of every stream. It is not safe for concurrent use.

func NewParallelReader

func NewParallelReader(xz io.ReaderAt, size int64) (r *ParallelReader, err error)

NewParallelReader creates a reader that decodes the blocks of an xz file concurrently using the default parameters. See ParallelReader for the conditions under which this actually parallelizes.

func (*ParallelReader) Close

func (r *ParallelReader) Close() error

Close stops the background workers. It must be called when the reader is abandoned before io.EOF was reached; it is a no-op otherwise. The error is always nil.

func (*ParallelReader) Read

func (r *ParallelReader) Read(p []byte) (n int, err error)

Read reads the uncompressed data stream. The blocks are decoded concurrently but delivered in order.

func (*ParallelReader) Size

func (r *ParallelReader) Size() int64

Size returns the total number of uncompressed bytes in the file, as recorded in the stream indexes.

func (*ParallelReader) WriteTo

func (r *ParallelReader) WriteTo(w io.Writer) (n int64, err error)

WriteTo writes the whole remaining uncompressed data stream to w. It avoids the intermediate copy of the Read interface by handing the decoded block buffers directly to the writer.

type ParallelReaderConfig

type ParallelReaderConfig struct {
	DictCap int
	Workers int
}

ParallelReaderConfig defines the parameters for the parallel xz reader. Workers is the number of blocks decoded concurrently; values below 1 select runtime.GOMAXPROCS(0).

func (ParallelReaderConfig) NewParallelReader

func (c ParallelReaderConfig) NewParallelReader(xz io.ReaderAt, size int64) (r *ParallelReader, err error)

NewParallelReader creates a new parallel reader using the given configuration. It reads and verifies the stream headers, footers and indexes, but does not decode any block data yet.

func (*ParallelReaderConfig) Verify

func (c *ParallelReaderConfig) Verify() error

Verify checks the configuration for errors. Zero values will be replaced by default values.

type Reader

type Reader struct {
	ReaderConfig
	// contains filtered or unexported fields
}

Reader supports the reading of one or multiple xz streams.

Example

package main

import (
	"bufio"
	"io"
	"log"
	"os"

	"github.com/forkcloser/xz"
)

func main() {
	f, err := os.Open("fox.xz")
	if err != nil {
		log.Fatalf("os.Open(%q) error %s", "fox.xz", err)
	}
	defer f.Close()
	r, err := xz.NewReader(bufio.NewReader(f))
	if err != nil {
		log.Fatalf("xz.NewReader(f) error %s", err)
	}
	if _, err = io.Copy(os.Stdout, r); err != nil {
		log.Fatalf("io.Copy error %s", err)
	}
}

Output:
The quick brown fox jumps over the lazy dog.

func NewReader

func NewReader(xz io.Reader) (r *Reader, err error)

NewReader creates a new xz reader using the default parameters. The function reads and checks the header of the first XZ stream. The reader will process multiple streams including padding.

func (*Reader) Read

func (r *Reader) Read(p []byte) (n int, err error)

Read reads uncompressed data from the stream.

type ReaderConfig

type ReaderConfig struct {
	DictCap      int
	SingleStream bool
}

ReaderConfig defines the parameters for the xz reader. The SingleStream parameter requests the reader to assume that the underlying stream contains only a single stream.

func (ReaderConfig) NewReader

func (c ReaderConfig) NewReader(xz io.Reader) (r *Reader, err error)

NewReader creates an xz stream reader. The created reader will be able to process multiple streams and padding unless a SingleStream has been set in the reader configuration c.

func (*ReaderConfig) Verify

func (c *ReaderConfig) Verify() error

Verify checks the reader parameters for Validity. Zero values will be replaced by default values.

type Writer

type Writer struct {
	WriterConfig
	// contains filtered or unexported fields
}

Writer compresses data written to it. It is an io.WriteCloser.

Example

package main

import (
	"fmt"
	"log"
	"os"

	"github.com/forkcloser/xz"
)

func main() {
	f, err := os.Create("example.xz")
	if err != nil {
		log.Fatalf("os.Open(%q) error %s", "example.xz", err)
	}
	defer f.Close()
	w, err := xz.NewWriter(f)
	if err != nil {
		log.Fatalf("xz.NewWriter(f) error %s", err)
	}
	defer w.Close()
	_, err = fmt.Fprintln(w, "The brown fox jumps over the lazy dog.")
	if err != nil {
		log.Fatalf("fmt.Fprintln error %s", err)
	}
	if err = w.Close(); err != nil {
		log.Fatalf("w.Close() error %s", err)
	}
}

func NewWriter

func NewWriter(xz io.Writer) (w *Writer, err error)

NewWriter creates a new xz writer using default parameters.

func (*Writer) Close

func (w *Writer) Close() error

Close closes the writer and adds the footer to the Writer. Close doesn't close the underlying writer.

func (*Writer) Write

func (w *Writer) Write(p []byte) (n int, err error)

Write compresses the uncompressed data provided.

type WriterConfig

type WriterConfig struct {
	Properties *lzma.Properties
	DictCap    int
	BufSize    int
	BlockSize  int64
	// checksum method: CRC32, CRC64 or SHA256 (default: CRC64)
	CheckSum byte
	// Forces NoChecksum (default: false)
	NoCheckSum bool
	// match algorithm
	Matcher lzma.MatchAlgorithm
}

WriterConfig describe the parameters for an xz writer.

func (WriterConfig) NewWriter

func (c WriterConfig) NewWriter(xz io.Writer) (w *Writer, err error)

NewWriter creates a new Writer using the given configuration parameters.

func (*WriterConfig) Verify

func (c *WriterConfig) Verify() error

Verify checks the configuration for errors. Zero values will be replaced by default values.