smallzstd

Documentation

Overview

Package smallzstd produces zstd encoders and decoders optimized for low memory usage, at the expense of compression efficiency.

This package is optimized primarily for the memory cost of compressing and decompressing data. We reduce this cost in two major ways: disable parallelism within the library (i.e. don't use multiple CPU cores to decompress), and drop the compression window down from the defaults of 4-16MiB, to 8kiB.

Decompressors cost 2x the window size in RAM to run, so by using an 8kiB window, we can run ~1000 more decompressors per unit of memory than with the defaults.

Depending on context, the benefit is either being able to run more decoders (e.g. in our logs processing system), or having a lower memory footprint when using compression in network protocols (e.g. in tailscaled, which should have a minimal RAM cost).

Index

• Constants
• func NewDecoder(r io.Reader, options ...zstd.DOption) (*zstd.Decoder, error)
• func NewEncoder(w io.Writer, options ...zstd.EOption) (*zstd.Encoder, error)

Constants

const WindowSize = 8 << 10 // 8kiB

WindowSize is the window size used for zstd compression. Decoder memory usage scales linearly with WindowSize.

Functions

func NewDecoder

func NewDecoder(r io.Reader, options ...zstd.DOption) (*zstd.Decoder, error)

NewDecoder returns a zstd.Decoder configured for low memory usage, at the expense of decompression performance.

func NewEncoder

func NewEncoder(w io.Writer, options ...zstd.EOption) (*zstd.Encoder, error)

NewEncoder returns a zstd.Encoder configured for low memory usage, both during compression and at decompression time, at the expense of performance and compression efficiency.