engine

Documentation

Overview

Package engine defines the compression engines that zrecipe searches over, and the parameter types recorded in Params.

Index

• Constants
• func Feed(w io.Writer, src io.Reader) (int64, error)
• func LevelOrder(xfl byte) []int
• func ModuleVersion(path string) string
• type DeflateEngine
• type DeflateParams
• type Engine
  • func ByName(engines []Engine, name string) (Engine, bool)
• type Format
• type GzipParams
• type ZstdEngine
• type ZstdParams

Constants

const (
	FormatNone = format.None
	FormatGzip = format.Gzip
	FormatZstd = format.Zstd
)

const (
	StrategyDefault     = "default"
	StrategyFiltered    = "filtered"
	StrategyHuffmanOnly = "huffman_only"
	StrategyRLE         = "rle"
	StrategyFixed       = "fixed"
)

zlib strategy names.

const FeedSize = 32 << 10

FeedSize is the size of the writes Feed hands to an engine writer.

Functions

func Feed

func Feed(w io.Writer, src io.Reader) (int64, error)

Feed copies src into w in writes of exactly FeedSize bytes; only the last write may be shorter. It never takes an io.WriterTo or io.ReaderFrom fast path, and it fills each chunk completely before writing it, so the write shape the engine sees depends neither on the type nor on the read granularity of src.

The search that picks parameters and Recompress that later applies them must both feed engine writers through Feed: an engine whose output depends on how its input is split across Writes (zlib at level 0 sizes stored blocks that way) is otherwise verified under one shape and rebuilt under another, and the recorded Params fail to reproduce the file.

func LevelOrder

func LevelOrder(xfl byte) []int

LevelOrder returns deflate levels 0..9 ordered by likelihood given the gzip XFL byte: 2 marks maximum compression, 4 marks fastest.

func ModuleVersion

func ModuleVersion(path string) string

ModuleVersion returns the version of a dependency module recorded in the binary's build info, or "(devel)" when unknown.

Types

type DeflateEngine

type DeflateEngine interface {
	Engine
	// Candidates returns parameter sets grouped by tier, most likely first.
	Candidates(h *format.GzipHeader, uncompressedSize int64) [][]DeflateParams
	// NewWriter returns a writer that compresses into w. Close flushes.
	NewWriter(w io.Writer, p DeflateParams) (io.WriteCloser, error)
}

DeflateEngine produces raw deflate streams (no gzip header or trailer).

type DeflateParams

type DeflateParams struct {
	Level      int    `json:"level"`
	Strategy   string `json:"strategy,omitempty"`    // zlib and pigz (pigz: huffman_only and rle only)
	WindowBits int    `json:"window_bits,omitempty"` // zlib only: 9..15
	MemLevel   int    `json:"mem_level,omitempty"`   // zlib only: 1..9
	Rsyncable  bool   `json:"rsyncable,omitempty"`   // gnu-gzip and pigz: --rsyncable

	// pigz and pgzip: the block the input is cut into, in KiB; 0 means the
	// engine's default (pigz -b 128, pgzip 1024).
	BlockSize int `json:"block_size,omitempty"`

	// pigz only.
	Independent  bool `json:"independent,omitempty"`   // -i: blocks are compressed without the preceding history
	SingleThread bool `json:"single_thread,omitempty"` // -p 1: pigz's single-thread code path, which flushes differently
}

DeflateParams configures a raw deflate stream.

type Engine

type Engine interface {
	// Name is the stable identifier stored in Params, for example "zlib".
	Name() string
	// Version identifies the implementation build, for example "1.3.2".
	Version() string
	// Format is the container format this engine produces payloads for.
	Format() Format
}

Engine is a compression implementation with a stable name and a version that determines its output.

func ByName

func ByName(engines []Engine, name string) (Engine, bool)

ByName finds an engine by name.

type Format

type Format = format.Format

Format is re-exported from package format.

type GzipParams

type GzipParams struct {
	HeaderB64 string `json:"header_b64"`
	DeflateParams
}

GzipParams is DeflateParams plus the verbatim gzip header.

type ZstdEngine

type ZstdEngine interface {
	Engine
	Candidates(h *format.ZstdFrameHeader, uncompressedSize int64) [][]ZstdParams
	NewWriter(w io.Writer, p ZstdParams, uncompressedSize int64) (io.WriteCloser, error)
}

ZstdEngine produces complete zstd frames.

type ZstdParams

type ZstdParams struct {
	Level         int  `json:"level"`
	WindowLog     int  `json:"window_log,omitempty"` // 0 means the engine default
	Checksum      bool `json:"checksum"`
	ContentSize   bool `json:"content_size"`   // frame header carries the content size
	PledgedSize   bool `json:"pledged_size"`   // encoder knew the source size
	SingleSegment bool `json:"single_segment"` // frame has no window descriptor
	Workers       int  `json:"workers"`        // 0: single-thread path, 1: job-based path
	Long          bool `json:"long,omitempty"` // long distance matching
	// EndWithData records that the producer passed its final input chunk
	// together with the end directive (known-size producers such as the
	// zstd CLI reading a file). When false, end was signalled after all
	// input, which makes libzstd emit an empty last block. Single-thread
	// path only.
	EndWithData bool `json:"end_with_data,omitempty"`
	// EncodeAll records that the klauspost-zstd engine used its one-shot
	// EncodeAll encoding path (which buffers the whole input and encodes it
	// in a single call) rather than its streaming writer.
	EncodeAll bool `json:"encode_all,omitempty"`
}

ZstdParams configures a zstd frame.