codec

Documentation

Overview

Package codec wraps a stream in the compression the source uses.

It is the one place a codec is chosen. The package had two switches over the same nine codecs, one reached through filesql.CompressionType and one through the parser package's fused file types, and a codec added to either was missing from the other.

Index

• Constants
• Variables
• type Codec
  • func (c Codec) CannotWrite() error
  • func (c Codec) Extension() string
  • func (c Codec) NewReader(reader io.Reader) (io.Reader, func() error, error)
  • func (c Codec) NewWriter(writer io.Writer) (io.Writer, func() error, error)
  • func (c Codec) String() string

Constants

const (
	ExtGZ     = ".gz"
	ExtBZ2    = ".bz2"
	ExtXZ     = ".xz"
	ExtZSTD   = ".zst"
	ExtZLIB   = ".z"
	ExtSNAPPY = ".snappy"
	ExtS2     = ".s2"
	ExtLZ4    = ".lz4"
)

File extensions, one per codec.

Variables

var ErrDeclaredSizeTooLarge = errors.New("the stream asks for more working memory than this package will hold for it")

ErrDeclaredSizeTooLarge reports a stream whose header asks for more working memory than this package agrees to hold for it.

xz and zstd both put the size of the buffer their decoder must allocate in the header, and both decoders allocate exactly what the header asks for before reading a byte of compressed data. The number is therefore set by what the file asserts rather than by anything it contains: a 28-byte xz stream can ask for a 4 GiB dictionary and a 14-byte zstd frame for a 512 MiB window, and a caller that opens such a file repeatedly pays it every time. What the bound gives is not the Parquet reader's rule, that a file costs no more than its own size; it is weaker, a fixed ceiling in place of whatever the header names.

var ErrNoBZ2Writer = errors.New("bzip2 compression is not supported for writing")

ErrNoBZ2Writer reports a request to write bzip2, which the standard library can read but not write. It is a sentinel so a caller can report it as an unsupported format rather than as a failed compressor.

Types

type Codec

type Codec int

Codec is a compression format.

const (
	// None is an uncompressed stream, which passes through unchanged.
	None Codec = iota
	// GZ is gzip.
	GZ
	// BZ2 is bzip2, which can be read but not written: the standard library
	// has no bzip2 writer.
	BZ2
	// XZ is xz.
	XZ
	// ZSTD is Zstandard.
	ZSTD
	// ZLIB is zlib.
	ZLIB
	// SNAPPY is the Snappy framing format.
	SNAPPY
	// S2 is S2, Snappy's extension.
	S2
	// LZ4 is LZ4.
	LZ4
)

func (Codec) CannotWrite

func (c Codec) CannotWrite() error

CannotWrite returns the error NewWriter would give for this codec, or nil when the codec has a writer. It answers before there is anything to write, so a caller can refuse a save that would end in ErrNoBZ2Writer.

func (Codec) Extension

func (c Codec) Extension() string

Extension returns the file extension for the codec, empty for None.

func (Codec) NewReader

func (c Codec) NewReader(reader io.Reader) (io.Reader, func() error, error)

NewReader wraps reader so it reads the decompressed bytes, and returns the function that releases the decompressor.

The close function is never nil, so a caller can defer it without asking whether this codec has anything to release.

func (Codec) NewWriter

func (c Codec) NewWriter(writer io.Writer) (io.Writer, func() error, error)

NewWriter wraps writer so what is written to it is compressed, and returns the function that flushes and releases the compressor. The close function is never nil.

func (Codec) String

func (c Codec) String() string

String returns the codec's short name, which is the name the command line and the file-type constants use for it.