zstd

package module
v0.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 9, 2025 License: MIT Imports: 8 Imported by: 0

README

zstd-purego

Go Reference

A pure Go binding for the Zstandard (zstd) compression library that doesn't use CGo, based on purego.

Features

  • No CGo dependency - significantly simplifies cross-compilation
  • Embedded shared libraries for supported platforms (Linux amd64, macOS arm64)
  • Full API support: simple, context, and streaming operations
  • Dictionary compression/decompression support
  • Memory-safe wrapper around the C API

Installation

go get github.com/develerltd/zstd-purego

## Supported Platforms
- Linux amd64 (with glibc 2.17+)
- macOS arm64 (Apple Silicon)

## Basic Usage

package main

import ( "fmt"

"github.com/develerltd/zstd-purego"

)

func main() { // Compress data with default compression level data := []byte("Hello, zstd!") compressed, err := zstd.Compress(data) if err != nil { panic(err) }

fmt.Printf("Compressed: %d bytes\n", len(compressed))

// Decompress
decompressed, err := zstd.Decompress(compressed, 0)
if err != nil {
	panic(err)
}

fmt.Printf("Decompressed: %s\n", decompressed)

}


## Streaming API

// Compressing var compressedBuf bytes.Buffer writer, _ := zstd.NewWriter(&compressedBuf) writer.Write([]byte("Data to compress")) writer.Close() // Important to flush any remaining data

// Decompressing reader, _ := zstd.NewReader(&compressedBuf) decompressed, _ := io.ReadAll(reader) reader.Close()


## Dictionary Compression

// Load a pre-trained dictionary z, _ := zstd.New() defer z.Close()

dict, _ := z.LoadDictionary(dictData) compressed, _ := z.CompressUsingDict(data, dict, zstd.DefaultCompression) decompressed, _ := z.DecompressUsingDict(compressed, dict, 0)


## Advanced Usage

// Create a custom instance of the library z, err := zstd.New() if err != nil { panic(err) } defer z.Close() // Important to free resources

// Get library version fmt.Printf("Zstandard version: %s\n", z.VersionString())

// Use best compression compressed, err := z.Compress(data, zstd.BestCompression) if err != nil { panic(err) }

// Estimate decompressed size bound := z.CompressBound(len(data)) fmt.Printf("Maximum compressed size: %d bytes\n", bound)


## License
This project is licensed under the MIT License - see the LICENSE file for details.
The Zstandard library is licensed under a dual BSD/GPLv2 license. For more information, see the Zstandard repository.

## Additional Notes About Library Files

For the library to work, you need to include the actual shared libraries in the `libs` directory:

1. For Linux amd64: `libs/linux_amd64_glibc2.17/libzstd.so.1`
2. For macOS arm64: `libs/darwin_arm64/libzstd.dylib`

You can obtain these libraries as mentioned earlier:

### For macOS arm64

On an Apple Silicon Mac:
```bash
# Using Homebrew
brew install zstd
mkdir -p libs/darwin_arm64
cp /opt/homebrew/lib/libzstd.dylib libs/darwin_arm64/

Refresh library linux:

docker run --rm -v $(pwd):/work -w /work ubuntu:16.04 /bin/bash -c '
    apt-get update && apt-get install -y curl build-essential cmake
    curl -L https://github.com/facebook/zstd/archive/refs/tags/v1.5.5.tar.gz -o zstd.tar.gz
    tar -xzf zstd.tar.gz
    cd zstd-1.5.5
    make
    mkdir -p /work/libs/linux_amd64_glibc2.17
    cp lib/libzstd.so.1 /work/libs/linux_amd64_glibc2.17/

Refresh on mac:

# 1. Install build tools if you haven't already
xcode-select --install
brew install cmake  # Optional but useful

# 2. Download and extract the zstd source
curl -L https://github.com/facebook/zstd/archive/refs/tags/v1.5.5.tar.gz -o zstd.tar.gz
tar -xzf zstd.tar.gz
cd zstd-1.5.5

# 3. Build the library
make

# 4. Copy the library to the project directory
mkdir -p ../libs/darwin_arm64
cp lib/libzstd.1.dylib ../libs/darwin_arm64/libzstd.dylib

# 5. Clean up (optional)
cd ..
rm -rf zstd-1.5.5 zstd.tar.gz

Documentation

Overview

Package zstd provides Go bindings to the Zstandard (zstd) compression library using purego to avoid CGo dependencies.

This package embeds the zstd shared libraries for supported platforms and extracts them at runtime, allowing for easy cross-compilation and deployment without external dependencies.

Currently supported platforms: - Linux amd64 (glibc 2.17+) - macOS arm64 (Apple Silicon)

Index

Constants

View Source 
const (
	// Fast compression levels (negative values)
	BestSpeed    = 1
	FastSpeed    = 1
	DefaultSpeed = 3

	// Regular compression levels
	DefaultCompression = 3
	BetterCompression  = 7
	BestCompression    = 19 // Highest practical level, very slow
	UltraCompression   = 22 // Maximum possible level
)

Constants defining Zstandard compression levels

View Source 
const (
	// End operation modes for compressStream2
	EndContinue = 0 // More data to come
	EndFlush    = 1 // Flush pending data
	EndEnd      = 2 // End the frame

)

Constants for stream operations

View Source 
const (
	// Version is the semantic version of the library
	Version = "0.1.0"

	// CommitHash can be set during build using ldflags
	CommitHash = "unknown"
)

Version information for the library

Variables

View Source 
var (
	ErrInvalidLevel    = fmt.Errorf("zstd: invalid compression level")
	ErrCompression     = fmt.Errorf("zstd: compression error")
	ErrDecompression   = fmt.Errorf("zstd: decompression error")
	ErrOutputTooSmall  = fmt.Errorf("zstd: output buffer too small")
	ErrInputTooLarge   = fmt.Errorf("zstd: input too large")
	ErrContextCreation = fmt.Errorf("zstd: failed to create context")
	ErrEmptyInput      = fmt.Errorf("zstd: empty input, nothing to compress")
	ErrMaxSizeExceeded = fmt.Errorf("zstd: maximum size exceeded")
	ErrUnsupported     = fmt.Errorf("zstd: unsupported platform")
	ErrAlreadyClosed   = fmt.Errorf("zstd: already closed")
)

Common errors

Functions

func Compress 

func Compress(src []byte) ([]byte, error)

Compress compresses the input data using the default compression level (3).

func CompressBest 

func CompressBest(src []byte) ([]byte, error)

CompressBest compresses the input data using a high compression level (19).

func CompressFast 

func CompressFast(src []byte) ([]byte, error)

CompressFast compresses the input data using the fastest compression level (1).

func CompressLevel 

func CompressLevel(src []byte, level int) ([]byte, error)

CompressLevel compresses the input data using the specified compression level. Level should be between 1 (fastest) and 22 (highest compression ratio).

func Decompress 

func Decompress(src []byte, maxSize int) ([]byte, error)

Decompress decompresses the input data. The maxSize parameter limits the maximum size of the decompressed data to prevent decompression bombs. Use 0 for the library default max size.

func IsError 

func IsError(code uint64) bool

IsError returns true if the code represents an error condition

func NewErrorReader 

func NewErrorReader(err error) io.Reader

NewErrorReader creates a reader that always returns the specified error

func NewErrorWriter 

func NewErrorWriter(err error) io.Writer

NewErrorWriter creates a writer that always returns the specified error

func NewReader 

func NewReader(r io.Reader) (io.ReadCloser, error)

NewReader creates an io.ReadCloser for decompressing data from the provided reader. The returned reader should be closed with Close() when done.

func NewWriter 

func NewWriter(w io.Writer) (io.WriteCloser, error)

NewWriter creates an io.WriteCloser for compressing data to the provided writer using the default compression level. The returned writer should be closed with Close() when done.

func NewWriterLevel 

func NewWriterLevel(w io.Writer, level int) (io.WriteCloser, error)

NewWriterLevel creates an io.WriteCloser for compressing data to the provided writer using the specified compression level. The returned writer should be closed with Close() when done.

func VersionInfo 

func VersionInfo() string

VersionInfo returns a formatted string with version and build info

Types

type Dictionary 

type Dictionary struct {
	// contains filtered or unexported fields
}

Dictionary represents a pre-trained compression dictionary

func (*Dictionary) ID 

func (d *Dictionary) ID() uint32

ID returns the dictionary ID

type Error 

type Error struct {
	Code    uint64
	Message string
}

Error represents a Zstandard error

func (*Error) Error 

func (e *Error) Error() string

Error implements the error interface

type Options 

type Options struct {
	CompressionLevel  int   // Compression level (1-22, default 3)
	WindowSize        int   // Window size limit (0 = default)
	ReadBufferSize    int   // Read buffer size for streaming operations
	WriteBufferSize   int   // Write buffer size for streaming operations
	MaxDecompressSize int64 // Maximum size limit for decompression (0 = no limit)
}

Options contains configuration options for the Zstd compressor/decompressor

func BestOptions 

func BestOptions() Options

BestOptions returns options optimized for compression ratio

func DefaultOptions 

func DefaultOptions() Options

DefaultOptions returns the default compression options

func FastOptions 

func FastOptions() Options

FastOptions returns options optimized for speed

type Reader 

type Reader struct {
	// contains filtered or unexported fields
}

Reader implements an io.ReadCloser for reading and decompressing data

func (*Reader) Close 

func (r *Reader) Close() error

Close implements the io.Closer interface

func (*Reader) Read 

func (r *Reader) Read(p []byte) (int, error)

Read implements the io.Reader interface

type Writer 

type Writer struct {
	// contains filtered or unexported fields
}

Writer implements an io.WriteCloser for compressing and writing data

func (*Writer) Close 

func (w *Writer) Close() error

Close implements the io.Closer interface

func (*Writer) Flush 

func (w *Writer) Flush() error

Flush flushes any pending data to the underlying writer

func (*Writer) Write 

func (w *Writer) Write(p []byte) (int, error)

Write implements the io.Writer interface

type Zstd 

type Zstd struct {
	// contains filtered or unexported fields
}

Zstd represents an instance of the Zstandard library.

func New 

func New() (*Zstd, error)

New creates a new Zstandard instance. It handles loading the appropriate library for the current platform. The returned instance should be closed with Close() when done.

func (*Zstd) Close 

func (z *Zstd) Close() error

Close releases all resources associated with the Zstd instance. After Close is called, the Zstd instance cannot be used anymore.

func (*Zstd) Compress 

func (z *Zstd) Compress(src []byte, level int) ([]byte, error)

Compress compresses the data from src and returns the compressed data. Level can be between 1 (fastest) and 22 (highest compression ratio).

func (*Zstd) CompressBound 

func (z *Zstd) CompressBound(srcSize int) int

CompressBound returns the maximum compressed size in the worst case scenario.

func (*Zstd) CompressUsingDict 

func (z *Zstd) CompressUsingDict(src []byte, dict *Dictionary, level int) ([]byte, error)

CompressUsingDict compresses data using the dictionary

func (*Zstd) Decompress 

func (z *Zstd) Decompress(src []byte, maxSize int) ([]byte, error)

Decompress decompresses the data from src and returns the decompressed data. The maxSize parameter limits the maximum size of the decompressed data to prevent decompression bombs. Use 0 for the library default max size.

func (*Zstd) DecompressUsingDict 

func (z *Zstd) DecompressUsingDict(src []byte, dict *Dictionary, maxSize int) ([]byte, error)

DecompressUsingDict decompresses data using the dictionary

func (*Zstd) LoadDictionary 

func (z *Zstd) LoadDictionary(dictData []byte) (*Dictionary, error)

LoadDictionary loads a pre-trained dictionary for compression/decompression

func (*Zstd) NewReader 

func (z *Zstd) NewReader(r io.Reader) io.ReadCloser

NewReader creates an io.ReadCloser for decompressing data from the provided reader. It will read and decompress data on demand.

func (*Zstd) NewWriter 

func (z *Zstd) NewWriter(w io.Writer, level int) io.WriteCloser

NewWriter creates an io.WriteCloser for compressing data to the provided writer. The compressed data will be written to the provided writer. The caller must call Close() when done to ensure all data is flushed.

func (*Zstd) Version 

func (z *Zstd) Version() uint32

Version returns the library version as an integer

func (*Zstd) VersionString 

func (z *Zstd) VersionString() string

VersionString returns the library version as a string (e.g., "1.5.5")

type ZstdInBuffer 

type ZstdInBuffer struct {
	Src  unsafe.Pointer
	Size uint64
	Pos  uint64
}

ZstdInBuffer represents a buffer for zstd input operations

type ZstdOutBuffer 

type ZstdOutBuffer struct {
	Dst  unsafe.Pointer
	Size uint64
	Pos  uint64
}

ZstdOutBuffer represents a buffer for zstd output operations

Source Files

View all Source files

Directories

Path Synopsis
examples
simple command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.