zbytes

README

zbytes

A go library to transparently harness the power of zstandard custom-dictionary compression.

    /* create a wrapper. It will collect the first 100000 bytes of wrapped
       content, creating an 8k custom compression dictionary from it,
       and use the default (0) compression level.
       Before the 100k is collected, bytes will be stored uncompressed.
       Afterwards, the custom dictionary will be used to achieve very good
       compression on even very short individual messages
     */
    wrapper := CreateWrapper(8192, 100_000, 0)

    // this opaque object holds the possibly-compressed data
    wrappedBytes := wrapper.Wrap([]byte("A sample message"))

    // a copy of the original byte array can easily be retrieved
    originalBytes := wrapper.Unwrap(wrappedBytes)

why?

If you hold a large number of uncompressed objects in memory, this is for you.

considerations

• the initial wrapped data (100k in the above example) will not be compressed. Even after the compression dictionary is formed, the data will not be retrospectively modified; only new wrapped data will see the memory usage improvements.
• during the initial phase, additional memory is used to retain copies of the wrapped bytes. Once sufficient data has been collected to build the dictionary, this memory is freed.
• care need to be taken if the wrapped data is to be persisted to disk. It will require the original compression dictionary to be available. The Wrapper object includes a Backup method, allowing the same compression dictionary to be used across restarts.

Documentation

Index

• func CreateWrapper(dictionarySize int, sourceDataSize int, compressionLevel int) *wrapper
• func RestoreWrapper(reader io.Reader) (*wrapper, error)
• type Stats
  • func (s *Stats) CollectStats(prefix string, stats *statsd.Client, tags ...statsd.Tag)
• type Wrapped
  • func (w *Wrapped) IsCompressed() bool
• type Wrapper

Functions

func CreateWrapper

func CreateWrapper(dictionarySize int, sourceDataSize int, compressionLevel int) *wrapper

Creates a Wrapper, capable of wrapping byte arrays. A zstandard compression dictionary will be constructed from the first sourceDataSize bytes of input; prior to this, data will be stored internally, uncompressed. The dictionary's size when built will be dictionarySize bytes.

func RestoreWrapper

func RestoreWrapper(reader io.Reader) (*wrapper, error)

Recreate a Wrapper from a backup.

Types

type Stats

type Stats struct {
	UncompressedMessages int64
	CompressedMessages   int64
	UncompressedBytes    int64 // total size of messages prior to compression
	CompressedBytes      int64 // total size of messages after compression (or if not compressed, the original size)
	// contains filtered or unexported fields
}

func (*Stats) CollectStats

func (s *Stats) CollectStats(prefix string, stats *statsd.Client, tags ...statsd.Tag)

Generate a number of Gauge statistics relating to the volume of messages and level of compression since the last call.

type Wrapped

type Wrapped struct {
	// contains filtered or unexported fields
}

func (*Wrapped) IsCompressed

func (w *Wrapped) IsCompressed() bool

Returns whether the wrapped data is compressed.

type Wrapper

type Wrapper interface {
	Unwrap(*Wrapped) []byte
	Wrap([]byte) *Wrapped
	PeriodicStatsCollector(prefix string, stats *statsd.Client, period time.Duration, tags ...statsd.Tag)
	GetStats() Stats
	Backup(io.Writer) error
	Release()
}

A Wrapper manages the wrapping and unwrapping of []byte objects, transparently handling the creation and application of a compression dictionary which is shared between subsequent Wrap operations. This is safe for use by multiple goroutines.