lzo1z

README

lzo1z

Pure Go implementation of LZO1Z compression and decompression.

Overview

LZO1Z is a variant of the LZO1X compression algorithm used in real-time data feeds and other applications requiring fast compression/decompression.

This package provides both compression and decompression, fully compatible with the liblzo2 library.

Features

• Pure Go - no CGO, no external dependencies
• Zero allocations per call
• ~420 MB/s compression, ~1 GB/s decompression
• Compatible with liblzo2
• Cross-compilation friendly

Installation

go get github.com/rhnvrm/lzo1z

Usage

Compression

input := []byte("Hello, World! Hello, World! Hello, World!")

// Allocate buffer for compressed data
compressed := make([]byte, lzo1z.MaxCompressedSize(len(input)))

// Compress
n, err := lzo1z.Compress(input, compressed)
if err != nil {
    log.Fatal(err)
}
compressed = compressed[:n]

fmt.Printf("%d bytes -> %d bytes\n", len(input), n)
// Output: 41 bytes -> 21 bytes

Decompression

// Allocate buffer (must know or estimate decompressed size)
output := make([]byte, expectedSize)

// Decompress
n, err := lzo1z.Decompress(compressed, output)
if err != nil {
    log.Fatal(err)
}
result := output[:n]

Buffer Sizing

Use MaxCompressedSize to allocate compression buffers:

bufSize := lzo1z.MaxCompressedSize(inputLen)
// Returns: inputLen + inputLen/16 + 64 + 3

For decompression, you must know or estimate the output size. LZO does not store the decompressed size in the stream.

Performance

Benchmarks on Intel i7-1355U:

BenchmarkCompress-12         114550    10700 ns/op    420 MB/s    0 B/op    0 allocs/op
BenchmarkDecompress-12     13102768       94 ns/op   1060 MB/s    0 B/op    0 allocs/op

Compression Ratios

Input	Size	Compressed	Ratio
Repeated "A"	40 B	9 B	4.4x
"ABCD" x 100	400 B	16 B	25x
English text	17 KB	322 B	53x
Random bytes	256 B	261 B	0.98x

LZO1Z vs LZO1X

LZO1Z differs from LZO1X in offset encoding:

Aspect	LZO1Z	LZO1X
Offset encoding	(b0 << 6) + (b1 >> 2)	(b0 >> 2) + (b1 << 6)
M2 offset reuse	Yes	No
M2_MAX_OFFSET	1792	2048

These differences mean LZO1X and LZO1Z are not compatible.

Limitations

• Buffer sizing - caller must provide appropriately sized buffers
• No streaming - data must fit in memory

Testing

go test -v ./...           # Run tests
go test -bench=. -benchmem # Run benchmarks

Test vectors are verified against liblzo2 for both compression and decompression.

Credits

Based on the LZO algorithm by Markus Franz Xaver Johannes Oberhumer.

License

MIT License - see LICENSE.

Documentation

Overview

Package lzo1z implements LZO1Z compression and decompression in pure Go.

LZO1Z is a variant of the LZO1X compression algorithm with different offset encoding, used primarily for real-time data compression. This package is compatible with the liblzo2 library.

Compression

Compress data using the Compress function:

input := []byte("Hello, World!")
compressed := make([]byte, lzo1z.MaxCompressedSize(len(input)))

n, err := lzo1z.Compress(input, compressed)
if err != nil {
    log.Fatal(err)
}
compressed = compressed[:n]

Use MaxCompressedSize to determine the required buffer size for worst-case compression (incompressible data).

Decompression

Decompress LZO1Z data:

output := make([]byte, expectedSize)

n, err := lzo1z.Decompress(compressed, output)
if err != nil {
    log.Fatal(err)
}
result := output[:n]

Buffer Sizing

The caller must provide appropriately sized buffers:

• For compression: use MaxCompressedSize(inputLen)
• For decompression: you must know or estimate the output size

LZO does not store the decompressed size in the compressed stream, so the caller must track this separately.

Thread Safety

Both Compress and Decompress are safe for concurrent use - they have no global state and perform zero allocations.

Performance

The implementation achieves approximately 420 MB/s compression and 1 GB/s decompression on modern hardware with zero allocations.

Package lzo1z implements LZO1Z decompression in pure Go.

LZO1Z is a variant of the LZO1X compression algorithm with different offset encoding, used in real-time data feeds and embedded systems.

This implementation is based on the original LZO library by Markus Franz Xaver Johannes Oberhumer (http://www.oberhumer.com/opensource/lzo/).

Key differences from LZO1X:

• Offset encoding uses (ip[0] << 6) + (ip[1] >> 2) instead of (ip[0] >> 2) + (ip[1] << 6)
• M2 matches can reuse the last match offset when (t & 0x1f) >= 0x1c
• Different M2_MAX_OFFSET constant (0x0700 vs 0x0800)

Example

package main

import (
	"fmt"
	"log"

	"github.com/rhnvrm/lzo1z"
)

func main() {
	// Compress some data
	input := []byte("Hello, World! Hello, World! Hello, World!")

	compressed := make([]byte, lzo1z.MaxCompressedSize(len(input)))
	compLen, err := lzo1z.Compress(input, compressed)
	if err != nil {
		log.Fatal(err)
	}
	compressed = compressed[:compLen]

	// Decompress it back
	output := make([]byte, len(input)+100)
	decompLen, err := lzo1z.Decompress(compressed, output)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("Original: %d bytes\n", len(input))
	fmt.Printf("Compressed: %d bytes\n", compLen)
	fmt.Printf("Decompressed: %s\n", string(output[:decompLen]))
}

Output:

Original: 41 bytes
Compressed: 21 bytes
Decompressed: Hello, World! Hello, World! Hello, World!

Index

• Variables
• func Compress(src, dst []byte) (int, error)
• func Decompress(src, dst []byte) (int, error)
• func DecompressSafe(src, dst []byte) (int, error)
• func MaxCompressedSize(n int) int

Examples

• Package
• Compress
• Decompress
• Decompress (Repeated)
• MaxCompressedSize

Variables

var (
	ErrInputOverrun      = errors.New("lzo1z: input buffer overrun")
	ErrOutputOverrun     = errors.New("lzo1z: output buffer overrun")
	ErrLookbehindOverrun = errors.New("lzo1z: lookbehind overrun (match references before output start)")
	ErrCorrupted         = errors.New("lzo1z: corrupted input data")
	ErrInputNotConsumed  = errors.New("lzo1z: input not fully consumed (extra bytes after EOF marker)")
)

Errors returned by Decompress

Functions

func Compress

func Compress(src, dst []byte) (int, error)

Compress compresses src using LZO1Z algorithm and writes to dst. Returns the number of bytes written to dst. dst must be large enough to hold the compressed data. Worst case size is: len(src) + len(src)/16 + 64 + 3

This is a greedy compressor optimized for speed over compression ratio.

Example

package main

import (
	"bytes"
	"fmt"
	"log"

	"github.com/rhnvrm/lzo1z"
)

func main() {
	// Compress repetitive data
	input := bytes.Repeat([]byte("ABCD"), 100) // 400 bytes

	compressed := make([]byte, lzo1z.MaxCompressedSize(len(input)))
	n, err := lzo1z.Compress(input, compressed)
	if err != nil {
		log.Fatal(err)
	}

	ratio := float64(len(input)) / float64(n)
	fmt.Printf("Input: %d bytes, Compressed: %d bytes, Ratio: %.1fx\n", len(input), n, ratio)
}

Output:

Input: 400 bytes, Compressed: 16 bytes, Ratio: 25.0x

func Decompress

func Decompress(src, dst []byte) (int, error)

Decompress decompresses LZO1Z compressed data from src into dst. Returns the number of bytes written to dst.

The dst buffer must be large enough to hold the decompressed data. If dst is too small, ErrOutputOverrun is returned along with the number of bytes successfully written.

This function is compatible with data compressed by lzo1z_999_compress() from the liblzo2 library.

Example

package main

import (
	"fmt"
	"log"

	"github.com/rhnvrm/lzo1z"
)

func main() {
	// Compressed "AAAAAAAAAA" (10 repeated 'A' characters)
	compressed := []byte{0x1b, 0x41, 0x41, 0x41, 0x41, 0x41, 0x41, 0x41, 0x41, 0x41, 0x41, 0x11, 0x00, 0x00}

	output := make([]byte, 20)
	n, err := lzo1z.Decompress(compressed, output)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("Decompressed %d bytes: %s\n", n, string(output[:n]))
}

Output:

Decompressed 10 bytes: AAAAAAAAAA

Example (Repeated)

package main

import (
	"fmt"
	"log"

	"github.com/rhnvrm/lzo1z"
)

func main() {
	// Highly compressed data: 40 repeated 'A' characters
	// Demonstrates LZO1Z's match copying
	compressed := []byte{0x12, 0x41, 0x20, 0x06, 0x00, 0x00, 0x11, 0x00, 0x00}

	output := make([]byte, 50)
	n, err := lzo1z.Decompress(compressed, output)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("Compressed: %d bytes -> Decompressed: %d bytes\n", len(compressed), n)
	fmt.Printf("Ratio: %.1fx\n", float64(n)/float64(len(compressed)))
}

Output:

Compressed: 9 bytes -> Decompressed: 40 bytes
Ratio: 4.4x

func DecompressSafe

func DecompressSafe(src, dst []byte) (int, error)

DecompressSafe is an alias for Decompress that emphasizes bounds checking. All bounds are checked in Decompress, so this is functionally identical.

func MaxCompressedSize

func MaxCompressedSize(n int) int

MaxCompressedSize returns the maximum possible compressed size for input of length n. Use this to allocate the destination buffer.

Example

package main

import (
	"fmt"

	"github.com/rhnvrm/lzo1z"
)

func main() {
	inputSize := 1000

	// Always allocate enough space for worst-case compression
	bufferSize := lzo1z.MaxCompressedSize(inputSize)
	fmt.Printf("For %d byte input, allocate %d byte buffer\n", inputSize, bufferSize)
}

Output:

For 1000 byte input, allocate 1129 byte buffer