helper

package
v1.19.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 4, 2026 License: MIT Imports: 8 Imported by: 0

README

Archive Helper

License Go Version Coverage

Streaming compression and decompression helpers providing unified io.ReadWriteCloser interfaces for transparent data transformation.

Table of Contents

Overview

The helper package provides a simplified, unified interface for compression and decompression operations on streaming data. It acts as an adapter layer between standard Go io interfaces and compression algorithms from github.com/nabbar/golib/archive/compress.

Design Philosophy
  1. Interface Simplicity: Single Helper interface (io.ReadWriteCloser) for all operations
  2. Algorithm Agnostic: Works with any compression algorithm without algorithm-specific code
  3. Streaming First: Process data in chunks without loading entire streams into memory
  4. Resource Safety: Proper cleanup through Close() and automatic wrapper conversions
  5. Type Safety: Compile-time guarantees through interface-based design
Key Features
  • ✅ Unified Interface: Single Helper interface for compression and decompression
  • ✅ Bidirectional Operations: Compress/decompress on read or write
  • ✅ Algorithm Support: Works with GZIP, ZLIB, DEFLATE, BZIP2, LZ4, ZSTD, Snappy
  • ✅ Automatic Wrappers: Auto-converts io.Reader/Writer to io.ReadCloser/WriteCloser
  • ✅ Streaming Architecture: Minimal memory overhead with chunk-based processing
  • ✅ Thread-Safe: Safe for single-instance use (one Helper per goroutine)
  • ✅ Zero Dependencies: Only standard library and golib packages

Architecture

Component Diagram
┌──────────────────────────────────────────────┐
│               Helper Interface               │
│             (io.ReadWriteCloser)             │
└──────────────────────┬───────────────────────┘
                       │
          ┌────────────┴────────────┐
          │                         │
   ┌──────▼──────┐           ┌──────▼──────┐
   │  Compress   │           │  Decompress │
   └──────┬──────┘           └──────┬──────┘
     ┌────┴────┐               ┌────┴────┐
     ▼         ▼               ▼         ▼
 ┌────────┬────────┐       ┌────────┬────────┐
 │ Reader │ Writer │       │ Reader │ Writer │
 └────────┴────────┘       └────────┴────────┘
  compressReader         deCompressReader
  compressWriter         deCompressWriter
Data Flow

Compression Read Flow (compressReader):

  1. Client calls Read(p) on Helper
  2. Helper reads raw data from source reader
  3. Data is compressed using algorithm's Writer
  4. Compressed data is buffered internally
  5. Compressed chunks are returned to client

Compression Write Flow (compressWriter):

  1. Client calls Write(p) on Helper
  2. Data is passed to algorithm's Writer
  3. Compressed data is written to destination
  4. Close() finalizes compression stream

Decompression Read Flow (deCompressReader):

  1. Client calls Read(p) on Helper
  2. Compressed data is read from source
  3. Algorithm's Reader decompresses data
  4. Decompressed data is returned to client

Decompression Write Flow (deCompressWriter):

  1. Client calls Write(p) on Helper
  2. Compressed data is buffered
  3. Background goroutine reads from buffer
  4. Algorithm's Reader decompresses data
  5. Decompressed data is written to destination
Implementation Types
Type Direction Operation Use Case
compressReader Read Compress Read file → compress → output
compressWriter Write Compress Input → compress → write file
deCompressReader Read Decompress Read compressed → decompress → output
deCompressWriter Write Decompress Input compressed → decompress → write file

Performance

Benchmarks

Based on actual test results from the comprehensive test suite (Go 1.25, AMD64):

Operation Median Mean Max Throughput
Compress Read (Small) 100µs 300µs 7.3ms ~100 ops/sec
Compress Read (Medium) 200µs 300µs 800µs ~500 ops/sec
Compress Read (Large) 500µs 600µs 900µs ~200 ops/sec
Compress Write (Small) 100µs 200µs 1ms ~500 ops/sec
Compress Write (Medium) 100µs 200µs 500µs ~500 ops/sec
Compress Write (Large) 500µs 600µs 1ms ~200 ops/sec
Decompress Read <100µs <100µs 100µs ~1000 ops/sec
Round-trip 200µs 300µs 1.3ms ~500 ops/sec

Notes:

  • Small: ~100 bytes
  • Medium: ~1KB
  • Large: ~10KB
  • Actual performance depends on data characteristics and chosen algorithm
Memory Usage
Base overhead:        ~100 bytes (struct)
compressReader:       512 bytes (chunkSize buffer)
deCompressWriter:     + 1 goroutine (~2KB stack)
Per operation:        Minimal allocations

Memory Efficiency:

  • O(1) memory usage regardless of input size
  • No buffer growth or dynamic allocation
  • Streaming operations with constant memory footprint
Scalability

Concurrent Operations:

  • Create separate Helper instances per goroutine
  • No shared state between instances
  • Thread-safe for single-instance usage

Data Size:

  • Tested with files up to 10MB
  • Constant memory regardless of data size
  • Streaming architecture enables unlimited data processing

Use Cases

1. Compressed File I/O

Problem: Read and write compressed files transparently.

h, _ := helper.NewWriter(compress.GZIP, helper.Compress, file)
defer h.Close()
h.Write(data) // Automatically compressed

Real-world: Log file compression, archive creation, data storage.

2. Network Data Compression

Problem: Compress data before sending over network.

h, _ := helper.NewWriter(compress.LZ4, helper.Compress, conn)
defer h.Close()
h.Write(payload) // Compressed before transmission

Real-world: API data transfer, socket communication, bandwidth optimization.

3. Transparent Decompression

Problem: Read compressed data as if uncompressed.

h, _ := helper.NewReader(compress.GZIP, helper.Decompress, compressedStream)
defer h.Close()
scanner := bufio.NewScanner(h) // Works with any io.Reader consumer

Real-world: Processing compressed logs, reading archives, data ingestion.

4. Format Conversion

Problem: Convert between compression formats.

// Read GZIP, write LZ4
src, _ := helper.NewReader(compress.GZIP, helper.Decompress, gzipFile)
dst, _ := helper.NewWriter(compress.LZ4, helper.Compress, lz4File)
defer src.Close()
defer dst.Close()
io.Copy(dst, src)

Real-world: Migration between formats, compatibility adaptation.

5. Streaming Processing

Problem: Process compressed data incrementally.

h, _ := helper.NewReader(compress.ZSTD, helper.Decompress, compressedData)
defer h.Close()
processInChunks(h) // Read and process incrementally

Real-world: Large file processing, ETL pipelines, real-time data transformation.

Quick Start

Installation
go get github.com/nabbar/golib/archive/helper
Compress While Reading

Read from a source and get compressed data:

package main

import (
    "io"
    "os"
    
    "github.com/nabbar/golib/archive/compress"
    "github.com/nabbar/golib/archive/helper"
)

func main() {
    input, _ := os.Open("input.txt")
    defer input.Close()
    
    output, _ := os.Create("output.txt.gz")
    defer output.Close()
    
    // Create compression reader
    h, _ := helper.NewReader(compress.Gzip, helper.Compress, input)
    defer h.Close()
    
    // Copy compressed data to output
    io.Copy(output, h)
}
Compress While Writing

Write data and have it compressed automatically:

package main

import (
    "os"
    
    "github.com/nabbar/golib/archive/compress"
    "github.com/nabbar/golib/archive/helper"
)

func main() {
    output, _ := os.Create("output.txt.gz")
    defer output.Close()
    
    // Create compression writer
    h, _ := helper.NewWriter(compress.Gzip, helper.Compress, output)
    defer h.Close()
    
    // Write data - it will be compressed automatically
    data := []byte("Hello, World!")
    h.Write(data)
}
Decompress While Reading

Read compressed data and get decompressed output:

package main

import (
    "io"
    "os"
    
    "github.com/nabbar/golib/archive/compress"
    "github.com/nabbar/golib/archive/helper"
)

func main() {
    input, _ := os.Open("compressed.gz")
    defer input.Close()
    
    // Create decompression reader
    h, _ := helper.NewReader(compress.Gzip, helper.Decompress, input)
    defer h.Close()
    
    // Read decompressed data
    data, _ := io.ReadAll(h)
    os.Stdout.Write(data)
}
Decompress While Writing

Write compressed data and have it decompressed:

package main

import (
    "io"
    "os"
    "strings"
    
    "github.com/nabbar/golib/archive/compress"
    "github.com/nabbar/golib/archive/helper"
)

func main() {
    output, _ := os.Create("decompressed.txt")
    defer output.Close()
    
    // First, get some compressed data
    var buf strings.Builder
    cw, _ := helper.NewWriter(compress.Gzip, helper.Compress, &buf)
    cw.Write([]byte("Hello, World!"))
    cw.Close()
    
    // Now decompress while writing
    h, _ := helper.NewWriter(compress.Gzip, helper.Decompress, output)
    defer h.Close()
    
    // Write compressed data - it will be decompressed
    h.Write([]byte(buf.String()))
}
Automatic Type Detection

The New() function auto-detects reader vs writer:

// Source is io.Reader - creates a reader helper
input, _ := os.Open("file.txt")
h1, _ := helper.New(compress.Gzip, helper.Compress, input)
data, _ := io.ReadAll(h1)

// Source is io.Writer - creates a writer helper
output, _ := os.Create("file.gz")
h2, _ := helper.New(compress.Gzip, helper.Compress, output)
h2.Write(data)

Best Practices

Testing

The package includes a comprehensive test suite with 84.5% code coverage and 76 test specifications using BDD methodology (Ginkgo v2 + Gomega).

Key test coverage:

  • ✅ All public APIs and constructor functions
  • ✅ Concurrent operations with race detector (zero races detected)
  • ✅ Performance benchmarks (compression/decompression throughput)
  • ✅ Error handling and edge cases
  • ✅ Multiple compression algorithms

For detailed test documentation, see TESTING.md.

✅ DO

Always Close Resources:

h, err := helper.NewWriter(algo, helper.Compress, dst)
if err != nil {
    return err
}
defer h.Close() // Ensures compression finalization

Check Error Returns:

n, err := h.Write(data)
if err != nil {
    return fmt.Errorf("compression failed: %w", err)
}
if n != len(data) {
    return fmt.Errorf("incomplete write: %d of %d bytes", n, len(data))
}

Choose Appropriate Algorithms:

// Fast compression for temporary data
helper.NewWriter(compress.LZ4, helper.Compress, dst)

// Maximum compression for archival
helper.NewWriter(compress.Zstd, helper.Compress, dst)

// Compatibility with external tools
helper.NewWriter(compress.Gzip, helper.Compress, dst)

Stream Large Data:

// ✅ GOOD: Streaming, constant memory
h, _ := helper.NewReader(algo, helper.Compress, largeFile)
defer h.Close()
io.Copy(destination, h)
❌ DON'T

Don't forget to close:

// ❌ BAD: Resource leak
h, _ := helper.NewWriter(compress.Gzip, helper.Compress, file)
h.Write(data) // Missing defer h.Close()

// ✅ GOOD: Proper cleanup
h, _ := helper.NewWriter(compress.Gzip, helper.Compress, file)
defer h.Close()
h.Write(data)

Don't share instances across goroutines:

// ❌ BAD: Race condition
go func() { h.Write(data1) }()
go func() { h.Write(data2) }() // Concurrent writes!

// ✅ GOOD: Separate instances
go func() {
    h1, _ := helper.NewWriter(algo, helper.Compress, dst1)
    defer h1.Close()
    h1.Write(data1)
}()
go func() {
    h2, _ := helper.NewWriter(algo, helper.Compress, dst2)
    defer h2.Close()
    h2.Write(data2)
}()

Don't use wrong operation:

// ❌ BAD: Read() on writer returns error
h, _ := helper.NewWriter(compress.Gzip, helper.Compress, dst)
h.Read(buf) // Returns ErrInvalidSource

// ✅ GOOD: Use correct operation
h, _ := helper.NewReader(compress.Gzip, helper.Compress, src)
h.Read(buf)

API Reference

Helper Interface
type Helper interface {
    io.ReadWriteCloser
}

The Helper interface combines:

  • io.Reader: Read compressed/decompressed data
  • io.Writer: Write data for compression/decompression
  • io.Closer: Release resources and finalize streams

Note: Read() and Write() are mutually exclusive based on Helper type. Compression/decompression readers don't support Write(), and writers don't support Read().

Constructor Functions
New
func New(algo arccmp.Algorithm, ope Operation, src any) (Helper, error)

Creates a Helper with automatic type detection based on source.

Parameters:

  • algo: Compression algorithm (from github.com/nabbar/golib/archive/compress)
  • ope: Operation type (Compress or Decompress)
  • src: Data source (io.Reader or io.Writer)

Returns:

  • Helper: New Helper instance
  • error: ErrInvalidSource if src is neither io.Reader nor io.Writer
NewReader
func NewReader(algo arccmp.Algorithm, ope Operation, src io.Reader) (Helper, error)

Creates a Helper for reading data from an io.Reader.

Parameters:

  • algo: Compression algorithm
  • ope: Compress to compress while reading, Decompress to decompress while reading
  • src: Source reader

Returns:

  • Helper: Reader Helper instance
  • error: ErrInvalidOperation for invalid operation
NewWriter
func NewWriter(algo arccmp.Algorithm, ope Operation, dst io.Writer) (Helper, error)

Creates a Helper for writing data to an io.Writer.

Parameters:

  • algo: Compression algorithm
  • ope: Compress to compress while writing, Decompress to decompress while writing
  • dst: Destination writer

Returns:

  • Helper: Writer Helper instance
  • error: ErrInvalidOperation for invalid operation
Operation Types
type Operation uint8

const (
    Compress   Operation = iota  // Compress data
    Decompress                   // Decompress data
)
Error Codes
var (
    ErrInvalidSource    = errors.New("invalid source")
    ErrClosedResource   = errors.New("closed resource")
    ErrInvalidOperation = errors.New("invalid operation")
)

Error Conditions:

  • ErrInvalidSource: Returned when source is neither io.Reader nor io.Writer, or when calling Read() on a writer/Write() on a reader
  • ErrClosedResource: Returned when writing to a closed deCompressWriter
  • ErrInvalidOperation: Returned for unsupported Operation values (not Compress or Decompress)

Contributing

Contributions are welcome! Please follow these guidelines:

  1. Code Quality

    • Follow Go best practices and idioms
    • Maintain or improve code coverage (target: >80%)
    • Pass all tests including race detector
    • Use gofmt and golint

  2. AI Usage Policy

    • ❌ AI must NEVER be used to generate package code or core functionality
    • ✅ AI assistance is limited to:
      • Testing (writing and improving tests)
      • Debugging (troubleshooting and bug resolution)
      • Documentation (comments, README, TESTING.md)
    • All AI-assisted work must be reviewed and validated by humans

  3. Testing

    • Add tests for new features
    • Use Ginkgo v2 / Gomega for test framework
    • Use gmeasure for performance benchmarks
    • Ensure zero race conditions with go test -race

  4. Documentation

    • Update GoDoc comments for public APIs
    • Add examples for new features
    • Update README.md and TESTING.md if needed

  5. Pull Request Process

    • Fork the repository
    • Create a feature branch
    • Write clear commit messages
    • Ensure all tests pass
    • Update documentation
    • Submit PR with description of changes

Improvements & Security

Current Status

The package is production-ready with no urgent improvements or security vulnerabilities identified.

Code Quality Metrics
  • ✅ 84.5% test coverage (target: >80%)
  • ✅ Zero race conditions detected with -race flag
  • ✅ Thread-safe per instance (one Helper per goroutine)
  • ✅ Memory-safe with proper resource cleanup
  • ✅ Standard interfaces for maximum compatibility
Future Enhancements (Non-urgent)

The following enhancements could be considered for future versions:

  1. Configurable Buffer Sizes: Allow users to specify custom chunk sizes for performance tuning
  2. Progress Callbacks: Optional progress reporting for long-running operations
  3. Compression Level Control: Expose algorithm-specific compression levels
  4. Metrics Integration: Built-in performance metrics collection

These are optional improvements and not required for production use. The current implementation is stable and performant for its intended use cases.

Resources

Package Documentation

  • GoDoc - Complete API reference with function signatures, method descriptions, and runnable examples. Essential for understanding the public interface and usage patterns.

  • doc.go - In-depth package documentation including design philosophy, architecture diagrams, data flow, implementation details, and best practices. Provides comprehensive explanations of internal mechanisms and compression algorithm integration.

  • TESTING.md - Comprehensive test suite documentation covering test architecture, BDD methodology with Ginkgo v2, 84.5% coverage analysis, performance benchmarks, and guidelines for writing new tests. Includes troubleshooting and CI integration examples.

  • github.com/nabbar/golib/archive/compress - Compression algorithm implementations (GZIP, ZLIB, DEFLATE, BZIP2, LZ4, ZSTD, Snappy). The helper package wraps these algorithms with unified streaming interfaces.

  • github.com/nabbar/golib/ioutils/nopwritecloser - io.WriteCloser wrapper utilities used internally for automatic interface conversions. Provides no-op Close() implementations for writers that don't natively support closing.

External References

  • Effective Go - Official Go programming guide covering best practices for interfaces, error handling, and I/O patterns. The helper package follows these conventions for idiomatic Go code.

  • Go I/O Patterns - Official Go blog article explaining pipeline patterns and streaming I/O. Relevant for understanding how helper fits into larger data processing pipelines with compression/decompression.

  • compress Package - Standard library compression packages. The helper package provides a unified interface that works with both standard library and custom compression implementations.

AI Transparency

In compliance with EU AI Act Article 50.4: AI assistance was used for testing, documentation, and bug resolution under human supervision. All core functionality is human-designed and validated.

License

MIT License - See LICENSE file for details.

Copyright (c) 2025 Nicolas JUHEL

Maintained by: Nicolas JUHEL
Package: github.com/nabbar/golib/archive/helper
Version: See releases for versioning

Documentation

Overview

Package helper provides streaming compression and decompression helpers that wrap compression algorithms with unified io.ReadWriteCloser interfaces.

Overview

This package simplifies working with compression algorithms by providing a consistent interface for both compression and decompression operations on streaming data. It acts as an adapter layer between standard io interfaces and compression algorithms from github.com/nabbar/golib/archive/compress.

Key capabilities:

  • Transparent compression/decompression during read or write operations
  • Support for any compression algorithm that implements arccmp.Algorithm
  • Automatic handling of io.Reader/io.Writer to io.ReadCloser/io.WriteCloser conversions
  • Streaming operations with minimal memory overhead
  • Thread-safe for single-instance usage

Design Philosophy

1. Interface Simplicity: Provide a single Helper interface (io.ReadWriteCloser) for all operations 2. Algorithm Agnostic: Work with any compression algorithm without algorithm-specific code 3. Streaming First: Process data in chunks without loading entire streams into memory 4. Resource Safety: Proper cleanup through Close() and automatic wrapper conversions 5. Type Safety: Compile-time guarantees through interface-based design

Architecture

The package provides four internal implementation types, selected automatically based on operation type and I/O direction: 

	┌──────────────────────────────────────────────┐
	│               Helper Interface               │
	│             (io.ReadWriteCloser)             │
	└──────────────────────┬───────────────────────┘
	                       │
	          ┌────────────┴────────────┐
	          │                         │
	   ┌──────▼──────┐           ┌──────▼──────┐
	   │  Compress   │           │  Decompress │
	   └──────┬──────┘           └──────┬──────┘
	     ┌────┴────┐               ┌────┴────┐
      ▼         ▼               ▼         ▼
	 ┌────────┬────────┐       ┌────────┬────────┐
	 │        │        │       │        │        │
	 │ Reader │ Writer │       │ Reader │ Writer │
	 │        │        │       │        │        │
	 └────────┴────────┘       └────────┴────────┘
	   compressReader        deCompressReader
	   compressWriter        deCompressWriter

Component Responsibilities:

  • compressReader: Reads from source, compresses data, provides compressed output
  • compressWriter: Accepts data, compresses it, writes compressed data to destination
  • deCompressReader: Reads compressed data from source, provides decompressed output
  • deCompressWriter: Accepts compressed data, decompresses it, writes to destination

Data Flow

Compression Read Flow (compressReader):

  1. Client calls Read(p) on Helper
  2. Helper reads raw data from source reader
  3. Data is compressed using algorithm's Writer
  4. Compressed data is buffered internally
  5. Compressed chunks are returned to client

Compression Write Flow (compressWriter):

  1. Client calls Write(p) on Helper
  2. Data is passed to algorithm's Writer
  3. Compressed data is written to destination
  4. Close() finalizes compression stream

Decompression Read Flow (deCompressReader):

  1. Client calls Read(p) on Helper
  2. Compressed data is read from source
  3. Algorithm's Reader decompresses data
  4. Decompressed data is returned to client

Decompression Write Flow (deCompressWriter):

  1. Client calls Write(p) on Helper
  2. Compressed data is buffered
  3. Background goroutine reads from buffer
  4. Algorithm's Reader decompresses data
  5. Decompressed data is written to destination

Basic Usage

Compress data while reading from a source: 

import (
    "io"
    "os"
    "github.com/nabbar/golib/archive/compress"
    "github.com/nabbar/golib/archive/helper"
)

func compressFile(inputPath, outputPath string) error {
    input, err := os.Open(inputPath)
    if err != nil {
        return err
    }
    defer input.Close()

    output, err := os.Create(outputPath)
    if err != nil {
        return err
    }
    defer output.Close()

    // Create compression reader
    h, err := helper.NewReader(compress.GZIP, helper.Compress, input)
    if err != nil {
        return err
    }
    defer h.Close()

    // Copy compressed data to output
    _, err = io.Copy(output, h)
    return err
}

Compress data while writing to a destination: 

func writeCompressed(data []byte, outputPath string) error {
    output, err := os.Create(outputPath)
    if err != nil {
        return err
    }
    defer output.Close()

    // Create compression writer
    h, err := helper.NewWriter(compress.GZIP, helper.Compress, output)
    if err != nil {
        return err
    }
    defer h.Close()

    // Write data - it will be compressed automatically
    _, err = h.Write(data)
    return err
}

Decompress data while reading: 

func decompressFile(inputPath string) ([]byte, error) {
    input, err := os.Open(inputPath)
    if err != nil {
        return nil, err
    }
    defer input.Close()

    // Create decompression reader
    h, err := helper.NewReader(compress.GZIP, helper.Decompress, input)
    if err != nil {
        return nil, err
    }
    defer h.Close()

    // Read decompressed data
    return io.ReadAll(h)
}

Decompress data while writing: 

func writeDecompressed(compressedData []byte, outputPath string) error {
    output, err := os.Create(outputPath)
    if err != nil {
        return err
    }
    defer output.Close()

    // Create decompression writer
    h, err := helper.NewWriter(compress.GZIP, helper.Decompress, output)
    if err != nil {
        return err
    }
    defer h.Close()

    // Write compressed data - it will be decompressed automatically
    _, err = h.Write(compressedData)
    return err
}

Automatic Source Type Detection

The New() function automatically determines the operation type based on source type: 

// Source is io.Reader - creates a reader helper
h1, _ := helper.New(compress.GZIP, helper.Compress, input)
data, _ := io.ReadAll(h1)

// Source is io.Writer - creates a writer helper
h2, _ := helper.New(compress.GZIP, helper.Compress, output)
h2.Write(data)

Supported Compression Algorithms

The package works with any algorithm implementing arccmp.Algorithm from github.com/nabbar/golib/archive/compress. Common algorithms include:

  • compress.GZIP: Standard gzip compression (RFC 1952)
  • compress.ZLIB: Zlib compression (RFC 1950)
  • compress.DEFLATE: Raw deflate compression (RFC 1951)
  • compress.BZIP2: Bzip2 compression
  • compress.LZ4: LZ4 fast compression
  • compress.ZSTD: Zstandard compression
  • compress.SNAPPY: Snappy compression

See the compress package documentation for the complete list and algorithm details.

Implementation Details

Thread Safety:

  • Each Helper instance is safe for single-goroutine use
  • Do not share Helper instances across goroutines
  • Create separate instances for concurrent operations

Memory Management:

  • compressReader uses a 512-byte internal buffer (chunkSize)
  • deCompressWriter spawns a single background goroutine
  • bufNoEOF implements backpressure through wait loops
  • Close() must be called to release resources properly

Atomicity:

  • compressReader uses atomic.Bool for close state
  • deCompressWriter uses atomic.Bool for close and run states
  • Prevents race conditions in concurrent read/write scenarios

Error Handling

The package defines three error types:

  • ErrInvalidSource: Returned when source is neither io.Reader nor io.Writer
  • ErrClosedResource: Returned when writing to a closed deCompressWriter
  • ErrInvalidOperation: Returned for unsupported Operation values

Errors from underlying readers/writers and compression algorithms are propagated unchanged to the caller.

Read() and Write() methods of incorrect types (e.g., Read() on a writer) return ErrInvalidSource with zero bytes to clearly indicate the misuse.

Performance Considerations

Buffer Sizing:

  • Default chunkSize is 512 bytes
  • compressReader allocates buffers based on caller's buffer size
  • Larger read buffers reduce overhead for large data transfers

Compression Trade-offs:

  • Compression adds CPU overhead but reduces I/O size
  • Choose algorithms based on speed vs compression ratio needs
  • GZIP: Good balance, widely compatible
  • LZ4/Snappy: Fast compression, lower ratios
  • ZSTD/BZIP2: Slower, higher compression ratios

Decompression Writer Overhead:

  • deCompressWriter uses a background goroutine
  • Adds minimal latency but enables streaming
  • Wait group ensures proper cleanup on Close()

Use Cases

1. Compressed File I/O

Read and write compressed files transparently: 

h, _ := helper.NewWriter(compress.GZIP, helper.Compress, file)
defer h.Close()
h.Write(data) // Automatically compressed

2. Network Data Compression

Compress data before sending over network: 

h, _ := helper.NewWriter(compress.LZ4, helper.Compress, conn)
defer h.Close()
h.Write(payload) // Compressed before transmission

3. Transparent Decompression

Read compressed data as if uncompressed: 

h, _ := helper.NewReader(compress.GZIP, helper.Decompress, compressedStream)
defer h.Close()
scanner := bufio.NewScanner(h) // Works with any io.Reader consumer

4. Format Conversion

Convert between compression formats: 

// Read GZIP, write LZ4
src, _ := helper.NewReader(compress.GZIP, helper.Decompress, gzipFile)
dst, _ := helper.NewWriter(compress.LZ4, helper.Compress, lz4File)
defer src.Close()
defer dst.Close()
io.Copy(dst, src)

5. Streaming Processing

Process compressed data in streams without full decompression: 

h, _ := helper.NewReader(compress.ZSTD, helper.Decompress, compressedData)
defer h.Close()
processInChunks(h) // Read and process incrementally

Limitations

Read/Write Type Restrictions:

  • Compression/decompression readers do not support Write()
  • Compression/decompression writers do not support Read()
  • Attempting wrong operation returns ErrInvalidSource

Concurrency:

  • Helper instances are not safe for concurrent Read/Write
  • Use separate instances per goroutine
  • Close() should be called from a single goroutine

Algorithm Requirements:

  • Algorithm must implement arccmp.Algorithm interface
  • Algorithm must provide working Reader() and Writer() methods
  • Invalid algorithms cause creation errors

Backpressure:

  • deCompressWriter uses time.Sleep for backpressure (100µs)
  • Not suitable for hard real-time systems
  • May cause latency spikes under memory pressure

Best Practices

Always Close Resources: 

h, err := helper.NewWriter(algo, helper.Compress, dst)
if err != nil {
    return err
}
defer h.Close() // Ensures compression finalization

Check Error Returns: 

n, err := h.Write(data)
if err != nil {
    return fmt.Errorf("compression failed: %w", err)
}
if n != len(data) {
    return fmt.Errorf("incomplete write: %d of %d bytes", n, len(data))
}

Choose Appropriate Algorithms: 

// Fast compression for temporary data
helper.NewWriter(compress.LZ4, helper.Compress, dst)

// Maximum compression for archival
helper.NewWriter(compress.ZSTD, helper.Compress, dst)

// Compatibility with external tools
helper.NewWriter(compress.GZIP, helper.Compress, dst)

Handle Large Data Efficiently: 

// Stream large files instead of loading into memory
h, _ := helper.NewReader(algo, helper.Compress, largeFile)
defer h.Close()
io.Copy(destination, h) // Streaming, constant memory

This package integrates with:

  • github.com/nabbar/golib/archive/compress: Compression algorithm implementations
  • github.com/nabbar/golib/ioutils/nopwritecloser: io.WriteCloser wrapper utilities

Standard library compatibility:

  • io.Reader, io.Writer, io.Closer: Fully compatible
  • io.Copy: Works transparently with Helper instances
  • bufio: Can wrap Helper for buffered operations

Testing

The package includes comprehensive tests covering:

  • Constructor functions (New, NewReader, NewWriter)
  • Compression read/write operations
  • Decompression read/write operations
  • Error handling and edge cases
  • Resource cleanup and close semantics
  • Concurrent operation safety
  • Integration with various compression algorithms

See TESTING.md for detailed test documentation and coverage reports.

Example (ChainedCompression)

Example_chainedCompression demonstrates chaining compression operations. Shows converting between different compression algorithms. 

package main

import (
	"bytes"
	"fmt"

	arccmp "github.com/nabbar/golib/archive/compress"
	"github.com/nabbar/golib/archive/helper"
)

func main() {
	original := "Hello, World!"

	var buf1 bytes.Buffer
	h1, _ := helper.NewWriter(arccmp.Gzip, helper.Compress, &buf1)
	h1.Write([]byte(original))
	h1.Close()

	fmt.Printf("Original: %d bytes\n", len(original))
	fmt.Printf("GZIP compressed: %d bytes\n", buf1.Len())
}

Output:

Original: 13 bytes
GZIP compressed: 37 bytes
Example (CompressStream)

Example_compressStream demonstrates compressing a data stream. This is useful for processing large amounts of data efficiently. 

package main

import (
	"bytes"
	"fmt"
	"io"
	"log"
	"strings"

	arccmp "github.com/nabbar/golib/archive/compress"
	"github.com/nabbar/golib/archive/helper"
)

func main() {
	input := strings.NewReader("Line 1\nLine 2\nLine 3\n")
	var output bytes.Buffer

	h, err := helper.NewReader(arccmp.Gzip, helper.Compress, input)
	if err != nil {
		log.Fatal(err)
	}
	defer h.Close()

	n, err := io.Copy(&output, h)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("Compressed %d bytes\n", n)
}

Output:

Compressed 37 bytes
Example (DecompressStream)

Example_decompressStream demonstrates decompressing a data stream. Shows how to transparently decompress while reading. 

package main

import (
	"bytes"
	"fmt"
	"io"
	"log"

	arccmp "github.com/nabbar/golib/archive/compress"
	"github.com/nabbar/golib/archive/helper"
)

func main() {
	original := "Hello"
	var buf bytes.Buffer
	cw, _ := helper.NewWriter(arccmp.Gzip, helper.Compress, &buf)
	cw.Write([]byte(original))
	cw.Close()

	h, err := helper.NewReader(arccmp.Gzip, helper.Decompress, &buf)
	if err != nil {
		log.Fatal(err)
	}
	defer h.Close()

	data, err := io.ReadAll(h)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("Decompressed: %s\n", data)
}

Output:

Decompressed: Hello
Example (ErrorHandling)

Example_errorHandling demonstrates proper error handling with helpers. 

package main

import (
	"fmt"
	"io"
	"strings"

	arccmp "github.com/nabbar/golib/archive/compress"
	"github.com/nabbar/golib/archive/helper"
)

func main() {
	h, err := helper.NewReader(arccmp.Gzip, helper.Compress, strings.NewReader("test"))
	if err != nil {
		fmt.Printf("Creation error: %v\n", err)
		return
	}
	defer h.Close()

	buf := make([]byte, 10)
	n, err := h.Read(buf)
	if err != nil && err != io.EOF {
		fmt.Printf("Read error: %v\n", err)
		return
	}

	fmt.Printf("Read %d bytes successfully\n", n)
}

Output:

Read 10 bytes successfully
Example (InvalidOperation)

Example_invalidOperation demonstrates error handling for invalid operations. 

package main

import (
	"fmt"
	"strings"

	arccmp "github.com/nabbar/golib/archive/compress"
	"github.com/nabbar/golib/archive/helper"
)

func main() {
	_, err := helper.NewReader(arccmp.Gzip, 99, strings.NewReader("test"))
	if err != nil {
		fmt.Printf("Error: %v\n", err)
	}
}

Output:

Error: invalid operation
Example (InvalidSource)

Example_invalidSource demonstrates error handling for invalid sources. 

package main

import (
	"fmt"

	arccmp "github.com/nabbar/golib/archive/compress"
	"github.com/nabbar/golib/archive/helper"
)

func main() {
	_, err := helper.New(arccmp.Gzip, helper.Compress, "invalid source")
	if err != nil {
		fmt.Printf("Error: %v\n", err)
	}
}

Output:

Error: invalid source
Example (LargeData)

Example_largeData demonstrates handling larger data efficiently. 

package main

import (
	"bytes"
	"fmt"
	"strings"

	arccmp "github.com/nabbar/golib/archive/compress"
	"github.com/nabbar/golib/archive/helper"
)

func main() {
	data := strings.Repeat("test data ", 100)

	var buf bytes.Buffer
	h, _ := helper.NewWriter(arccmp.Gzip, helper.Compress, &buf)
	h.Write([]byte(data))
	h.Close()

	ratio := float64(len(data)) / float64(buf.Len())
	fmt.Printf("Compression ratio: %.2fx\n", ratio)
}

Output:

Compression ratio: 23.81x
Example (MultipleWrites)

Example_multipleWrites demonstrates writing data in multiple chunks. 

package main

import (
	"bytes"
	"fmt"
	"io"
	"log"

	arccmp "github.com/nabbar/golib/archive/compress"
	"github.com/nabbar/golib/archive/helper"
)

func main() {
	var buf bytes.Buffer

	h, err := helper.NewWriter(arccmp.Gzip, helper.Compress, &buf)
	if err != nil {
		log.Fatal(err)
	}

	chunks := []string{"Hello", " ", "World", "!"}
	for _, chunk := range chunks {
		h.Write([]byte(chunk))
	}
	h.Close()

	r, _ := helper.NewReader(arccmp.Gzip, helper.Decompress, &buf)
	defer r.Close()

	data, _ := io.ReadAll(r)
	fmt.Printf("%s\n", data)
}

Output:

Hello World!
Example (WriterToReader)

Example_writerToReader demonstrates reading compressed data from a writer operation. 

package main

import (
	"bytes"
	"fmt"
	"io"
	"log"

	arccmp "github.com/nabbar/golib/archive/compress"
	"github.com/nabbar/golib/archive/helper"
)

func main() {
	var buf bytes.Buffer

	w, err := helper.NewWriter(arccmp.Gzip, helper.Compress, &buf)
	if err != nil {
		log.Fatal(err)
	}

	w.Write([]byte("Data to compress"))
	w.Close()

	r, err := helper.NewReader(arccmp.Gzip, helper.Decompress, &buf)
	if err != nil {
		log.Fatal(err)
	}
	defer r.Close()

	data, _ := io.ReadAll(r)
	fmt.Printf("%s\n", data)
}

Output:

Data to compress

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	// ErrInvalidSource is returned when the provided source is not an io.Reader or io.Writer.
	ErrInvalidSource = errors.New("invalid source")
	// ErrClosedResource is returned when attempting to write to a closed resource.
	ErrClosedResource = errors.New("closed resource")
	// ErrInvalidOperation is returned when an unsupported operation is requested.
	ErrInvalidOperation = errors.New("invalid operation")
)

Functions

This section is empty.

Types

type Helper 

type Helper interface {
	io.ReadWriteCloser
}

Helper provides a unified interface for compression and decompression operations. It implements io.ReadWriteCloser to enable transparent compression/decompression in streaming scenarios.

func New 

func New(algo arccmp.Algorithm, ope Operation, src any) (h Helper, err error)

New creates a new Helper instance based on the provided algorithm, operation, and source.

Parameters:

  • algo: The compression algorithm to use (from github.com/nabbar/golib/archive/compress)
  • ope: The operation type (Compress or Decompress)
  • src: The data source, must be either io.Reader or io.Writer

Returns:

  • Helper: A new Helper instance for compression/decompression operations
  • error: ErrInvalidSource if src is neither io.Reader nor io.Writer

The function automatically determines whether to create a reader or writer based on the type of src. For io.Reader, it creates a Helper that can be read from. For io.Writer, it creates a Helper that can be written to.

Example

ExampleNew demonstrates creating a Helper using automatic type detection. This is the simplest way to create a Helper when the source type determines usage. 

package main

import (
	"bytes"
	"fmt"
	"log"

	arccmp "github.com/nabbar/golib/archive/compress"
	"github.com/nabbar/golib/archive/helper"
)

func main() {
	data := []byte("test data")
	src := bytes.NewReader(data)

	h, err := helper.New(arccmp.Gzip, helper.Compress, src)
	if err != nil {
		log.Fatal(err)
	}
	defer h.Close()

	fmt.Printf("Helper created: %T\n", h)
}

Output:

Helper created: *helper.compressReader

func NewReader 

func NewReader(algo arccmp.Algorithm, ope Operation, src io.Reader) (Helper, error)

NewReader creates a new Helper instance for reading data from the provided io.Reader.

Parameters:

  • algo: The compression algorithm to use
  • ope: The operation type (Compress to compress while reading, Decompress to decompress while reading)
  • src: The source reader to read data from

Returns:

  • Helper: A new Helper instance that wraps the source reader
  • error: ErrInvalidOperation if the operation is not Compress or Decompress

When operation is Compress, reading from the returned Helper will compress data from src. When operation is Decompress, reading from the returned Helper will decompress data from src.

Example (Compress)

ExampleNewReader_compress demonstrates compressing data while reading. Data is read from source, compressed, and returned to the caller. 

package main

import (
	"fmt"
	"io"
	"log"
	"strings"

	arccmp "github.com/nabbar/golib/archive/compress"
	"github.com/nabbar/golib/archive/helper"
)

func main() {
	input := strings.NewReader("Hello, World!")

	h, err := helper.NewReader(arccmp.Gzip, helper.Compress, input)
	if err != nil {
		log.Fatal(err)
	}
	defer h.Close()

	compressed, err := io.ReadAll(h)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("Original: 13 bytes, Compressed: %d bytes\n", len(compressed))
}

Output:

Original: 13 bytes, Compressed: 37 bytes
Example (Decompress)

ExampleNewReader_decompress demonstrates decompressing data while reading. Compressed data is read from source and returned decompressed. 

package main

import (
	"bytes"
	"fmt"
	"io"
	"log"

	arccmp "github.com/nabbar/golib/archive/compress"
	"github.com/nabbar/golib/archive/helper"
)

func main() {
	original := "Hello"
	var buf bytes.Buffer
	cw, _ := helper.NewWriter(arccmp.Gzip, helper.Compress, &buf)
	cw.Write([]byte(original))
	cw.Close()

	h, err := helper.NewReader(arccmp.Gzip, helper.Decompress, &buf)
	if err != nil {
		log.Fatal(err)
	}
	defer h.Close()

	data, err := io.ReadAll(h)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("%s\n", data)
}

Output:

Hello

func NewWriter 

func NewWriter(algo arccmp.Algorithm, ope Operation, dst io.Writer) (Helper, error)

NewWriter creates a new Helper instance for writing data to the provided io.Writer.

Parameters:

  • algo: The compression algorithm to use
  • ope: The operation type (Compress to compress while writing, Decompress to decompress while writing)
  • dst: The destination writer to write data to

Returns:

  • Helper: A new Helper instance that wraps the destination writer
  • error: ErrInvalidOperation if the operation is not Compress or Decompress

When operation is Compress, writing to the returned Helper will compress data to dst. When operation is Decompress, writing to the returned Helper will decompress data to dst.

Example (Compress)

ExampleNewWriter_compress demonstrates compressing data while writing. Data written to the Helper is compressed and written to the destination. 

package main

import (
	"bytes"
	"fmt"
	"log"

	arccmp "github.com/nabbar/golib/archive/compress"
	"github.com/nabbar/golib/archive/helper"
)

func main() {
	var buf bytes.Buffer

	h, err := helper.NewWriter(arccmp.Gzip, helper.Compress, &buf)
	if err != nil {
		log.Fatal(err)
	}

	n, err := h.Write([]byte("Hello, World!"))
	if err != nil {
		log.Fatal(err)
	}

	if err := h.Close(); err != nil {
		log.Fatal(err)
	}

	fmt.Printf("Wrote %d bytes, compressed size: %d bytes\n", n, buf.Len())
}

Output:

Wrote 13 bytes, compressed size: 37 bytes
Example (Decompress)

ExampleNewWriter_decompress demonstrates decompressing data while writing. Compressed data written to the Helper is decompressed and written to destination. 

package main

import (
	"bytes"
	"fmt"
	"log"

	arccmp "github.com/nabbar/golib/archive/compress"
	"github.com/nabbar/golib/archive/helper"
)

func main() {
	original := "Hello"
	var compressedBuf bytes.Buffer
	temp, _ := helper.NewWriter(arccmp.Gzip, helper.Compress, &compressedBuf)
	temp.Write([]byte(original))
	temp.Close()
	compressed := compressedBuf.Bytes()

	var buf bytes.Buffer

	h, err := helper.NewWriter(arccmp.Gzip, helper.Decompress, &buf)
	if err != nil {
		log.Fatal(err)
	}

	n, err := h.Write(compressed)
	if err != nil {
		log.Fatal(err)
	}

	if err := h.Close(); err != nil {
		log.Fatal(err)
	}

	fmt.Printf("Wrote %d compressed bytes, decompressed: %s\n", n, buf.String())
}

Output:

Wrote 29 compressed bytes, decompressed: Hello

type Operation 

type Operation uint8

Operation defines the type of operation to perform on data streams. It specifies whether data should be compressed or decompressed. 

const (
	// Compress indicates that data should be compressed using the specified algorithm.
	Compress Operation = iota
	// Decompress indicates that data should be decompressed using the specified algorithm.
	Decompress
)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.