vcdiff

package module
v0.0.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 23, 2025 License: Apache-2.0 Imports: 4 Imported by: 1

README

VCDIFF Go Decoder

A Go implementation of a VCDIFF (RFC 3284) decoder library and command-line tool for efficient binary differencing and compression.

Overview

This repository contains both a Go library and a command-line interface (CLI) for working with VCDIFF delta files. The library provides a VCDIFF decoder that can decode delta files created according to RFC 3284 - The VCDIFF Generic Differencing and Compression Data Format. VCDIFF is a format for expressing one data stream as a variant of another data stream, commonly used for binary differencing, compression, and patch applications.

The CLI tool can be used to apply VCDIFF deltas to reconstruct files, as well as to inspect and analyze the structure of VCDIFF delta files.

Features

  • Go Library: RFC 3284 compliant VCDIFF decoding with clean, idiomatic API
  • Command-Line Tool: Apply deltas and inspect VCDIFF file structure
  • Comprehensive Validation: Support for all VCDIFF instruction types (ADD, COPY, RUN)
  • Address Caching: Efficient decoding with proper address cache implementation
  • Checksum Validation: Full Adler-32 checksum validation support
  • Robust Error Handling: Detailed error messages for debugging malformed files
  • Extensive Testing: 94 test cases with reference implementation validation

Limitations

  • Application Headers: This implementation does not handle application header information
  • Secondary Compression: This decoder does not support secondary compression (e.g., gzip, bzip2)
  • Compatibility: Works with VCDIFF deltas created using xdelta3 -e -S -A (no secondary compression, no application header)

Checksum Support

  • VCD_ADLER32: This implementation detects and parses the VCD_ADLER32 extension (bit 0x04 in window indicator)
  • Non-standard Extension: The Adler-32 checksum is not part of RFC 3284 but is supported by some implementations
  • Validation: Full Adler-32 checksum validation is implemented and performed during decoding
  • Display: Checksums are displayed in the CLI output as Adler32: 0x########

Installation

go get github.com/ably/vcdiff-go

Quick Start

Library Usage
package main

import (
    "fmt"
    "io/ioutil"
    "log"
    
    "github.com/ably/vcdiff-go"
)

func main() {
    // Read the source file
    source, err := ioutil.ReadFile("original.txt")
    if err != nil {
        log.Fatal(err)
    }
    
    // Read the VCDIFF delta file
    deltaData, err := ioutil.ReadFile("changes.vcdiff")
    if err != nil {
        log.Fatal(err)
    }
    
    // Apply the delta to reconstruct the target
    result, err := vcdiff.Decode(source, deltaData)
    if err != nil {
        log.Fatal(err)
    }
    
    fmt.Printf("Decoded result: %s\n", result)
}
CLI Usage

Build the CLI tool:

go build -o vcdiff ./cmd/vcdiff

Apply a VCDIFF delta:

./vcdiff apply -b source.txt -d changes.vcdiff -o result.txt

Inspect a VCDIFF delta file:

./vcdiff parse -d changes.vcdiff

Analyze a VCDIFF delta with source context:

./vcdiff analyze -b source.txt -d changes.vcdiff

API Reference

Core Functions
vcdiff.Decode(source []byte, delta []byte) ([]byte, error)

Decodes a VCDIFF delta file using the provided source data and returns the reconstructed target data.

Parameters:

  • source: The original source data (may be empty for deltas that don't reference source)
  • delta: The VCDIFF delta file data

Returns:

  • Decoded target data as byte slice
  • Error if decoding fails (malformed delta, checksum validation failure, etc.)
vcdiff.NewDecoder(source []byte) Decoder

Creates a new decoder instance with the specified source data. Useful for decoding multiple deltas against the same source.

Parameters:

  • source: The source data for decoding operations

Returns:

  • A Decoder interface that can be used to decode multiple deltas
decoder.Decode(delta []byte) ([]byte, error)

Decodes a single VCDIFF delta using the decoder's source data.

Error Handling

The decoder provides detailed error messages for various failure conditions:

  • Invalid VCDIFF format or magic bytes
  • Malformed varint encoding
  • Out-of-bounds memory access attempts
  • Checksum validation failures
  • Truncated or corrupted delta files

Command-Line Interface

The CLI provides three main commands:

apply - Apply VCDIFF Delta

Applies a VCDIFF delta to a source file to produce the target file.

./vcdiff apply -b <source-file> -d <delta-file> -o <output-file>

Flags:

  • -b, --base: Source/base file path (required)
  • -d, --delta: VCDIFF delta file path (required)
  • -o, --output: Output file path (required)
parse - Inspect VCDIFF Structure

Parses and displays the internal structure of a VCDIFF delta file.

./vcdiff parse -d <delta-file>

Flags:

  • -d, --delta: VCDIFF delta file path (required)

Output includes:

  • Header information (magic bytes, version, flags)
  • Window details (source segments, target length, checksums)
  • Instruction breakdown (ADD, COPY, RUN operations)
  • Address cache usage
  • Data section analysis
analyze - Analyze with Source Context

Analyzes a VCDIFF delta file with access to the source data, providing additional insights.

./vcdiff analyze -b <source-file> -d <delta-file>

Flags:

  • -b, --base: Source/base file path (required)
  • -d, --delta: VCDIFF delta file path (required)

Additional features:

  • Validates actual address references
  • Shows source data context for COPY operations
  • Provides compression ratio analysis

License

This project is licensed under the Apache License 2.0. See the LICENSE file for details.

References

Documentation

Index

Constants

View Source 
const (
	SelfMode = 0
	HereMode = 1
)
View Source 
const (
	VCDIFFMagic1  = 0xD6 // First magic byte: 'V' with high bit set
	VCDIFFMagic2  = 0xC3 // Second magic byte: 'C' with high bit set
	VCDIFFMagic3  = 0xC4 // Third magic byte: 'D' with high bit set
	VCDIFFVersion = 0x00 // Version 0 as defined in RFC 3284
)

VCDIFF magic bytes and version - RFC 3284 Section 4.1

View Source 
const (
	VCDDecompress = 0x01 // VCD_DECOMPRESS: secondary compression used
	VCDCodetable  = 0x02 // VCD_CODETABLE: custom instruction table used
	VCDAppHeader  = 0x04 // VCD_APPHEADER: application header present
)

Header indicator flags - RFC 3284 Section 4.1

View Source 
const (
	VCDSource  = 0x01 // VCD_SOURCE: window uses source data
	VCDTarget  = 0x02 // VCD_TARGET: window uses target data
	VCDAdler32 = 0x04 // VCD_ADLER32: window includes Adler-32 checksum (non-standard extension)
)

Window indicator flags - RFC 3284 Section 4.2

View Source 
const (
	VarintContinuationBit = 0x80 // High bit indicates continuation
	VarintValueMask       = 0x7F // Mask for 7-bit value portion
	VarintMaxShift        = 32   // Maximum shift to prevent overflow
	VarintShiftIncrement  = 7    // Bits to shift for each byte
)

Variable-length integer encoding constants - RFC 3284 Section 2

View Source 
const (
	RunInstructionMin  = 0   // RUN instructions: 0-17
	RunInstructionMax  = 17  // RUN instructions: 0-17
	AddInstructionMin  = 18  // ADD instructions: 18-161
	AddInstructionMax  = 161 // ADD instructions: 18-161
	CopyInstructionMin = 162 // COPY instructions: 162-255
	CopyInstructionMax = 255 // COPY instructions: 162-255
)

Instruction code ranges - RFC 3284 Section 5

View Source 
const (
	NearCacheSize        = 4       // Size of "near" address cache
	SameCacheSize        = 3 * 256 // Size of "same" address cache
	InstructionTableSize = 256     // Size of instruction code table
)

Address cache configuration - RFC 3284 Section 5.3

View Source 
const (
	VCDAdd = iota
	VCDCopy
	VCDRun
	VCDNoop
)
View Source 
const (
	MinimumFileSize = 4 // Minimum VCDIFF file size (magic + version)
)

File format validation constants

Variables

View Source 
var (
	ErrInvalidMagic    = errors.New("invalid VCDIFF magic bytes")
	ErrInvalidVersion  = errors.New("unsupported VCDIFF version")
	ErrInvalidFormat   = errors.New("invalid VCDIFF format")
	ErrCorruptedData   = errors.New("corrupted VCDIFF data")
	ErrInvalidChecksum = errors.New("invalid checksum")
)
View Source 
var DefaultCodeTable = BuildDefaultCodeTable()

DefaultCodeTable is the default code table instance

View Source 
var VCDIFFMagic = [3]byte{VCDIFFMagic1, VCDIFFMagic2, VCDIFFMagic3}

VCDIFFMagic is the expected magic number sequence - RFC 3284 Section 4.1

Functions

func ComputeChecksum 

func ComputeChecksum(initial uint32, data []byte) uint32

ComputeChecksum computes the Adler32 checksum for the given data

func Decode 

func Decode(source []byte, delta []byte) ([]byte, error)

func ReadVarint 

func ReadVarint(reader *bytes.Reader) (uint32, error)

ReadVarint reads a variable-length integer as defined in RFC 3284 Section 2 Follows the same algorithm as the C# MiscUtil reference implementation

Types

type AddressCache 

type AddressCache struct {
	// contains filtered or unexported fields
}

AddressCache manages address encoding/decoding for COPY instructions

func NewAddressCache 

func NewAddressCache(nearSize, sameSize int) *AddressCache

NewAddressCache creates a new address cache with the specified sizes

func (*AddressCache) DecodeAddress 

func (ac *AddressCache) DecodeAddress(here uint32, mode byte) (uint32, error)

DecodeAddress decodes an address using the specified mode

func (*AddressCache) Reset 

func (ac *AddressCache) Reset(addresses []byte)

Reset resets the address cache for a new window

func (*AddressCache) Update 

func (ac *AddressCache) Update(address uint32)

Update updates the address cache with a new address

type Adler32 

type Adler32 struct{}

Adler32 implements the Adler-32 checksum algorithm

type CodeTable 

type CodeTable struct {
	// contains filtered or unexported fields
}

CodeTable represents the VCDIFF instruction code table

func BuildDefaultCodeTable 

func BuildDefaultCodeTable() *CodeTable

BuildDefaultCodeTable creates the default code table specified in RFC 3284

func (*CodeTable) Get 

func (ct *CodeTable) Get(code byte, slot int) Instruction

Get returns the instruction at the given code and slot

type Decoder 

type Decoder interface {
	Decode(delta []byte) ([]byte, error)
}

func NewDecoder 

func NewDecoder(source []byte) Decoder
type Header struct {
	Magic     [3]byte
	Version   byte
	Indicator byte
}

type Instruction 

type Instruction struct {
	Type InstructionType
	Size byte
	Mode byte
}

Instruction represents a single VCDIFF instruction from the code table

func NewInstruction 

func NewInstruction(instrType InstructionType, size byte, mode byte) Instruction

NewInstruction creates a new instruction

type InstructionEntry 

type InstructionEntry struct {
	Type1 byte
	Size1 byte
	Mode1 byte
	Type2 byte
	Size2 byte
	Mode2 byte
}

type InstructionTable 

type InstructionTable struct {
	Entries [InstructionTableSize]InstructionEntry
}

type InstructionType 

type InstructionType byte

InstructionType represents the type of VCDIFF instruction 

const (
	NoOp InstructionType = 0
	Add  InstructionType = 1
	Run  InstructionType = 2
	Copy InstructionType = 3
)

func (InstructionType) String 

func (it InstructionType) String() string

String returns string representation of instruction type

type LegacyInstruction 

type LegacyInstruction struct {
	Type byte
	Size uint32
	Mode byte
	Addr uint32
	Data []byte
}

Legacy instruction type for backwards compatibility

type ParsedDelta 

type ParsedDelta struct {
	Header       Header
	Windows      []Window
	Instructions []RuntimeInstruction
}

func ParseDelta 

func ParseDelta(delta []byte) (*ParsedDelta, error)

ParseDelta parses a VCDIFF delta and returns a structured representation

type RuntimeInstruction 

type RuntimeInstruction struct {
	Type InstructionType
	Size uint32
	Mode byte
	Addr uint32
	Data []byte
}

RuntimeInstruction represents an instruction with resolved size during decoding

type Window 

type Window struct {
	WinIndicator             byte   // Win_Indicator - RFC 3284 Section 4.2
	SourceSegmentSize        uint32 // Source segment size - RFC 3284 Section 4.2
	SourceSegmentPosition    uint32 // Source segment position - RFC 3284 Section 4.2
	TargetWindowLength       uint32 // Length of the target window - RFC 3284 Section 4.3
	DeltaEncodingLength      uint32 // Length of the delta encoding - RFC 3284 Section 4.3
	DeltaIndicator           byte   // Delta_Indicator - RFC 3284 Section 4.3
	DataSectionLength        uint32 // Length of data for ADDs and RUNs - RFC 3284 Section 4.3
	InstructionSectionLength uint32 // Length of instructions section - RFC 3284 Section 4.3
	AddressSectionLength     uint32 // Length of addresses for COPYs - RFC 3284 Section 4.3
	DataSection              []byte // Data section for ADDs and RUNs - RFC 3284 Section 4.3
	InstructionSection       []byte // Instructions and sizes section - RFC 3284 Section 4.3
	AddressSection           []byte // Addresses section for COPYs - RFC 3284 Section 4.3
	Checksum                 uint32 // Adler-32 checksum of target window (VCD_ADLER32 extension)
	HasChecksum              bool   // Whether VCD_ADLER32 bit is set in WinIndicator
}

Source Files

View all Source files

Directories

Path Synopsis
cmd
vcdiff command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.