vcdiff

README

vcdiff

A pure-Go, xdelta3-interoperable implementation of the VCDIFF (RFC 3284) delta format. Part of the go-deltasync family of delta-sync tools.

encode  [-s SOURCE] TARGET DELTA   compute the delta from SOURCE to TARGET
decode  [-s SOURCE] DELTA  OUTPUT  apply the delta to SOURCE, producing OUTPUT

VCDIFF is the lingua franca of delta encoding — the vcdiff HTTP content-encoding, Chrome's SDCH, and xdelta3 all speak it. The bytes this tool emits use the standard wire layout (file version 0x00, the RFC default code table, separate data/instructions/addresses sections, no secondary compression), so deltas are interchangeable with xdelta3 — the actively maintained reference implementation. Cross-impl interop is verified both directions against xdelta3 -S (secondary compression disabled) under -tags=compat, alongside a comparative performance test (go test -tags=compat -run Perf) that reports encode/decode throughput and delta size against xdelta3.

Install

go install github.com/go-deltasync/vcdiff/cmd/vcdiff@latest

Usage

vcdiff encode -s old.bin new.bin patch.vcdiff   # delta against a dictionary
vcdiff decode -s old.bin patch.vcdiff out.bin
cmp new.bin out.bin                             # identical

vcdiff encode new.bin patch.vcdiff              # no source: pure compression
vcdiff decode patch.vcdiff out.bin

- means stdin/stdout for the streamable TARGET/DELTA/OUTPUT argument. The optional source (dictionary) is always read fully into memory because COPY instructions reference it at arbitrary offsets.

Library

The package is importable for use in other Go programs (pure Go, no cgo):

import "github.com/go-deltasync/vcdiff"

delta := vcdiff.EncodeBytes(source, target)        // []byte
var out bytes.Buffer
_, err := vcdiff.Decode(source, bytes.NewReader(delta), &out) // out == target

Encode(source, target, w io.Writer) streams the delta, and Decode accepts any standard RFC 3284 delta (including those produced by xdelta3). For batch workloads, reuse a vcdiff.NewEncoder() across calls (see Performance).

How it works

The encoder searches the address space U = SOURCE ‖ TARGET with a greedy longest-match index (4-byte seed hash chains) and turns matches into COPY instructions, repeated bytes into RUN, and everything else into ADD literals. It emits a single window per call using only the always-standard SELF / HERE / near address modes. A COPY that begins in the source window is kept within it (matches that would cross into the target window are capped), so the delta works with decoders — like xdelta3 — that read a source-window copy straight from the source buffer; target-window self-references may still overlap freely.

The decoder is a full RFC 3284 reader: it handles every default-table opcode (including the double ADD+COPY / COPY+ADD forms and size-baked opcodes), all nine address modes (SELF, HERE, four near-cache, three same-cache), VCD_SOURCE and VCD_TARGET windows, the application header, and the optional VCD_CHECKSUM Adler-32. It rejects secondary compression (VCD_DECOMPRESS) and custom code tables (VCD_CODETABLE) with explicit errors.

Current limitations

• Single window at encode time. The encoder loads SOURCE and TARGET fully into memory and emits one window covering the whole target. (The decoder accepts arbitrary multi-window deltas.)
• No secondary compression or custom code tables on output, and no interleaved-format output (open-vcdiff's 0x53 extension). The decoder still reads the standard layout these tools produce by default.

Performance

The encoder indexes the address space with zlib-style head/prev hash chains (flat slices, no per-position allocation) capped at a fixed chain depth, matches 8 bytes at a time, and does not re-index already-copied spans (identical content is still reachable through the source index). The decoder applies a COPY with a single bulk copy when it lies wholly within the source or the produced target, falling back to byte-by-byte only for run-length overlap or boundary-crossing spans.

Protocol

TestPerfVsXdelta3 (under -tags=compat) builds the Go CLI and invokes it and xdelta3 -S as subprocesses on identical files (apples-to-apples; each pays process startup + file I/O). The input is an 8 MiB incompressible source with 32 scattered edits plus an insertion as the target, so COPY matching dominates. Delta size is exact; wall-clock is the best of 3 runs. Reproduce with go test -tags=compat -v -run Perf ./internal/vcdiff/.

Results

Measured on an Apple M4 Max, Go 1.26, xdelta3 3.1.0:

impl	encode	decode	delta bytes
go-vcdiff	272 MB/s	873 MB/s	389
xdelta3 -S	296 MB/s	830 MB/s	450

Encode throughput is at parity with the C reference, decode is a touch faster, and the delta is slightly smaller — close on every axis for a pure-Go, cgo-free implementation. Numbers are machine-dependent and indicative.

For batch workloads (encoding many pairs in one process), reuse an Encoder to amortize the per-call buffer allocation: enc := NewEncoder() then enc.EncodeBytes(src, tgt) in a loop. On 256 KiB pairs this is ~1.5× faster and allocates ~145× less per encode than the one-shot EncodeBytes, with identical output.

License

BSD-3-Clause. See LICENSE.

Documentation

Overview

Package vcdiff is a pure-Go, cgo-free, xdelta3-interoperable implementation of the VCDIFF (RFC 3284) delta format. It encodes a delta from a source ("dictionary") to a target and decodes it back, using the standard wire layout (default code table, separate sections, no secondary compression) so deltas interoperate with xdelta3 and other RFC 3284 tools.

delta := vcdiff.EncodeBytes(source, target)
var out bytes.Buffer
_, _ = vcdiff.Decode(source, bytes.NewReader(delta), &out) // out == target

For batch workloads, reuse an Encoder to amortize buffer allocation:

enc := vcdiff.NewEncoder()
for _, p := range pairs {
	delta := enc.EncodeBytes(p.source, p.target)
	// ...
}

Index

• func Decode(source []byte, delta io.Reader, out io.Writer) (int64, error)
• func Encode(source, target []byte, out io.Writer) error
• func EncodeBytes(source, target []byte) []byte
• type Encoder
  • func NewEncoder() *Encoder

Functions

func Decode

func Decode(source []byte, delta io.Reader, out io.Writer) (int64, error)

Decode applies a VCDIFF delta to source, writes the reconstructed target to out, and returns the number of target bytes written. The delta may use any standard RFC 3284 construct, so deltas produced by xdelta3 are accepted.

func Encode

func Encode(source, target []byte, out io.Writer) error

Encode writes a VCDIFF delta that reconstructs target from source to out. A nil or empty source produces a self-referential ("pure compression") delta.

func EncodeBytes

func EncodeBytes(source, target []byte) []byte

EncodeBytes returns a VCDIFF delta that reconstructs target from source. A nil or empty source produces a self-referential ("pure compression") delta.

Types

type Encoder

type Encoder = impl.Encoder

Encoder is a reusable VCDIFF encoder. Reusing one across many Encode calls in a batch or server workload avoids re-allocating its internal buffers on every call. An Encoder must not be used concurrently.

func NewEncoder

func NewEncoder() *Encoder

NewEncoder returns a reusable Encoder.