delta

README

go-rsync

Go implementation of a binary delta-transfer algorithm — with AVX2/AVX-512 accelerated MD5 (8-way + 16-way SIMD). Rolling checksum matching, block signature generation, file reconstruction, and a binary wire protocol.

Built to power Shuttle, my own Windows-native file sync tool — this library was extracted from Shuttle and is its core delta-transfer engine.

Features

• 8-way AVX2 MD5 — 8 blocks hashed in parallel via AVX2 assembly (YMM), VPGATHERDD gather load + transpose.
• 16-way AVX-512 MD5 — 16 blocks in parallel (ZMM), blockSize ≥ 2KB.
• 8-way AVX2 SHA-256 — same SIMD approach as MD5, for integrity-critical workloads.
• 3-tier checksum engine — AVX2 (64B/iter) → SSE2 (32B/iter) → pure Go 128B batch. Auto-detects CPU at runtime; opt-in AVX-512 (Checksum1AVX512).
• Pluggable strong hash — md5, sha256, xxh64, xxh3-128 built-in. Register your own with FastSum support.
• Binary wire protocol — compact big-endian encoding, ready for SSH pipes.
• Streaming I/O — GenerateSignatureReader, SearchReader, stream decode — O(blockSize) memory for multi-GB files.
• Parallel APIs — GenerateSignatureParallel, SearchParallel, SearchReaderParallel (near-linear speedup on many cores; streaming variant is O(windowSize) memory).
• Rolling checksum — CHAR_OFFSET=31, uint32 natural-overflow arithmetic.
• Well tested — roundtrip, fuzz, parity (AVX2 vs SSE2 vs pure Go), MD5 8-way + 16-way (AVX2 + AVX-512 vs stdlib).

📦 Install

go get github.com/henryborner/go-rsync

🚀 Quick start

package main

import (
    "os"
    delta "github.com/henryborner/go-rsync"
)

func main() {
    oldFile, _ := os.ReadFile("v1.bin")
    newFile, _ := os.ReadFile("v2.bin")

    blockSize := delta.CalculateBlockSize(int64(len(oldFile)))

    // One-liner: compute delta and reconstruct
    result, err := delta.RoundTrip(oldFile, newFile, blockSize, "md5")
    if err != nil {
        panic(err)
    }
    os.WriteFile("v2_reconstructed.bin", result, 0644)
}

For network use, split into sender/receiver:

// --- Sender side ---
insts := delta.Delta(oldFile, newFile, blockSize, "md5")
delta.WireEncodeInstructions(conn, insts)

// --- Receiver side ---
delta.ApplyDeltaStream(oldFile, conn, outputFile, blockSize, "md5")

📊 Benchmarks

All numbers below: AMD Ryzen 9 8940HX (Zen 4, 16 cores / 32 threads), Windows 11, Go 1.26.5 — the same machine used for the full tables in docs/benchmarks.md.

GenerateSignature (1MB data, blockSize=700, single-threaded):

Hash	Path	Time	Throughput
md5	AVX2 8-way	~341 µs	2.93 GB/s
xxh64	cespare/xxhash	~151 µs	6.62 GB/s
xxh3	zeebo/xxh3	~116 µs	8.62 GB/s
sha256	stdlib SHA-NI	~616 µs	1.62 GB/s

GenerateSignatureParallel (md5, 100MB, blockSize≈10KB, 32-thread):

Benchmark	Path	Time	Throughput
GenerateSignatureParallel	AVX2 8-way	~2.37 ms	44.3 GB/s

GenerateSignature prefers the stdlib SHA-NI hardware path when the CPU has SHA-NI; the 8-way AVX2 SHA-256 core is only used on non-SHA-NI CPUs.

MD5 SIMD cores (zero-alloc):

Benchmark	Time	Throughput
MD5x8_Bulk (AVX2 8-way, 32KB)	~7.7 µs	4.26 GB/s
MD5x8Core_Bulk (AVX2 raw, 1000×64B×8)	~81 µs	6.31 GB/s
MD5x16Core_Bulk (AVX-512 raw, 1000×64B×16)	~91 µs	11.24 GB/s

Checksum1 (rolling weak checksum, AVX2 path):

Data size	Throughput
1 KB	78 GB/s
64 KB	109 GB/s
1 MB	106 GB/s

Opt-in AVX-512: Checksum1AVX512(data []byte) forces the single-ZMM 64 B/iter path. Faster than AVX2 only on Intel server Xeons (full-width 512-bit integer units) for blocks ≥ 16 KB (up to +27% at 256 KB); on AMD Zen 4 it is slower, and on CPUs without AVX-512 it falls back to Checksum1. The default Checksum1 is unchanged (auto AVX2 → SSE2 → Go). Not guaranteed on all Intel CPUs — only measured on one Cascade Lake Xeon; benchmark on your own hardware before enabling. See docs/benchmarks.md → "AVX-512 rolling checksum experiment".

Per-size curves, allocation counts (B/op, allocs/op), the streaming SignatureReader results, and the rsync AVX2 comparison (deterministic + random data): docs/benchmarks.md.

Run on your own machine:

go test -bench='BenchmarkSignature$|BenchmarkSignatureXXH64$|BenchmarkSignatureXXH3$|BenchmarkSignatureSHA256$|BenchmarkSignatureParallel$|BenchmarkMD5x8_Bulk$|BenchmarkMD5x8Core_Bulk$|BenchmarkMD5x16Core_Bulk$|BenchmarkChecksum1$|BenchmarkSignatureReader$' -benchmem -count=3 .

📁 Package layout

File	Purpose
match.go	Block matching engine, signature generation
reconstruct.go	File reconstruction from instruction stream
wire.go	Binary wire protocol encode/decode
registry.go	Pluggable strong-hash registry
api.go	High-level convenience API: Delta, ApplyDelta, RoundTrip, ApplyDeltaStream
rolling.go	Rolling checksum (RollingSum, Checksum1)
rolling_amd64.s	AVX2 checksum assembly (64B/iter, conditional prefetch) + checksum1PackedAVX2
rolling_sse2_amd64.s	SSE2/SSSE3 checksum assembly (32B/iter, conditional prefetch)
rolling_avx512_amd64.s	AVX-512 single-ZMM checksum (64B/iter, opt-in)
rolling_avx512_decl.go	Opt-in Checksum1AVX512 with AVX-512 CPU guard
rolling_avx512_test.go	AVX-512 parity test (skips without AVX-512)
rolling_fast_amd64.go	Tiered dispatch: AVX2 → SSE2 → Go, inlined Checksum1
rolling_generic.go	Portable byte-by-byte checksum (non-amd64 fallback)
rolling_fast_arm64.go	ARM64 NEON tiered dispatch: UDOT (dotprod) → VUMULL → Go
rolling_neon_dotprod_arm64.s	UDOT checksum assembly (ARMv8.2+dotprod, 4 insns/64B)
rolling_neon_arm64.s	VUMULL NEON checksum assembly (ARM64 fallback)
md5x8_amd64.s	Generated — 64-step unrolled AVX2 MD5 core (8-way)
md5x8_transpose_fast_amd64.s	Register-shuffle transpose (~91 vs ~288 VPINSRD instructions)
md5x8_load_transpose_amd64.s	VPINSRD scalar load+transpose (~288 insn/chunk, test cross-validation)
md5x8_amd64.go	Go-side glue: md5Hash8wayAVX2, md5Finalize8way
md5x8_common.go	Shared MD5 constants + md5FinalLane
md5x8_generic.go	Stubs for non-amd64 (!amd64)
md5x8_purego.go	Correct pure-Go 8-way MD5 reference (test/validation only)
md5x8_gather_amd64.s	VPGATHERDD gather load + transpose (8-way AVX2)
md5x16_amd64.s	Generated — AVX-512 MD5 core (16-way, ≥2KB blocks)
md5x16_amd64.go	Go-side glue for AVX-512 path
md5x16_gather_amd64.s	ZMM VPGATHERDD load+transpose (k-mask reloaded per gather)
sha256x8_amd64.s	Generated — AVX2 SHA-256 core (8-way)
sha256x8_amd64.go	Go-side glue for SHA-256 8-way
sha256x8_common.go	Shared SHA-256 constants
md5x4_arm64.s	Generated — 4-way NEON MD5 core (64 unrolled steps)
md5x4_arm64.go	Go-side glue for 4-way NEON MD5
md5x4_generic.go	Stubs for non-arm64
md5x4_ref.go	Pure-Go 4-way MD5 reference (validation)
gen_md5x4/main.go	Code generator for md5x4_arm64.s
registry_stdlib.go	Built-in hash constructors + FastSum implementations
md5x8_test.go	Tests: 8-way + 16-way MD5 parity, gather verification
md5x8_rand_test.go	Randomized MD5 parity (100 iterations × 2–8 random-length blocks)
delta_test.go	Core roundtrip, identical-file, reconstruction tests
fuzz_test.go	Fuzz tests: roundtrip, wire encode/decode, checksum parity
gen_md5x8/main.go	Code generator for md5x8_amd64.s
gen_md5x16/main.go	Code generator for md5x16_amd64.s
gen_sha256x8/main.go	Code generator for sha256x8_amd64.s
docs/checksum-engine.md	Checksum engine: algorithm, loop structure, conventions, optimization history, SSE2 appendix
docs/md5-simd.md	MD5 SIMD reference: architecture, techniques, safety checklist
docs/neon-checksum.md	ARM64 NEON rolling checksum: UDOT/VUMULL tiers, WORD encodings, performance
docs/benchmarks.md	Full benchmark tables with per-op allocations (B/op, allocs/op)

📚 Documentation

• Checksum Engine — Rolling checksum algorithm, AVX2/SSE2 loop structure, Go Plan 9 conventions, optimization history (v0→v6), register map, test coverage, performance data.
• MD5 SIMD — AVX2/AVX-512 parallel MD5 architecture, gather/transpose techniques, assembly notes, safety checklist.
• NEON Checksum — ARM64 NEON rolling checksum: UDOT/VUMULL dispatch, WORD encodings, QEMU/CI verification.
• Benchmarks — Complete benchmark tables with per-operation allocation counts.

🔗 Related

• rsync — the original C implementation
• md5-simd — MinIO's AVX2/AVX-512 MD5 (multi-stream server use case)
• md5vec — first Go AVX2 8-way MD5 (2018, unmaintained)
• Shuttle — Windows sync tool using this library

Documentation

Overview

Package delta implements a binary delta-transfer algorithm in Go.

It provides a complete toolkit for binary delta compression: rolling checksum matching, block signature generation, instruction-based file reconstruction, and a binary wire protocol for sending deltas over the network.

Key features

• Tiered checksum engine: AVX2 (64B/iter, ~80 GB/s) → SSE2 (32B/iter, ~39 GB/s) → pure Go 128B batch on amd64; ARM64 NEON (rolling checksum + 4-way MD5); byte-by-byte elsewhere. Automatic CPU dispatch at runtime.
• Pluggable strong hash: md5, sha256, xxh64, xxh3-128 built-in. Register your own via Register.
• Streaming I/O: generate signatures from io.Reader, decode instructions one-at-a-time with DecodeInstructionsStream, and reconstruct files with minimal memory.
• Binary wire format: compact big-endian encoding for signatures and instructions — ready to pipe over SSH or any io.Writer.

Quick start

// --- Sender side ---
sig := delta.GenerateSignature(oldFile, blockSize, "md5")
eng := delta.NewMatchEngine(blockSize, "md5")
eng.LoadSignature(sig)
insts := eng.Search(newFile)
delta.WireEncodeInstructions(os.Stdout, insts)

// --- Receiver side ---
sig, err := delta.WireDecodeSignature(os.Stdin)
if err != nil {
    log.Fatal(err)
}
recon, err := delta.NewReconstructor(oldFile, blockSize, "md5")
delta.DecodeInstructionsStream(os.Stdin, func(inst delta.MatchResult) error {
    return recon.WriteInstruction(os.Stdout, inst)
})

Relationship to rsync

The rolling checksum uses CHAR_OFFSET=31. (Note: standard rsync builds use CHAR_OFFSET=0; not wire-compatible.) The hash-table matching and block-size goal (~√N blocks) follow the same approach. This package is NOT a wire-compatible rsync client — it provides the building blocks so you can embed delta compression in your own tools.

Performance (amd64, Ryzen 9 8940HX)

checksum1 throughput:
  AVX2      ~111 GB/s
  SSE2      ~57 GB/s
  pure Go    ~2.1 GB/s

See the project README for full benchmark tables.

Package delta allows registering custom hash algorithms, replacing hard-coded switches. 允许用户注册自定义哈希算法，替换硬编码的 switch。

Example

Example demonstrates a complete roundtrip: signature → match → reconstruct.

package main

import (
	"bytes"
	"fmt"

	delta "github.com/henryborner/go-rsync"
)

func main() {
	oldFile := []byte("The quick brown fox jumps over the lazy dog. " +
		"This is an example of delta transfer.")
	newFile := []byte("The quick brown fox jumps over the lazy dog. " +
		"INSERTED CONTENT HERE. " +
		"This is an example of delta transfer.")

	blockSize := int32(32)

	sig := delta.GenerateSignature(oldFile, blockSize, "md5")
	eng, _ := delta.NewMatchEngine(blockSize, "md5")
	eng.LoadSignature(sig)
	instructions := eng.Search(newFile)

	recon, _ := delta.NewReconstructor(oldFile, blockSize, "md5")
	result, _ := recon.Reconstruct(instructions)

	fmt.Println(bytes.Equal(result, newFile))
	fmt.Printf("Transfer: %.0f%%\n",
		float64(eng.LiteralBytes)/float64(len(newFile))*100)

}

Output:
true
Transfer: 70%

Example (Parallel)

Example_parallel demonstrates parallel search for multi-core speedup.

package main

import (
	"bytes"
	"fmt"

	delta "github.com/henryborner/go-rsync"
)

func main() {
	oldFile := make([]byte, 100*1024) // 100KB
	for i := range oldFile {
		oldFile[i] = byte(i % 256)
	}
	newFile := make([]byte, len(oldFile))
	copy(newFile, oldFile)

	blockSize := delta.CalculateBlockSize(int64(len(oldFile)))
	sig := delta.GenerateSignature(oldFile, blockSize, "md5")

	eng, _ := delta.NewMatchEngine(blockSize, "md5")
	eng.LoadSignature(sig)
	results := eng.SearchParallel(newFile, 4) // 4 workers

	recon, _ := delta.NewReconstructor(oldFile, blockSize, "md5")
	result, _ := recon.Reconstruct(results)

	fmt.Println(bytes.Equal(result, newFile))
	// For identical files, almost all bytes are matched (near-zero transfer).
	fmt.Printf("transfer: %d / %d bytes\n", eng.LiteralBytes, len(newFile))

}

Output:
true
transfer: 200 / 102400 bytes

Example (Streaming)

Example_streaming demonstrates the low-memory streaming API. Instead of loading the entire new file, SearchReader reads from an io.Reader and delivers results via callback as they are found.

package main

import (
	"bytes"
	"fmt"

	delta "github.com/henryborner/go-rsync"
)

func main() {
	oldFile := []byte("The quick brown fox jumps over the lazy dog. " +
		"This is an example of delta transfer.")
	newFile := []byte("The quick brown fox jumps over the lazy dog. " +
		"INSERTED CONTENT HERE. " +
		"This is an example of delta transfer.")

	blockSize := int32(32)
	sig := delta.GenerateSignature(oldFile, blockSize, "md5")
	eng, _ := delta.NewMatchEngine(blockSize, "md5")
	eng.LoadSignature(sig)

	var instructions []delta.MatchResult
	eng.SearchReader(bytes.NewReader(newFile), int64(len(newFile)),
		func(mr delta.MatchResult) error {
			cp := mr
			if mr.IsLiteral {
				cp.Data = make([]byte, len(mr.Data))
				copy(cp.Data, mr.Data)
			}
			instructions = append(instructions, cp)
			return nil
		})

	recon, _ := delta.NewReconstructor(oldFile, blockSize, "md5")
	result, _ := recon.Reconstruct(instructions)

	fmt.Println(bytes.Equal(result, newFile))
	fmt.Printf("Transfer: %.0f%%\n",
		float64(eng.LiteralBytes)/float64(len(newFile))*100)

}

Output:
true
Transfer: 70%

Index

• Constants
• func ApplyDelta(basisFile []byte, insts []MatchResult, blockSize int32, algo string) ([]byte, error)
• func ApplyDeltaStream(basisFile []byte, r io.Reader, w io.Writer, blockSize int32, algo string) error
• func CalculateBlockSize(fileSize int64) int32
• func Checksum1(data []byte) uint32
• func Checksum1AVX512(data []byte) uint32
• func Checksum1Components(data []byte) (s1, s2 uint32)
• func DecodeInstructionsStream(r io.Reader, fn func(inst MatchResult) error) error
• func DecodeInstructionsStreamAll(r io.Reader, fn func(inst MatchResult) error) error
• func DeltaFromWire(r io.Reader, newFile []byte, algo string) ([]MatchResult, *MatchEngine, error)
• func DeltaFromWireStream(sigR io.Reader, newR io.Reader, newFileSize int64, algo string, ...) error
• func DeltaStream(oldFile []byte, newFileR io.Reader, newFileSize int64, blockSize int32, ...) error
• func GetDefault() string
• func ListAlgos() []string
• func MD5x8CoreForBench(x *[16][8]uint32, state *[4][8]uint32)
• func MD5x16CoreForBench(x *[16][16]uint32, state *[4][16]uint32)
• func MD5x16available() bool
• func NewHash(algoName string) (hash.Hash, error)
• func Register(algo ChecksumAlgo)
• func RoundTrip(oldFile, newFile []byte, blockSize int32, algo string) ([]byte, error)
• func SHA256x8Available() bool
• func SetDefault(name string) error
• func WireEncodeInstructions(w io.Writer, insts []MatchResult) error
• func WireEncodeSignature(w io.Writer, sig *Signature) error
• type BlockSum
• type ChecksumAlgo
  • func GetAlgo(name string) (ChecksumAlgo, error)
  • func MustGet(name string) ChecksumAlgo
• type MatchEngine
  • func NewMatchEngine(blockSize int32, strongAlgo string) (*MatchEngine, error)
  • func (me *MatchEngine) LoadSignature(sig *Signature)
  • func (me *MatchEngine) Search(data []byte) []MatchResult
  • func (me *MatchEngine) SearchParallel(data []byte, numWorkers int) []MatchResult
  • func (me *MatchEngine) SearchReader(r io.Reader, fileSize int64, fn func(MatchResult) error) error
  • func (me *MatchEngine) SearchReaderParallel(r io.Reader, fileSize int64, windowSize, workers int, ...) error
• type MatchResult
  • func Delta(oldFile, newFile []byte, blockSize int32, algo string) ([]MatchResult, error)
  • func WireDecodeInstructions(r io.Reader) ([]MatchResult, error)
• type Reconstructor
  • func NewReconstructor(basisFile []byte, blockSize int32, strongAlgo string, blockLens ...[]int32) (*Reconstructor, error)
  • func (rc *Reconstructor) Reconstruct(instructions []MatchResult) ([]byte, error)
  • func (rc *Reconstructor) Verify(result []byte, expectedSum []byte) bool
  • func (rc *Reconstructor) WriteInstruction(w io.Writer, inst MatchResult) error
• type RollingSum
  • func NewRollingSum(data []byte) *RollingSum
  • func (rs *RollingSum) Reset(data []byte)
  • func (rs *RollingSum) Roll(oldByte, newByte byte, blockLen int32)
  • func (rs *RollingSum) S1() uint32
  • func (rs *RollingSum) S2() uint32
  • func (rs *RollingSum) Value() uint32
• type Signature
  • func GenerateSignature(data []byte, blockSize int32, strongAlgo string) *Signature
  • func GenerateSignatureParallel(data []byte, blockSize int32, strongAlgo string) (*Signature, error)
  • func GenerateSignatureReader(r io.Reader, fileSize int64, blockSize int32, strongAlgo string) (*Signature, error)
  • func WireDecodeSignature(r io.Reader) (*Signature, error)

Examples

• Package
• Package (Parallel)
• Package (Streaming)

Constants

const (
	CHAR_OFFSET = 31
)

const CHUNK_SIZE = 32 * 1024

CHUNK_SIZE is the maximum literal chunk size (32KB). Large literals are split into CHUNK_SIZE pieces to ensure the receiver never allocates more than 32KB at once (safe for low-memory servers). CHUNK_SIZE 字面量分块上限（32KB）。 大字面量拆分为多个 CHUNK_SIZE 块，确保接收端单次缓冲区分配不超过此值。

Functions

func ApplyDelta

func ApplyDelta(basisFile []byte, insts []MatchResult, blockSize int32, algo string) ([]byte, error)

ApplyDelta reconstructs newFile from basisFile and an instruction stream. Wraps NewReconstructor + Reconstruct.

ApplyDelta 根据基础文件和指令流重建新文件。 等同于 NewReconstructor + Reconstruct。

func ApplyDeltaStream

func ApplyDeltaStream(basisFile []byte, r io.Reader, w io.Writer, blockSize int32, algo string) error

ApplyDeltaStream reads wire-encoded instructions from r, applies them to basisFile, and writes the reconstructed file to w. Uses streaming I/O so neither the instruction list nor the output is fully buffered in memory. Suitable for piping over SSH.

ApplyDeltaStream 从 r 读取 wire 格式的指令流，应用到 basisFile 并将结果 写入 w。使用流式 I/O，指令和输出都不会完全缓存在内存中。适用于 SSH 管道场景。

func CalculateBlockSize

func CalculateBlockSize(fileSize int64) int32

func Checksum1

func Checksum1(data []byte) uint32

Checksum1 zero-alloc fast path for amd64 with AVX2. Inlines the AVX2 checksum directly to avoid the checksum1 wrapper call overhead.

func Checksum1AVX512

func Checksum1AVX512(data []byte) uint32

Checksum1AVX512 computes the rolling checksum on the AVX-512 path.

This is an explicit opt-in: Checksum1 auto-dispatches (AVX2 → SSE2 → Go) and is the right choice on most machines. The single-ZMM 64 B/iter loop is measurably faster than AVX2 only on Intel server Xeons with full-width 512-bit integer units, and only for blocks ≥ 16 KB (up to +27% at 256 KB); on AMD Zen 4 it is slower, and on CPUs without AVX-512 it would crash. See docs/benchmarks.md → "AVX-512 rolling checksum experiment".

⚠️ Not guaranteed faster on all Intel CPUs — measured on one Cascade Lake Xeon only; benchmark on your own hardware before enabling.

Falls back to Checksum1 when the CPU lacks AVX-512 or the asm refuses.

func Checksum1Components

func Checksum1Components(data []byte) (s1, s2 uint32)

Checksum1Components returns the raw (s1, s2) components of the rolling checksum, following the same dispatch as Checksum1. On amd64 the 16-bit-lane assembly returns components truncated to 16 bits (only the low 16 bits are meaningful — the wire format packs s1|(s2<<16)); on arm64 and generic platforms they are full 32-bit. Currently unused by production code.

func DecodeInstructionsStream

func DecodeInstructionsStream(r io.Reader, fn func(inst MatchResult) error) error

DecodeInstructionsStream decodes instructions one at a time, calling fn for each. The MatchResult.Data passed to fn is only valid during the callback (reuses a single buffer). Do not retain the reference. Designed for low-memory receivers: avoids loading all instructions + literal data into memory at once.

Note: MatchResult.Offset is set to the instruction index (not file byte offset). The receiver cannot compute file offsets without the block size (which is in the signature, not the instruction stream). Reconstruction functions (Reconstruct, WriteInstruction) do not use Offset — it exists only for sender-side ordering.

DecodeInstructionsStream 流式解码指令，每读取一条指令就回调 fn。 fn 收到的 MatchResult.Data 仅回调期间有效（使用可复用缓冲区），不得持有引用。 用于低内存接收端：避免将全部指令+字面量数据加载到内存。

注意：MatchResult.Offset 设为指令序号（非文件字节偏移）。接收端没有 blockSize 信息（在 signature 中，不在 instruction stream 中），无法计算文件偏移。 Reconstruct / WriteInstruction 不使用 Offset。Offset 仅用于发送端排序。

func DecodeInstructionsStreamAll

func DecodeInstructionsStreamAll(r io.Reader, fn func(inst MatchResult) error) error

DecodeInstructionsStreamAll reads multiple batches of instructions from r, calling fn for each. Batches are prefixed with a 4-byte big-endian count; a count of 0 signals end-of-stream. This is the receiver-side counterpart to the batched-send pattern used by streaming senders.

MatchResult.Offset is set to the instruction index within the batch (not a file byte offset — the receiver cannot compute file offsets without knowing the block size from the signature). Reconstruction functions do not use Offset.

DecodeInstructionsStreamAll 从 r 读取多个批次的指令。 每批以 4 字节 count 开头，count=0 表示结束。 与流式发送端的分批发送模式配对使用。

func DeltaFromWire

func DeltaFromWire(r io.Reader, newFile []byte, algo string) ([]MatchResult, *MatchEngine, error)

DeltaFromWire reads a wire-encoded signature from r, then computes the delta against newFile. Returns both the instruction stream and the MatchEngine so callers can inspect LiteralBytes for perfect-match optimisations.

DeltaFromWire 从 r 读取 wire 格式的签名，然后计算与 newFile 的 delta。 同时返回指令流和 MatchEngine，以便调用者检查 LiteralBytes 实现完全匹配优化。

func DeltaFromWireStream

func DeltaFromWireStream(sigR io.Reader, newR io.Reader, newFileSize int64, algo string, fn func(MatchResult) error) error

DeltaFromWireStream reads a wire-encoded signature from sigR, then streams the delta of newFile (from newR) via fn. Neither the signature blocks (beyond the hash table) nor the new file are fully buffered.

DeltaFromWireStream 从 sigR 读取 wire 格式签名，然后流式匹配 newR 中的新文件。 签名块（除哈希表外）和新文件均不全量缓存，内存友好。

func DeltaStream

func DeltaStream(oldFile []byte, newFileR io.Reader, newFileSize int64, blockSize int32, algo string, fn func(MatchResult) error) error

DeltaStream computes the delta between oldFile (in memory) and newFile (streamed from r), delivering each instruction via fn as it is discovered. oldFile is loaded fully (the basis); newFile is read in chunks and never fully buffered. Suitable for servers with limited RAM.

DeltaStream 计算 oldFile 与 newFile 的流式 delta。 oldFile 全部加载（基础文件）；newFile 通过 r 流式读取，不占用额外内存。 每发现一条指令即回调 fn。适合内存受限服务器。

func GetDefault

func GetDefault() string

GetDefault returns the current default algorithm name. GetDefault 获取默认算法名称。

func ListAlgos

func ListAlgos() []string

func MD5x8CoreForBench

func MD5x8CoreForBench(x *[16][8]uint32, state *[4][8]uint32)

MD5x8CoreForBench is an exported wrapper for benchmarking the pure AVX2 core.

func MD5x16CoreForBench

func MD5x16CoreForBench(x *[16][16]uint32, state *[4][16]uint32)

MD5x16CoreForBench is an exported wrapper for benchmarking the pure AVX512 core.

func MD5x16available

func MD5x16available() bool

MD5x16available is the exported version for external use.

func NewHash

func NewHash(algoName string) (hash.Hash, error)

func Register

func Register(algo ChecksumAlgo)

Register registers a checksum algorithm (thread-safe). Register 注册一个校验和算法（线程安全）。

func RoundTrip

func RoundTrip(oldFile, newFile []byte, blockSize int32, algo string) ([]byte, error)

RoundTrip computes the delta between oldFile and newFile, then applies it to reconstruct the result. Returns an error if the reconstruction does not match newFile byte-for-byte. Useful for testing and validation.

RoundTrip 计算 delta 并重建结果，验证与 newFile 逐字节一致。 主要用于测试和验证。

func SHA256x8Available

func SHA256x8Available() bool

SHA256x8Available is the exported version for external use.

func SetDefault

func SetDefault(name string) error

SetDefault sets the default algorithm. SetDefault 设置默认算法。

func WireEncodeInstructions

func WireEncodeInstructions(w io.Writer, insts []MatchResult) error

WireEncodeInstructions encodes an instruction sequence as a binary stream. WireEncodeInstructions 将指令序列编码为二进制流。

func WireEncodeSignature

func WireEncodeSignature(w io.Writer, sig *Signature) error

WireEncodeSignature encodes a signature as a binary stream. WireEncodeSignature 将签名编码为二进制流。

Types

type BlockSum

type BlockSum struct {
	Index  int    // block index / 块索引
	Sum1   uint32 // weak rolling checksum / 弱滚动校验和
	Sum2   []byte // strong checksum (MD5/SHA256) / 强校验和
	Offset int64  // byte offset within the file / 块在文件中的偏移
	Length int32  // actual block length (last block may be shorter) / 块长（末块可能更短）
}

BlockSum represents a single block checksum from file B. BlockSum 表示文件 B 中一个块的校验和。

type ChecksumAlgo

type ChecksumAlgo struct {
	Name   string           // algorithm name e.g. "md5", "sha256", "xxh3" / 算法名称
	New    func() hash.Hash // hash constructor / 哈希构造函数
	Length int              // output bytes / 输出字节数

	// FastSum is an optional zero-allocation fast path for in-memory hashing.
	// It computes the hash of data and writes the result to out[:Length].
	// out is a pre-allocated buffer with at least Length bytes of capacity.
	// Returns out[:Length] (may be a subslice of out).
	// If nil, the hash.Hash path (New+Write+Sum) is used instead.
	// FastSum 是可选的零分配快速哈希路径。直接计算 data 的哈希写入 out[:Length]，
	// out 是预分配的缓冲区（容量 ≥ Length）。返回 out[:Length]。若为 nil，走 hash.Hash 路径。
	FastSum func(out, data []byte) []byte
}

ChecksumAlgo describes a checksum algorithm. ChecksumAlgo 描述一个校验和算法。

func GetAlgo

func GetAlgo(name string) (ChecksumAlgo, error)

GetAlgo retrieves a registered algorithm. GetAlgo 获取已注册的算法。

func MustGet

func MustGet(name string) ChecksumAlgo

type MatchEngine

type MatchEngine struct {

	// stats / 统计
	HashHits     int
	FalseAlarms  int
	Matches      int
	LiteralBytes int64
	// contains filtered or unexported fields
}

MatchEngine is the delta match engine. MatchEngine 增量匹配引擎。

func NewMatchEngine

func NewMatchEngine(blockSize int32, strongAlgo string) (*MatchEngine, error)

NewMatchEngine creates a new match engine. NewMatchEngine 创建匹配引擎。

func (*MatchEngine) LoadSignature

func (me *MatchEngine) LoadSignature(sig *Signature)

func (*MatchEngine) Search

func (me *MatchEngine) Search(data []byte) []MatchResult

Search searches source data for matches, returning an instruction sequence. Search 在源数据中搜索匹配，返回指令序列。

func (*MatchEngine) SearchParallel

func (me *MatchEngine) SearchParallel(data []byte, numWorkers int) []MatchResult

SearchParallel splits data into numWorkers overlapping segments and runs Search on each in parallel. The hash table and checksum list are shared read-only across workers; each worker gets its own strong-hash instance.

Results are merged in file order. For identical files, speedup is ~5-7× on 8-core machines.

SearchParallel 将数据拆分为 numWorkers 个重叠段，并行运行 Search。 哈希表和校验和列表跨 worker 只读共享。结果按文件顺序合并。

If numWorkers ≤ 1 or data is too small to split, falls back to Search.

func (*MatchEngine) SearchReader

func (me *MatchEngine) SearchReader(r io.Reader, fileSize int64, fn func(MatchResult) error) error

SearchReader performs streaming delta matching from an io.Reader. Results are delivered via fn callback as they are discovered. MatchResult.Data points into internal buffers and is only valid during fn.

Memory usage is O(blockSize), independent of file size: at most max(2×blockSize + CHUNK_SIZE, 4×CHUNK_SIZE) bytes of buffered data (the 4×CHUNK_SIZE floor guards tiny block sizes). Suitable for low-memory servers and multi-GB files piped over SSH.

SearchReader 从 io.Reader 流式执行增量匹配。 每发现一条匹配/字面量指令就回调 fn，数据仅回调期间有效。

内存占用 O(blockSize)，与文件大小无关：最多缓冲 max(2×blockSize + CHUNK_SIZE, 4×CHUNK_SIZE) 字节（4×CHUNK_SIZE 下限保护小块大小）。 适合内存受限的服务器和 SSH 管道传输超大文件。

func (*MatchEngine) SearchReaderParallel

func (me *MatchEngine) SearchReaderParallel(r io.Reader, fileSize int64, windowSize, workers int, fn func(MatchResult) error) error

SearchReaderParallel streams newFile from r and matches it in fixed-size in-memory windows with SearchParallel. Results are delivered in file order via fn; as with SearchReader, literal Data is only valid during the callback and is overwritten by the next window.

Memory is O(windowSize + per-window result slices), independent of fileSize. A match that would cross a window boundary is not found (the window does not read past the boundary); both windows still reconstruct byte-for-byte correctly and simply send that prefix as literals. For 32-64MiB windows this costs at most a block or two of compression per boundary, which is negligible in practice.

SearchReaderParallel 以固定大小的内存窗口流式读取 newFile，并在每个窗口 内用 SearchParallel 并行匹配。结果按文件顺序通过 fn 回调；与 SearchReader 相同，literal Data 仅在回调期间有效，下一窗口会覆盖缓冲。

内存占用为 O(windowSize + 每窗口结果切片)，与文件总大小无关。跨窗口边界 的匹配不会被上一窗口发现，因此该前缀按 literal 发送；重建结果仍逐字节 正确。对 32-64MiB 窗口，每个边界最多损失一两个块的压缩率，实际可忽略。

type MatchResult

type MatchResult struct {
	IsLiteral bool   // true = literal data, false = block reference / true=字面量, false=块引用
	Data      []byte // literal payload / 字面量数据
	BlockIdx  int    // matched block index / 匹配的块索引
	Offset    int64  // source offset (for ordering) / 来源中的偏移（用于排序）
}

func Delta

func Delta(oldFile, newFile []byte, blockSize int32, algo string) ([]MatchResult, error)

Delta computes the instruction stream (delta) between oldFile and newFile. It combines GenerateSignature + NewMatchEngine + LoadSignature + Search into a single call.

Delta 计算 oldFile 与 newFile 之间的指令流（delta）。 等同于 GenerateSignature + NewMatchEngine + LoadSignature + Search 的组合。

func WireDecodeInstructions

func WireDecodeInstructions(r io.Reader) ([]MatchResult, error)

type Reconstructor

type Reconstructor struct {
	// contains filtered or unexported fields
}

Reconstructor rebuilds files on the receiver side. Reconstructor 文件重建器（接收端）。

func NewReconstructor

func NewReconstructor(basisFile []byte, blockSize int32, strongAlgo string, blockLens ...[]int32) (*Reconstructor, error)

NewReconstructor creates a reconstructor. blockLens is optional: pass per-block actual lengths (from Signature.BlockSums[].Length) to avoid copying extra bytes from the last block. Pass nil to use blockSize uniformly, truncating by file length. NewReconstructor 创建重建器。 blockLens 可选：传入每个块的实际长度（通常来自 Signature.BlockSums[].Length）， 避免最后一个块复制多余字节。传 nil 则全部使用 blockSize 并靠文件长度截断。

func (*Reconstructor) Reconstruct

func (rc *Reconstructor) Reconstruct(instructions []MatchResult) ([]byte, error)

Reconstruct rebuilds a file from an instruction sequence. Reconstruct 根据指令序列重建文件。

func (*Reconstructor) Verify

func (rc *Reconstructor) Verify(result []byte, expectedSum []byte) bool

Verify checks the reconstructed result against an expected strong checksum. Verify 验证重建结果。

func (*Reconstructor) WriteInstruction

func (rc *Reconstructor) WriteInstruction(w io.Writer, inst MatchResult) error

WriteInstruction writes the output of a single instruction to w (streaming reconstruction, low memory). Literal instructions write Data directly; match instructions copy the corresponding block from the basis file. WriteInstruction 将单条指令的输出写入 w（流式重建，低内存占用）。 Literal 指令直接写 Data，Match 指令从 basisFile 复制对应块。

type RollingSum

type RollingSum struct {
	// contains filtered or unexported fields
}

func NewRollingSum

func NewRollingSum(data []byte) *RollingSum

func (*RollingSum) Reset

func (rs *RollingSum) Reset(data []byte)

func (*RollingSum) Roll

func (rs *RollingSum) Roll(oldByte, newByte byte, blockLen int32)

Roll advances the rolling window: removes one old byte, adds one new byte, and updates s1/s2. All arithmetic is uint32 with natural overflow — no modulo, no widening, no floating point. Roll 滚动窗口：移除一个旧字节，加入一个新字节，更新 s1/s2。 全部使用 uint32 算术并自然溢出——无取模、无拓宽、无浮点。

func (*RollingSum) S1

func (rs *RollingSum) S1() uint32

S1 returns the lower 16 bits of s1. S1 返回 s1 低 16 位。

func (*RollingSum) S2

func (rs *RollingSum) S2() uint32

S2 returns the lower 16 bits of s2. S2 返回 s2 低 16 位。

func (*RollingSum) Value

func (rs *RollingSum) Value() uint32

type Signature

type Signature struct {
	BlockSize int32      // block size / 块大小
	BlockSums []BlockSum // all block checksums / 所有块的校验和
	FileSize  int64      // original file size / 文件原始大小
}

func GenerateSignature

func GenerateSignature(data []byte, blockSize int32, strongAlgo string) *Signature

GenerateSignature generates block signatures from in-memory data. GenerateSignature 从内存数据生成块签名。

This is a convenience wrapper around GenerateSignatureReader. For large files, prefer GenerateSignatureReader to stream from disk and avoid holding the entire file in memory.

func GenerateSignatureParallel

func GenerateSignatureParallel(data []byte, blockSize int32, strongAlgo string) (*Signature, error)

GenerateSignatureParallel generates block signatures from in-memory data using multiple goroutines. Blocks are distributed evenly across workers; each worker computes Checksum1 + strong hash independently.

Uses the same SIMD-accelerated Checksum1 (AVX2/SSE2/pure-Go) and FastSum paths as the serial version. Speedup is near-linear for large files since blocks are independent and data is read-only.

GenerateSignatureParallel 使用多 goroutine 并行生成块签名。 块均匀分配给 worker，每个 worker 独立计算 Checksum1 + 强校验和。 复用 Checksum1 的 SIMD 加速路径。大文件加速比接近线性。

func GenerateSignatureReader

func GenerateSignatureReader(r io.Reader, fileSize int64, blockSize int32, strongAlgo string) (*Signature, error)

GenerateSignatureReader generates block signatures from an io.Reader, avoiding loading the entire file into memory. Uses 16-way AVX-512, 8-way AVX2, or 4-way NEON acceleration for md5 when available. GenerateSignatureReader 从 io.Reader 流式生成块签名，避免全量读入内存。 md5 + AVX512 可用时使用 16 路 SIMD；否则 AVX2 8 路；否则标量回退。

func WireDecodeSignature

func WireDecodeSignature(r io.Reader) (*Signature, error)