delta

package module
v0.4.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 28, 2026 License: MIT Imports: 14 Imported by: 0

README

go-rsync

Go Reference Go License

Go implementation of the rsync delta-transfer algorithm — with AVX2/AVX-512 accelerated MD5 (8-way + 16-way SIMD). Rolling checksum matching, block signature generation, file reconstruction, and a binary wire protocol.

Built to power Shuttle, my own Windows-native file sync tool — this library was extracted from Shuttle and is its core delta-transfer engine.

Features

  • 8-way AVX2 MD5 — 8 blocks hashed in parallel via AVX2 assembly (YMM), VPGATHERDD gather load + transpose.
  • 16-way AVX-512 MD5 — 16 blocks in parallel (ZMM), blockSize ≥ 2KB.
  • 8-way AVX2 SHA-256 — same SIMD approach as MD5, for integrity-critical workloads.
  • 3-tier checksum engine — AVX2 (64B/iter) → SSE2 (32B/iter) → pure Go 128B batch. Auto-detects CPU at runtime.
  • Pluggable strong hash — md5, sha256, xxh64, xxh3-128 built-in. Register your own with FastSum support.
  • Binary wire protocol — compact big-endian encoding, ready for SSH pipes.
  • Streaming I/O — GenerateSignatureReader, SearchReader, stream decode — O(blockSize) memory for multi-GB files.
  • Parallel APIs — GenerateSignatureParallel, SearchParallel (5-7× on 8 cores).
  • Rolling checksum — CHAR_OFFSET=31, uint32 natural-overflow arithmetic.
  • Well tested — roundtrip, fuzz, parity (AVX2 vs SSE2 vs pure Go), MD5 8-way + 16-way (AVX2 + AVX-512 vs stdlib).

📦 Install

go get github.com/henryborner/go-rsync

🚀 Quick start

package main

import (
    "os"
    delta "github.com/henryborner/go-rsync"
)

func main() {
    oldFile, _ := os.ReadFile("v1.bin")
    newFile, _ := os.ReadFile("v2.bin")

    blockSize := delta.CalculateBlockSize(int64(len(oldFile)))

    // One-liner: compute delta and reconstruct
    result, err := delta.RoundTrip(oldFile, newFile, blockSize, "md5")
    if err != nil {
        panic(err)
    }
    os.WriteFile("v2_reconstructed.bin", result, 0644)
}

For network use, split into sender/receiver:

// --- Sender side ---
insts := delta.Delta(oldFile, newFile, blockSize, "md5")
delta.WireEncodeInstructions(conn, insts)

// --- Receiver side ---
delta.ApplyDeltaStream(oldFile, conn, outputFile, blockSize, "md5")

📊 Benchmarks

AMD Ryzen 9 8940HX (Zen 4), single-threaded, 1MB data, blockSize≈700:

Benchmark Time Throughput
GenerateSignature (md5) ~345 µs 2.90 GB/s
GenerateSignature (xxh64) ~152 µs 6.57 GB/s
GenerateSignature (xxh3) ~231 µs 4.33 GB/s
GenerateSignature (sha256) ~617 µs 1.62 GB/s
GenerateSignatureParallel (100MB, 32-thread) ~2.63 ms 39.9 GB/s

Intel Xeon Platinum @ 2.5GHz, AVX-512 enabled:

Benchmark Time Throughput
MD5x8_Bulk (AVX2 8-way, 32KB) 11.5 µs 2.84 GB/s
MD5x8Core_Bulk (AVX2 raw, 1000×64B×8) 145 µs 3.54 GB/s
MD5x16Core_Bulk (AVX-512 raw, 1000×64B×16) 94 µs 10.86 GB/s
SignatureReader/10MB_32KB 2.88 ms 3.64 GB/s

Checksum1 (rolling weak checksum) throughput:

Data size AVX2 (Ryzen) AVX2 (Xeon) rsync-AVX2 (Xeon)
1 KB 63 GB/s 37 GB/s 43 GB/s
64 KB 77 GB/s 44 GB/s 44 GB/s
1 MB 77 GB/s 44 GB/s —

64KB within 1.4% of rsync on Xeon. AVX-512 raw MD5 core hits 10.9 GB/s.

Run on your own machine:

go test -bench='BenchmarkSignature$|BenchmarkMD5x8_Bulk|BenchmarkChecksum1' -benchmem .

📁 Package layout

File Purpose
match.go Block matching engine, signature generation
reconstruct.go File reconstruction from instruction stream
wire.go Binary wire protocol encode/decode
registry.go Pluggable strong-hash registry
api.go High-level convenience API: Delta, ApplyDelta, RoundTrip, ApplyDeltaStream
rolling.go Rolling checksum (RollingSum, Checksum1)
rolling_amd64.s AVX2 checksum assembly (64B/iter) + checksum1PackedAVX2
rolling_sse2_amd64.s SSE2/SSSE3 checksum assembly (32B/iter)
rolling_fast_amd64.go Tiered dispatch: AVX2 → SSE2 → Go, inlined Checksum1
rolling_generic.go Portable byte-by-byte checksum (non-amd64 fallback)
md5x8_amd64.s Generated — 64-step unrolled AVX2 MD5 core (8-way)
md5x8_transpose_fast_amd64.s Register-shuffle transpose (~80 vs ~320 VPINSRD instructions)
md5x8_transpose.s Contiguous 8×64→16 transposed YMMs (tail finalization)
md5x8_load_transpose_amd64.s VPINSRD scalar load+transpose (~288 insn/chunk, correct fallback)
md5x8_amd64.go Go-side glue: md5Hash8wayAVX2, md5Finalize8way
md5x8_common.go Shared MD5 constants + md5FinalLane
md5x8_generic.go Stubs for non-amd64 (darwin/arm64)
md5x8_purego.go Correct pure-Go 8-way MD5 reference (fallback / validation)
md5x8_gather_amd64.s VPGATHERDD gather load + transpose (8-way AVX2)
md5x16_amd64.s Generated — AVX-512 MD5 core (16-way, ≥2KB blocks)
md5x16_amd64.go Go-side glue for AVX-512 path
md5x16_gather_amd64.s ZMM VPGATHERDD load+transpose (k-mask reloaded per gather)
sha256x8_amd64.s Generated — AVX2 SHA-256 core (8-way)
sha256x8_amd64.go Go-side glue for SHA-256 8-way
sha256x8_common.go Shared SHA-256 constants
registry_stdlib.go Built-in hash constructors + FastSum implementations
md5x8_test.go Tests: 8-way + 16-way MD5 parity, gather verification
md5x8_rand_test.go Randomized MD5 parity (100 random-length blocks)
delta_test.go Core roundtrip, identical-file, reconstruction tests
fuzz_test.go Fuzz tests: roundtrip, wire encode/decode, checksum parity
gen_md5x8/main.go Code generator for md5x8_amd64.s
gen_md5x16/main.go Code generator for md5x16_amd64.s
gen_sha256x8/main.go Code generator for sha256x8_amd64.s
docs/checksum-engine.md Checksum engine: algorithm, loop structure, conventions, optimization history, SSE2 appendix
docs/md5-simd.md MD5 SIMD reference: architecture, techniques, safety checklist

📚 Documentation

  • Checksum Engine — Rolling checksum algorithm, AVX2/SSE2 loop structure, Go Plan 9 conventions, optimization history (v0→v6), register map, test coverage, performance data.
  • MD5 SIMD — AVX2/AVX-512 parallel MD5 architecture, gather/transpose techniques, assembly notes, safety checklist.
  • rsync — the original C implementation
  • md5-simd — MinIO's AVX2/AVX-512 MD5 (multi-stream server use case)
  • md5vec — first Go AVX2 8-way MD5 (2018, unmaintained)
  • Shuttle — Windows sync tool using this library

Documentation

Overview

Package delta implements the rsync delta-transfer algorithm in Go.

It provides a complete toolkit for binary delta compression: rolling checksum matching (rsync-style), block signature generation, instruction-based file reconstruction, and a binary wire protocol for sending deltas over the network.

Key features

  • Tiered checksum engine: AVX2 (64B/iter, ~77 GB/s) → SSE2 (32B/iter, ~39 GB/s) → pure Go 128B batch on amd64; portable byte-by-byte on other architectures. Automatic CPU dispatch at runtime.
  • Pluggable strong hash: md5, sha256, xxh64, xxh3-128 built-in. Register your own via Register.
  • Streaming I/O: generate signatures from io.Reader, decode instructions one-at-a-time with DecodeInstructionsStream, and reconstruct files with minimal memory.
  • Binary wire format: compact big-endian encoding for signatures and instructions — ready to pipe over SSH or any io.Writer.

Quick start 

// --- Sender side ---
sig := delta.GenerateSignature(oldFile, blockSize, "md5")
eng := delta.NewMatchEngine(blockSize, "md5")
eng.LoadSignature(sig)
insts := eng.Search(newFile)
delta.WireEncodeInstructions(os.Stdout, insts)

// --- Receiver side ---
sig, _ := delta.WireDecodeSignature(os.Stdin)
recon := delta.NewReconstructor(oldFile, blockSize, "md5")
delta.DecodeInstructionsStream(os.Stdin, func(inst delta.MatchResult) error {
    return recon.WriteInstruction(os.Stdout, inst)
})

Relationship to rsync

The rolling checksum uses CHAR_OFFSET=31. (Note: standard rsync builds use CHAR_OFFSET=0; not wire-compatible.) The hash-table matching and block-size goal (~√N blocks) follow the same approach. This package is NOT a wire-compatible rsync client — it provides the building blocks so you can embed delta compression in your own tools.

Performance (amd64, Ryzen 9 8940HX) 

checksum1 throughput:
  AVX2      ~77 GB/s
  SSE2      ~39 GB/s
  pure Go    ~1.9 GB/s

See the project README for full benchmark tables.

Package delta allows registering custom hash algorithms, replacing hard-coded switches. 允许用户注册自定义哈希算法，替换硬编码的 switch。

Example

Example demonstrates a complete roundtrip: signature → match → reconstruct. 

package main

import (
	"bytes"
	"fmt"

	delta "github.com/henryborner/go-rsync"
)

func main() {
	oldFile := []byte("The quick brown fox jumps over the lazy dog. " +
		"This is an example of rsync-style delta transfer.")
	newFile := []byte("The quick brown fox jumps over the lazy dog. " +
		"INSERTED CONTENT HERE. " +
		"This is an example of rsync-style delta transfer.")

	blockSize := int32(32)

	sig := delta.GenerateSignature(oldFile, blockSize, "md5")
	eng := delta.NewMatchEngine(blockSize, "md5")
	eng.LoadSignature(sig)
	instructions := eng.Search(newFile)

	recon := delta.NewReconstructor(oldFile, blockSize, "md5")
	result, _ := recon.Reconstruct(instructions)

	fmt.Println(bytes.Equal(result, newFile))
	fmt.Printf("Transfer: %.0f%%\n",
		float64(eng.LiteralBytes)/float64(len(newFile))*100)

}

Output:
true
Transfer: 73%
Example (Parallel)

Example_parallel demonstrates parallel search for multi-core speedup. 

package main

import (
	"bytes"
	"fmt"

	delta "github.com/henryborner/go-rsync"
)

func main() {
	oldFile := make([]byte, 100*1024) // 100KB
	for i := range oldFile {
		oldFile[i] = byte(i % 256)
	}
	newFile := make([]byte, len(oldFile))
	copy(newFile, oldFile)

	blockSize := delta.CalculateBlockSize(int64(len(oldFile)))
	sig := delta.GenerateSignature(oldFile, blockSize, "md5")

	eng := delta.NewMatchEngine(blockSize, "md5")
	eng.LoadSignature(sig)
	results := eng.SearchParallel(newFile, 4) // 4 workers

	recon := delta.NewReconstructor(oldFile, blockSize, "md5")
	result, _ := recon.Reconstruct(results)

	fmt.Println(bytes.Equal(result, newFile))
	// For identical files, almost all bytes are matched (near-zero transfer).
	fmt.Printf("transfer: %d / %d bytes\n", eng.LiteralBytes, len(newFile))

}

Output:
true
transfer: 200 / 102400 bytes
Example (Streaming)

Example_streaming demonstrates the low-memory streaming API. Instead of loading the entire new file, SearchReader reads from an io.Reader and delivers results via callback as they are found. 

package main

import (
	"bytes"
	"fmt"

	delta "github.com/henryborner/go-rsync"
)

func main() {
	oldFile := []byte("The quick brown fox jumps over the lazy dog. " +
		"This is an example of rsync-style delta transfer.")
	newFile := []byte("The quick brown fox jumps over the lazy dog. " +
		"INSERTED CONTENT HERE. " +
		"This is an example of rsync-style delta transfer.")

	blockSize := int32(32)
	sig := delta.GenerateSignature(oldFile, blockSize, "md5")
	eng := delta.NewMatchEngine(blockSize, "md5")
	eng.LoadSignature(sig)

	var instructions []delta.MatchResult
	eng.SearchReader(bytes.NewReader(newFile), int64(len(newFile)),
		func(mr delta.MatchResult) error {
			cp := mr
			if mr.IsLiteral {
				cp.Data = make([]byte, len(mr.Data))
				copy(cp.Data, mr.Data)
			}
			instructions = append(instructions, cp)
			return nil
		})

	recon := delta.NewReconstructor(oldFile, blockSize, "md5")
	result, _ := recon.Reconstruct(instructions)

	fmt.Println(bytes.Equal(result, newFile))
	fmt.Printf("Transfer: %.0f%%\n",
		float64(eng.LiteralBytes)/float64(len(newFile))*100)

}

Output:
true
Transfer: 73%

Index

Examples

Constants

View Source 
const (
	CHAR_OFFSET = 31
)
View Source 
const CHUNK_SIZE = 32 * 1024

CHUNK_SIZE is the maximum literal chunk size (32KB). Large literals are split into CHUNK_SIZE pieces to ensure the receiver never allocates more than 32KB at once (safe for low-memory servers). CHUNK_SIZE 字面量分块上限，同 rsync 的 32KB。 大字面量拆分为多个 CHUNK_SIZE 块，确保接收端单次缓冲区分配不超过此值。

Variables

This section is empty.

Functions

func ApplyDelta added in v0.3.0 

func ApplyDelta(basisFile []byte, insts []MatchResult, blockSize int32, algo string) ([]byte, error)

ApplyDelta reconstructs newFile from basisFile and an instruction stream. Wraps NewReconstructor + Reconstruct.

ApplyDelta 根据基础文件和指令流重建新文件。 等同于 NewReconstructor + Reconstruct。

func ApplyDeltaStream added in v0.3.0 

func ApplyDeltaStream(basisFile []byte, r io.Reader, w io.Writer, blockSize int32, algo string) error

ApplyDeltaStream reads wire-encoded instructions from r, applies them to basisFile, and writes the reconstructed file to w. Uses streaming I/O so neither the instruction list nor the output is fully buffered in memory. Suitable for piping over SSH.

ApplyDeltaStream 从 r 读取 wire 格式的指令流，应用到 basisFile 并将结果 写入 w。使用流式 I/O，指令和输出都不会完全缓存在内存中。适用于 SSH 管道场景。

func CalculateBlockSize 

func CalculateBlockSize(fileSize int64) int32

func Checksum1 

func Checksum1(data []byte) uint32

Checksum1 zero-alloc fast path for amd64 with AVX2. Inlines the AVX2 checksum directly to avoid the checksum1 wrapper call overhead.

func Checksum1Components added in v0.4.0 

func Checksum1Components(data []byte) (s1, s2 uint32)

Checksum1Components returns the raw (s1, s2) components of the rolling checksum, following the same dispatch as Checksum1. Useful for cross-machine parity verification where the full 32-bit values matter (the packed Checksum1 discards the upper 16 bits of each component).

func DecodeInstructionsStream 

func DecodeInstructionsStream(r io.Reader, fn func(inst MatchResult) error) error

DecodeInstructionsStream decodes instructions one at a time, calling fn for each. The MatchResult.Data passed to fn is only valid during the callback (reuses a single buffer). Do not retain the reference. Designed for low-memory receivers: avoids loading all instructions + literal data into memory at once.

DecodeInstructionsStream 流式解码指令，每读取一条指令就回调 fn。 fn 收到的 MatchResult.Data 仅回调期间有效（使用可复用缓冲区），不得持有引用。 用于低内存接收端：避免将全部指令+字面量数据加载到内存。

func DecodeInstructionsStreamAll added in v0.3.0 

func DecodeInstructionsStreamAll(r io.Reader, fn func(inst MatchResult) error) error

DecodeInstructionsStreamAll reads multiple batches of instructions from r, calling fn for each. Batches are prefixed with a 4-byte big-endian count; a count of 0 signals end-of-stream. This is the receiver-side counterpart to the batched-send pattern used by streaming senders.

DecodeInstructionsStreamAll 从 r 读取多个批次的指令。 每批以 4 字节 count 开头，count=0 表示结束。 与流式发送端的分批发送模式配对使用。

func DeltaFromWire added in v0.3.0 

func DeltaFromWire(r io.Reader, newFile []byte, algo string) ([]MatchResult, *MatchEngine, error)

DeltaFromWire reads a wire-encoded signature from r, then computes the delta against newFile. Returns both the instruction stream and the MatchEngine so callers can inspect LiteralBytes for perfect-match optimisations.

DeltaFromWire 从 r 读取 wire 格式的签名，然后计算与 newFile 的 delta。 同时返回指令流和 MatchEngine，以便调用者检查 LiteralBytes 实现完全匹配优化。

func DeltaFromWireStream added in v0.3.0 

func DeltaFromWireStream(sigR io.Reader, newR io.Reader, newFileSize int64, algo string, fn func(MatchResult) error) error

DeltaFromWireStream reads a wire-encoded signature from sigR, then streams the delta of newFile (from newR) via fn. Neither the signature blocks (beyond the hash table) nor the new file are fully buffered.

DeltaFromWireStream 从 sigR 读取 wire 格式签名，然后流式匹配 newR 中的新文件。 签名块（除哈希表外）和新文件均不全量缓存，内存友好。

func DeltaStream added in v0.3.0 

func DeltaStream(oldFile []byte, newFileR io.Reader, newFileSize int64, blockSize int32, algo string, fn func(MatchResult) error) error

DeltaStream computes the delta between oldFile (in memory) and newFile (streamed from r), delivering each instruction via fn as it is discovered. oldFile is loaded fully (the basis); newFile is read in chunks and never fully buffered. Suitable for servers with limited RAM.

DeltaStream 计算 oldFile 与 newFile 的流式 delta。 oldFile 全部加载（基础文件）；newFile 通过 r 流式读取，不占用额外内存。 每发现一条指令即回调 fn。适合内存受限服务器。

func GetDefault 

func GetDefault() string

GetDefault returns the current default algorithm name. GetDefault 获取默认算法名称。

func ListAlgos 

func ListAlgos() []string

func MD5x8CoreForBench added in v0.2.2 

func MD5x8CoreForBench(x *[16][8]uint32, state *[4][8]uint32)

MD5x8CoreForBench is an exported wrapper for benchmarking the pure AVX2 core.

func MD5x16CoreForBench added in v0.2.2 

func MD5x16CoreForBench(x *[16][16]uint32, state *[4][16]uint32)

MD5x16CoreForBench is an exported wrapper for benchmarking the pure AVX512 core.

func MD5x16available added in v0.2.2 

func MD5x16available() bool

MD5x16available is the exported version for external use.

func NewHash 

func NewHash(algoName string) (hash.Hash, error)

func Register 

func Register(algo ChecksumAlgo)

Register registers a checksum algorithm (thread-safe). Register 注册一个校验和算法（线程安全）。

func RoundTrip added in v0.3.0 

func RoundTrip(oldFile, newFile []byte, blockSize int32, algo string) ([]byte, error)

RoundTrip computes the delta between oldFile and newFile, then applies it to reconstruct the result. Returns an error if the reconstruction does not match newFile byte-for-byte. Useful for testing and validation.

RoundTrip 计算 delta 并重建结果，验证与 newFile 逐字节一致。 主要用于测试和验证。

func SHA256x8Available added in v0.3.0 

func SHA256x8Available() bool

SHA256x8Available is the exported version for external use.

func SetDefault 

func SetDefault(name string) error

SetDefault sets the default algorithm. SetDefault 设置默认算法。

func WireEncodeInstructions 

func WireEncodeInstructions(w io.Writer, insts []MatchResult) error

WireEncodeInstructions encodes an instruction sequence as a binary stream. WireEncodeInstructions 将指令序列编码为二进制流。

func WireEncodeSignature 

func WireEncodeSignature(w io.Writer, sig *Signature) error

WireEncodeSignature encodes a signature as a binary stream. WireEncodeSignature 将签名编码为二进制流。

Types

type BlockSum 

type BlockSum struct {
	Index  int    // block index / 块索引
	Sum1   uint32 // weak rolling checksum / 弱滚动校验和
	Sum2   []byte // strong checksum (MD5/SHA256) / 强校验和
	Offset int64  // byte offset within the file / 块在文件中的偏移
	Length int32  // actual block length (last block may be shorter) / 块长（末块可能更短）
}

BlockSum represents a single block checksum from file B. BlockSum 表示文件 B 中一个块的校验和。

type ChecksumAlgo 

type ChecksumAlgo struct {
	Name   string           // algorithm name e.g. "md5", "sha256", "xxh3" / 算法名称
	New    func() hash.Hash // hash constructor / 哈希构造函数
	Length int              // output bytes / 输出字节数

	// FastSum is an optional zero-allocation fast path for in-memory hashing.
	// It computes the hash of data and writes the result to out[:Length].
	// out is a pre-allocated buffer with at least Length bytes of capacity.
	// Returns out[:Length] (may be a subslice of out).
	// If nil, the hash.Hash path (New+Write+Sum) is used instead.
	// FastSum 是可选的零分配快速哈希路径。直接计算 data 的哈希写入 out[:Length]，
	// out 是预分配的缓冲区（容量 ≥ Length）。返回 out[:Length]。若为 nil，走 hash.Hash 路径。
	FastSum func(out, data []byte) []byte
}

ChecksumAlgo describes a checksum algorithm. ChecksumAlgo 描述一个校验和算法。

func GetAlgo 

func GetAlgo(name string) (ChecksumAlgo, error)

GetAlgo retrieves a registered algorithm. GetAlgo 获取已注册的算法。

func MustGet 

func MustGet(name string) ChecksumAlgo

type MatchEngine 

type MatchEngine struct {

	// stats / 统计
	HashHits     int
	FalseAlarms  int
	Matches      int
	LiteralBytes int64
	// contains filtered or unexported fields
}

MatchEngine is the delta match engine. MatchEngine 增量匹配引擎。

func NewMatchEngine 

func NewMatchEngine(blockSize int32, strongAlgo string) *MatchEngine

NewMatchEngine creates a new match engine. NewMatchEngine 创建匹配引擎。

func (*MatchEngine) LoadSignature 

func (me *MatchEngine) LoadSignature(sig *Signature)

func (*MatchEngine) Search 

func (me *MatchEngine) Search(data []byte) []MatchResult

Search searches source data for matches, returning an instruction sequence. Search 在源数据中搜索匹配，返回指令序列。

func (*MatchEngine) SearchParallel added in v0.3.0 

func (me *MatchEngine) SearchParallel(data []byte, numWorkers int) []MatchResult

SearchParallel splits data into numWorkers overlapping segments and runs Search on each in parallel. The hash table and checksum list are shared read-only across workers; each worker gets its own strong-hash instance.

Results are merged in file order. For identical files, speedup is ~5-7× on 8-core machines.

SearchParallel 将数据拆分为 numWorkers 个重叠段，并行运行 Search。 哈希表和校验和列表跨 worker 只读共享。结果按文件顺序合并。

If numWorkers ≤ 1 or data is too small to split, falls back to Search.

func (*MatchEngine) SearchReader added in v0.3.0 

func (me *MatchEngine) SearchReader(r io.Reader, fileSize int64, fn func(MatchResult) error) error

SearchReader performs streaming delta matching from an io.Reader. Results are delivered via fn callback as they are discovered. MatchResult.Data points into internal buffers and is only valid during fn.

Memory usage is O(blockSize), independent of file size: at most 2×blockSize + CHUNK_SIZE bytes of buffered data. Suitable for low-memory servers and multi-GB files piped over SSH.

SearchReader 从 io.Reader 流式执行增量匹配。 每发现一条匹配/字面量指令就回调 fn，数据仅回调期间有效。

内存占用 O(blockSize)，与文件大小无关：最多缓冲 2×blockSize + CHUNK_SIZE 字节。 适合内存受限的服务器和 SSH 管道传输超大文件。

type MatchResult 

type MatchResult struct {
	IsLiteral bool   // true = literal data, false = block reference / true=字面量, false=块引用
	Data      []byte // literal payload / 字面量数据
	BlockIdx  int    // matched block index / 匹配的块索引
	Offset    int64  // source offset (for ordering) / 来源中的偏移（用于排序）
}

func Delta added in v0.3.0 

func Delta(oldFile, newFile []byte, blockSize int32, algo string) []MatchResult

Delta computes the instruction stream (delta) between oldFile and newFile. It combines GenerateSignature + NewMatchEngine + LoadSignature + Search into a single call.

Delta 计算 oldFile 与 newFile 之间的指令流（delta）。 等同于 GenerateSignature + NewMatchEngine + LoadSignature + Search 的组合。

func WireDecodeInstructions 

func WireDecodeInstructions(r io.Reader) ([]MatchResult, error)

type Reconstructor 

type Reconstructor struct {
	// contains filtered or unexported fields
}

Reconstructor rebuilds files on the receiver side. Reconstructor rebuilds files on the receiver side. Reconstructor 文件重建器（接收端）。

func NewReconstructor 

func NewReconstructor(basisFile []byte, blockSize int32, strongAlgo string, blockLens ...[]int32) *Reconstructor

NewReconstructor creates a reconstructor. blockLens is optional: pass per-block actual lengths (from Signature.BlockSums[].Length) to avoid copying extra bytes from the last block. Pass nil to use blockSize uniformly, truncating by file length. NewReconstructor creates a reconstructor. blockLens is optional: pass per-block actual lengths (from Signature.BlockSums[].Length) to avoid copying extra bytes from the last block. Pass nil to use blockSize uniformly, truncating by file length. NewReconstructor 创建重建器。 blockLens 可选：传入每个块的实际长度（通常来自 Signature.BlockSums[].Length）， 避免最后一个块复制多余字节。传 nil 则全部使用 blockSize 并靠文件长度截断。

func (*Reconstructor) Reconstruct 

func (rc *Reconstructor) Reconstruct(instructions []MatchResult) ([]byte, error)

Reconstruct rebuilds a file from an instruction sequence. Reconstruct rebuilds a file from an instruction sequence. Reconstruct 根据指令序列重建文件。

func (*Reconstructor) Verify 

func (rc *Reconstructor) Verify(result []byte, expectedSum []byte) bool

Verify checks the reconstructed result against an expected strong checksum. Verify checks the reconstructed result against an expected strong checksum. Verify 验证重建结果。

func (*Reconstructor) WriteInstruction 

func (rc *Reconstructor) WriteInstruction(w io.Writer, inst MatchResult) error

WriteInstruction writes the output of a single instruction to w (streaming reconstruction, low memory). Literal instructions write Data directly; match instructions copy the corresponding block from the basis file. WriteInstruction writes the output of a single instruction to w (streaming reconstruction, low memory). Literal instructions write Data directly; match instructions copy the corresponding block from the basis file. WriteInstruction 将单条指令的输出写入 w（流式重建，低内存占用）。 Literal 指令直接写 Data，Match 指令从 basisFile 复制对应块。

type RollingSum 

type RollingSum struct {
	// contains filtered or unexported fields
}

func NewRollingSum 

func NewRollingSum(data []byte) *RollingSum

func (*RollingSum) Reset 

func (rs *RollingSum) Reset(data []byte)

func (*RollingSum) Roll 

func (rs *RollingSum) Roll(oldByte, newByte byte, blockLen int32)

Roll advances the rolling window: removes one old byte, adds one new byte, and updates s1/s2. Uses uint32 natural overflow, no modulo, no type conversions. Roll 滚动窗口：移除一个旧字节，加入一个新字节，更新 s1/s2。 使用 uint32 自然溢出（同 rsync），无取模、无类型转换。

func (*RollingSum) S1 

func (rs *RollingSum) S1() uint32

S1 returns the lower 16 bits of s1. S1 返回 s1 低 16 位。

func (*RollingSum) S2 

func (rs *RollingSum) S2() uint32

S2 returns the lower 16 bits of s2. S2 返回 s2 低 16 位。

func (*RollingSum) Value 

func (rs *RollingSum) Value() uint32

type Signature 

type Signature struct {
	BlockSize int32      // block size / 块大小
	BlockSums []BlockSum // all block checksums / 所有块的校验和
	FileSize  int64      // original file size / 文件原始大小
}

func GenerateSignature 

func GenerateSignature(data []byte, blockSize int32, strongAlgo string) *Signature

GenerateSignature generates block signatures from in-memory data. GenerateSignature 从内存数据生成块签名。

This is a convenience wrapper around GenerateSignatureReader. For large files, prefer GenerateSignatureReader to stream from disk and avoid holding the entire file in memory.

func GenerateSignatureParallel added in v0.2.2 

func GenerateSignatureParallel(data []byte, blockSize int32, strongAlgo string) *Signature

GenerateSignatureParallel generates block signatures from in-memory data using multiple goroutines. Blocks are distributed evenly across workers; each worker computes Checksum1 + strong hash independently.

Uses the same SIMD-accelerated Checksum1 (AVX2/SSE2/pure-Go) and FastSum paths as the serial version. Speedup is near-linear for large files since blocks are independent and data is read-only.

GenerateSignatureParallel 使用多 goroutine 并行生成块签名。 块均匀分配给 worker，每个 worker 独立计算 Checksum1 + 强校验和。 复用 Checksum1 的 SIMD 加速路径。大文件加速比接近线性。

func GenerateSignatureReader 

func GenerateSignatureReader(r io.Reader, fileSize int64, blockSize int32, strongAlgo string) *Signature

GenerateSignatureReader generates block signatures from an io.Reader, avoiding loading the entire file into memory. Uses 8-way AVX2 acceleration for md5 when available. GenerateSignatureReader 从 io.Reader 流式生成块签名，避免全量读入内存。 md5 + AVX512 可用时使用 16 路 SIMD；否则 AVX2 8 路；否则标量回退。

func WireDecodeSignature 

func WireDecodeSignature(r io.Reader) (*Signature, error)

Source Files

View all Source files

Directories

Path Synopsis
Code generator for 4-way MD5 ARM64 NEON.
 Code generator for 4-way MD5 ARM64 NEON.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.