README
¶
shard-common
Part of the BSV Layered Multicast open-source project — see the main repository for the full architecture, design docs, and BRC specifications.
Shared protocol primitives for the BSV transaction sharding pipeline. Imported
by shard-proxy, shard-listener, retry-endpoint, subtx-generator, and
shard-manifest.
Packages
| Package | Purpose |
|---|---|
frame |
Wire format codec: BRC-12/124/128 frames, BRC-127 announce, BRC-130 fragments, BRC-131 blocks, BRC-132 subtree data, BRC-134 anchor txs, BRC-135 block headers, BRC-139 shard manifest |
shard |
TxID → IPv6 multicast group derivation (consistent-hash); control groups |
seqhash |
XXH64 per-flow HashKey computation |
sequence |
Per-shard monotonic counters (sync/atomic, zero-alloc) |
cache |
Modular TTL'd KV backend with atomic SetNX (create-only). Implementations: memory, redis (Redis/Valkey/Dragonfly/Cluster), aerospike (Community Edition). Shared by txidset (tier-2) and the retry-endpoint frame store. See docs/cache-backend.md. |
txidset |
Two-tier TxID dedup: tier-1 in-process LRU (zero-alloc hot path) + optional tier-2 cache.Backend SETNX; fail-open on backend errors |
netjoin |
IPv6 multicast Join/Leave — branches IPV6_JOIN_GROUP (ASM) and MCAST_JOIN_SOURCE_GROUP (SSM, RFC 3678) by source-list presence; token-bucket Limiter and Jitter helper for cold-start storm protection at scale. Powers the SSM join sites in every receiver. |
bootstrap |
DNS-resolving source-set tracker for SSM (S,G) bootstrap lists: fail-closed startup, last-good retention on refresh failures, diff-callback for join/leave plumbing. |
manifest |
BRC-139 auto-shard-config consumer: TTL-bounded Registry keyed on (SrcIPv6, InstanceID) and an Evaluator implementing the normative consumer profile (Authoritative quorum, hysteresis, ±1 ShardBits shift bound, manual-pin precedence, divergence telemetry, Successor-block adoption for live re-sharding). Powers proxy and listener auto-config when -manifest-consumer-enabled is set. |
logging |
Unified structured-logging entrypoint for every binary: Init installs a process-wide slog default with the service.{name,instance.id,version} identity triple (shared with OTLP metrics) and returns a runtime *slog.LevelVar; LevelHandler (HTTP /loglevel), InstallSIGHUPToggle, and a Throttle (log-once-then-count) for the data-plane/error log-economy discipline. JSON→stdout, text→stderr. See docs/logging.md. |
hostinfo |
One-shot host inventory (Gather → host.inventory log event): OS/kernel, CPU, memory, per-NIC facts incl. both IPv4 and IPv6 addresses + link speed/driver (Linux sysfs), and the multicast-gating sysctls. Pure-Go (gopsutil); best-effort, never fails startup. |
tracing |
Opt-in OpenTelemetry tracer for control-plane flows. Init returns a no-op tracer when sampling ≤ 0 or no OTLP endpoint (zero cost); otherwise an OTLP/gRPC exporter mirroring the metrics OTLP path. MUST NOT be wired into the packet hot path. |
The shard package also exposes the SSM addressing helpers: SourceMode
(asm|ssm), Scope (site|global), and Prefix(mode, scope) which yields
FF05 (ASM site), FF35 (SSM site), FF3E (SSM global) and rejects ASM
at global scope per RFC 8815.
The frame.ShardManifest codec implements BRC-139's SSM and
live-resharding extensions: Flags.SourceModeSSM (bit 3),
Flags.SourcesValid (bit 4), Flags.PilotOnly (bit 5),
Flags.SuccessorValid (bit 6), SourceCount at bytes [42:44], the
trailing SourceCount × 16-byte sources payload, and the 24-byte
SuccessorBlock carrying (GenerationID, ShardBits, Flags, TransitionEpoch)
for in-flight generation transitions. See the
SSM Support Plan
and the
Automatic Shard Configuration Plan
for the system-level designs.
Documentation
- Wire Protocol Specification — BRC-124/BRC-128 frame format, legacy BRC-12, shard derivation, proxy forward rules
Requirements
- Go 1.25 or later
Build
go build ./...
go test ./...
License
Apache 2.0 — see LICENSE.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package bootstrap maintains a live set of IPv6 source addresses resolved from a list of DNS names and/or IPv6 literals.
|
Package bootstrap maintains a live set of IPv6 source addresses resolved from a list of DNS names and/or IPv6 literals. |
|
Package cache defines a modular, TTL'd key-value backend with an atomic create-only operation (SetNX).
|
Package cache defines a modular, TTL'd key-value backend with an atomic create-only operation (SetNX). |
|
aerospike
Package aerospike provides an Aerospike Community Edition [cache.Backend].
|
Package aerospike provides an Aerospike Community Edition [cache.Backend]. |
|
memory
Package memory provides an in-process [cache.Backend]: zero infrastructure, used for dev/CI and for per-instance frame storage in the retry endpoint.
|
Package memory provides an in-process [cache.Backend]: zero infrastructure, used for dev/CI and for per-instance frame storage in the retry endpoint. |
|
redis
Package redis provides a Redis-protocol [cache.Backend].
|
Package redis provides a Redis-protocol [cache.Backend]. |
|
BRC-135 block header frame encode/decode.
|
BRC-135 block header frame encode/decode. |
|
Package hostinfo gathers a one-shot snapshot of the static facts of the machine a component runs on (OS, CPU, memory, NICs incl.
|
Package hostinfo gathers a one-shot snapshot of the static facts of the machine a component runs on (OS, CPU, memory, NICs incl. |
|
Package logging provides a single, uniform structured-logging entrypoint shared by every binary in the multicast fleet (shard-proxy, shard-listener, retry-endpoint, shard-manifest, subtx-generator).
|
Package logging provides a single, uniform structured-logging entrypoint shared by every binary in the multicast fleet (shard-proxy, shard-listener, retry-endpoint, shard-manifest, subtx-generator). |
|
Package manifest implements the BRC-139 shard-manifest consumer profile: a TTL-bounded registry of received manifests, a quorum + hysteresis evaluator that produces an adopted view of (ShardBits, SourceMode, Successor) fields, and a deduplicated source-set view.
|
Package manifest implements the BRC-139 shard-manifest consumer profile: a TTL-bounded registry of received manifests, a quorum + hysteresis evaluator that produces an adopted view of (ShardBits, SourceMode, Successor) fields, and a deduplicated source-set view. |
|
Package netjoin provides socket-level helpers for joining IPv6 multicast groups, branching between ASM ([IPV6_JOIN_GROUP]) and SSM ([MCAST_JOIN_SOURCE_GROUP], RFC 3678) by the presence of a source list.
|
Package netjoin provides socket-level helpers for joining IPv6 multicast groups, branching between ASM ([IPV6_JOIN_GROUP]) and SSM ([MCAST_JOIN_SOURCE_GROUP], RFC 3678) by the presence of a source list. |
|
Package seqhash provides the hash function used to compute HashKey values in BRC-124/BRC-128 frames.
|
Package seqhash provides the hash function used to compute HashKey values in BRC-124/BRC-128 frames. |
|
Package sequence provides per-shard monotonic sequence counters.
|
Package sequence provides per-shard monotonic sequence counters. |
|
Package shard derives IPv6 multicast group addresses from BSV transaction IDs for deterministic packet-level sharding across the BSV sharding pipeline.
|
Package shard derives IPv6 multicast group addresses from BSV transaction IDs for deterministic packet-level sharding across the BSV sharding pipeline. |
|
Package tracing provides an opt-in OpenTelemetry tracer shared by the fleet.
|
Package tracing provides an opt-in OpenTelemetry tracer shared by the fleet. |
|
Package txidset implements a two-tier TxID claim store used by both the proxy ingress dedup gate and the listener per-deployment egress dedup gate.
|
Package txidset implements a two-tier TxID claim store used by both the proxy ingress dedup gate and the listener per-deployment egress dedup gate. |
Click to show internal directories.
Click to hide internal directories.