shard-listener

command module
v1.5.9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 12, 2026 License: Apache-2.0 Imports: 31 Imported by: 0

README

shard-listener

CI CodeQL Release Go Reference Go Report Card License

Part of the BSV Layered Multicast open-source project — see the main repository for the full architecture, design docs, and BRC specifications.

Multicast subscriber and forwarder for the BSV transaction sharding pipeline. Receives BRC-124/BRC-128 frames from the shard-proxy multicast fabric, applies shard and subtree filters, forwards matching frames to a configurable downstream consumer over unicast UDP/TCP and/or multicast egress (domain bridging), and performs NORM-inspired NACK-based gap recovery via HashKey/SeqNum per-flow sequence tracking with BRC-126 beacon-discovered retry endpoints and tier-based escalation.

FF05::<shard>:9001  ──multicast──►  shard-listener  ──UDP/TCP──►  downstream :9100
[Control Groups]    ──multicast──►  (BRC-127 SubtreeGroupAnnounce) └─multicast►  FF02::<shard>
                                           │  shard + subtree filter
                                     gap detected
                                           │
                          NACK (BRC-126) ──▼──────►  [nack-addr]:9300
                                           ◄─── ACK / MISS

Features

  • SO_REUSEPORT multi-worker receive with kernel-level source affinity
  • Shard filter — subscribe to a subset of shard groups (empty = all)
  • Subtree filter — include/exclude by 32-byte SubtreeID (BRC-124/BRC-128 frames)
  • BRC-127 subtree group announcements — dynamic group-based filtering via multicast SubtreeGroupAnnounce datagrams with TTL eviction and sender ACLs
  • Gap tracking — per-flow HashKey/SeqNum monotonic counter gap detection (BRC-124/BRC-128)
  • NACK dispatch — 64-byte NACK datagrams (HashKey + StartSeq/EndSeq + SubtreeID) with 16-byte ACK/MISS response handling
  • Beacon discovery — dynamic retry endpoint registry via BRC-126 ADVERT beacons
  • Tier escalation — MISS → immediate advance to next endpoint; ACK → gap cancelled
  • Semaphore-bounded dispatch — concurrent NACK goroutines with configurable limit
  • Egress UDP or TCP with optional strip-header mode (payload-only)
  • Multicast egress — optional domain bridging; re-emits filtered frames onto a separate multicast address space with configurable scope, interface, port, and hop limit
  • BRC-135 header egress — optional; re-emits the 80-byte block header carried in BRC-131 BlockAnnounce frames as a 172-byte BRC-135 frame on a dedicated egress group (0xFFFA) for SPV-style consumers
  • Per-deployment egress TxID dedup — optional shared store (Redis SETNX) to suppress duplicate egress when multiple listeners cover the same shard subset; honours optional ingress courtesy marks from shard-proxy
  • Prometheus + OTLP metrics, /healthz, /readyz
  • Graceful shutdown with configurable drain window
  • SSM (RFC 4607) opt-in — -source-mode=ssm branches every join site (data plane, beacon, subtree-announce) between IPV6_JOIN_GROUP and MCAST_JOIN_SOURCE_GROUP via shard-common/netjoin; per-control-group bootstrap source lists resolve DNS names or IPv6 literals through shard-common/bootstrap.Resolver (fail-closed startup, last-good retention on refresh failures); ASM is the default. See SSM Support Plan.

Quick start

# Subscribe to all groups; forward to localhost:9100 over UDP (ASM, default)
shard-listener \
  -iface eth0 \
  -shard-bits 2 \
  -egress-addr 127.0.0.1:9100

SSM (Posture C — requires PIM-SSM in the fabric and raised net.ipv6.mld_max_msf):

shard-listener \
  -iface eth0 \
  -shard-bits 2 \
  -scope site \
  -source-mode ssm \
  -ssm-bootstrap-manifest shard-manifest-headless.svc.cluster.local \
  -ssm-bootstrap-beacon   retry-endpoint-headless.svc.cluster.local \
  -egress-addr 127.0.0.1:9100

Build

make build       # -> build/shard-listener
make test        # unit tests (race detector)
make test-e2e    # end-to-end tests (see Testing below)
make docker      # build Docker image

Testing

Unit tests cover the filter, NACK tracker, egress, and frame-decode paths:

make test

The E2E suite (test/run-e2e.sh) starts a listener instance, injects frames as unicast UDP directly to the listener's bound port, and verifies delivery via a sink-test-frames UDP sink. Three scenarios are exercised sequentially:

  1. Basic delivery — all frames forwarded; verified by sink count and bsl_frames_forwarded_total Prometheus metric.
  2. Shard filter — -shard-include 0 passes only the group-0 frame.
  3. Strip-header — -strip-header forwards raw payload bytes; sink counts raw datagrams.

The suite requires shard-proxy checked out at ../shard-proxy (for send-test-frames). make test-e2e builds all binaries fresh before running:

make test-e2e

Documentation

Dependencies

  • github.com/lightwebinc/shard-common — frame, shard packages
  • Prometheus client + OpenTelemetry SDK (same versions as proxy)
  • golang.org/x/net/ipv6 — multicast group join
  • golang.org/x/sys/unix — SO_REUSEPORT

Container image

The Dockerfile produces a gcr.io/distroless/static:nonroot image with the single static binary at /usr/local/bin/shard-listener. Configure via Helm values.yaml, container environment variables, or CLI flags.

Helm chart

A Kubernetes Helm chart is published from a dedicated chart repository:

  • Repository: lightwebinc/shard-listener-helm
  • HTTPS: 
    helm repo add bsl https://lightwebinc.github.io/shard-listener-helm
helm install listener bsl/shard-listener
  • OCI: helm install listener oci://ghcr.io/lightwebinc/charts/shard-listener --version 0.1.0

Supports workloadType=Deployment (default) and workloadType=DaemonSet. Every flag accepted by this binary is exposed under .config in the chart's values.yaml. The chart hardcodes NUM_WORKERS=1 to avoid SO_REUSEPORT multicast duplication. See the chart README for the full reference.

License

See LICENSE.

Documentation

Overview

shard-listener receives IPv6 multicast BSV transaction frames, filters by shard and/or subtree, forwards matching frames to a configurable downstream unicast host:port over UDP or TCP, and performs NACK-based gap recovery for BRC-124/BRC-128 frames.

Source Files

View all Source files

Directories

Path Synopsis
cmd
sink-test-frames command
Command sink-test-frames listens on a UDP port and counts incoming BSV transaction frames delivered by shard-listener.
 Command sink-test-frames listens on a UDP port and counts incoming BSV transaction frames delivered by shard-listener.
Package config loads and validates runtime configuration for shard-listener.
 Package config loads and validates runtime configuration for shard-listener.
Package dedup implements a fixed-capacity sliding-window deduplicator for downstream egress in shard-listener.
 Package dedup implements a fixed-capacity sliding-window deduplicator for downstream egress in shard-listener.
Package discovery implements ADVERT beacon decoding and retry endpoint registry management for shard-listener (BRC-126).
 Package discovery implements ADVERT beacon decoding and retry endpoint registry management for shard-listener (BRC-126).
Package egress implements the unicast forwarding sink for shard-listener.
 Package egress implements the unicast forwarding sink for shard-listener.
Package fanout implements decode-once, subscription-indexed egress fan-out for shard-listener.
 Package fanout implements decode-once, subscription-indexed egress fan-out for shard-listener.
Package filter implements allocation-free shard and subtree filtering for shard-listener.
 Package filter implements allocation-free shard and subtree filtering for shard-listener.
Package listener implements the multicast receive workers for shard-listener.
 Package listener implements the multicast receive workers for shard-listener.
Package manifest hosts the listener-side applier that consumes the BRC-139 manifest evaluator's adopted view and translates it into concrete actions on the listener's join state and metrics.
 Package manifest hosts the listener-side applier that consumes the BRC-139 manifest evaluator's adopted view and translates it into concrete actions on the listener's join state and metrics.
Package metrics initialises an OpenTelemetry MeterProvider backed by both a Prometheus exporter (for scraping) and an optional OTLP gRPC exporter (for push-based delivery to any OTel-compatible backend).
 Package metrics initialises an OpenTelemetry MeterProvider backed by both a Prometheus exporter (for scraping) and an optional OTLP gRPC exporter (for push-based delivery to any OTel-compatible backend).
Package nack implements NORM-inspired multicast gap recovery for shard-listener.
 Package nack implements NORM-inspired multicast gap recovery for shard-listener.
Package reassembly implements BRC-130 fragment reassembly for the listener.
 Package reassembly implements BRC-130 fragment reassembly for the listener.
Package subtreegroup provides a thread-safe, time-bounded registry that maps 128-bit group IDs to sets of 32-byte subtree IDs.
 Package subtreegroup provides a thread-safe, time-bounded registry that maps 128-bit group IDs to sets of 32-byte subtree IDs.
Package txdedup provides per-deployment Redis-backed TxID egress dedup for shard-listener, plus optional courtesy marking of the proxy's ingress namespace ("network seen" set).
 Package txdedup provides per-deployment Redis-backed TxID egress dedup for shard-listener, plus optional courtesy marking of the proxy's ingress namespace ("network seen" set).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.