Help us shape the future of Go! Take the 2026 Go Developer Survey

forge

package module
v0.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 4, 2026 License: Apache-2.0 Imports: 20 Imported by: 0

README

go-p2p-forge

A Netty-inspired Go framework for building performant libp2p server applications.

go-p2p-forge extracts common patterns from building production libp2p services into a reusable toolkit — middleware pipelines, buffer pooling, structured lifecycle management, and a server builder that gets you from zero to a working P2P server in minutes.

Features

  • Middleware Pipeline — Composable handler chain with bidirectional processing (inspired by Netty's ChannelPipeline). Each middleware wraps the next, enabling both request pre-processing and response post-processing in a single function.
  • Buffer Pooling — Tiered sync.Pool buffer management (4KB / 64KB / 1MB) for frame I/O, reducing GC pressure under high throughput.
  • Server Builder — Option-pattern server construction (inspired by Netty's ServerBootstrap) with sensible defaults for identity, DHT, GossipSub, relay, and transport.
  • Rate Limiting — Per-peer sliding window rate limiter with single-bucket and dual-bucket (read/write) variants, usable as standalone or as pipeline middleware.
  • Service Lifecycle — Ordered startup with rollback-on-failure, reverse-order shutdown, and a TickerService helper for periodic background tasks.
  • P2P Stack — Configurable libp2p host creation (pluggable transports, Noise security, Yamux muxer, relay, AutoNAT) and a DHT + GossipSub node wrapper with topic management.
  • Test Helpers — MockStream and MockConn for testing pipelines and middleware without a real libp2p network.

Quick Start

package main

import (
    "context"
    "log/slog"
    "os"
    "os/signal"

    forge "github.com/stephanfeb/go-p2p-forge"
    "github.com/stephanfeb/go-p2p-forge/codec"
    "github.com/stephanfeb/go-p2p-forge/middleware"
)

type PingRequest struct {
    Message string `json:"message"`
}

type PingResponse struct {
    Reply string `json:"reply"`
}

func main() {
    logger := slog.New(slog.NewTextHandler(os.Stdout, nil))
    pool := codec.NewBufferPool()
    limiter := middleware.NewSingleBucket(time.Minute, 100)

    // Build a protocol pipeline
    pipeline := forge.NewPipeline(logger,
        middleware.Recovery(),
        forge.JSONResponseWriter(),
        forge.FrameDecodeMiddleware(pool),
        middleware.RateLimitMiddleware(limiter),
        forge.JSONDeserialize[PingRequest](),
        func(sc *forge.StreamContext, next func()) {
            req := sc.Request.(*PingRequest)
            sc.Response = &PingResponse{Reply: "pong: " + req.Message}
        },
    )

    // Build and start the server
    srv := forge.NewServer(
        forge.WithPort(9000),
        forge.WithLogger(logger),
    )
    srv.Handle("/my-app/ping/1.0.0", pipeline.StreamHandler())

    ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt)
    defer cancel()

    if err := srv.ListenAndServe(ctx); err != nil {
        logger.Error("server error", "error", err)
    }
}

Architecture

                    Inbound                          Outbound
                    ───────►                         ◄───────

              ┌──────────────┐
              │   Recovery   │  catches panics
              └──────┬───────┘
                     │
              ┌──────▼───────┐
              │  Response    │  marshals sc.Response after handler returns
              │  Writer      │
              └──────┬───────┘
                     │
              ┌──────▼───────┐
              │  Frame       │  reads length-prefixed frame into sc.RawBytes
              │  Decode      │  (pooled buffers)
              └──────┬───────┘
                     │
              ┌──────▼───────┐
              │  Rate        │  per-peer sliding window check
              │  Limiter     │
              └──────┬───────┘
                     │
              ┌──────▼───────┐
              │  JSON        │  unmarshals sc.RawBytes → sc.Request
              │  Deserialize │
              └──────┬───────┘
                     │
              ┌──────▼───────┐
              │  Handler     │  your business logic
              │  (app code)  │  sets sc.Response
              └──────────────┘

Each middleware calls next() to proceed. Code before next() runs on the inbound path; code after runs on the outbound path. Not calling next() short-circuits the pipeline (e.g., rate limit rejection).

Packages

Package Description
forge (root) Server, Pipeline, StreamContext, Middleware, codec middleware, config
codec Length-prefixed frame codec, tiered buffer pool
middleware Rate limiter, panic recovery, pipeline middleware adapters
service Lifecycle manager, TickerService for periodic tasks
host libp2p host creation, Ed25519 identity management
node DHT + GossipSub wrapper, topic management
forgetest MockStream, MockConn, GenerateTestPeerID()

Testing

go test ./...

All packages include tests. Use forgetest.MockStream to test your pipelines without a network:

func TestMyHandler(t *testing.T) {
    reqData, _ := json.Marshal(MyRequest{Value: 42})
    stream := forgetest.NewMockStreamWithFrame(forgetest.GenerateTestPeerID(), reqData)

    pipeline.HandleStream(stream)

    respData, _ := stream.ReadResponseFrame()
    var resp MyResponse
    json.Unmarshal(respData, &resp)
    // assert on resp...
}

License

MIT

Documentation

Overview

Package forge provides a framework for building performant libp2p server applications.

Index

Constants

View Source
const (
	// DefaultIdleTimeout is how long a pipeline waits for a peer to make
	// progress on a frame before abandoning the stream. It is refreshed as
	// bytes arrive, so it bounds stalls, not transfer time.
	DefaultIdleTimeout = 30 * time.Second

	// DefaultRequestTimeout bounds the whole life of an inbound stream: read,
	// handler and response. It is the backstop against a peer that trickles
	// one byte inside every idle window, and it is the deadline handlers see
	// on sc.Ctx.
	DefaultRequestTimeout = 5 * time.Minute
)

Variables

View Source
var (
	// ErrPanic indicates a handler panicked and was recovered by the Recovery middleware.
	ErrPanic = errors.New("handler panicked")

	// ErrRateLimited indicates a request was rejected due to rate limiting.
	ErrRateLimited = errors.New("rate limit exceeded")

	// ErrServerNotStarted indicates an operation requires the server to be started.
	ErrServerNotStarted = errors.New("server not started")
)

Sentinel errors used across the framework.

Functions

func LoadConfigFromFile

func LoadConfigFromFile(path string, base *Config) error

LoadConfigFromFile reads a YAML config file and applies non-zero values on top of the provided base config.

func Service

func Service[T any](r *Registry, key string) (T, bool)

Service retrieves a typed value from the registry. Returns the zero value of T and false if the key is missing or the type does not match.

func ServiceFrom

func ServiceFrom[T any](sc *StreamContext, key string) (T, bool)

ServiceFrom retrieves a typed server-level singleton from the StreamContext's registry. Returns the zero value and false if the registry is nil, key is missing, or the type assertion fails.

Types

type Codec

type Codec interface {
	// Marshal encodes a value into bytes.
	Marshal(v any) ([]byte, error)
	// Unmarshal decodes bytes into the provided pointer.
	Unmarshal(data []byte, v any) error
	// ContentType returns a human-readable name for logging (e.g., "json", "protobuf").
	ContentType() string
}

Codec defines a symmetric encoding/decoding interface for stream payloads. Implement this interface to support custom serialization formats (e.g., Protobuf).

type Config

type Config struct {
	// Host configuration (network, relay, transport)
	Host forgehost.Config `yaml:"host"`

	// Node configuration (DHT, PubSub)
	Node node.Config `yaml:"node"`

	// Identity
	IdentityFile  string `yaml:"identity_file"`
	DataDirectory string `yaml:"data_directory"`

	// Rate limiting defaults (can be overridden per-protocol)
	RateLimitWindow      time.Duration `yaml:"rate_limit_window"`
	MaxRequestsPerWindow int           `yaml:"max_requests_per_window"`

	// ShutdownTimeout is the maximum time to wait for active streams to drain
	// during graceful shutdown. Zero means no waiting (immediate close).
	ShutdownTimeout time.Duration `yaml:"shutdown_timeout"`
}

Config holds framework-level server configuration. Application-specific config (storage, domain logic) lives in the application.

func DefaultConfig

func DefaultConfig() *Config

DefaultConfig returns sensible defaults for a forge server.

func (*Config) Validate

func (c *Config) Validate() error

Validate checks the configuration for errors.

type FrameIterator

type FrameIterator interface {
	Next() ([]byte, error)
}

FrameIterator produces a sequence of response frames. Implementations return each frame's bytes via Next(). Return io.EOF to signal the end of the sequence.

When sc.Response implements FrameIterator, ResponseWriterMiddleware writes each frame individually instead of marshaling a single object.

type JSONCodec

type JSONCodec struct{}

JSONCodec implements Codec using encoding/json.

func (JSONCodec) ContentType

func (JSONCodec) ContentType() string

ContentType returns "json".

func (JSONCodec) Marshal

func (JSONCodec) Marshal(v any) ([]byte, error)

Marshal encodes v as JSON.

func (JSONCodec) Unmarshal

func (JSONCodec) Unmarshal(data []byte, v any) error

Unmarshal decodes JSON data into v.

type MetricsCollector

type MetricsCollector interface {
	// StreamStarted is called when a stream begins processing.
	StreamStarted(protocol string, peerID string)

	// StreamCompleted is called when a stream finishes processing.
	StreamCompleted(protocol string, peerID string, durationMs float64, err error)

	// RateLimitHit is called when a request is rejected by the rate limiter.
	RateLimitHit(peerID string)

	// ActiveStreams reports the current number of in-flight streams.
	ActiveStreams(count int64)

	// BufferPoolStats reports buffer pool hit/miss counters.
	BufferPoolStats(hits, misses int64)
}

MetricsCollector is the interface framework users implement to capture operational metrics using their preferred backend (Prometheus, StatsD, etc.).

type Middleware

type Middleware func(sc *StreamContext, next func())

Middleware processes a StreamContext and calls next to continue the chain. If next is not called, the pipeline is short-circuited (e.g., rate limit rejection). Middleware can perform both pre-processing and post-processing by executing code before and after calling next(), similar to Netty's ChannelHandler.

func DeserializeMiddleware

func DeserializeMiddleware[T any](c Codec) Middleware

DeserializeMiddleware returns a middleware that unmarshals sc.RawBytes into type T using the provided codec, storing the result in sc.Request.

func FrameDecodeMiddleware

func FrameDecodeMiddleware(pool *codec.BufferPool) Middleware

FrameDecodeMiddleware reads a length-prefixed frame from the stream and stores the raw bytes in sc.RawBytes. Uses buffer pooling to reduce GC pressure.

func JSONDeserialize

func JSONDeserialize[T any]() Middleware

JSONDeserialize returns a middleware that unmarshals sc.RawBytes into type T and stores it in sc.Request. Uses Go generics for type safety. This is equivalent to DeserializeMiddleware[T](JSONCodec{}).

func JSONResponseWriter

func JSONResponseWriter() Middleware

JSONResponseWriter returns a middleware that marshals sc.Response and writes it as a length-prefixed frame after the downstream handler completes. This is equivalent to ResponseWriterMiddleware(JSONCodec{}).

Supports both single-frame responses and multi-frame responses via FrameIterator.

func ResponseWriterMiddleware

func ResponseWriterMiddleware(c Codec) Middleware

ResponseWriterMiddleware returns a middleware that marshals sc.Response using the provided codec and writes it as a length-prefixed frame after the downstream handler completes.

If sc.Response implements FrameIterator, each frame is written individually, enabling multi-frame compound responses (e.g., metadata + N result frames). The FrameIterator produces raw bytes, so the codec is not involved in the multi-frame path — marshaling is the iterator's responsibility.

Partial-stream behavior: if an error occurs mid-stream (iterator failure on frame N, or a network write error), the client will have received frames 0 through N-1 with no trailing sentinel. sc.Err is set so upstream middleware can detect the incomplete response. Callers that need guaranteed atomicity should pre-marshal all frames before setting sc.Response.

type NoopMetrics

type NoopMetrics struct{}

NoopMetrics is the default no-op MetricsCollector that discards all metrics.

func (NoopMetrics) ActiveStreams

func (NoopMetrics) ActiveStreams(int64)

func (NoopMetrics) BufferPoolStats

func (NoopMetrics) BufferPoolStats(int64, int64)

func (NoopMetrics) RateLimitHit

func (NoopMetrics) RateLimitHit(string)

func (NoopMetrics) StreamCompleted

func (NoopMetrics) StreamCompleted(string, string, float64, error)

func (NoopMetrics) StreamStarted

func (NoopMetrics) StreamStarted(string, string)

type Option

type Option func(*Server)

Option configures a Server.

func OnPeerConnected

func OnPeerConnected(fn func(peer.ID)) Option

OnPeerConnected registers a callback invoked when a new peer connection is established. Multiple callbacks can be registered; they execute in registration order.

func OnPeerDisconnected

func OnPeerDisconnected(fn func(peer.ID)) Option

OnPeerDisconnected registers a callback invoked when a peer disconnects. Multiple callbacks can be registered; they execute in registration order.

func WithBootstrapPeers

func WithBootstrapPeers(peers ...string) Option

WithBootstrapPeers sets the DHT bootstrap peers.

func WithConfig

func WithConfig(cfg *Config) Option

WithConfig sets the server configuration.

func WithIdentity

func WithIdentity(priv crypto.PrivKey) Option

WithIdentity sets the server's private key.

func WithListenAddrs

func WithListenAddrs(addrs ...string) Option

WithListenAddrs sets the multiaddrs to listen on.

func WithLogger

func WithLogger(logger *slog.Logger) Option

WithLogger sets the structured logger.

func WithMetrics

func WithMetrics(m MetricsCollector) Option

WithMetrics sets a MetricsCollector for observability. If not set, a no-op collector is used and no metrics overhead is incurred.

func WithPort

func WithPort(port int) Option

WithPort sets the listen port.

func WithPreset

func WithPreset(preset Preset) Option

WithPreset applies a configuration preset.

func WithShutdownTimeout

func WithShutdownTimeout(d time.Duration) Option

WithShutdownTimeout sets the maximum time to wait for active streams to drain during graceful shutdown. Zero disables draining (immediate close).

func WithTransport

func WithTransport(t libp2p.Option) Option

WithTransport adds a libp2p transport option.

type Pipeline

type Pipeline struct {
	// contains filtered or unexported fields
}

Pipeline is an ordered chain of middleware that processes libp2p streams. It implements the Netty ChannelPipeline pattern: middleware executes in order, each wrapping the downstream call via next(). This enables bidirectional processing (request + response) in a single middleware function.

func NewPipeline

func NewPipeline(logger *slog.Logger, mw ...Middleware) *Pipeline

NewPipeline creates a pipeline with the given middleware chain. Middleware executes in the order provided.

Streams are bounded by DefaultIdleTimeout and DefaultRequestTimeout unless WithIdleTimeout or WithRequestTimeout say otherwise.

func (*Pipeline) HandleStream

func (p *Pipeline) HandleStream(s network.Stream)

HandleStream is the entry point that satisfies libp2p's network.StreamHandler. It creates a StreamContext, runs the middleware chain, and handles cleanup.

func (*Pipeline) StreamHandler

func (p *Pipeline) StreamHandler() network.StreamHandler

StreamHandler returns the pipeline's HandleStream method as a network.StreamHandler.

func (*Pipeline) Use

func (p *Pipeline) Use(mw ...Middleware) *Pipeline

Use appends middleware to the pipeline and returns the pipeline for chaining.

func (*Pipeline) WithActiveStreams

func (p *Pipeline) WithActiveStreams(wg *sync.WaitGroup) *Pipeline

WithActiveStreams sets a WaitGroup that tracks in-flight stream processing. The pipeline calls Add(1) on stream entry and Done() on exit, enabling graceful shutdown draining in Server.Stop().

func (*Pipeline) WithIdleTimeout

func (p *Pipeline) WithIdleTimeout(d time.Duration) *Pipeline

WithIdleTimeout sets how long a frame read or write may wait for the peer to make progress. Zero disables the deadline.

func (*Pipeline) WithRegistry

func (p *Pipeline) WithRegistry(r *Registry) *Pipeline

WithRegistry sets the service registry for the pipeline. The registry is injected into each StreamContext, making server-level singletons accessible to handlers via ServiceFrom[T].

func (*Pipeline) WithRequestTimeout

func (p *Pipeline) WithRequestTimeout(d time.Duration) *Pipeline

WithRequestTimeout bounds the whole life of a stream and is the deadline on sc.Ctx. Zero disables it, leaving only the idle timeout.

type Preset

type Preset func(cfg *Config)

Preset modifies a Config with predefined values.

var DevelopmentPreset Preset = func(cfg *Config) {
	cfg.RateLimitWindow = time.Minute
	cfg.MaxRequestsPerWindow = 1000
}

DevelopmentPreset configures for local development.

var ProductionPreset Preset = func(cfg *Config) {
	cfg.RateLimitWindow = time.Minute
	cfg.MaxRequestsPerWindow = 100
}

ProductionPreset configures for production deployment.

type RateLimitedError

type RateLimitedError struct {
	// RetryAfter is how long until the limiter would admit the same request.
	// Zero means the limiter could not say — including the case of a request
	// that costs more than the bucket can ever hold, which no amount of
	// waiting will admit.
	RetryAfter time.Duration
}

RateLimitedError reports a rate-limit rejection along with how long the caller should wait before trying again.

It exists because "rate limit exceeded" on its own leaves a client with nothing to do but guess, and clients that guess converge on pacing themselves far below what the limiter would actually allow. The limiter knows precisely when the next token arrives; this is how it says so.

It unwraps to ErrRateLimited, so every existing errors.Is check keeps working and callers can opt into the detail with errors.As.

func (*RateLimitedError) Error

func (e *RateLimitedError) Error() string

func (*RateLimitedError) Unwrap

func (e *RateLimitedError) Unwrap() error

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry is a type-safe container for server-level singletons. It is populated during server setup via Server.Provide() and made available to handlers through StreamContext.

func NewRegistry

func NewRegistry() *Registry

NewRegistry creates an empty registry.

func (*Registry) Provide

func (r *Registry) Provide(key string, value any)

Provide stores a named value in the registry. Panics if the key is already registered (fail-fast on duplicate bindings).

type Server

type Server struct {
	// contains filtered or unexported fields
}

Server is the main go-p2p-forge server, analogous to Netty's ServerBootstrap. It manages a libp2p host, P2P node (DHT + GossipSub), protocol handlers, and background services with structured lifecycle management.

func NewServer

func NewServer(opts ...Option) *Server

NewServer creates a new server with the given options. Call ListenAndServe to start it.

func (*Server) ActiveStreams

func (s *Server) ActiveStreams() *sync.WaitGroup

ActiveStreams returns the WaitGroup tracking in-flight streams. Pipelines should call Add(1) on stream entry and Done() on stream exit to enable graceful draining.

func (*Server) AddService

func (s *Server) AddService(svc service.Service)

AddService registers a background service that participates in the lifecycle.

func (*Server) Handle

func (s *Server) Handle(id protocol.ID, handler network.StreamHandler)

Handle registers a protocol handler.

func (*Server) Host

func (s *Server) Host() host.Host

Host returns the underlying libp2p host (available after Start).

func (*Server) ListenAndServe

func (s *Server) ListenAndServe(ctx context.Context) error

ListenAndServe starts the server and blocks until ctx is cancelled.

func (*Server) Metrics

func (s *Server) Metrics() MetricsCollector

Metrics returns the server's MetricsCollector (NoopMetrics if none was configured).

func (*Server) Node

func (s *Server) Node() *node.Node

Node returns the underlying P2P node (available after Start).

func (*Server) OpenStream

func (s *Server) OpenStream(ctx context.Context, peerID peer.ID, proto protocol.ID) (*StreamContext, error)

OpenStream opens a new stream to the given peer for the specified protocol and returns a StreamContext ready for writing. The caller is responsible for closing the stream when done.

This enables server-initiated communication (e.g., push notifications). Use codec.WriteFrame() and codec.ReadFrame() on sc.Stream for I/O.

func (*Server) PeerID

func (s *Server) PeerID() peer.ID

PeerID returns the server's peer ID (available after Start).

func (*Server) Provide

func (s *Server) Provide(key string, value any)

Provide registers a server-level singleton accessible to all pipeline handlers via ServiceFrom[T]. Must be called before Start(). Panics on duplicate keys.

func (*Server) Registry

func (s *Server) Registry() *Registry

Registry returns the server's service registry for use with Pipeline.WithRegistry().

func (*Server) Start

func (s *Server) Start(ctx context.Context) error

Start initializes the P2P stack, registers handlers, and starts services.

func (*Server) Stop

func (s *Server) Stop() error

Stop gracefully shuts down the server. If ShutdownTimeout is configured, it waits for active streams to drain before closing the node.

type SliceIterator

type SliceIterator struct {
	// contains filtered or unexported fields
}

SliceIterator implements FrameIterator by marshaling a slice of objects into individual frames using the provided Codec, one per call to Next().

func NewSliceIterator

func NewSliceIterator(c Codec, items ...any) *SliceIterator

NewSliceIterator creates a FrameIterator from a slice of response objects. Each object is marshaled into a separate frame using the provided codec.

func (*SliceIterator) Next

func (si *SliceIterator) Next() ([]byte, error)

Next returns the next frame's bytes, or io.EOF when all items have been yielded.

type StreamContext

type StreamContext struct {
	// Ctx is the request-scoped context, cancelled on stream close.
	Ctx context.Context

	// Stream is the underlying libp2p stream.
	Stream network.Stream

	// PeerID is the remote peer's identity.
	PeerID peer.ID

	// Logger is pre-tagged with peer and protocol info.
	Logger *slog.Logger

	// RawBytes holds the raw frame payload (set by FrameDecode middleware).
	RawBytes []byte

	// Request holds the decoded request object (set by deserialize middleware).
	Request any

	// Response holds the response to be encoded (set by the handler).
	Response any

	// Err signals an error that short-circuits the pipeline.
	Err error

	// PoolBuf tracks a pooled buffer for lifecycle management.
	PoolBuf *codec.PoolBuffer

	// Registry provides access to server-level singletons registered via Server.Provide().
	Registry *Registry

	// IdleTimeout is how long a frame read or write may wait for the peer to
	// make progress before the stream is abandoned. The pipeline sets it;
	// WriteFrame and FrameDecodeMiddleware honour it. Zero means no deadline.
	IdleTimeout time.Duration
	// contains filtered or unexported fields
}

StreamContext carries request-scoped data through the middleware pipeline. Analogous to Netty's ChannelHandlerContext.

func (*StreamContext) Get

func (sc *StreamContext) Get(key any) (any, bool)

Get retrieves a value by key.

func (*StreamContext) Operation

func (sc *StreamContext) Operation() string

Operation returns the routed operation name, or the empty string when the request never reached a route — because it carried no operation field, or named one that does not exist.

Middleware downstream of next() can read it to attribute timing, logging or metrics to a specific operation. Because only a matched route is recorded, the set of values is bounded by the routing table rather than by what a caller sends, which is what makes it safe to use as a metric label.

func (*StreamContext) Set

func (sc *StreamContext) Set(key, val any)

Set stores a key-value pair for inter-middleware communication.

func (*StreamContext) SetOperation

func (sc *StreamContext) SetOperation(op string)

SetOperation records the operation this request was routed to. OperationRouter calls it once a route has matched.

func (*StreamContext) WriteFrame

func (sc *StreamContext) WriteFrame(data []byte) error

WriteFrame writes data to the stream as one length-prefixed frame, bounded by the context's IdleTimeout so a peer that stops reading cannot hold the handler for the whole response. Response writers should use it instead of codec.WriteFrame on sc.Stream directly.

Directories

Path Synopsis
Package codec provides wire-format encoding/decoding for libp2p stream protocols.
Package codec provides wire-format encoding/decoding for libp2p stream protocols.
Package forgetest provides test helpers for the forge framework.
Package forgetest provides test helpers for the forge framework.
Package host provides libp2p host creation and identity management.
Package host provides libp2p host creation and identity management.
Package middleware provides reusable middleware components for libp2p stream pipelines.
Package middleware provides reusable middleware components for libp2p stream pipelines.
Package node wraps a libp2p host with DHT and GossipSub.
Package node wraps a libp2p host with DHT and GossipSub.
Package service provides lifecycle management for background services.
Package service provides lifecycle management for background services.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL