scuttlebutt

README

Scuttlebutt

A Go library that facilitates cluster membership and state propagation using the Scuttlebutt protocol.

The Scuttlebutt protocol is an eventually consistent, gossip-based anti-entropy mechanism, as described in this paper.

The state of a node is represented by a set of application-defined key-value pairs, including the node type, status, and routing information.

The state of each node is disseminated to all other nodes in the cluster, allowing each node to develop an eventually consistent view of the entire cluster. This state is presented as a key-value store on each node, but typically, applications subscribe to updates about node join/leave and state changes.

Failed nodes in the cluster are detected using the phi accrual failure detector.

The implementation is described in docs/.

Usage

The full API docs can be viewed with go doc --all.

Create a gossip node

Creates a new node, which will start listening for updates from nodes in the cluster. If SeedCB is given it will attempt to join the cluster by gossiping with these nodes.

Whenever the node doesn't know about any other peers it will re-seed by calling SeedCB to get a new list of seeds.

node := scuttlebutt.Create(
	"0.0.0.0:8229",
	scuttlebutt.WithSeedCB(func() []string {
		return myconfig.Seeds
	}),
	scuttlebutt.WithOnJoin(...),
	scuttlebutt.WithOnLeave(...),
	scuttlebutt.WithOnUpdate(...),
)

See options.go for the full set of options.

Update our nodes state

Updates our nodes local state, which will be propagated to other nodes in the cluster and notify their subscribes of the update.

node.UpdateLocal("routing.addr", "10.25.104.42:5112")
node.UpdateLocal("state", "ready")

Note the keys and values are limitted to 256 bytes.

Lookup the known state of another node

Looks up the state of the peer as known by this node. Since the cluster membership is eventually consistent this may be out of date with the actual peer, though should converge quickly.

Note typically you'll subscribe to updates with OnUpdate rather than querying directly.

addr, ok := node.Lookup("10.26.104.82:7188", "routing.addr")
if !ok {
	// ...
}

Building

Assuming you have Go installed, simply build with

$ go build

API Docs

Show the API docs with

$ go doc --all

Testing

Tests are split into unit and system tests.

Unit tests test small units in isolation, with no goroutines or networking, so should finish very fast. These sit alongside the code under test so can be ran with

$ go test

System tests spin up a cluster of nodes to test. These are kept in the tests/ directory so can be run with

$ go test ./...

Evaluation

Theres a CLI tool in eval/ that can be used to evaluate the cluster. Such as the time it takes to propagate an update to all nodes in a cluster with 64 nodes.

Documentation

Index

• Constants
• type Option
  • func WithConvictionThreshold(convictionThreshold float64) Option
  • func WithInterval(interval time.Duration) Option
  • func WithLogger(logger *zap.Logger) Option
  • func WithMaxMessageSize(size int) Option
  • func WithOnJoin(cb func(peerAddr string)) Option
  • func WithOnLeave(cb func(peerAddr string)) Option
  • func WithOnUpdate(cb func(peerAddr string, key string, value string)) Option
  • func WithSeedCB(seedCB func() []string) Option
• type Options
• type Scuttlebutt
  • func Create(addr string, options ...Option) (*Scuttlebutt, error)
  • func (s *Scuttlebutt) Addrs() []string
  • func (s *Scuttlebutt) BindAddr() string
  • func (s *Scuttlebutt) Lookup(addr string, key string) (string, bool)
  • func (s *Scuttlebutt) Shutdown() error
  • func (s *Scuttlebutt) UpdateLocal(key string, value string)

Constants

const (
	DefaultMaxMessageSize      = 512
	DefaultConvictionThreshold = 8.0
	DefaultInterval            = time.Millisecond * 500
)

Types

type Option

type Option func(*Options)

func WithConvictionThreshold

func WithConvictionThreshold(convictionThreshold float64) Option

func WithInterval

func WithInterval(interval time.Duration) Option

func WithLogger

func WithLogger(logger *zap.Logger) Option

func WithMaxMessageSize

func WithMaxMessageSize(size int) Option

func WithOnJoin

func WithOnJoin(cb func(peerAddr string)) Option

func WithOnLeave

func WithOnLeave(cb func(peerAddr string)) Option

func WithOnUpdate

func WithOnUpdate(cb func(peerAddr string, key string, value string)) Option

func WithSeedCB

func WithSeedCB(seedCB func() []string) Option

type Options

type Options struct {
	// SeedCB is a callback that returns a list of seed addresses to use to
	// join the cluster. This will be called whenever the node does not know
	// about any other nodes in the cluster. If nil the node will not attempt
	// to seed and must wait for the other nodes to contact it instead.
	SeedCB func() []string

	// OnJoin is invoked when a peer joins the cluster.
	OnJoin func(peerAddr string)

	// OnLeave is invoked when a peer leaves the cluster or is considered
	// inactive.
	OnLeave func(peerAddr string)

	// OnUpdate is invoked when a peers state is updated.
	OnUpdate func(peerAddr string, key string, value string)

	// MaxMessageSize is the maximum allowed UDP payload for gossip messages.
	// If the MTU is known this should be increased to the maximum size. If not
	// set default to 512 bytes.
	MaxMessageSize int

	// ConvictionThreshold is the value if phi in the failure detector to
	// consider a node down. If not set defaults to 8.0.
	ConvictionThreshold float64

	// Interval is the time between gossip rounds, when the node selects
	// a random peer to sync with.
	// If not set defaults to 500ms.
	Interval time.Duration

	Logger *zap.Logger
}

type Scuttlebutt

type Scuttlebutt struct {
	// contains filtered or unexported fields
}

Scuttlebutt handles cluster membership using the scuttlebutt protocol. This is thread safe.

func Create

func Create(addr string, options ...Option) (*Scuttlebutt, error)

Create will create a new Scuttlebutt using the given configuration. This will start listening on the network to allow other nodes to join, though will not attempt to join the cluster itself unless contacted by another node. After this the given configuration should not be modified again.

func (*Scuttlebutt) Addrs

func (s *Scuttlebutt) Addrs() []string

Addrs returns the addresses of the peers known by this node (including ourselves).

func (*Scuttlebutt) BindAddr

func (s *Scuttlebutt) BindAddr() string

BindAddr returns the address the transport listener is bound to. Note this may be different from the configured bind addr if the system chooses the addr (such as using a port of 0).

func (*Scuttlebutt) Lookup

func (s *Scuttlebutt) Lookup(addr string, key string) (string, bool)

Lookup looks up the given key in the known state of the peer with the address. Since the cluster state is eventually consistent, this isn't guaranteed to be up to date with the actual state of the peer, though should converge quickly.

func (*Scuttlebutt) Shutdown

func (s *Scuttlebutt) Shutdown() error

Shutdown closes all background networking and stops gossiping its state to the cluster.

func (*Scuttlebutt) UpdateLocal

func (s *Scuttlebutt) UpdateLocal(key string, value string)

UpdateLocal updates this nodes state with the given key-value pair. This will be propagated to the other nodes in the cluster.