ringpop

README

ringpop-go

Ringpop is a library that brings cooperation and coordination to distributed applications. It maintains a consistent hash ring on top of a membership protocol and provides request forwarding as a routing convenience. It can be used to shard your application in a way that's scalable and fault tolerant.

Getting started

To install ringpop-go:

go get github.com/uber/ringpop-go

Developing

Run the tests by doing:

make

To run the test you need both thirft and thrift-gen on your path. On OSX you can install them with the following commands:

brew install thrift
go get github.com/uber/tchannel-go/thrift/thrift-gen

Documentation

Interested in where to go from here? Read the docs at ringpop.readthedocs.org

Documentation

Overview

Package ringpop is a library that maintains a consistent hash ring atop a gossip-based membership protocol. It can be used by applications to arbitrarily shard data in a scalable and fault-tolerant manner.

Index

• Variables
• func DeserializeThrift(b []byte, s athrift.TStruct) error
• func SerializeThrift(s athrift.TStruct) ([]byte, error)
• type Arg
• type HashRing
• type HashRingConfiguration
• type IdentityResolver
• type Interface
• type Option
  • func Channel(ch shared.TChannel) Option
  • func HashRingConfig(c *HashRingConfiguration) Option
  • func Identity(hostport string) Option
  • func IdentityResolverFunc(resolver IdentityResolver) Option
  • func Logger(l log.Logger) Option
  • func Statter(s log.StatsReporter) Option
• type Ringpop
  • func New(app string, opts ...Option) (*Ringpop, error)
  • func (rp *Ringpop) App() string
  • func (rp *Ringpop) Bootstrap(userBootstrapOpts *swim.BootstrapOptions) ([]string, error)
  • func (rp *Ringpop) Checksum() (uint32, error)
  • func (rp *Ringpop) CountReachableMembers() (int, error)
  • func (rp *Ringpop) Destroy()
  • func (rp *Ringpop) Forward(dest string, keys []string, request []byte, service, endpoint string, ...) ([]byte, error)
  • func (rp *Ringpop) GetReachableMembers() ([]string, error)
  • func (rp *Ringpop) HandleEvent(event events.Event)
  • func (rp *Ringpop) HandleOrForward(key string, request []byte, response *[]byte, service, endpoint string, ...) (bool, error)
  • func (rp *Ringpop) Lookup(key string) (string, error)
  • func (rp *Ringpop) LookupN(key string, n int) ([]string, error)
  • func (rp *Ringpop) Ready() bool
  • func (rp *Ringpop) RegisterListener(l events.EventListener)
  • func (rp *Ringpop) Uptime() (time.Duration, error)
  • func (rp *Ringpop) WhoAmI() (string, error)

Variables

var ErrNotBootstrapped = errors.New("ringpop is not bootstrapped")

ErrNotBootstrapped is returned by public methods which require the ring to be bootstrapped before they can operate correctly.

Functions

func DeserializeThrift

func DeserializeThrift(b []byte, s athrift.TStruct) error

DeserializeThrift takes a byte slice and attempts to write it into the given thrift struct using the thrift binary protocol. This is a temporary measure before frames can forwarded directly past the endpoint to the proper destinaiton.

func SerializeThrift

func SerializeThrift(s athrift.TStruct) ([]byte, error)

SerializeThrift takes a thrift struct and returns the serialized bytes of that struct using the thrift binary protocol. This is a temporary measure before frames can forwarded directly past the endpoint to the proper destinaiton.

Types

type Arg

type Arg struct{}

Arg is a blank arg

type HashRing

type HashRing interface {
	AddRemoveServers(add []string, remove []string) bool
	Checksum() uint32
	GetServers() []string
	HasServer(address string) bool
	Lookup(key string) (string, bool)
	LookupN(key string, n int) []string
	RemoveServer(address string)
}

HashRing is an interface for a hash ring that Ringpop uses for consistent hashing.

type HashRingConfiguration

type HashRingConfiguration struct {
	// ReplicaPoints is the number of positions a node will be assigned on the
	// ring. A bigger number will provide better key distribution, but require
	// more computation when building or traversing the ring (typically on
	// lookups or membership changes).
	ReplicaPoints int
}

HashRingConfiguration is a configuration struct that can be passed to the Ringpop constructor to customize hash ring options.

type IdentityResolver

type IdentityResolver func() (string, error)

IdentityResolver is a function that returns the listen interface/port that Ringpop should identify as.

type Interface

type Interface interface {
	Destroy()
	App() string
	WhoAmI() (string, error)
	Uptime() (time.Duration, error)
	RegisterListener(l events.EventListener)
	Bootstrap(opts *swim.BootstrapOptions) ([]string, error)
	Checksum() (uint32, error)
	Lookup(key string) (string, error)
	LookupN(key string, n int) ([]string, error)
	GetReachableMembers() ([]string, error)
	CountReachableMembers() (int, error)

	HandleOrForward(key string, request []byte, response *[]byte, service, endpoint string, format tchannel.Format, opts *forward.Options) (bool, error)
	Forward(dest string, keys []string, request []byte, service, endpoint string, format tchannel.Format, opts *forward.Options) ([]byte, error)
}

Interface specifies the public facing methods a user of ringpop is able to use.

type Option

type Option func(*Ringpop) error

An Option is a modifier functions that configure/modify a real Ringpop object.

There are typically two types of runtime options you can provide: flags (functions that modify the object) and value options (functions the accept user-specific arguments and then return a function that modifies the object).

For more information, see: http://commandcenter.blogspot.com/2014/01/self-referential-functions-and-design.html

func Channel

func Channel(ch shared.TChannel) Option

Channel is used to provide a TChannel instance that Ringpop should use for all communication.

Example:

rp, err := ringpop.New("my-app", ringpop.Channel(myChannel))

Channel is a required option. The constructor will throw an error if this option is not present.

func HashRingConfig

func HashRingConfig(c *HashRingConfiguration) Option

HashRingConfig takes a `HashRingConfiguration` struct that can be used to configure the hash ring.

Example:

rp, err := ringpop.New("my-app",
    ringpop.Channel(myChannel),
    ringpop.HashRingConfig(&HashRingConfiguration{
        ReplicaPoints: 100,
    }),
)

See documentation on the `HashRingConfiguration` struct for more information about what options are available.

func Identity

func Identity(hostport string) Option

Identity is used to specify a static hostport string as this Ringpop instance's identity.

Example:

ringpop.New("my-app",
    ringpop.Channel(myChannel),
    ringpop.Identity("10.32.12.2:21130"),
)

You should make sure the identity matches the listening address of the TChannel object.

By default, you do not need to provide an identity. If you do not provide one, the identity will be resolved automatically by the default resolver.

func IdentityResolverFunc

func IdentityResolverFunc(resolver IdentityResolver) Option

IdentityResolverFunc is used to specify a function that will called when the Ringpop instance needs to resolve its identity (typically, on bootstrap).

func Logger

func Logger(l log.Logger) Option

Logger is used to specify a bark-compatible logger that will be used for all Ringpop logging. If a logger is not provided, one will be created automatically.

func Statter

func Statter(s log.StatsReporter) Option

Statter is used to specify a bark-compatible (bark.StatsReporter) stats reporter that will be used to record ringpop stats. If a statter is not provided, stats will be emitted to a null stats-reporter.

type Ringpop

type Ringpop struct {
	// contains filtered or unexported fields
}

Ringpop is a consistent hashring that uses a gossip protocol to disseminate changes around the ring.

func New

func New(app string, opts ...Option) (*Ringpop, error)

New returns a new Ringpop instance.

func (*Ringpop) App

func (rp *Ringpop) App() string

App returns the name of the application this Ringpop instance belongs to. The application name is set in the constructor when the Ringpop instance is created.

func (*Ringpop) Bootstrap

func (rp *Ringpop) Bootstrap(userBootstrapOpts *swim.BootstrapOptions) ([]string, error)

Bootstrap starts communication for this Ringpop instance.

When Bootstrap is called, this Ringpop instance will attempt to contact other instances from a seed list provided either in the BootstrapOptions or as a JSON file.

If no seed hosts are provided, a single-node cluster will be created.

func (*Ringpop) Checksum

func (rp *Ringpop) Checksum() (uint32, error)

Checksum returns the current checksum of this Ringpop instance's hashring.

func (*Ringpop) CountReachableMembers

func (rp *Ringpop) CountReachableMembers() (int, error)

CountReachableMembers returns the number of members currently in this instance's membership list that aren't faulty.

func (*Ringpop) Destroy

func (rp *Ringpop) Destroy()

Destroy stops all communication. Note that this does not close the TChannel instance that was passed to Ringpop in the constructor. Once an instance is destroyed, it cannot be restarted.

func (*Ringpop) Forward

func (rp *Ringpop) Forward(dest string, keys []string, request []byte, service, endpoint string,
	format tchannel.Format, opts *forward.Options) ([]byte, error)

Forward forwards the request to given destination host and returns the response.

func (*Ringpop) GetReachableMembers

func (rp *Ringpop) GetReachableMembers() ([]string, error)

GetReachableMembers returns a slice of members currently in this instance's membership list that aren't faulty.

func (*Ringpop) HandleEvent

func (rp *Ringpop) HandleEvent(event events.Event)

HandleEvent is used to satisfy the swim.EventListener interface. No touchy.

func (*Ringpop) HandleOrForward

func (rp *Ringpop) HandleOrForward(key string, request []byte, response *[]byte, service, endpoint string,
	format tchannel.Format, opts *forward.Options) (bool, error)

HandleOrForward returns true if the request should be handled locally, or false if it should be forwarded to a different node. If false is returned, forwarding is taken care of internally by the method, and, if no error has occured, the response is written in the provided response field.

func (*Ringpop) Lookup

func (rp *Ringpop) Lookup(key string) (string, error)

Lookup returns the address of the server in the ring that is responsible for the specified key. It returns an error if the Ringpop instance is not yet initialized/bootstrapped.

func (*Ringpop) LookupN

func (rp *Ringpop) LookupN(key string, n int) ([]string, error)

LookupN returns the addresses of all the servers in the ring that are responsible for the specified key. It returns an error if the Ringpop instance is not yet initialized/bootstrapped.

func (*Ringpop) Ready

func (rp *Ringpop) Ready() bool

Ready returns whether or not ringpop is bootstrapped and ready to receive requests.

func (*Ringpop) RegisterListener

func (rp *Ringpop) RegisterListener(l events.EventListener)

RegisterListener adds a listener to the ringpop. The listener's HandleEvent method should be thread safe.

func (*Ringpop) Uptime

func (rp *Ringpop) Uptime() (time.Duration, error)

Uptime returns the amount of time that this Ringpop instance has been bootstrapped for.

func (*Ringpop) WhoAmI

func (rp *Ringpop) WhoAmI() (string, error)

WhoAmI returns the address of the current/local Ringpop node. It returns an error if Ringpop is not yet initialized/bootstrapped.