consistent

README

go-consistent

go-consistent is a Go implementation of the consistent hashing algorithm.

Consistent hashing minimizes rehashing when nodes are added or removed from a distributed system, ensuring balanced data distribution with minimal key remapping.

Features

• Thread-safe for concurrent use
• Configurable replication factor (virtual nodes)
• Simple API: Add, Remove, Get, Nodes

Installation

go get github.com/kcxlee/go-consistent

Usage

package main

import (
	"fmt"
	"log"

	"github.com/kcxlee/go-consistent"
)

func main() {
	nodes := []string{"node1", "node2", "node3"}
	hash := consistent.NewWithNodes(nodes, consistent.Config{
		ReplicationFactor: 200,
	})

	node, err := hash.Get("my-key")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("my-key is assigned to %s\n", node)
}

Performance

Distribution of 64,000 topics across 10 nodes with varying replica counts:

Replicas	Duration (ms)	StdDev
50	430	1141.21
100	449	504.31
200	463	450.69
400	497	233.30

Higher replica counts improve distribution at the cost of slightly longer add/remove operations.

API

• New(config Config) *ConsistentHash – creates a new consistent hash ring.
• NewWithNodes(nodes []string, config Config) *ConsistentHash – creates a ring and adds the given nodes.
• Add(node string) – adds a physical node to the ring.
• Remove(node string) – removes a physical node from the ring.
• Get(key string) (string, error) – returns the node responsible for the given key.
• Nodes() []string – returns all physical nodes currently in the ring.
• NodeCount() int – returns the number of physical nodes.
• Distribute(keys []string) map[string][]string – distributes a slice of keys across the ring and returns the mapping of node → keys.
• Distribution(keys []string) map[string]int – returns the number of keys assigned to each node.

Documentation

Index

• type Config
• type ConsistentHash
  • func New(config Config) *ConsistentHash
  • func NewWithNodes(nodes []string, config Config) *ConsistentHash
  • func (c *ConsistentHash) Add(node string)
  • func (c *ConsistentHash) Distribute(keys []string) map[string][]string
  • func (c *ConsistentHash) Distribution(keys []string) map[string]int
  • func (c *ConsistentHash) Get(key string) (string, error)
  • func (c *ConsistentHash) NodeCount() int
  • func (c *ConsistentHash) Nodes() []string
  • func (c *ConsistentHash) Remove(node string)

Examples

• ConsistentHash

Types

type Config

type Config struct {
	// ReplicationFactor is the number of virtual nodes (replicas) per physical node.
	ReplicationFactor int
}

Config holds configuration for ConsistentHash.

type ConsistentHash

type ConsistentHash struct {
	// contains filtered or unexported fields
}

ConsistentHash implements consistent hashing. It is safe for concurrent use.

Example

nodes := []string{"node1", "node2"}
c := consistent.NewWithNodes(nodes, consistent.Config{ReplicationFactor: 16})

node, err := c.Get("my-key")
if err != nil {
	log.Fatal(err)
}
fmt.Println(node)

func New

func New(config Config) *ConsistentHash

New creates a new ConsistentHash instance.

func NewWithNodes

func NewWithNodes(nodes []string, config Config) *ConsistentHash

NewWithNodes creates a new ConsistentHash instance and adds the given nodes.

func (*ConsistentHash) Add

func (c *ConsistentHash) Add(node string)

Add adds a physical node to the ring. Duplicate adds are ignored.

func (*ConsistentHash) Distribute

func (c *ConsistentHash) Distribute(keys []string) map[string][]string

Distribute assigns each key to its responsible node and returns a mapping of node -> assigned keys. The returned map contains an entry for every node in the ring, even if it receives no keys.

func (*ConsistentHash) Distribution

func (c *ConsistentHash) Distribution(keys []string) map[string]int

Distribution returns the number of keys assigned to each node.

func (*ConsistentHash) Get

func (c *ConsistentHash) Get(key string) (string, error)

Get returns the node responsible for the given key. If the ring is empty, an error is returned.

func (*ConsistentHash) NodeCount

func (c *ConsistentHash) NodeCount() int

NodeCount returns the number of physical nodes in the ring.

func (*ConsistentHash) Nodes

func (c *ConsistentHash) Nodes() []string

Nodes returns a slice of all physical node names currently in the ring.

func (*ConsistentHash) Remove

func (c *ConsistentHash) Remove(node string)

Remove removes a physical node from the ring.