goldflake

README

Goldflake

Generator to generate Bigint ID or unique key for 174 years (Approx 200k unique Id/sec/machine on 8k distributed machines at once) inspired by Twitter's Snowflake and Sonyflake

Goldflake is a distributed unique ID generator inspired by Twitter's Snowflake & Sonyflake

Goal:

To become application & database oriented unique bigint id generator for lifetime without compromising performance.

Why Goldflake id instead of Sonyflake?

1. Sonyflake generator is good, if you are generating all Ids at application end.
2. Sonyflake has a limit of 25K id/sec on single node, which seems low in today's scenarios when we are talking about cloud & databases can be created with high system configurations easily.
3. I have tried & tested Sonyflake bits configuration on Postgres-12 on my machine (Dell XPS, Intel® Core™ i7-9750H Processor 6 Core 12 Threads, 16GB RAM, SSD) & observed duplicate key conflict, which is a very basic configuration for Databases now a days.
4. Goldflake is usnig 11 bits for sequence instead of 8 (as used by Sonyflake) which gives more unique ID generation on databases as well.

So it has a different bit assignment from Snowflake & Sonyflake. A Goldflake ID is composed of

39 bits for time in units of 10 msec                      // Similar to Sonyflake
13 bits for a machine id or app instance id or shard id   // Balance between Snowflake and Sonyflake (If you can live with 8192 machines)
11 bits for a sequence number                             // To generate 2^11=2048 unique ids per 10 millisecond or 204800 keys/sec on one node

For PostgreSQL -https://github.com/AmreeshTyagi/goldflake-pg

Comparison chart

	Twitter Snowflake	Sonyflake	Goldflake
Lifetime to generate unique id on single machine	69	174	174
No. of distributed machines	1024	65536	8192
No. of unique IDs per 10 milliseconds	40960	256	2048
Max No. of unique IDs per second on single node	4096000 = 4096k	25600 = 25.6k	204800 = 204.8k
Max No. of unique IDs per second on all nodes	4194304000	1677721600	1677721600
Database oriented	Yes	No	Yes
Application oriented	Yes	Yes	Yes
Comment	Less number of machines	DB will return duplicate soon in case of heavy concurrency	50% application generated IDs & 50% database generated IDs makes it a best fit to use it on app server and db server. I believe 4096 database shards are enough to handle potential large load. Though it depends on solution design as well.

Documentation

Overview

Package goldflake implements Goldflake, a distributed unique ID generator inspired by Twitter's Snowflake & Sonyflake

A Goldflake ID is composed of

39 bits for time in units of 10 msec
11 bits for a sequence number
13 bits for a machine id

Index

• Constants
• Variables
• func Decompose(id uint64) map[string]uint64
• type Goldflake
  • func NewGoldflake(st Settings) *Goldflake
  • func (gf *Goldflake) NextID() (uint64, error)
• type Settings

Constants

const (
	BitLenTime      = 39                               // bit length of time
	BitLenSequence  = 11                               // bit length of sequence number
	BitLenMachineID = 63 - BitLenTime - BitLenSequence // bit length of machine id 63-39-11=13
)

These constants are the bit lengths of Goldflake ID parts.

const DefaultMachineID = 8191

DefaultMachineID Default MachineID

Variables

var DefaultStartTime = time.Date(2020, 9, 1, 0, 0, 0, 0, time.UTC)

DefaultStartTime Default Start Time

Functions

func Decompose

func Decompose(id uint64) map[string]uint64

Decompose returns a set of Goldflake ID parts.

Types

type Goldflake

type Goldflake struct {
	// contains filtered or unexported fields
}

Goldflake is a distributed unique ID generator.

func NewGoldflake

func NewGoldflake(st Settings) *Goldflake

NewGoldflake returns a new Goldflake configured with the given Settings. NewGoldflake returns nil in the following cases: - Settings.StartTime is ahead of the current time. - Settings.MachineID returns an error. - Settings.CheckMachineID returns false.

func (*Goldflake) NextID

func (gf *Goldflake) NextID() (uint64, error)

NextID generates a next unique ID. After the Goldflake time overflows, NextID returns an error.

type Settings

type Settings struct {
	StartTime      time.Time
	MachineID      func() (uint16, error)
	CheckMachineID func(uint16) bool
}

Settings configures Goldflake:

StartTime is the time since which the Goldflake time is defined as the elapsed time. If StartTime is 0, the start time of the Goldflake is set to "2020-09-01 00:00:00 +0000 UTC". If StartTime is ahead of the current time, Goldflake is not created.

MachineID returns the unique ID of the Goldflake instance. If MachineID returns an error, Goldflake is not created. If MachineID is nil, default MachineID is used. Default MachineID is the max possible value based of 2^13 i.e. 8191

CheckMachineID validates the uniqueness of the machine ID. If CheckMachineID returns false, Goldflake is not created. If CheckMachineID is nil, no validation is done.