snowflake

package module
Version: v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 6, 2021 License: BSD-2-Clause Imports: 7 Imported by: 0

README

Update

  • Split 10 bits of Nodes into: 5 bits for IDCs(max to 32 IDCs), 7 bits for nodes(max to 128 nodes)
    An IDC can be power down, so better use some IDCs than setting too many nodes into one.

  • Decrease 12 bits of Step/Sequence(max to 4096) to 8 bit (max to 256 in 1 ms)
    As we tested with following code, it actually only need about 3 step\sequence numbers in 1 ms.

for i := 0; i < 10; i++ {
   fmt.Printf("ID       : %s\n", node.Generate().Base2())
}

In real case, your service gonna do a lot of things after getting an ID. So, 256 is more than enough for a node.

  • Increase 41 bits of Time Bits(max to 69 years) to 43 bits(max to 278 years)

snowflake

GoDoc Go report Coverage Build Status Discord Gophers

snowflake is a Go package that provides

  • A very simple Twitter snowflake generator.
  • Methods to parse existing snowflake IDs.
  • Methods to convert a snowflake ID into several other data types and back.
  • JSON Marshal/Unmarshal functions to easily use snowflake IDs within a JSON API.
  • Monotonic Clock calculations protect from clock drift.

For help with this package or general Go discussion, please join the Discord Gophers chat server.

Status

This package should be considered stable and completed. Any additions in the future will strongly avoid API changes to existing functions.

ID Format

By default, the ID format follows the original Twitter snowflake format.

  • The ID as a whole is a 63 bit integer stored in an int64
  • 43 bits are used to store a timestamp with millisecond precision, using a custom epoch.
  • 5 bits are used to store a idc id - a range from 0 through 31.
  • 7 bits are used to store a node id - a range from 0 through 127.
  • 8 bits are used to store a sequence number - a range from 0 through 255.
Custom Format

You can alter the number of bits used for the idc id, node id and step number (sequence) by setting the snowflake.IDCBits snowflake.NodeBits and snowflake.StepBits values.
For now, there is a maximum of 20 bits available that can be shared between these two values. You can change that.

Custom Epoch

By default this package uses the Twitter Epoch of 1288834974657 or Nov 04 2010 01:42:54.
I updated it to 1612562862000 or 2021-02-06 06:07:42 in milliseconds.
You can set your own epoch value by setting snowflake.Epoch to a time in milliseconds to use as the epoch.

Custom Notes

When setting custom epoch or bit values you need to set them prior to calling any functions on the snowflake package, including NewNode(). Otherwise the custom values you set will not be applied correctly.

How it Works.

Each time you generate an ID, it works, like this.

  • A timestamp with millisecond precision is stored using 43 bits of the ID.
  • Then the IDC ID is added in subsequent bits.
  • Then the Node ID is added in subsequent bits.
  • Then the Sequence Number is added, starting at 0 and incrementing for each ID generated in the same millisecond. If you generate enough IDs in the same millisecond that the sequence would roll over or overfill then the generate function will pause until the next millisecond.

The default Twitter format shown below.

+--------------------------------------------------------------------------+
| 1 Bit Unused | 41 Bit Timestamp |  10 Bit NodeID  |   12 Bit Sequence ID |
+--------------------------------------------------------------------------+

My Updated format shown below.

+---------------------------------------------------------------------------------------+
| 1 Bit Unused | 43 Bit Timestamp |  5 Bit IDCID |  7 Bit NodeID  |   8 Bit Sequence ID |
+---------------------------------------------------------------------------------------+

Using the default settings, this allows for 256 unique IDs to be generated every millisecond, per Node.

Getting Started

Installing

This assumes you already have a working Go environment, if not please see this page first.

go get github.com/butaixianran/snowflake
Usage

Import the package into your project then construct a new snowflake Node using a unique node number. The default settings permit a node number range from 0 to 1023. If you have set a custom NodeBits value, you will need to calculate what your node number range will be. With the node object call the Generate() method to generate and return a unique snowflake ID.

Keep in mind that each node you create must have a unique node number, even across multiple servers. If you do not keep node numbers unique the generator cannot guarantee unique IDs across all nodes.

Example Program:

package main

import (
	"fmt"

	"github.com/butaixianran/snowflake"
)

func main() {

	// Create a new Node with a IDC number of 3 and a Node number of 1
	node, err := snowflake.NewNode(3, 1)
	if err != nil {
		fmt.Println(err)
		return
	}

	// Generate a snowflake ID.
	id := node.Generate()

	// Print out the ID in a few different ways.
	fmt.Printf("Int64  ID: %d\n", id)
	fmt.Printf("String ID: %s\n", id)
	fmt.Printf("Base2  ID: %s\n", id.Base2())
	fmt.Printf("Base64 ID: %s\n", id.Base64())

	// Print out the ID's timestamp
	fmt.Printf("ID Time  : %d\n", id.Time())

	// Print out the ID's idc number
	fmt.Printf("ID IDC   : %d\n", id.IDC())

	// Print out the ID's node number
	fmt.Printf("ID Node  : %d\n", id.Node())

	// Print out the ID's sequence number
	fmt.Printf("ID Step  : %d\n", id.Step())

	// Generate and print, all in one.
	fmt.Printf("ID       : %d\n", node.Generate().Int64())
  
	// Or do it with a loop
	for i := 0; i < 10; i++ {
		fmt.Printf("ID       : %s\n", node.Generate().Base2())
	}
}
Performance

With twitter's default settings, this snowflake generator should be sufficiently fast enough on most systems to generate 4096 unique ID's per millisecond. This is the maximum that the snowflake ID format supports. That is, around 243-244 nanoseconds per operation.

Since the snowflake generator is single threaded the primary limitation will be the maximum speed of a single processor on your system.

To benchmark the generator on your system run the following command inside the snowflake package directory.

go test -run=^$ -bench=.

If your curious, check out this commit that shows benchmarks that compare a few different ways of implementing a snowflake generator in Go.

Documentation

Index

Constants

This section is empty.

Variables

View Source
var (
	// Epoch is set to the twitter snowflake epoch of Nov 04 2010 01:42:54 UTC in milliseconds
	// You may customize this to set a different epoch for your application.
	// Epoch int64 = 1288834974657
	// butaixianran: Set start time to 2021-02-06 06:07:42 in milliseconds
	Epoch int64 = 1612562862000

	// IDCBits holds the number of bits to use for IDC
	// NodeBits holds the number of bits to use for Node
	// Remember, you have a total 20 bits to use with IDC+Node+Step
	IDCBits  uint8 = 5
	NodeBits uint8 = 7

	// StepBits holds the number of bits to use for Step
	// Remember, you have a total 20 bits to use with IDC+Node+Step
	StepBits uint8 = 8
)
View Source
var ErrInvalidBase32 = errors.New("invalid base32")

ErrInvalidBase32 is returned by ParseBase32 when given an invalid []byte

View Source
var ErrInvalidBase58 = errors.New("invalid base58")

ErrInvalidBase58 is returned by ParseBase58 when given an invalid []byte

Functions

This section is empty.

Types

type ID

type ID int64

An ID is a custom type used for a snowflake ID. This is used so we can attach methods onto the ID.

func ParseBase2 added in v0.3.0

func ParseBase2(id string) (ID, error)

ParseBase2 converts a Base2 string into a snowflake ID

func ParseBase32

func ParseBase32(b []byte) (ID, error)

ParseBase32 parses a base32 []byte into a snowflake ID NOTE: There are many different base32 implementations so becareful when doing any interoperation.

func ParseBase36 added in v0.3.0

func ParseBase36(id string) (ID, error)

ParseBase36 converts a Base36 string into a snowflake ID

func ParseBase58

func ParseBase58(b []byte) (ID, error)

ParseBase58 parses a base58 []byte into a snowflake ID

func ParseBase64 added in v0.3.0

func ParseBase64(id string) (ID, error)

ParseBase64 converts a base64 string into a snowflake ID

func ParseBytes added in v0.3.0

func ParseBytes(id []byte) (ID, error)

ParseBytes converts a byte slice into a snowflake ID

func ParseInt64 added in v0.3.0

func ParseInt64(id int64) ID

ParseInt64 converts an int64 into a snowflake ID

func ParseIntBytes added in v0.3.0

func ParseIntBytes(id [8]byte) ID

ParseIntBytes converts an array of bytes encoded as big endian integer as a snowflake ID

func ParseString added in v0.3.0

func ParseString(id string) (ID, error)

ParseString converts a string into a snowflake ID

func (ID) Base2

func (f ID) Base2() string

Base2 returns a string base2 of the snowflake ID

func (ID) Base32

func (f ID) Base32() string

Base32 uses the z-base-32 character set but encodes and decodes similar to base58, allowing it to create an even smaller result string. NOTE: There are many different base32 implementations so becareful when doing any interoperation.

func (ID) Base36

func (f ID) Base36() string

Base36 returns a base36 string of the snowflake ID

func (ID) Base58

func (f ID) Base58() string

Base58 returns a base58 string of the snowflake ID

func (ID) Base64

func (f ID) Base64() string

Base64 returns a base64 string of the snowflake ID

func (ID) Bytes

func (f ID) Bytes() []byte

Bytes returns a byte slice of the snowflake ID

func (ID) IDC added in v0.4.0

func (f ID) IDC() int64

IDC returns an int64 of the snowflake ID IDC number

func (ID) Int64

func (f ID) Int64() int64

Int64 returns an int64 of the snowflake ID

func (ID) IntBytes

func (f ID) IntBytes() [8]byte

IntBytes returns an array of bytes of the snowflake ID, encoded as a big endian integer.

func (ID) MarshalJSON

func (f ID) MarshalJSON() ([]byte, error)

MarshalJSON returns a json byte array string of the snowflake ID.

func (ID) Node

func (f ID) Node() int64

Node returns an int64 of the snowflake ID node number

func (ID) Step

func (f ID) Step() int64

Step returns an int64 of the snowflake step (or sequence) number

func (ID) String

func (f ID) String() string

String returns a string of the snowflake ID

func (ID) Time

func (f ID) Time() int64

Time returns an int64 unix timestamp in milliseconds of the snowflake ID time

func (*ID) UnmarshalJSON

func (f *ID) UnmarshalJSON(b []byte) error

UnmarshalJSON converts a json byte array of a snowflake ID into an ID type.

type JSONSyntaxError

type JSONSyntaxError struct {
	// contains filtered or unexported fields
}

A JSONSyntaxError is returned from UnmarshalJSON if an invalid ID is provided.

func (JSONSyntaxError) Error

func (j JSONSyntaxError) Error() string

type NodeInfo added in v0.4.0

type NodeInfo struct {
	// contains filtered or unexported fields
}

A NodeInfo struct holds the basic information needed for a snowflake generator nodeInfo

func NewNode

func NewNode(idc int64, node int64) (*NodeInfo, error)

NewNode returns a new snowflake node that can be used to generate snowflake IDs

func (*NodeInfo) Generate added in v0.4.0

func (n *NodeInfo) Generate() ID

Generate creates and returns a unique snowflake ID To help guarantee uniqueness - Make sure your system is keeping accurate system time - Make sure you never have multiple nodes running with the same node ID

Source Files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
t or T : Toggle theme light dark auto
y or Y : Canonical URL