tsid

package module
v1.0.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 7, 2024 License: Apache-2.0 Imports: 10 Imported by: 0

README

TSID Generator

Go library for generating Time-Sorted Unique Identifiers (TSID) Implementation inspired from f4b6a3/tsid-creator

It brings together ideas from Twitter's Snowflake and ULID Spec.

In summary:

  • Sorted by generation time;
  • Can be stored as an integer of 64 bits;
  • Can be stored as a string of 13 chars;
  • String format is encoded to Crockford's base32;
  • String format is URL safe, is case insensitive, and has no hyphens;
  • Shorter than UUID, ULID and KSUID.

Recommended readings:

Usage

Create a TSID Factory:

tsidFactory, err := TsidFactoryBuilder().
    WithNodeBits(nodeBits). // max 20
    WithNode(nodeId). // max 2^nodeBits
    WithCustomEpoch(customEpoch).
    WithClock(clock).
    WithRandom(random).
    Build()

Default node Id is 0 & node bits is 0. Epoch and clock are optional, and Random when not provided it internally uses random from math/rand to generate random values

[!NOTE] .Build() creates / returns the existing instance of the tsid factory, this is useful where single instance of tsid factory needs to be shared across go routines. But if you need a new instance use .NewInstance()

Generate TSID

tsid, err := tsidFactory.Generate()

Get TSID as int64:

number := tsid.ToNumber(); // 122390644586507544

Get TSID as string:

tsidStr := tsid.ToString(); // 03CPHMJ76HV8R

The TSID generator is thread-safe.

Dependency

Run the following command:

go get github.com/vishal-bihani/go-tsid
TSID as Number

The tsid.ToNumber() method simply unwraps the internal int64 value of a TSID.

id := tsid.ToNumber()

Sequence of TSIDs:

38352658567418867
38352658567418868
38352658567418869
38352658567418870
38352658567418871
38352658567418872
38352658567418873
38352658567418874
38352658573940759 < millisecond changed
38352658573940760
38352658573940761
38352658573940762
38352658573940763
38352658573940764
38352658573940765
38352658573940766
         ^      ^ look

|--------|------|
   time   random
TSID as String

The tsid.ToString() method encodes a TSID to Crockford's base 32 encoding. The returned string is 13 characters long.

idStr := tsid.ToString();

Sequence of TSID strings:

01226N0640J7K
01226N0640J7M
01226N0640J7N
01226N0640J7P
01226N0640J7Q
01226N0640J7R
01226N0640J7S
01226N0640J7T
01226N0693HDA < millisecond changed
01226N0693HDB
01226N0693HDC
01226N0693HDD
01226N0693HDE
01226N0693HDF
01226N0693HDG
01226N0693HDH
        ^   ^ look

|-------|---|
   time random

The string format can be useful for languages that store numbers in double-precision 64-bit binary format IEEE 754, such as Javascript.

TSID Structure

The term TSID stands for (roughly) Time-Sorted ID. A TSID is a number that is formed by the creation time along with random bits.

The TSID has 2 components:

  • Time component (42 bits)
  • Random component (22 bits)

The time component is the count of milliseconds since 2020-01-01 00:00:00 UTC.

The Random component has 2 sub-parts:

  • Node ID (0 to 20 bits)
  • Counter (2 to 22 bits)

The counter bits depend on the node bits. If the node bits are 10, the counter bits are limited to 12. In this example, the maximum node value is 2^10-1 = 1023 and the maximum counter value is 2^12-1 = 4095. So the maximum TSIDs that can be generated per millisecond is 4096.

The node identifier uses 10 bits of the random component by default in the TsidFactory. It's possible to adjust the node bits to a value between 0 and 20. The counter bits are affected by the node bits.

This is the default TSID structure:

                                            adjustable
                                           <---------->
|------------------------------------------|----------|------------|
       time (msecs since 2020-01-01)           node       counter
                42 bits                       10 bits     12 bits

- time:    2^42 = ~69 years or ~139 years (with adjustable epoch)
- node:    2^10 = 1,024 (with adjustable bits)
- counter: 2^12 = 4,096 (initially random)

Notes:
The node is adjustable from 0 to 20 bits.
The node bits affect the counter bits.
The time component can be used for ~69 years if stored in a SIGNED 64 bits integer field.
The time component can be used for ~139 years if stored in a UNSIGNED 64 bits integer field.

The time component can be 1 ms or more ahead of the system time when necessary to maintain monotonicity and generation speed.

Node identifier

A simple way to avoid collisions is to make sure that each generator has its exclusive node ID. A "node" as we call it in this library can be a physical machine, a virtual machine, a container, a k8s pod, a running process, a database instance number, etc.

Notes:

  1. As a reference, 6,000 tweets are posted on Twitter every second as of 2022;
  2. According to the hostname manual:
    • --ip-address or -i (lowercase): Display the network address(es) of the host name. Note that this works only if the host name can be resolved. Avoid using this option; use hostname --all-ip-addresses instead.
    • --all-ip-addresses or -I (uppercase): Display all network addresses of the host. This option enumerates all configured addresses on all network interfaces. The loopback interface and IPv6 link-local addresses are omitted. Contrary to option -i, this option does not depend on name resolution. Do not make any assumptions about the order of the output.
More Examples

Create a quick TSID:

tsid := tsid.Fast();

Create a quick TSID from canonical string (13 chars)

tsid := tsid.FromString("03CPHMJ76HV8R")

Get the creation unix millis of the tsid

millis := tsid.GetUnixMillis()

A TsidFactory with a FIXED node identifier and CUSTOM node bits:

// setup a factory for up to 64 nodes and 65536 ID/ms.
tsidFactory, err := TsidFactoryBuilder()
    .WithNodeBits(6)      // max: 20
    .WithNode(63)         // max: 2^nodeBits
    .Build();

// use the factory
tsid, err := tsidFactory.Generate()

A TsidFactory with a CUSTOM epoch:

epoch := time.Now().UnixMilli()
tsidFactory, err := TsidFactoryBuilder().
    WithCustomEpoch(epoch).
    build();

// use the factory
tsid, err := tsidFactory.Generate()

A TsidFactory with Crypto Random value generator

// using crypto/rand for security use cases
supplier := NewCryptoRandomSupplier()

// creating random value generator
intRandom := NewIntRandom(supplier)

tsidFactory, err := TsidFactoryBuilder().
    WithRandom(intRandom).
    build();

You can use custom random value suppliers either by implementing IntSupplier, ByteSupplier or using NewIntRandomWithSupplierFunc

Ports, forks and other OSS

Ports and forks:

Language Name
Java vladmihalcea/hypersistence-tsid
.NET kgkoutis/TSID.Creator.NET
PHP odan/tsid
Python. luismedel/tsid-python

Other OSS:

Language Name
Java fillumina/id-encryptor
.NET ullmark/hashids.net

License

This library is Open Source software released under the Apache-2.0 license.

Documentation

Index

Constants

View Source 
const (
	BYTE_SIZE        = 8
	INTEGER_SIZE_32  = 32
	INTEGER_BYTES_32 = 4
)
View Source 
const (
	TSID_EPOCH     int64 = 1672531200000 // 2023-01-01T00:00:00.000Z
	TSID_BYTES     int32 = 8
	TSID_CHARS     int32 = 13 // ToString returns a string of length 13
	RANDOM_BITS    int32 = 22
	RANDOM_MASK    int32 = 0x003fffff
	NODE_BITS_1024 int32 = 10
)

Variables

View Source 
var ALPHABET_LOWERCASE []rune = []rune("0123456789abcdefghjkmnpqrstvwxyz")
View Source 
var ALPHABET_UPPERCASE []rune = []rune("0123456789ABCDEFGHJKMNPQRSTVWXYZ")
View Source 
var ALPHABET_VALUES []int64

Functions

func IsValidRuneArray 

func IsValidRuneArray(arr []rune) bool

IsValidRuneArray validates the rune array.

func NewByteRandom 

func NewByteRandom(randomSupplier RandomSupplier) *byteRandom

func NewByteRandomWithSupplierFunc 

func NewByteRandomWithSupplierFunc(randomSupplierFunc func(length int32) ([]byte, error)) *byteRandom

func NewCryptoRandomSupplier 

func NewCryptoRandomSupplier() *cryptoRandomSupplier

func NewIntRandom 

func NewIntRandom(intSupplier IntSupplier) *intRandom

func NewIntRandomWithSupplierFunc 

func NewIntRandomWithSupplierFunc(intSupplierFunc func() (int32, error)) *intRandom

func NewMathRandomSupplier 

func NewMathRandomSupplier() *mathRandomSupplier

func ToRuneArray 

func ToRuneArray(str string) []rune

ToRuneArray converts the given string to rune array. It also performs validations on the rune array

func TsidFactoryBuilder 

func TsidFactoryBuilder() *tsidFactoryBuilder

TsidFactoryBuilder should be used to get instance of tsidFactory

Types

type ByteSupplier 

type ByteSupplier interface {
	GetBytes(length int32) ([]byte, error)
}

type Clock 

type Clock interface {
	UnixMilli() int64
}

type IntSupplier 

type IntSupplier interface {
	GetInt() (int32, error)
}

type Random 

type Random interface {
	NextInt() (int32, error)
	NextBytes(length int32) ([]byte, error)
}

type RandomSupplier 

type RandomSupplier interface {
	GetInt() (int32, error)
	GetBytes(length int32) ([]byte, error)
}

Suppliers

type Tsid added in v1.0.5 

type Tsid struct {
	// contains filtered or unexported fields
}

func Fast 

func Fast() *Tsid

Fast returns a pointer to new random tsid

func FromBytes 

func FromBytes(bytes []byte) *Tsid

FromBytes returns pointer to tsid by converting the given bytes to number

func FromNumber 

func FromNumber(number int64) *Tsid

FromNumber returns pointer to tsid using the given number

func FromString 

func FromString(str string) *Tsid

FromString returns pointer to tsid by converting the given string to number. It validates the string before conversion.

func NewTsid 

func NewTsid(number int64) *Tsid

NewTsid returns pointer to new tsid

func (*Tsid) GetRandom added in v1.0.5 

func (t *Tsid) GetRandom() int64

GetRandom returns random component (node + counter) of the tsid

func (*Tsid) GetUnixMillis added in v1.0.5 

func (t *Tsid) GetUnixMillis() int64

GetUnixMillis returns time of creation in millis since 1970-01-01

func (*Tsid) GetUnixMillisWithCustomEpoch added in v1.0.5 

func (t *Tsid) GetUnixMillisWithCustomEpoch(epoch int64) int64

GetUnixMillis returns time of creation in millis since 1970-01-01

func (*Tsid) IsValid added in v1.0.5 

func (t *Tsid) IsValid(str string) bool

IsValid checks if the given tsid string is valid or not

func (*Tsid) ToBytes added in v1.0.5 

func (t *Tsid) ToBytes() []byte

ToBytes converts the number to bytes and returns the byte array

func (*Tsid) ToLowerCase added in v1.0.5 

func (t *Tsid) ToLowerCase() string

ToLowerCase converts the number to a canonical string in lower case. The output is 13 characters long and only contains characters from Crockford's base32 alphabets

func (*Tsid) ToNumber added in v1.0.5 

func (t *Tsid) ToNumber() int64

ToNumber returns the numerical component of the tsid

func (*Tsid) ToString added in v1.0.5 

func (t *Tsid) ToString() string

ToString converts the number to a canonical string. The output is 13 characters long and only contains characters from Crockford's base32 alphabets

func (*Tsid) ToStringWithAlphabets added in v1.0.5 

func (t *Tsid) ToStringWithAlphabets(alphabets []rune) string

ToStringWithAlphabets converts the number to string using the given alphabets and returns it

type TsidFactory added in v1.0.5 

type TsidFactory struct {
	// contains filtered or unexported fields
}

tsidFactory is a singleton which should be used to generate random tsid

func (*TsidFactory) Generate added in v1.0.5 

func (factory *TsidFactory) Generate() (*Tsid, error)

Generate will return a tsid with random number

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.