README
¶
Custom Range-based ID Generator for Go
crid is a range-based, Snowflake-variant distributed ID generator for Go. It produces
unique 63-bit integer IDs by combining a timestamp with a sequence number drawn from a
block reserved out of a shared registry.
Uniqueness comes entirely from the registry, which hands out non-overlapping blocks of sequence numbers per timestamp. Nodes need no identity of their own and require no coordination with one another beyond the registry they share.
Features
- 63-bit IDs that fit in an
int64and are always non-negative. - High throughput: a node reserves a block of sequence numbers at a time and serves them from memory, hitting the registry only once per block.
- Asynchronous pre-allocation: the next block is reserved in the background before the current one runs out, so steady-state generation does not block on the registry.
- Pluggable registry: an in-memory implementation ships with the library, a durable
Postgres-backed one is available under
registry/postgres, and any backend that satisfies theregistry.Registryinterface works. - Configurable bit layout, epoch, block size, and pre-allocation threshold.
Install
go get -u github.com/from-cero/crid
Requires Go 1.26 or newer.
Quick start
package main
import (
"context"
"fmt"
"log"
"github.com/from-cero/crid"
"github.com/from-cero/crid/registry/memory"
)
func main() {
ctx := context.Background()
// All nodes that must not collide share one registry.
reg := memory.New()
node, err := crid.New(reg)
if err != nil {
log.Fatal(err)
}
parser, err := crid.NewParser()
if err != nil {
log.Fatal(err)
}
id, err := node.Generate(ctx)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s -> %s\n", id, parser.Parse(id))
}
How it works
- A
Nodeasks theRegistryto reserveblockSizesequence numbers for the current second (Allocate(ctx, secondsSinceEpoch, blockSize)), receiving the start of the block. Generatehands out IDs from that block in memory:id = (timestamp << sequenceBits) | sequence.- When the remaining count in the block drops below
threshold, the node reserves the next block in a background goroutine. When the current block is exhausted it swaps in the pre-allocated block, or reserves one synchronously if the background reservation has not finished.
Because the registry guarantees non-overlapping blocks per timestamp, two IDs can only collide if they share both timestamp and sequence number, which never happens. This holds across every node sharing the registry, with no per-node ID.
ID layout
An ID is a 63-bit value packed into an int64 (the sign bit is always zero, so valid IDs
are non-negative):
63 sequenceBits 0
+--+-------------------+------------------------+
|0 | timestamp | sequence |
+--+-------------------+------------------------+
The default layout is 31 timestamp bits and 32 sequence bits:
- 31 timestamp bits in seconds give roughly 68 years of range from the epoch.
- 32 sequence bits allow about 4.29 billion IDs per second across all nodes sharing the registry.
timestampBits + sequenceBits must always equal 63.
Working with IDs
id.Int64() // the raw int64
id.String() // decimal string
ID marshals to and from JSON as a quoted decimal string, avoiding precision loss in
environments (such as JavaScript) that cannot represent 63-bit integers exactly:
{
"id": "123456789012345"
}
Parsing
A Parser decodes an ID back into its timestamp and sequence without a running node:
parsed := parser.Parse(id)
parsed.Timestamp // time.Time
parsed.Sequence // int64
[!IMPORTANT] A
Parsermust be created with the same epoch and format as theNodethat generated the IDs. A mismatch produces silently incorrect results, not an error.
Configuration
Pass options to New and NewParser:
| Option | Default | Description |
|---|---|---|
WithFormat(WithTimestampBits, WithSequenceBits) |
31 / 32 | Bit split; must sum to 63. |
WithEpoch(time.Time) |
2026-01-01 00:00:00 UTC | Reference time. Must not be in the future. |
WithBlockSize(int64) |
10,000 | Sequence numbers reserved per registry call. Must be in [1, 2^sequenceBits]. |
WithThreshold(int64) |
5,000 | Remaining count that triggers background pre-allocation. Must not exceed the block size; a value below 1 disables pre-allocation. |
node, err := crid.New(reg,
crid.WithEpoch(time.Date(2020, 1, 1, 0, 0, 0, 0, time.UTC)),
crid.WithFormat(crid.WithTimestampBits(41), crid.WithSequenceBits(22)),
crid.WithBlockSize(1_000),
crid.WithThreshold(200),
)
Tuning block size and threshold
A larger blockSize means fewer registry round-trips but coarser allocation (and, with a
remote registry, larger permanently-wasted gaps if a node restarts mid-block). The
threshold should leave enough headroom to reserve the next block before the current one
is exhausted. As a rule of thumb, keep:
threshold >= peak_generation_rate * registry_allocation_latency
so background pre-allocation stays ahead of demand and Generate does not fall back to a
synchronous registry call.
Registry
type Registry interface {
// Allocate reserves blockSize sequence numbers for timestamp and returns the
// starting value of the reserved block. Successive calls for the same timestamp
// must return non-overlapping blocks.
Allocate(ctx context.Context, timestamp, blockSize int64) (start int64, err error)
}
The bundled registry/memory implementation is suitable for tests, examples, and
single-process deployments. It does not persist allocations across restarts.
Memory
import "github.com/from-cero/crid/registry/memory"
reg := memory.New()
Postgres
For durable, multi-process deployments, registry/postgres backs the registry with
PostgreSQL. Allocations are persisted, so a block is never handed out twice even across
restarts, and uniqueness holds for every node sharing the same database. It is a separate
module (so the core stays dependency-free) and uses the pgx
driver:
go get -u github.com/from-cero/crid/registry/postgres
import (
"github.com/from-cero/crid"
"github.com/from-cero/crid/registry/postgres"
"github.com/jackc/pgx/v5/pgxpool"
)
pool, err := pgxpool.New(ctx, dsn)
if err != nil {
log.Fatal(err)
}
defer pool.Close()
reg, err := postgres.New(pool)
if err != nil {
log.Fatal(err)
}
// Create the table once at startup, or manage it with migrations instead.
if err := reg.EnsureSchema(ctx); err != nil {
log.Fatal(err)
}
node, err := crid.New(reg)
New accepts any value satisfying its DB interface (*pgxpool.Pool, *pgx.Conn,
or pgx.Tx). The table name defaults to crid_allocations and can be overridden with
postgres.WithTable("schema.table"). Each Allocate is a single atomic UPSERT, so
concurrent callers across all processes receive non-overlapping blocks. The schema is:
CREATE TABLE IF NOT EXISTS crid_allocations (
ts BIGINT PRIMARY KEY,
next_seq BIGINT NOT NULL
);
A durable registry on any other backend can be added the same way, by implementing the
registry.Registry interface; the node algorithm is unchanged.
Guarantees and caveats
- Uniqueness is guaranteed for all nodes sharing one registry, as long as the registry honors its contract (non-overlapping blocks per timestamp).
- IDs are not time-ordered. The embedded timestamp is the time the block was reserved, not the time an individual ID was generated. Because the next block is reserved ahead of time, an ID can carry a timestamp from slightly earlier than its actual generation moment. Do not rely on these IDs being monotonic or k-sortable.
- Error sentinel. On failure
GeneratereturnsID(-1). Valid IDs are always non-negative, so a negative result unambiguously signals an error. Always check the returned error; do not treat0as "no ID" (zero is a valid ID). - Clock. Timestamps come from the system wall clock. A backward clock jump below the
epoch causes
ErrClockBeforeEpoch; uniqueness is never affected by clock movement because the registry tracks sequences per timestamp.
Errors
New/NewParser validation errors all wrap ErrInvalidConfig: ErrInvalidBitFormat,
ErrEpochInFuture, ErrInvalidBlockSize, ErrInvalidThreshold. New also returns
ErrNilRegistry.
Generate may return ErrClockBeforeEpoch, ErrTimestampOverflow, ErrSequenceOverflow
(too many IDs in one second), or ErrInvalidSequence, in addition to any error surfaced by
the registry. Use errors.Is to match them.
Concurrency
A Node is safe for concurrent use by multiple goroutines. Background pre-allocation runs
in its own goroutine and synchronizes on the node's lock.
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ErrInvalidConfig is returned when New is called with a config that fails validation. ErrInvalidConfig = errors.New("invalid config") // ErrInvalidBitFormat is returned when format.timestampBits + format.sequenceBits != 63. ErrInvalidBitFormat = errors.New("bit format must sum to 63") // ErrEpochInFuture is returned when cfg.epoch is the time // after the current system time at the time of Node creation. ErrEpochInFuture = errors.New("configured epoch is in the future") // ErrInvalidBlockSize is returned when cfg.blockSize is not a positive integer // or exceeds the maximum sequence number for the given format (2^format.sequenceBits). ErrInvalidBlockSize = errors.New("block size must be positive and <= max sequence number for the given format") // ErrInvalidThreshold is returned when cfg.threshold exceeds cfg.blockSize. ErrInvalidThreshold = errors.New("threshold must not exceed block size") // ErrNilRegistry is returned when New is called with a nil Registry. ErrNilRegistry = errors.New("registry cannot be nil") // ErrClockBeforeEpoch is returned when the system clock is behind the configured epoch. ErrClockBeforeEpoch = errors.New("system clock is before the configured epoch") // ErrTimestampOverflow is returned when the current time // exceeds the maximum representable timestamp for the given format. ErrTimestampOverflow = errors.New("timestamp exceeds maximum for the given format") // ErrSequenceOverflow is returned when too many IDs are generated in the same second // and the sequence number exceeds the maximum for the given format. ErrSequenceOverflow = errors.New("too many IDs generated in the same second") // ErrInvalidSequence is returned when the sequence number acquired from the registry // is out of range for the given format.sequenceBits. ErrInvalidSequence = errors.New("invalid sequence number for given format") )
Functions ¶
This section is empty.
Types ¶
type FormatOption ¶
type FormatOption func(*format)
FormatOption configures the bit layout of generated IDs.
func WithSequenceBits ¶
func WithSequenceBits(bits uint8) FormatOption
WithSequenceBits sets the number of bits used for the sequence component.
func WithTimestampBits ¶
func WithTimestampBits(bits uint8) FormatOption
WithTimestampBits sets the number of bits used for the timestamp component.
type ID ¶
type ID int64
ID is a 63-bit Snowflake-style distributed identifier.
func (ID) MarshalJSON ¶
MarshalJSON encodes the ID as a quoted decimal string to avoid precision loss in JavaScript, which cannot represent 63-bit integers exactly.
func (*ID) UnmarshalJSON ¶
UnmarshalJSON decodes the ID from a quoted decimal string.
type Node ¶
type Node struct {
// contains filtered or unexported fields
}
Node generates unique IDs by combining the timestamp at which a block of sequence numbers was reserved from the Registry with the sequence numbers in that block. The embedded timestamp therefore reflects block-allocation time, not the moment an individual ID is generated, so IDs are not strictly ordered by generation time. A Node is safe for concurrent use by multiple goroutines.
type Option ¶
type Option func(*config)
Option configures a Node or Parser at creation time.
func WithBlockSize ¶
WithBlockSize sets the number of sequence numbers a Node reserves from the registry per allocation. It must be positive. The default is 10,000.
func WithEpoch ¶
WithEpoch sets the reference time from which timestamps are measured. The epoch must not be in the future. The default is 2026-01-01 00:00:00 UTC.
func WithFormat ¶
func WithFormat(opts ...FormatOption) Option
WithFormat sets the bit layout of generated IDs using the given format options. By default, the timestamp/sequence split from defaultFormat is used.
func WithThreshold ¶
WithThreshold sets how many sequence numbers must remain in the current block before the Node asynchronously pre-allocates the next one. It must not exceed the block size. A value below 1 disables pre-allocation. The default is 5,000.
Source Files
Directories