README
¶
Distributed ID Generator | 中文文档
A high-performance Snowflake-based distributed ID generator in Go, with three bit-layout variants and a shard pool for near-lock-free throughput.
Three Variants at a Glance
| Variant | Timestamp bits | Precision | Machine ID bits | Sequence bits | Time span | Single-instance | Shard pool |
|---|---|---|---|---|---|---|---|
| v1 (this project) | 40 | 1 ms | 12 (4096 nodes) | 11 (2048/ms) | ~34 yr | ~890K/s | ~9.76M/s |
| v2 | 43 | 1 ms | 12 (4096 nodes) | 8 (256/ms) | ~278 yr | ~120K/s | ~1.02M/s |
| v3 | 42 | 1 ms | 12 (4096 nodes) | 9 (512/ms) | ~139 yr | ~250K/s | ~1.70M/s |
| bwmarrin/snowflake | 41 | 1 ms | 10 (1024 nodes) | 12 (4096/ms) | ~69 yr | ~4M/s | — |
| sony/sonyflake | 39 | 10 ms | 16 (65536 nodes) | 8 (256/10ms) | ~174 yr | ~25K/s | — |
| Twitter original | 41 | 1 ms | 10 (1024 nodes) | 12 (4096/ms) | ~69 yr | — | — |
All three variants use 12-bit machine IDs and 63 effective bits (int64 minus sign bit). The trade-off is fixed: timestamp + machine ID + sequence = 63 bits.
ID Structure (v1)
63 62 23 22 11 10 0
| | | | | | |
0 [ 40-bit ms timestamp ] [ 12-bit machine ID ] [ 11-bit sequence ]
| Field | Bits | Range | Notes |
|---|---|---|---|
| Sign | 1 | fixed 0 | guarantees positive int64 |
| Timestamp | 40 | 0 ~ 2⁴⁰-1 | ms offset from epoch (2024-01-01), ~34 years |
| Machine ID | 12 | 0 ~ 4095 | up to 4096 distributed nodes |
| Sequence | 11 | 0 ~ 2047 | up to 2048 IDs per millisecond |
Bit assembly
id := tick<<23 | machineID<<11 | sequence
Because the timestamp occupies the high bits, IDs are naturally time-ordered — ORDER BY id replaces ORDER BY created_at.
Time span
2⁴⁰ - 1 = 1,099,511,627,775 ms ≈ 34.8 years (from 2024-01-01, until ~2058)
When the epoch approaches expiry, update it to a more recent date to extend the range.
NextID Logic
Call NextID()
│
▼
Lock (sync.Mutex)
│
▼
tick = time.Now().UnixMilli() - epoch
│
├─ tick < lastStamp ──► clock rollback: spin until tick > lastStamp
│
├─ tick == lastStamp ─► same millisecond
│ │
│ ▼
│ sequence = (sequence + 1) & 0x7FF
│ │
│ └─ sequence == 0 ──► sequence exhausted: spin to next tick
│
└─ tick > lastStamp ──► new millisecond, reset sequence to 0
│
▼
lastStamp = tick
id = tick<<23 | machineID<<11 | sequence
Unlock, return id
Clock rollback — caused by NTP sync or VM clock drift. The generator spins until the clock catches up. Suitable for small rollback amounts.
Sequence exhaustion — on the 2049th call within the same millisecond, the sequence wraps to 0 and the goroutine spins (while holding the lock) until the next millisecond. This is the single-instance bottleneck under high concurrency.
Shard Pool Design
Root cause
sync.Mutex is a global serialization point. Under N concurrent goroutines, only one executes at a time — the higher the concurrency, the worse the lock contention.
Solution: shard by physical CPU count
┌─ shard[0] machineID = base+0 own lock
goroutine 0,8,16 ──►│
├─ shard[1] machineID = base+1 own lock
goroutine 1,9,17 ──►│
├─ shard[2] machineID = base+2 own lock
goroutine 2,10,18──►│
│ ...
└─ shard[N-1] own lock
Each shard is an independent Snowflake instance with its own lock, sequence counter, and lastStamp. They never block each other.
shard = idx % size
machineID = (baseID + shardIndex) & 0xFFF // unique per shard, globally unique IDs
Shard count = runtime.NumCPU(), aligned with GOMAXPROCS. One shard per CPU core minimizes lock contention.
Benchmark (4-core machine)
| Scenario | Latency | Notes |
|---|---|---|
| Single-instance, serial | 343 ns/op | baseline lock overhead, no contention |
| Single-instance, concurrent | 435 ns/op | 8 goroutines competing |
| Shard pool, serial | 353 ns/op | modulo overhead, on par with single |
| Shard pool, concurrent | 50 ns/op | contention spread, near lock-free, 8.6× faster |
Bit Trade-off Rules
timestamp bits +1 → time span ×2, machine ID or sequence bits -1
sequence bits -1 → throughput /2, timestamp bits +1 available
machine ID bits -1 → node count /2, timestamp bits +1 available
UUID v4 vs ULID vs Snowflake
UUID v4 — unordered
Fully random 128-bit value, no time information.
Generated order:
[0] e2649244-8c88-4a46-8011-6e104351a0f4
[1] a384de78-6503-460b-9a74-cee55201ffe5
[2] 13942ecc-8133-4f05-8c07-77288c8e6a3e
...
Sorted order is completely different
- Random B-Tree insertions cause frequent page splits → poor write performance
- Cannot infer time range from ID range
- Requires a separate
created_atcolumn for ordering
ULID — lexicographically ordered
128-bit: high 48 bits = ms timestamp, low 80 bits = random. Lexicographic order = time order.
Generated order (10 ms apart):
[0] 01KP0FWTYQ1F7P10V3ET41KDN8 time part: 01KP0FWTYQ
[1] 01KP0FWTZ2PWP7YCH7EXDCV84D time part: 01KP0FWTZ2
...
In-order: true
String ordering is index-friendly compared to UUID, but storage and comparison cost is 2× that of an integer.
Snowflake — integer ordered
id1 < id2 ⟺ id1 was generated before id2
Integer index is the most compact. ORDER BY id directly replaces ORDER BY created_at, and the generation time can be decoded from the ID.
Comparison
| UUID v4 | ULID | Snowflake | |
|---|---|---|---|
| Type | string (36 chars) | string (26 chars) | int64 |
| Ordering | none | lexicographic | integer |
| Distributed | native | native | requires machine ID coordination |
| DB index | poor | good | best |
| Decode time | no | yes | yes |
| Storage | 16 bytes | 16 bytes | 8 bytes |
| Throughput | ~14M/s | ~28M/s | ~2M/s (single) / ~14.8M/s (shard pool) |
Spin vs Sleep Strategy
SonyflakeCompat uses the same bit layout as v1 but replaces spin-wait with sleep on sequence exhaustion:
// Spin (this project): holds lock, all other goroutines block
for t <= last { t = currentTick() }
// Sleep (SonyCompat): releases lock, other goroutines can proceed
s.elapsedTime++
time.Sleep(time.Duration(overtime) * time.Millisecond)
Under high concurrency, sleep can actually be faster: while spinning holds the lock and blocks everyone, sleeping releases it so other goroutines keep producing IDs in parallel.
The shard pool eliminates this problem entirely — contention drops to 1/N, sequence exhaustion becomes rare, and the two strategies converge.
Auto Machine ID Derivation
// Take the last two bytes of the first non-loopback NIC's MAC address, keep low 12 bits
val := int64(mac[len(mac)-2])<<8 | int64(mac[len(mac)-1])
return val & 0xFFF, nil
MAC addresses are globally unique, so collision probability within the same LAN is negligible. No manual configuration needed.
Note: getMachineID takes ~4.3 ms and allocates on the heap. Call it once at initialization — never on the hot path.
Quick Start
// Single instance
sf, err := snowflakeid.NewSnowflakeAuto()
id, err := sf.NextID()
// Shard pool (recommended for high concurrency)
pool, err := snowflakeid.NewShardPool(sf.MachineID())
id, err := pool.NextID(goroutineIndex)
Run Benchmark
go run ./cmd/benchmark
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type ShardPool ¶
type ShardPool struct {
// contains filtered or unexported fields
}
ShardPool 按物理核心数分片的 Snowflake 池，每个分片独立加锁，消除竞争。 物理核心数比逻辑核心数更准确，超线程共享执行单元，分片数对齐物理核心竞争最低。
ShardPool is a sharded pool of Snowflake generators, one shard per physical CPU core. Each shard has its own lock, eliminating cross-shard contention. Physical cores are preferred over logical cores because hyper-threads share execution units; aligning shard count to physical cores minimises lock contention.
func NewShardPool ¶
NewShardPool 创建分片池，baseID 为机器ID基础值，分片数等于物理核心数。 每个分片的 machineID = (baseID + shardIndex) & 0xFFF，保证全局唯一。
NewShardPool creates a shard pool. baseID is the base machine ID; each shard gets machineID = (baseID + shardIndex) & 0xFFF, ensuring global uniqueness.
type ShardPool2 ¶
type ShardPool2 struct {
// contains filtered or unexported fields
}
ShardPool2 本项目2的分片池（序列号8位，时间戳43位，~278年） ShardPool2 is the shard pool for Snowflake2 (8-bit sequence, 43-bit timestamp, ~278 years).
func NewShardPool2 ¶
func NewShardPool2(baseID int64) (*ShardPool2, error)
NewShardPool2 创建 Snowflake2 分片池，baseID 为机器ID基础值。 NewShardPool2 creates a shard pool for Snowflake2; baseID is the base machine ID.
func (*ShardPool2) NextID ¶
func (p *ShardPool2) NextID(idx int64) (int64, error)
NextID 根据调用方索引路由到对应分片生成 ID。 NextID routes to the shard at idx % size and generates an ID.
func (*ShardPool2) Size ¶
func (p *ShardPool2) Size() int64
Size 返回分片数（等于物理核心数）。 Size returns the number of shards (equals physical core count).
type ShardPool3 ¶
type ShardPool3 struct {
// contains filtered or unexported fields
}
ShardPool3 本项目3的分片池（序列号9位，时间戳42位，~139年） ShardPool3 is the shard pool for Snowflake3 (9-bit sequence, 42-bit timestamp, ~139 years).
func NewShardPool3 ¶
func NewShardPool3(baseID int64) (*ShardPool3, error)
NewShardPool3 创建 Snowflake3 分片池，baseID 为机器ID基础值。 NewShardPool3 creates a shard pool for Snowflake3; baseID is the base machine ID.
func (*ShardPool3) NextID ¶
func (p *ShardPool3) NextID(idx int64) (int64, error)
NextID 根据调用方索引路由到对应分片生成 ID。 NextID routes to the shard at idx % size and generates an ID.
func (*ShardPool3) Size ¶
func (p *ShardPool3) Size() int64
Size 返回分片数（等于物理核心数）。 Size returns the number of shards (equals physical core count).
type Snowflake ¶
type Snowflake struct {
// contains filtered or unexported fields
}
Snowflake 分布式ID生成器（毫秒精度） Snowflake is a distributed ID generator with millisecond precision.
func NewSnowflake ¶
NewSnowflake 创建生成器，machineID 范围 [0, 4095] NewSnowflake creates a generator; machineID must be in [0, 4095].
func NewSnowflakeAuto ¶
NewSnowflakeAuto 自动从本机MAC地址派生 machineID NewSnowflakeAuto derives the machineID automatically from the local MAC address.
type Snowflake2 ¶
type Snowflake2 struct {
// contains filtered or unexported fields
}
Snowflake2 压缩序列号版本：序列号8位，时间戳43位，可用约278年 Snowflake2 trades sequence bits for a longer timestamp: 8-bit sequence (256/ms), 43-bit timestamp (~278 years).
func NewSnowflake2 ¶
func NewSnowflake2(machineID int64) (*Snowflake2, error)
NewSnowflake2 创建 Snowflake2 生成器，machineID 范围 [0, 4095] NewSnowflake2 creates a Snowflake2 generator; machineID must be in [0, 4095].
func (*Snowflake2) MachineID ¶
func (s *Snowflake2) MachineID() int64
MachineID 返回当前实例使用的机器ID MachineID returns the machine ID used by this instance.
func (*Snowflake2) NextID ¶
func (s *Snowflake2) NextID() (int64, error)
NextID 生成下一个唯一ID NextID generates the next unique ID.
type Snowflake3 ¶
type Snowflake3 struct {
// contains filtered or unexported fields
}
Snowflake3 序列号9位版本：时间戳42位，可用约139年，每ms最多512个ID Snowflake3 is a balanced variant: 9-bit sequence (512/ms), 42-bit timestamp (~139 years).
func NewSnowflake3 ¶
func NewSnowflake3(machineID int64) (*Snowflake3, error)
NewSnowflake3 创建 Snowflake3 生成器，machineID 范围 [0, 4095] NewSnowflake3 creates a Snowflake3 generator; machineID must be in [0, 4095].
func (*Snowflake3) MachineID ¶
func (s *Snowflake3) MachineID() int64
MachineID 返回当前实例使用的机器ID MachineID returns the machine ID used by this instance.
func (*Snowflake3) NextID ¶
func (s *Snowflake3) NextID() (int64, error)
NextID 生成下一个唯一ID NextID generates the next unique ID.
type SonyflakeCompat ¶
type SonyflakeCompat struct {
// contains filtered or unexported fields
}
func NewSonyflakeCompat ¶
func NewSonyflakeCompat(machineID int64) *SonyflakeCompat
NewSonyflakeCompat 创建 SonyflakeCompat 实例，machineID 截取低12位。 NewSonyflakeCompat creates a SonyflakeCompat instance; only the low 12 bits of machineID are used.
func (*SonyflakeCompat) NextID ¶
func (s *SonyflakeCompat) NextID() (int64, error)
NextID 生成下一个唯一ID。 序列号耗尽时使用 sleep 让出调度（sony 风格），而非自旋持锁。
NextID generates the next unique ID. When the sequence is exhausted it sleeps (sony style) instead of spinning, releasing the lock so other goroutines can proceed.
Source Files
Directories