refid

README

// SPDX-License-Identifier: Apache-2.0 // Copyright (c) 2026 Lanka Software Foundation

Reference ID Generation System (refid)

refid is a shared Go package in the OpenNSW system that generates structured, sequential reference IDs based on YAML configuration. It allows any NSW service to define and issue custom reference IDs without writing custom ID generation code.

Features

• Config-Driven: Define ID structures for multiple Issuers and ID Types purely via YAML.
• Typed Segments: Concatenate literal, list, date, sequence, and random segments into custom ID formats.
• Durable Counters, Pluggable Backend: Atomic sequence increment via raw SQL (no ORM) against either the bundled PostgreSQL (refid/store/postgres) or SQLite (refid/store/sqlite) backend, or bring your own refid.SequenceStore implementation.
• Random Segments, Collision-Checked: Fixed-length random values (numeric/alpha/alphanumeric) reserved via a pluggable refid.RandomStore, retrying on collision.
• Flexible Resets: Scope key templates allow counters (and random value uniqueness sets) to reset daily ({yyyyMMdd}), monthly ({yyyyMM}), yearly ({yyyy}), or never.
• Fail-Fast & Side-Effect Free: Two-pass generation validates all caller parameters before executing database side-effects, and a format is capped at one stateful (sequence/random) segment so a later segment's failure can never orphan an earlier one's already-committed side effect.

---

Quickstart

package main

import (
	"context"
	"database/sql"
	"fmt"
	"log"

	"github.com/OpenNSW/core/refid"
	"github.com/OpenNSW/core/refid/store/postgres"

	_ "github.com/jackc/pgx/v5/stdlib" // registers the "pgx" database/sql driver
)

func main() {
	ctx := context.Background()

	// 1. Load configuration
	cfg, err := refid.LoadConfig("config.yaml")
	if err != nil {
		log.Fatalf("failed to load config: %v", err)
	}

	// 2. Connect DB and migrate the sequence table
	dsn := "host=localhost port=5432 user=postgres password=postgres dbname=postgres sslmode=disable"
	db, err := sql.Open("pgx", dsn)
	if err != nil {
		log.Fatalf("failed to open db: %v", err)
	}
	if err := postgres.MigrateSequence(ctx, db); err != nil {
		log.Fatalf("migration failed: %v", err)
	}

	// 3. Initialize Registry. WithSequenceStore/WithRandomStore are both
	// optional — supply only the store(s) backing segment types your config
	// actually uses (this example's config has no random segments).
	store, err := postgres.NewSequence(db)
	if err != nil {
		log.Fatalf("failed to create store: %v", err)
	}
	reg, err := refid.NewRegistry(cfg, refid.WithSequenceStore(store))
	if err != nil {
		log.Fatalf("failed to create registry: %v", err)
	}

	// 4. Generate reference ID
	id, err := reg.Generate(ctx, "RTA", "application_id", map[string]string{
		"officeCode": "COL",
	})
	if err != nil {
		log.Fatalf("generation failed: %v", err)
	}

	fmt.Println("Generated Reference ID:", id)
	// Output: RTA-APP-COL-20260818-000001
}

---

Segment Types

Segment	Description	Key Fields	Example
literal	Fixed text string	value: "FCAU-"	FCAU-
list	Parameter value validated against a controlled list	list: office_location, param: officeCode	COL
date	Current UTC date/time using Go reference layout	layout: "20060102"	20260818
sequence	Zero-padded durable counter	sequence: {scopeKey: "{issuer}:{idType}:{officeCode}:{yyyyMMdd}", padding: 6}	000042
random	Fixed-length random value, collision-checked via RandomStore	random: {scopeKey: "{issuer}:{idType}", charset: alphanumeric, length: 8}	7K2QQXAB

sequence and random are each configured under their own nested block (SequenceSegmentConfig/RandomSegmentConfig) rather than flat fields on the segment, and a format may contain at most one of them combined — see Stateful Segment Limit below.

random fields

Field	Description
scopeKey	Same template syntax as sequence — determines the uniqueness scope for generated values.
charset	One of numeric, alpha, alphanumeric.
length	Number of characters to generate (must be ≥ 1).
maxAttempts	Collision retries before Generate returns ErrRandomExhausted. Optional; defaults to 10, must be between 0 and 100.

---

Scope Key Placeholders & Reset Cadence

Sequence and random segments each resolve a scopeKey template per generation call — for sequence it scopes a durable counter, for random it scopes the set of previously issued values checked for collisions. Each unique scope key gets its own independent counter or uniqueness set.

Reserved placeholders:

• {issuer} — Issuing authority (e.g. "RTA")
• {idType} — Format identifier (e.g. "application_id")
• {yyyy} — Current 4-digit UTC year (Yearly reset)
• {yyyyMM} — Current UTC year + month (Monthly reset)
• {yyyyMMdd} — Current UTC year + month + day (Daily reset)
• {<param>} — Any caller-supplied param (e.g. {officeCode})

[!NOTE] Curly braces { and } are reserved syntax for placeholder delimiters in scopeKey templates.

---

Stateful Segment Limit

sequence and random are the only segment types with a side effect that persists to a store (a counter increment, a random value reservation) — literal/list/date are pure functions of the caller's params and the current time. Generate validates every segment first, then renders them in order with no rollback: if a format had two or more stateful segments and a later one failed during render (a sequence overflowing, a random segment exhausting its retries), an earlier one's already-committed side effect would be permanently orphaned — for a random segment, that permanently wastes one value from its bounded charset/length space with no ID ever returned.

To rule this out, NewRegistry rejects any format with more than one sequence/random segment combined. A format can still mix any number of literal/list/date segments with at most one of sequence or random.

---

Database Setup

refid.SequenceStore (Next(ctx, scopeKey, max) (int64, error)) and refid.RandomStore (Reserve(ctx, scopeKey, value) error) are pluggable interfaces; the package ships raw-SQL backends for both, each in its own subpackage. Wire in SequenceStore via refid.WithSequenceStore, RandomStore via refid.WithRandomStore — both optional, needed only if your config uses the corresponding segment type.

Neither backend registers a database/sql driver — they only issue SQL against the *sql.DB you hand them, so you import the driver, open the connection, and pass the result in. That keeps the driver choice yours, and avoids an init panic in a binary that already registers the same driver name.

PostgreSQL (refid/store/postgres)

SequenceStore uses a single table (refid_sequences by default) with row-level atomic upsert:

CREATE TABLE IF NOT EXISTS refid_sequences (
    scope_key  TEXT        NOT NULL PRIMARY KEY,
    counter    BIGINT      NOT NULL DEFAULT 0,
    updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

Initialize it automatically via postgres.MigrateSequence(ctx, db). To use a custom table name:

store, err := postgres.NewSequence(db, postgres.WithTableName("custom_sequences"))
err = postgres.MigrateSequence(ctx, db, postgres.WithTableName("custom_sequences"))

db is a *sql.DB you opened yourself — the queries use PostgreSQL's native $1 placeholders, so any PostgreSQL driver works. The Quickstart above uses pgx.

RandomStore uses a similarly shaped table (refid_random by default), keyed on (scope_key, value) rather than incrementing a counter — every random-segment format shares this one table, distinguished by scope_key:

CREATE TABLE IF NOT EXISTS refid_random (
    scope_key  TEXT        NOT NULL,
    value      TEXT        NOT NULL,
    issued_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (scope_key, value)
);

randomStore, err := postgres.NewRandom(db, postgres.WithTableName("custom_random"))
err = postgres.MigrateRandom(ctx, db, postgres.WithTableName("custom_random"))

SQLite (refid/store/sqlite)

Same schema shapes and API for both SequenceStore and RandomStore. Import a SQLite driver — modernc.org/sqlite is pure Go, no CGO:

import _ "modernc.org/sqlite" // registers the "sqlite" driver

db, err := sql.Open("sqlite", "refid.db")
if err := sqlite.MigrateSequence(ctx, db); err != nil { ... }
store, err := sqlite.NewSequence(db)

if err := sqlite.MigrateRandom(ctx, db); err != nil { ... }
randomStore, err := sqlite.NewRandom(db)

SQLite allows only one writer at a time, and nothing sets a busy timeout by default, so concurrent access can fail immediately with SQLITE_BUSY. Set a busy timeout in the DSN (e.g. sql.Open("sqlite", "file:refid.db?_busy_timeout=5000")) if you need Next/Reserve to wait instead of failing, or use refid/store/postgres for real concurrent-safe access. This makes SQLite a convenient choice for local development and tests.

Bring your own backend

Any type implementing refid.SequenceStore or refid.RandomStore works — a Redis-backed counter, an in-memory store for tests, etc. Whether Next/Reserve is safe under concurrent or multi-process use is entirely up to your implementation; the two bundled backends above sit at different points on that spectrum, so check their docs for what each one actually guarantees.

---

Error Handling

Check sentinel errors using errors.Is(err, refid.Err...):

• refid.ErrUnknownIssuer — Issuer not configured in registry.
• refid.ErrUnknownIDType — ID Type not found under specified issuer.
• refid.ErrInvalidParam — Required param missing, not in allowed list, or scopeKey placeholder un-substituted.
• refid.ErrCounterOverflow — Sequence counter value exceeds configured padding width.
• refid.ErrRandomExhausted — Random segment found no unreserved value within maxAttempts; widen the charset/length or narrow the scope key.
• refid.ErrRandomCollision — Returned by a RandomStore.Reserve implementation when a value is already reserved under a scope key; Generate retries internally on this, so callers of Generate don't normally see it directly.

Documentation

Overview

Package refid provides a shared, config-driven reference ID generation system for NSW services.

Overview

Each ID format is defined as an ordered list of typed segments (literal, list, date, sequence, random) that are concatenated at generation time. Formats are grouped by issuer and identified by an idType, and are looked up from a Registry by (issuer, idType). A format may contain at most one stateful segment (sequence or random) — see "Stateful segment limit" below.

Usage

cfg, err := refid.LoadConfig("refid_config.yaml")
store, err := postgres.NewSequence(db) // github.com/OpenNSW/core/refid/store/postgres
reg, err := refid.NewRegistry(cfg, refid.WithSequenceStore(store))

id, err := reg.Generate(ctx, "RTA", "application_id", map[string]string{
    "officeCode": "COL",
})
// id == "RTA-APP-COL-20260817-000042"

WithSequenceStore and WithRandomStore are both optional — only supply the one(s) backing segment types actually used in cfg. If cfg uses random segments, also pass a RandomStore:

randomStore, err := postgres.NewRandom(db)
reg, err := refid.NewRegistry(cfg, refid.WithSequenceStore(store), refid.WithRandomStore(randomStore))

Config format

See the package-level example_config.yaml for a fully annotated example.

Scope key placeholders

Sequence and random segments use a scopeKey template to determine the scope within which their values must be unique (a counter for sequence segments, the set of previously issued values for random segments). Curly braces '{' and '}' are reserved in scopeKey templates for placeholder delimiters. The following placeholders are resolved at generation time:

{issuer}    — the issuer identifier for this format
{idType}    — the ID type identifier for this format
{yyyy}      — four-digit year (UTC)
{yyyyMM}    — year and month (UTC)
{yyyyMMdd}  — year, month, and day (UTC)
{<param>}   — any caller-supplied param not already claimed above

Including {yyyyMMdd} in a scope key gives a daily-resetting counter (or daily-reset uniqueness set, for random segments); omitting all date components gives a scope that never resets.

Counter overflow

If a counter exceeds the maximum value representable with the configured padding width, Generate returns ErrCounterOverflow. This is intentional: a wider-than-expected ID would silently break any downstream system that validates ID length. Operations should be alerted and the scope key configuration reviewed.

Random segment exhaustion

A random segment generates a value from its charset/length and reserves it via RandomStore, retrying on collision up to maxAttempts (default 10). If every attempt collides, Generate returns ErrRandomExhausted — this means the charset/length combination is too small for the number of values already issued in that scope; widen the charset or length, or narrow the scope key (e.g. add {yyyyMMdd}).

Stateful segment limit

Generate has no transaction or rollback across segments: it validates all segments, then renders them in order, and a sequence or random segment's store call (a counter increment, a random value reservation) takes effect immediately as a side effect of rendering. If a format had two or more stateful segments and a later one failed during render (e.g. a sequence segment overflowing, or a random segment exhausting its retries), an earlier stateful segment's already-committed side effect would be permanently orphaned — for a random segment, that permanently wastes one value from its bounded charset/length space for no returned ID. To rule this out, NewRegistry rejects any format with more than one sequence or random segment combined.

Index

• Constants
• Variables
• type Config
  • func LoadConfig(path string) (Config, error)
• type FormatConfig
• type IssuerConfig
• type RandomSegmentConfig
• type RandomStore
• type Registry
  • func NewRegistry(cfg Config, opts ...RegistryOption) (Registry, error)
• type RegistryOption
  • func WithRandomStore(store RandomStore) RegistryOption
  • func WithSequenceStore(store SequenceStore) RegistryOption
• type SegmentConfig
• type SequenceSegmentConfig
• type SequenceStore

Constants

const (
	SegmentTypeLiteral  = "literal"
	SegmentTypeList     = "list"
	SegmentTypeDate     = "date"
	SegmentTypeSequence = "sequence"
	SegmentTypeRandom   = "random"
)

Segment type names accepted by SegmentConfig.Type.

const (
	CharsetNumeric      = "numeric"
	CharsetAlpha        = "alpha"
	CharsetAlphanumeric = "alphanumeric"
)

Charset names accepted by a random segment's Charset config field.

Variables

var (
	// ErrUnknownIssuer is returned when Generate is called with an issuer
	// that was not present in the config used to build the registry.
	ErrUnknownIssuer = errors.New("refid: unknown issuer")

	// ErrUnknownIDType is returned when Generate is called with an idType
	// that was not declared under the given issuer.
	ErrUnknownIDType = errors.New("refid: unknown id type")

	// ErrInvalidParam is returned when a list segment's required caller-supplied
	// param is missing or its value is not in the allowed list.
	ErrInvalidParam = errors.New("refid: invalid or missing param")

	// ErrCounterOverflow is returned when the sequence counter value for a scope
	// key exceeds the number of digits allowed by the segment's padding setting.
	// For example, a counter of 1,000,001 with padding:6 would produce a 7-digit
	// string, breaking the expected ID format. Callers should alert operations
	// when this occurs; the scope key is likely configured too broadly.
	ErrCounterOverflow = errors.New("refid: sequence counter exceeds padding width")

	// ErrRandomCollision is returned by RandomStore.Reserve when the given
	// value is already reserved under the same scope key. A random segment
	// treats this as a signal to generate a new value and retry, up to its
	// configured maxAttempts.
	ErrRandomCollision = errors.New("refid: random value already reserved for this scope")

	// ErrRandomExhausted is returned when a random segment could not find an
	// unreserved value within its configured maxAttempts. This usually means
	// the charset/length combination is too small for the volume of IDs being
	// issued in that scope; widen the charset or length, or narrow the scope.
	ErrRandomExhausted = errors.New("refid: random segment exhausted attempts without finding an unreserved value")
)

Sentinel errors returned by the refid package. Use errors.Is to check for these in calling code.

Types

type Config

type Config struct {
	// Issuers is the ordered list of issuer definitions. Each issuer owns one
	// or more ID type formats.
	Issuers []IssuerConfig `yaml:"issuers"`

	// Lists is a map of named value sets. Segments of type "list" reference
	// these by name, and the value supplied by the caller at generation time is
	// validated against the corresponding slice.
	Lists map[string][]string `yaml:"lists"`
}

Config is the top-level configuration for the ID generation system. It is typically loaded from a YAML file via LoadConfig, but can also be constructed programmatically (e.g. in tests or for embedded defaults).

func LoadConfig

func LoadConfig(path string) (Config, error)

LoadConfig reads and parses a YAML configuration file at the given path. It performs no semantic validation; call NewRegistry to validate the config against the full constraint set.

type FormatConfig

type FormatConfig struct {
	// IDType uniquely identifies this format within the issuer, e.g.
	// "application_id" or "permit_id". It is also available as the {idType}
	// placeholder in scope key templates.
	IDType string `yaml:"idType"`

	// Segments is the ordered list of typed segments whose rendered outputs are
	// concatenated to form the final ID string.
	Segments []SegmentConfig `yaml:"segments"`
}

FormatConfig describes how a single ID type is assembled from an ordered list of segments.

type IssuerConfig

type IssuerConfig struct {
	// Issuer is the unique identifier for this issuing authority, e.g. "RTA".
	// It is also available as the {issuer} placeholder in scope key templates.
	Issuer string `yaml:"issuer"`

	// Formats is the list of ID type formats owned by this issuer.
	Formats []FormatConfig `yaml:"formats"`
}

IssuerConfig groups all ID format definitions for a single issuer.

type RandomSegmentConfig

type RandomSegmentConfig struct {
	// ScopeKey is a template string determining the scope within which
	// generated values must be unique. Placeholders are resolved at
	// generation time; see the Registry documentation for the full
	// placeholder reference.
	ScopeKey string `yaml:"scopeKey"`

	// Charset selects the alphabet to draw characters from.
	// One of: "numeric", "alpha", "alphanumeric".
	Charset string `yaml:"charset"`

	// Length is the number of characters to generate.
	Length int `yaml:"length"`

	// MaxAttempts caps the number of collision retries before Generate
	// returns ErrRandomExhausted. Defaults to 10 if unset; must be between
	// 0 and 100.
	MaxAttempts int `yaml:"maxAttempts,omitempty"`
}

RandomSegmentConfig holds the settings for a random segment.

type RandomStore

type RandomStore interface {
	// Reserve atomically records value as issued under scopeKey. If value is
	// already reserved under the same scope key, Reserve returns
	// ErrRandomCollision without side effects, and the caller should generate
	// a new value and retry.
	Reserve(ctx context.Context, scopeKey, value string) error
}

RandomStore is the persistence interface for tracking issued random ID values so that collisions can be detected. Each distinct scope key has its own independent set of reserved values.

The refid/store/postgres and refid/store/sqlite subpackages each provide an implementation via their NewRandom constructor. Any caller that needs a different backend (Redis, in-memory for tests, etc.) can provide their own implementation.

type Registry

type Registry interface {
	// Generate produces a new ID for the given issuer and idType.
	//
	// params supplies caller-provided values consumed by list, sequence, and
	// random segments (e.g. map[string]string{"officeCode": "COL"}). Unused
	// keys are silently ignored; missing required keys return ErrInvalidParam.
	//
	// Errors:
	//   - ErrUnknownIssuer   — issuer not found in config
	//   - ErrUnknownIDType   — idType not found under the given issuer
	//   - ErrInvalidParam    — a required param is missing or has an invalid value
	//   - ErrCounterOverflow — sequence counter exceeds padding width
	//   - ErrRandomExhausted — random segment found no free value within maxAttempts
	Generate(ctx context.Context, issuer, idType string, params map[string]string) (string, error)
}

Registry is the entry point for generating IDs. Obtain one via NewRegistry.

Implementations must be safe for concurrent use by multiple goroutines.

func NewRegistry

func NewRegistry(cfg Config, opts ...RegistryOption) (Registry, error)

NewRegistry validates cfg and compiles all formats into an internal lookup table. It returns an error if the config contains:

• duplicate (issuer, idType) pairs
• a segment that references an undefined list name
• a segment with missing required fields (e.g. empty scopeKey, empty layout)
• an unrecognised segment type
• more than one stateful (sequence or random) segment in a single format

Fail-fast at startup: every error that would surface at generation time is caught here instead.

type RegistryOption

type RegistryOption func(*registryConfig)

RegistryOption configures optional behavior for NewRegistry.

func WithRandomStore

func WithRandomStore(store RandomStore) RegistryOption

WithRandomStore supplies the RandomStore used to back "random" segments. It is only required if the config contains at least one random segment; NewRegistry returns an error when called if one is used without this option set.

func WithSequenceStore

func WithSequenceStore(store SequenceStore) RegistryOption

WithSequenceStore supplies the SequenceStore used to back "sequence" segments. It is only required if the config contains at least one sequence segment; NewRegistry returns an error when called if one is used without this option set.

type SegmentConfig

type SegmentConfig struct {
	// Type is one of the SegmentType* constants: SegmentTypeLiteral,
	// SegmentTypeList, SegmentTypeDate, SegmentTypeSequence, SegmentTypeRandom.
	Type string `yaml:"type"`

	// Value is the fixed text for a literal segment.
	Value string `yaml:"value,omitempty"`

	// List is the name of a list defined in Config.Lists, used by list segments.
	List string `yaml:"list,omitempty"`

	// Param is the key the caller must supply in their params map, used by list
	// segments to look up the caller-provided value.
	Param string `yaml:"param,omitempty"`

	// Layout is a Go reference-date format string (e.g. "20060102"), used by
	// date segments.
	Layout string `yaml:"layout,omitempty"`

	// Sequence holds the settings for a sequence segment. Required (non-nil)
	// when Type is "sequence".
	Sequence *SequenceSegmentConfig `yaml:"sequence,omitempty"`

	// Random holds the settings for a random segment. Required (non-nil) when
	// Type is "random".
	Random *RandomSegmentConfig `yaml:"random,omitempty"`
}

SegmentConfig is the raw configuration for a single segment. Fields are interpreted according to Type; unused fields are ignored.

type SequenceSegmentConfig

type SequenceSegmentConfig struct {
	// ScopeKey is a template string determining the durable counter's scope.
	// Placeholders are resolved at generation time. Curly braces '{' and '}'
	// are reserved as placeholder delimiters in scope keys. See the Registry
	// documentation for the full placeholder reference.
	ScopeKey string `yaml:"scopeKey"`

	// Padding is the minimum number of digits for the counter. The counter is
	// zero-padded to this width. Must be between 1 and 18. If the counter
	// exceeds the maximum value representable with Padding digits,
	// ErrCounterOverflow is returned.
	Padding int `yaml:"padding"`
}

SequenceSegmentConfig holds the settings for a sequence segment.

type SequenceStore

type SequenceStore interface {
	// Next atomically increments the counter for the given scope key and returns
	// the new value, provided the current counter is less than max.
	// If the counter would exceed max, Next returns ErrCounterOverflow without
	// incrementing the counter in storage.
	Next(ctx context.Context, scopeKey string, max int64) (int64, error)
}

SequenceStore is the persistence interface for durable, atomic sequence counters. Each distinct scope key gets its own counter, starting at 1.

The refid/store/postgres and refid/store/sqlite subpackages each provide an implementation via their NewSequence constructor. Any caller that needs a different backend (Redis, in-memory for tests, etc.) can provide their own implementation.