Help us shape the future of Go! Take the 2026 Go Developer Survey

idempotent

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 29, 2026 License: MIT Imports: 9 Imported by: 0

README

go-idempotent logo

go-idempotent

A typed, storage-backed idempotency layer for Go APIs, workers, webhooks, and retryable workflows.

Go Reference CI Go Report Card MIT license

Overview

go-idempotent suppresses duplicate execution by atomically assigning one caller ownership of an idempotency key. Later callers receive the recorded outcome, wait for it, reject it, or return a pending state—according to policy.

It is designed for safely retryable HTTP endpoints, at-least-once queue consumers, webhook handlers, scheduled jobs, and payment-oriented application workflows.

Idempotency makes repeated requests converge on one stored outcome. It is not a universal exactly-once guarantee or a distributed transaction across external systems.

Highlights

  • Atomic ownership acquisition with owner and fencing tokens
  • Lease heartbeats and stale-owner recovery
  • Typed results with generic executors and pluggable codecs
  • Request fingerprints to reject conflicting key reuse
  • Configurable pending, TTL, failure, and panic semantics
  • In-memory and SQLite stores; standard-library net/http middleware
  • Lifecycle hooks, contract testing, fuzz targets, and benchmarks

Install

Requires Go 1.23 or later.

go get github.com/magnexis/go-idempotent

Quick start

records := memory.New[Order]()
defer records.Close()

executor := idempotent.New[Order](
    idempotent.WithStore[Order](records),
    idempotent.WithSuccessTTL[Order](24*time.Hour),
)

result, err := executor.DoWithOptions(ctx, "checkout:order-123", createOrder,
    idempotent.WithFingerprint(idempotent.FingerprintJSON(request)),
)

The first caller runs createOrder. Concurrent duplicates replay the completed result. A reused key with a different fingerprint returns ErrFingerprintMismatch.

How it works

caller → executor → fingerprint / policy / codec → store
                                              ├─ memory
                                              └─ SQLite
  1. The store atomically creates a pending record and grants its owner token.
  2. Only that owner runs the operation; automatic heartbeats renew its lease.
  3. Completion and failure compare the owner token and fencing token.
  4. A stale record may be reclaimed after its lease expires; the old owner can no longer write.

HTTP middleware

records := memory.New[middleware.Response]()
executor := idempotent.New[middleware.Response](idempotent.WithStore[middleware.Response](records))

handler := middleware.Idempotency(executor,
    middleware.WithMaxRequestBody(1<<20),
    middleware.WithMaxResponseBody(1<<20),
    middleware.WithFingerprintHeaders("X-API-Version"),
)(checkoutHandler)

For POST, PUT, and PATCH, the middleware fingerprints the method, path, normalized query, bounded body, and selected stable headers. It replays the stored status, body, and allowlisted headers. Conflicting requests receive 409 Conflict.

go run ./examples/http-api
curl -i -X POST http://localhost:8080/checkout -H "Idempotency-Key: checkout-123"

Do not fingerprint credentials, cookies, request IDs, timestamps, or tracing headers. Streaming, hijacked, and WebSocket responses should bypass this middleware.

Workers and ownership loss

result, err := executor.Do(ctx, idempotent.Key(event.ID), func(ctx context.Context) (Receipt, error) {
    _, ownership := idempotent.OwnershipFromContext(ctx)
    select {
    case <-ownership.Lost():
        return Receipt{}, idempotent.ErrOwnershipLost
    default:
        return processEvent(ctx, event)
    }
})

When heartbeat renewal fails, the operation context is canceled and Ownership.Lost() closes. Stop critical side effects immediately—another process may reclaim the record.

Store support

Store Status Intended use
Memory Available Tests and single-process services
SQLite Available Embedded and single-node applications
Redis Planned Shared distributed coordination
PostgreSQL Planned Shared relational coordination

SQLite enables WAL mode and a busy timeout, but it is not a multi-host coordination solution. See store guidance.

Failure and security semantics

Successful and failed results have separate TTLs. Failure caching is controlled by WithFailurePolicy; cached failures replay as *idempotent.StoredError, not an invented original error type.

Namespace untrusted keys and enforce key/result/request limits. Hash persistence keys when they may contain sensitive data. Authorize before idempotency middleware: replay must never become an authorization bypass.

Development

go test ./...
go test -race ./...
go vet ./...
go test -bench=. -benchmem ./...

See CONTRIBUTING.md, SECURITY.md, and the architecture notes.

Roadmap

  • Redis and PostgreSQL adapters with live integration tests
  • Expanded store-contract coverage and cancellation/failure injection
  • Prometheus and OpenTelemetry examples

License

MIT. See LICENSE.

Documentation

Overview

Package idempotent provides storage-backed duplicate suppression for operations.

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrInvalidKey          = errors.New("invalid idempotency key")
	ErrFingerprintMismatch = errors.New("idempotency fingerprint mismatch")
	ErrInProgress          = errors.New("operation already in progress")
	ErrWaitTimeout         = errors.New("timed out waiting for operation")
	ErrOwnershipLost       = errors.New("idempotency ownership lost")
	ErrRecordExpired       = errors.New("idempotency record expired")
	ErrResultTooLarge      = errors.New("idempotency result too large")
	ErrCodec               = errors.New("idempotency codec failure")
	ErrStoreUnavailable    = errors.New("idempotency store unavailable")
)

Functions

func FingerprintBytes 

func FingerprintBytes(data []byte) string

func FingerprintJSON 

func FingerprintJSON(value any) string

FingerprintJSON creates a deterministic digest for JSON-compatible values.

func FingerprintString 

func FingerprintString(value string) string

func ValidateKey 

func ValidateKey(key Key) error

ValidateKey rejects empty keys and keys containing control characters. Executors additionally enforce their configured length limits.

Types

type AcquireRequest 

type AcquireRequest struct {
	Key                Key
	Fingerprint        string
	Metadata           map[string]string
	LeaseDuration, TTL time.Duration
	Now                time.Time
}

type AcquireResult 

type AcquireResult[T any] struct {
	Record              Record[T]
	Acquired, Reclaimed bool
}

type CallOption 

type CallOption func(*callConfig)

func WithFingerprint 

func WithFingerprint(value string) CallOption

func WithMetadata 

func WithMetadata(value map[string]string) CallOption

type CompleteRequest 

type CompleteRequest[T any] struct {
	Key        Key
	OwnerToken OwnerToken
	Fence      Fence
	Value      []byte
	Now        time.Time
	TTL        time.Duration
}

type ConflictError 

type ConflictError struct {
	Key                                      Key
	ExistingFingerprint, IncomingFingerprint string
}

ConflictError describes reuse of a key with a different request fingerprint.

func (*ConflictError) Error 

func (e *ConflictError) Error() string

func (*ConflictError) Unwrap 

func (e *ConflictError) Unwrap() error

type ErrorClass 

type ErrorClass string
const (
	ErrorRetryable ErrorClass = "retryable"
	ErrorPermanent ErrorClass = "permanent"
	ErrorUnknown   ErrorClass = "unknown"
)

type ErrorClassifier 

type ErrorClassifier interface{ Classify(error) ErrorClass }

type Event 

type Event struct {
	KeyHash, Namespace string
	State              State
	Owner, Replayed    bool
	Attempt            int
	Duration           time.Duration
	Store              string
}

Event contains safe lifecycle metadata. KeyHash never includes the raw key.

type Executor 

type Executor[T any] struct {
	// contains filtered or unexported fields
}

Executor runs typed operations using one Store.

func New 

func New[T any](options ...Option[T]) *Executor[T]

func (*Executor[T]) Do 

func (e *Executor[T]) Do(ctx context.Context, key Key, operation func(context.Context) (T, error)) (Result[T], error)

func (*Executor[T]) DoWithOptions 

func (e *Executor[T]) DoWithOptions(ctx context.Context, key Key, operation func(context.Context) (T, error), options ...CallOption) (Result[T], error)

func (*Executor[T]) Renew 

func (e *Executor[T]) Renew(ctx context.Context, key Key, token OwnerToken, fence Fence) error

Renew explicitly extends the caller's lease. Most callers should use automatic heartbeats, but manual renewal is useful for externally scheduled work.

type FailRequest 

type FailRequest struct {
	Key        Key
	OwnerToken OwnerToken
	Fence      Fence
	Error      string
	Now        time.Time
	TTL        time.Duration
	Delete     bool
}

type FailurePolicy 

type FailurePolicy string
const (
	NeverCacheFailures     FailurePolicy = "never"
	CacheAllFailures       FailurePolicy = "all"
	CacheSelectedFailures  FailurePolicy = "selected"
	CachePermanentFailures FailurePolicy = "permanent"
)

type Fence 

type Fence uint64

type FingerprintPolicy 

type FingerprintPolicy string
const (
	FingerprintDisabled FingerprintPolicy = "disabled"
	FingerprintOptional FingerprintPolicy = "optional"
	FingerprintRequired FingerprintPolicy = "required"
)

type Hooks 

type Hooks[T any] struct {
	OnAcquire        func(Event)
	OnReplay         func(Event)
	OnConflict       func(Event)
	OnComplete       func(Event, T)
	OnFailure        func(Event, error)
	OnOwnershipLost  func(Event)
	OnWait           func(Event)
	OnStaleReclaimed func(Event)
}

Hooks receives lifecycle events. Panics from callbacks are recovered.

type Key 

type Key string

func JoinKey 

func JoinKey(parts ...string) Key

JoinKey creates a namespaced key. Empty segments are ignored.

type KeyHasher 

type KeyHasher func(Key) Key

type Option 

type Option[T any] func(*config[T])

func WithCodec 

func WithCodec[T any](value codec.Codec[T]) Option[T]

func WithErrorClassifier 

func WithErrorClassifier[T any](value ErrorClassifier) Option[T]

func WithFailurePolicy 

func WithFailurePolicy[T any](value FailurePolicy) Option[T]

func WithFailureTTL 

func WithFailureTTL[T any](value time.Duration) Option[T]

func WithFingerprintPolicy 

func WithFingerprintPolicy[T any](value FingerprintPolicy) Option[T]

func WithHeartbeatInterval 

func WithHeartbeatInterval[T any](value time.Duration) Option[T]

func WithHooks 

func WithHooks[T any](value Hooks[T]) Option[T]

WithHooks registers best-effort lifecycle callbacks.

func WithKeyHasher 

func WithKeyHasher[T any](value KeyHasher) Option[T]

func WithLeaseDuration 

func WithLeaseDuration[T any](value time.Duration) Option[T]

func WithMaxKeyLength 

func WithMaxKeyLength[T any](value int) Option[T]

func WithMaxResultSize 

func WithMaxResultSize[T any](value int) Option[T]

func WithMinKeyLength 

func WithMinKeyLength[T any](value int) Option[T]

WithMinKeyLength configures the shortest accepted persisted key.

func WithNamespace 

func WithNamespace[T any](value string) Option[T]

func WithPanicPolicy 

func WithPanicPolicy[T any](value PanicPolicy) Option[T]

func WithPendingPolicy 

func WithPendingPolicy[T any](value PendingPolicy) Option[T]

func WithStore 

func WithStore[T any](store Store[T]) Option[T]

func WithSuccessTTL 

func WithSuccessTTL[T any](value time.Duration) Option[T]

func WithTTL 

func WithTTL[T any](value time.Duration) Option[T]

func WithWaitTimeout 

func WithWaitTimeout[T any](value time.Duration) Option[T]

type OwnerToken 

type OwnerToken string

type Ownership 

type Ownership interface {
	Lost() <-chan struct{}
}

Ownership exposes loss of the lease held by the running operation. A closed Lost channel means the operation must stop any critical side effects.

func OwnershipFromContext 

func OwnershipFromContext(ctx context.Context) (context.Context, Ownership)

OwnershipFromContext obtains execution ownership information, when the context was supplied to an Executor operation.

type PanicPolicy 

type PanicPolicy string
const (
	PanicPropagate PanicPolicy = "propagate"
	PanicAsFailure PanicPolicy = "failure"
)

type PendingPolicy 

type PendingPolicy string
const (
	PendingWait   PendingPolicy = "wait"
	PendingReject PendingPolicy = "reject"
	PendingPoll   PendingPolicy = "poll"
	PendingReturn PendingPolicy = "return_pending"
)

type Record 

type Record[T any] struct {
	Key                                                          Key
	Fingerprint                                                  string
	Metadata                                                     map[string]string
	State                                                        State
	OwnerToken                                                   OwnerToken
	Fence                                                        Fence
	Value                                                        []byte
	Error                                                        string
	CreatedAt, UpdatedAt, CompletedAt, LeaseExpiresAt, ExpiresAt time.Time
	Attempt                                                      int
}

Record is the durable state maintained by a Store.

func (Record[T]) Expired 

func (r Record[T]) Expired(now time.Time) bool

Expired reports whether this record has passed its retention deadline.

func (Record[T]) LeaseExpired 

func (r Record[T]) LeaseExpired(now time.Time) bool

LeaseExpired reports whether a pending record may be reclaimed at now.

type RenewRequest 

type RenewRequest struct {
	Key           Key
	OwnerToken    OwnerToken
	Fence         Fence
	Now           time.Time
	LeaseDuration time.Duration
}

type Result 

type Result[T any] struct {
	Value                             T
	State                             State
	Replayed, Owner                   bool
	CreatedAt, CompletedAt, ExpiresAt time.Time
	Attempt                           int
}

Result reports both the value and how it was obtained.

func Do 

func Do[T any](ctx context.Context, executor *Executor[T], key Key, operation func(context.Context) (T, error)) (Result[T], error)

func (Result[T]) Terminal 

func (r Result[T]) Terminal() bool

Terminal reports whether this result has reached a durable final state.

type State 

type State string
const (
	StatePending   State = "pending"
	StateCompleted State = "completed"
	StateFailed    State = "failed"
	StateExpired   State = "expired"
)

type Store 

type Store[T any] interface {
	Acquire(ctx context.Context, request AcquireRequest) (AcquireResult[T], error)
	Complete(ctx context.Context, request CompleteRequest[T]) error
	Fail(ctx context.Context, request FailRequest) error
	Get(ctx context.Context, key Key) (Record[T], error)
	Renew(ctx context.Context, request RenewRequest) error
	Delete(ctx context.Context, key Key) error
	Cleanup(ctx context.Context, now time.Time, limit int) (int, error)
}

Store atomically grants execution ownership and fences state transitions.

type StoredError 

type StoredError struct{ Message string }

StoredError is a replayable representation of an operation error. Its original Go type is deliberately not reconstructed.

func (*StoredError) Error 

func (e *StoredError) Error() string

Source Files

View all Source files

Directories

Path Synopsis
examples
http-api command
worker command
Package middleware contains net/http idempotency middleware.
 Package middleware contains net/http idempotency middleware.
store
contract
Package contract provides a conformance suite for Store implementations.
 Package contract provides a conformance suite for Store implementations.
memory
Package memory provides a process-local Store implementation.
 Package memory provides a process-local Store implementation.
sqlite
Package sqlite provides a SQLite Store for single-node and embedded services.
 Package sqlite provides a SQLite Store for single-node and embedded services.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.