rqlitestorage

README

caddy-rqlite-storage

A CertMagic / Caddy storage backend on top of rqlite — distributed, Raft-replicated SQLite.

For a fleet of Caddy instances that need a shared certificate store: certificates and keys are shared, and ACME challenge tokens live in rqlite, so an ACME challenge landing on any instance can be solved, and a cert issued by one instance is immediately usable by the others.

Why rqlite

The hard requirement of a multi-instance Caddy cluster is correct distributed locking — two instances must never solve or issue the same certificate concurrently. rqlite's Raft consensus provides linearizable locking for that. If you already run an rqlite cluster, this backend reuses it — no separate datastore to operate — and gets stronger locking than a single Redis would.

Layout

Package	Imports	Purpose
. (rqlitestorage)	stdlib only	Transport-agnostic core: storage + locking logic, rqlite HTTP transport. Fully unit-tested.
./caddymodule	caddy/v2, certmagic	Registers the caddy.storage.rqlite Caddy module and adapts the core to certmagic.Storage.

The split keeps the core's tests fast and dependency-light (only an in-memory SQLite for the test fake), while the Caddy glue stays isolated.

Build into Caddy

xcaddy build \
  --with github.com/caddy-dns/rfc2136 \
  --with github.com/mholt/caddy-ratelimit \
  --with github.com/tissue-systems/caddy-rqlite-storage/caddymodule

Configure (Caddyfile global options)

{
    storage rqlite {
        url      http://127.0.0.1:4001   # this instance's local rqlite node
        username {$RQLITE_USER}           # optional HTTP basic-auth
        password {$RQLITE_PASSWORD}
        lock_ttl 60s                      # how long a held ACME lock survives before it may be stolen
        read_level weak                   # rqlite read consistency: weak (default) | none | linearizable | strong
    }
}

Each instance points at its local rqlite node; Raft replication makes the store identical everywhere. Lock acquisition is always a leader write (consistent). Cert/key lookups default to weak reads (routed through the Raft leader, so the store always sees its own writes, at the cost of one intra-cluster hop). read_level none (local FSM reads) is faster but is safe only on single-node deployments: on a cluster, a node that is not the leader can read its own just-forwarded write back as missing (follower FSM lag), which fails CertMagic's write-then-read storage preflight and aborts certificate obtains with failed storage check: file does not exist.

Schema

Created automatically on provision (idempotent):

CREATE TABLE caddy_storage (key TEXT PRIMARY KEY, value TEXT /*base64*/, size INTEGER, modified INTEGER);
CREATE TABLE caddy_locks   (name TEXT PRIMARY KEY, token TEXT, expires INTEGER);

Test

go test ./... -race

The core tests run the exact SQL the store emits against an in-memory SQLite, covering storage round-trips (incl. binary), not-found semantics, listing (recursive / non-recursive / LIKE-escaping), the lock primitives (acquire / steal-after-expiry / refresh / release / token-ownership), an N-goroutine mutual-exclusion contention test, and the rqlite HTTP wire format end-to-end.

License

MIT

Documentation

Overview

Package rqlitestorage implements a CertMagic/Caddy storage backend on top of rqlite (https://rqlite.io) — distributed, Raft-replicated SQLite.

It lets a fleet of Caddy instances share one certificate store: certificates and keys are shared, and ACME challenge tokens live in rqlite, so whichever instance a challenge lands on can answer it, and a cert issued by one instance is immediately usable by the others. rqlite's Raft consensus provides the linearizable distributed locking an ACME cluster needs — two instances must never solve or issue the same certificate concurrently — without standing up a separate datastore for it.

This file is the transport-agnostic core: all storage and locking logic, exercised in full by the unit tests against an in-memory SQLite. The production rqlite HTTP transport is in httpconn.go; the Caddy module / certmagic.Storage adapter lives in the ./caddymodule subpackage so this core stays dependency-light.

Index

• Variables
• type CaddyLocks
  • func (c CaddyLocks) Table() string
• type CaddyStorage
  • func (c CaddyStorage) Table() string
• type KeyInfo
• type Statement
• type Store
  • func NewHTTPStore(baseURL, username, password string) *Store
  • func NewStore(c conn) *Store
  • func (s *Store) Del(ctx context.Context, key string) error
  • func (s *Store) EnsureSchema(ctx context.Context) error
  • func (s *Store) Get(ctx context.Context, key string) ([]byte, error)
  • func (s *Store) Has(ctx context.Context, key string) bool
  • func (s *Store) List(ctx context.Context, prefix string, recursive bool) ([]string, error)
  • func (s *Store) Lock(ctx context.Context, name string) error
  • func (s *Store) Put(ctx context.Context, key string, value []byte) error
  • func (s *Store) SetLockTTL(d time.Duration)
  • func (s *Store) SetReadLevel(level string) error
  • func (s *Store) Stat(ctx context.Context, key string) (KeyInfo, error)
  • func (s *Store) Unlock(ctx context.Context, name string) error

Variables

var ErrNotExist = errors.New("rqlitestorage: does not exist")

ErrNotExist is returned by Get/Stat/Del when the key is absent. The certmagic adapter maps it to fs.ErrNotExist so CertMagic's errors.Is checks succeed.

Types

type CaddyLocks

type CaddyLocks struct {
	Name    string `db:"name" json:"name"`
	Token   string `db:"token" json:"token"`
	Expires int64  `db:"expires" json:"expires"`
}

func (CaddyLocks) Table

func (c CaddyLocks) Table() string

type CaddyStorage

type CaddyStorage struct {
	Key      string `db:"key" json:"key"`
	Value    string `db:"value" json:"value"`
	Size     int64  `db:"size" json:"size"`
	Modified int64  `db:"modified" json:"modified"`
}

func (CaddyStorage) Table

func (c CaddyStorage) Table() string

type KeyInfo

type KeyInfo struct {
	Key      string
	Modified time.Time
	Size     int64
}

KeyInfo mirrors the subset of certmagic.KeyInfo the store produces.

type Statement

type Statement struct {
	SQL  string
	Args []any
}

Statement is one parameterized SQL statement (`?` placeholders).

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store holds the storage + locking logic over a conn.

func NewHTTPStore

func NewHTTPStore(baseURL, username, password string) *Store

NewHTTPStore builds a Store backed by the rqlite node reachable at baseURL (e.g. "http://127.0.0.1:4001"). username/password may be empty when rqlite has no HTTP basic auth. This is the constructor the Caddy module uses.

func NewStore

func NewStore(c conn) *Store

NewStore builds a Store over c with default lock TTL/poll.

func (*Store) Del

func (s *Store) Del(ctx context.Context, key string) error

Del removes key, or returns ErrNotExist if it was absent.

func (*Store) EnsureSchema

func (s *Store) EnsureSchema(ctx context.Context) error

EnsureSchema creates the storage and lock tables if absent. Idempotent.

func (*Store) Get

func (s *Store) Get(ctx context.Context, key string) ([]byte, error)

Get returns the value stored under key, or ErrNotExist.

func (*Store) Has

func (s *Store) Has(ctx context.Context, key string) bool

Has reports whether key exists.

func (*Store) List

func (s *Store) List(ctx context.Context, prefix string, recursive bool) ([]string, error)

List returns the keys under prefix. With recursive=false only immediate children (one path segment beyond prefix, '/'-separated) are returned, deduplicated. An empty prefix lists the whole store.

func (*Store) Lock

func (s *Store) Lock(ctx context.Context, name string) error

Lock blocks until the named lock is acquired or ctx is done. Acquisition is a single atomic upsert (insert-if-absent, else steal-if-expired) serialized by rqlite's Raft, so at most one caller across the fleet holds the lock.

func (*Store) Put

func (s *Store) Put(ctx context.Context, key string, value []byte) error

Put stores value under key (upsert). Values are base64-encoded so arbitrary binary survives rqlite's JSON transport untouched.

func (*Store) SetLockTTL

func (s *Store) SetLockTTL(d time.Duration)

SetLockTTL overrides how long an acquired lock is held before it may be stolen by another instance (and how often a held lock is refreshed: TTL/3). No-op for d <= 0.

func (*Store) SetReadLevel

func (s *Store) SetReadLevel(level string) error

SetReadLevel sets the rqlite read-consistency level used for key lookups: "weak" (default), "none", "linearizable", or "strong". "weak" routes reads through the Raft leader, so the store always sees its own writes. "none" reads the local node's FSM — faster, but on a multi-node cluster a non-leader node can lag right after a write forwarded from it, which fails CertMagic's write-then-read storage preflight and aborts certificate obtains; only use "none" on single-node deployments. No-op (returns nil) when the Store is not backed by the rqlite HTTP transport.

func (*Store) Stat

func (s *Store) Stat(ctx context.Context, key string) (KeyInfo, error)

Stat returns size/modified metadata for key, or ErrNotExist.

func (*Store) Unlock

func (s *Store) Unlock(ctx context.Context, name string) error

Unlock releases a lock previously acquired by this Store (matched by its token, so it can never delete another holder's lock). A lock whose keeper was lost (e.g. process crash) is left to expire via its TTL.