Documentation
¶
Overview ¶
Package keystore provides database-backed signing key management with encryption at rest, automatic multi-pod refresh, and zero-downtime rotation. Keys are stored in PostgreSQL with the private key encrypted using AES-256-GCM (master key) and the kid as authenticated data (AAD).
Index ¶
- Constants
- Variables
- type KeyInfo
- type KeyRecord
- type KeyStore
- func (ks *KeyStore) ActiveKey() (*rsa.PrivateKey, string)
- func (ks *KeyStore) AllPublicKeys() map[string]*rsa.PublicKey
- func (ks *KeyStore) CleanupExpired(ctx context.Context) (int64, error)
- func (ks *KeyStore) EnsureKey(ctx context.Context, importKey *rsa.PrivateKey) error
- func (ks *KeyStore) Import(ctx context.Context, key *rsa.PrivateKey) (string, error)
- func (ks *KeyStore) KeyProvider() func() map[string]*rsa.PublicKey
- func (ks *KeyStore) ListKeys(ctx context.Context) ([]KeyInfo, error)
- func (ks *KeyStore) Refresh(ctx context.Context) error
- func (ks *KeyStore) Revoke(ctx context.Context, kid string) error
- func (ks *KeyStore) Rotate(ctx context.Context) (string, error)
- func (ks *KeyStore) RotateIfOlderThan(ctx context.Context, maxAge time.Duration) (string, error)
- func (ks *KeyStore) SetOnKeyChange(fn OnKeyChangeFunc)
- func (ks *KeyStore) StartRefreshLoop(ctx context.Context, interval time.Duration)
- func (ks *KeyStore) Stop()
- type OnKeyChangeFunc
- type Retention
- type Rotation
Constants ¶
const DefaultRotationInterval = 720 * time.Hour
DefaultRotationInterval is how old the active signing key may get before it is rotated. It matches docs/spec-draft.md's 30 days, which is the number the design specified and the number the shipped code never implemented.
const RotationCheckInterval = time.Hour
RotationCheckInterval is how often the scheduler asks whether the active signing key is due for rotation.
It is deliberately not the rotation interval itself. The decision is made against the ACTIVE KEY'S AGE, read from the database, not against this process's uptime, so the schedule survives a restart, a rollout and a replica count greater than one. An hourly check makes a 720-hour rotation land within an hour of when it is due, which is as precise as a monthly schedule needs to be.
const SweepInterval = 6 * time.Hour
SweepInterval is how often the expired-key sweeper runs.
The rows it removes have already stopped mattering: Refresh drops a retired key from the verification set the moment expires_at passes, so a row waiting for the next sweep is bytes and nothing else. Sweeping more often would buy nothing, and matching the audit and recovery sweepers leaves one number for an operator to reason about instead of three.
Variables ¶
var ErrClosed = errors.New("keystore: closed")
ErrClosed is returned by any operation needing the master key after Stop.
Failing closed matters more here than it looks. A zeroed master key is still 32 bytes, so AES-GCM accepts it and Import would encrypt a private key under all zeros and commit the row. The write succeeds, nothing reports an error, and the key is permanently undecryptable by the real master key. That is data destruction rather than a torn read.
var ErrNoActiveKey = errors.New("keystore: no active signing key")
ErrNoActiveKey is returned when no active signing key exists in the database.
var ErrRevokedKey = errors.New("keystore: key is revoked")
ErrRevokedKey is returned when an import would reactivate a revoked key. Revocation is terminal: the realistic reason to revoke a signing key is that its private material leaked, so it must never come back as active.
Functions ¶
This section is empty.
Types ¶
type KeyInfo ¶
type KeyInfo struct {
// KID is the key identifier as published in JWKS.
KID string `json:"kid"`
// Algorithm is the JWS signing algorithm, "RS256" for keys this store issues.
Algorithm string `json:"algorithm"`
// Status is "active", "retired" or "revoked"; see [KeyRecord.Status] for
// what each state permits.
Status string `json:"status"`
// CreatedAt is when the key was first stored.
CreatedAt time.Time `json:"created_at"`
// RetiredAt is when the key stopped being active, absent while it still is.
RetiredAt *time.Time `json:"retired_at,omitempty"`
// ExpiresAt is when a retired key stops verifying, absent for the active key.
ExpiresAt *time.Time `json:"expires_at,omitempty"`
}
KeyInfo is a metadata-only view of a signing key. It deliberately has no field for key material: it is the shape returned to admin API clients, and omitting the private key from the type makes leaking it a compile error rather than a review catch.
type KeyRecord ¶
type KeyRecord struct {
// KID is the key identifier published in JWKS and in the JWT kid header.
// It is derived from the public key, so the same key always yields the same
// kid and re-importing a key updates its row rather than creating a second.
KID string
// PrivateKey is the signing key, decrypted from the row's AES-256-GCM
// ciphertext under the master key with KID as AAD. Populated only for the
// active key; retired keys are loaded for verification only.
PrivateKey *rsa.PrivateKey
// PublicKey is the verification key published in JWKS.
PublicKey *rsa.PublicKey
// Algorithm is the JWS signing algorithm for this key. Only "RS256" is
// issued today.
Algorithm string
// Status is one of exactly "active", "retired" or "revoked". At most one
// key is "active" and it is the only one that signs; "retired" keys still
// verify until ExpiresAt so tokens outlive a rotation; "revoked" is
// terminal and drops the key from JWKS immediately, on the assumption that
// the private material leaked. Import refuses to move a row back out of
// "revoked" (see [ErrRevokedKey]).
Status string
// CreatedAt is when the key was first stored.
CreatedAt time.Time
// RetiredAt is when the key stopped being active, nil while it still is.
RetiredAt *time.Time
// ExpiresAt is when a retired key stops verifying and becomes eligible for
// deletion by CleanupExpired. Nil for the active key.
ExpiresAt *time.Time
}
KeyRecord is the decrypted, in-memory shape of an auth.signing_keys row. Unlike KeyInfo it carries private key material and must never be logged, serialized to a response, or written anywhere but memory.
type KeyStore ¶
type KeyStore struct {
// contains filtered or unexported fields
}
KeyStore manages signing keys stored in PostgreSQL with encryption at rest. It supports automatic refresh from the database, key rotation, and notifies subscribers when the active key changes.
func New ¶
New creates a new KeyStore. masterKey must be exactly 32 bytes (AES-256).
The key is copied rather than retained, because Stop wipes it and the caller's slice is shared. cmd/vault takes one working copy of the master key and hands that same slice to this constructor and to the service container, which passes it on to the identity, blob, service-document and TOTP paths. Retaining the caller's array made Stop zero the key all of those were still using, and 32 zero bytes is still a valid AES-256 key, so a request draining through shutdown encrypted successfully against it and wrote a row no later process could ever decrypt.
Owning the copy is what makes the wipe in Stop safe to perform at all: it destroys this keystore's key material and nobody else's.
func (*KeyStore) ActiveKey ¶
func (ks *KeyStore) ActiveKey() (*rsa.PrivateKey, string)
ActiveKey returns the current active signing key and its kid.
func (*KeyStore) AllPublicKeys ¶
AllPublicKeys returns all non-expired public keys (for JWKS).
func (*KeyStore) CleanupExpired ¶
CleanupExpired removes expired retired keys from the database.
Called by the keystore Retention sweeper (internal/keystore/retention.go) on its sweep interval; deletes retired keys whose expires_at has passed. It is never called from Refresh.
func (*KeyStore) EnsureKey ¶
EnsureKey loads keys from the database. If no active key exists, it either imports the provided key or generates a new one.
func (*KeyStore) Import ¶
Import encrypts and stores a key in the database as the active key. Any existing active key is retired.
func (*KeyStore) KeyProvider ¶
KeyProvider returns a function suitable for middleware.Auth that provides the current public key set. This avoids passing a static map.
func (*KeyStore) ListKeys ¶
ListKeys returns metadata about all signing keys (no private key material).
func (*KeyStore) Refresh ¶
Refresh loads all non-revoked, non-expired keys from the database and updates the in-memory state. Notifies OnKeyChange if the active key changed.
Every row is opened before any of it is published. A row is published only if its private_key decrypts under the master key with its kid as AAD and the decrypted key's public half is the public_key column. Nothing else proves the row is this vault's: auth.signing_keys is writable by vault_app, so a row that merely sits in the table proves only that someone could issue SQL. Publishing on that basis let anyone holding the app role INSERT a key of their own as 'retired' with a NULL expires_at and have it in JWKS within one refresh interval, at which point tokens they signed for any subject verified here and in every service that polls this issuer. The public-key comparison closes the same attack run as an UPDATE of public_key on a genuine row, where the ciphertext is the vault's own and decrypts perfectly.
The three partial-failure paths differ, deliberately, and none is silent to an operator reading logs:
A row whose public key will not parse, or is not RSA, is logged and skipped. The rest of the key set still loads, so one corrupt row cannot take the whole JWKS down; the cost is that the skipped kid disappears from JWKS and live tokens signed by it start failing verification. This is the intentional trade: a partial verification set beats none.
A non-active row that does not open is logged and skipped for the same reason, and the reason is stronger here: the row may well be hostile, and failing the whole refresh on it would hand anyone who can write one row a way to freeze the key set of every pod, then break the next pod to boot, since EnsureKey refuses to start without a Refresh.
A decrypt, parse or mismatch failure on the ACTIVE key aborts before applyKeys, so the previously loaded key set stays in memory untouched. The process keeps signing and verifying with what it already had rather than dropping to no keys at all. Skipping instead would leave it signing with a key absent from its own JWKS. Callers see the error; StartRefreshLoop only logs it, so a persistently failing active key surfaces as a stale-key warning in the log and not as an outage.
A successful Refresh always replaces the whole set: keys are never merged with what was loaded before.
Opening every row costs one AES-256-GCM decrypt and one PKCS#8 parse per key, measured at 151 microseconds for RSA-2048 on the CI-class machine this was written on, against a default refresh interval of 60 seconds. A deployment on the default one-hour retention holds an active key and one or two retired ones; even fifty would be 7.5 milliseconds a minute.
func (*KeyStore) Revoke ¶
Revoke immediately marks a key as revoked. It will be excluded from JWKS on the next refresh. Tokens signed with this key will fail validation.
func (*KeyStore) Rotate ¶
Rotate generates a new RSA-2048 key, stores it as active, and retires the old one.
func (*KeyStore) RotateIfOlderThan ¶ added in v1.0.3
RotateIfOlderThan rotates the active signing key when it is older than maxAge and returns the new kid, or "" when nothing was due.
Serialized across replicas by a session-scoped advisory lock. Rotation is not idempotent the way a sweep is: two replicas deciding simultaneously would each generate a key, each retire whatever was active when it looked, and the deployment would come out of one due date with two rotations and a key retired the instant it was created. A replica that does not get the lock returns ("", nil) and re-derives the decision on its next tick, where the key it sees is the freshly rotated one.
The age comes from auth.signing_keys.created_at, so it is the key's age and not the process's. No active key at all is not this function's problem: EnsureKey creates one at startup, and rotating into an empty store would race that.
The retire-terminal invariants hold by construction. Rotation reaches the database only through Import, whose retire statement always writes a concrete expires_at (now + retentionPeriod), so migration 027's `status <> 'retired' OR expires_at IS NOT NULL` CHECK is satisfied on every rotation, and migration 026's guard is never even reached: it fires only on writes to a row that is ALREADY retired, and Import only ever retires a row that was active.
func (*KeyStore) SetOnKeyChange ¶
func (ks *KeyStore) SetOnKeyChange(fn OnKeyChangeFunc)
SetOnKeyChange registers a callback invoked when the active key changes.
func (*KeyStore) StartRefreshLoop ¶
StartRefreshLoop starts a background goroutine that refreshes keys from the database at the given interval, which is how a pod picks up a rotation performed by another pod. Call Stop to terminate the loop.
A failing Refresh is logged and the loop continues. Since Refresh leaves the previous key set in place on an active-key failure, a pod whose refreshes keep failing serves stale keys indefinitely rather than losing the ability to sign; the log line is the only signal, so it belongs in an alert.
func (*KeyStore) Stop ¶
func (ks *KeyStore) Stop()
Stop terminates the refresh loop and zeros the master key.
It blocks until the refresh loop has exited. Refresh reads masterKey outside ks.mu (AES-GCM decrypt of the active key), so zeroing it while a refresh is still in flight is a data race on live key material — and a refresh that read the half-zeroed key would fail to decrypt. Stop is idempotent: a second call must not re-close stopCh.
type OnKeyChangeFunc ¶
type OnKeyChangeFunc func(activeKey *rsa.PrivateKey, kid string, allPublicKeys map[string]*rsa.PublicKey)
OnKeyChangeFunc is a callback invoked when the active key changes. It receives the new active private key, its kid, and all public keys.
type Retention ¶ added in v1.0.3
type Retention struct {
// contains filtered or unexported fields
}
Retention removes retired signing keys that have outlived their retention window.
CleanupExpired shipped with the DB-backed keystore and nothing ever called it. Its doc comment says it is "called periodically during refresh", which was true of no code path: the refresh loop only refreshes. So auth.signing_keys grew by one row per rotation forever, and every one of those rows carries an AES-256-GCM ciphertext of a private key that no longer verifies anything. Keeping decommissioned key material around indefinitely is the part that makes this more than table bloat.
Unlike the audit and recovery sweepers this one has no horizon of its own and no off switch. Those two default to disabled because destroying security logs or the only recoverable copy of an erased account is an operator's call. Here the horizon is already a per-row decision the keystore made at retirement time, expires_at is that decision written down, and a row past it verifies nothing for anyone. There is no judgment left to defer.
Revoked keys are not reapable and are not meant to be. Their row is the tombstone that keeps a leaked kid from being re-inserted, so it outlives everything; the database refuses to delete one whatever this sweeper asks.
func NewRetention ¶ added in v1.0.3
func NewRetention(ks expiredKeyReaper) *Retention
NewRetention builds a sweeper over ks.
A vault running the file-based signing key mode builds no keystore at all, so the caller may well have nothing to give: an inert sweeper is the answer rather than a second branch at the call site.
func (*Retention) Done ¶ added in v1.0.3
func (r *Retention) Done() <-chan struct{}
Done is closed once the sweep loop has exited, whether it ended via Stop or via its context being canceled. The channel never closes if Start was not called: a loop that never ran has nothing to wait for.
func (*Retention) Enabled ¶ added in v1.0.3
Enabled reports whether there is a keystore to sweep.
The typed-nil check is not paranoia. cmd/vault holds the keystore as a *KeyStore that stays nil in file-based mode, and a nil *KeyStore stored in an interface is not a nil interface, so `r.reaper == nil` is false for it and the first tick would dereference a nil pool inside a goroutine, where the panic takes the process down with no request to attribute it to.
func (*Retention) Start ¶ added in v1.0.3
Start runs the sweeper until Stop is called or ctx is canceled. It sweeps once immediately: a deployment that rolls its pods more often than the interval would otherwise never reach a tick and the reap would never happen.
func (*Retention) Stop ¶ added in v1.0.3
func (r *Retention) Stop()
Stop terminates the sweep loop and blocks until it has actually exited.
The wait is the point. Stop is deferred in cmd/vault above a deferred close of the database pool, so a Stop that only asked the loop to finish could return while a sweep was still inside its DELETE and have the pool torn out from under it. Safe to call more than once, and safe on a sweeper that was never started.
func (*Retention) Sweep ¶ added in v1.0.3
Sweep removes every retired key past its expiry and returns how many rows went.
It cannot shorten any key's verification life. Refresh loads a row only while `expires_at > NOW() OR (expires_at IS NULL AND status = 'active')`, and the reap's predicate is `expires_at IS NOT NULL AND expires_at < NOW()`. The two sets are disjoint, so every row this deletes is one the keystore already refuses to publish, and the tokens it signed had already stopped verifying at expires_at rather than here. What decides whether a live token outlives its key is VAULT_KEY_RETENTION_PERIOD against the access token TTL, which is a choice made at rotation time and nothing this sweeper can affect either way.
type Rotation ¶ added in v1.0.3
type Rotation struct {
// contains filtered or unexported fields
}
Rotation rotates the JWT signing key on a schedule.
Nothing did. docs/spec-draft.md specified rotation every 30 days with at most three keys in JWKS; docs/spec.md quietly redefined rotation as manual (POST /admin/keys/rotate and the rotate-jwks CLI) and no interval setting existed — only VAULT_KEY_REFRESH_INTERVAL, which is how often a pod re-reads the store, and VAULT_KEY_RETENTION_PERIOD, which is how long a retired key lingers. Neither rotates anything. A default install therefore signed with one key forever, and every token ever issued by that deployment verified under a single private key whose exposure window was the lifetime of the install.
The horizon is an operator's call in the same way the audit and escrow horizons are, so the interval is configurable; unlike those two it defaults to ON, because a key that is never rotated is not a retained record an operator might want kept, it is a control that does not exist.
func NewRotation ¶ added in v1.0.3
NewRotation builds a scheduler over ks. An interval of zero or less disables it, which is how an operator who rotates on their own schedule turns it off.
A vault running the file-based signing key mode builds no keystore at all, so the caller may well have nothing to give: an inert scheduler is the answer rather than a second branch at the call site.
func (*Rotation) Done ¶ added in v1.0.3
func (r *Rotation) Done() <-chan struct{}
Done is closed once the rotation loop has exited, whether it ended via Stop or via its context being canceled. The channel never closes if Start was not called: a loop that never ran has nothing to wait for.
func (*Rotation) Enabled ¶ added in v1.0.3
Enabled reports whether there is a keystore to rotate and a horizon to rotate against.
The typed-nil check is not paranoia. cmd/vault holds the keystore as a *KeyStore that stays nil in file-based mode, and a nil *KeyStore stored in an interface is not a nil interface, so `r.rotator == nil` is false for it and the first tick would dereference a nil pool inside a goroutine, where the panic takes the process down with no request to attribute it to.
func (*Rotation) Rotate ¶ added in v1.0.3
Rotate rotates the active key if it is older than the configured interval, and returns the new kid or "" when nothing was due.
func (*Rotation) Start ¶ added in v1.0.3
Start runs the scheduler until Stop is called or ctx is canceled. It checks once immediately, which is safe precisely because the check is against the stored key's age: a deployment that restarts more often than the check interval still rotates exactly when the key is old enough, and never because a pod booted.
Calling it more than once starts nothing further. Two loops would share one doneCh, and the second one to exit would close an already-closed channel: an unrecoverable panic raised from a deferred call in a background goroutine, which no handler can catch and which takes the process with it.
func (*Rotation) Stop ¶ added in v1.0.3
func (r *Rotation) Stop()
Stop terminates the rotation loop and blocks until it has actually exited.
The wait is the point. Stop is deferred in cmd/vault above a deferred close of the database pool, so a Stop that only asked the loop to finish could return while a rotation was still inside its transaction and have the pool torn out from under it. Safe to call more than once, and safe on a scheduler that was never started.
Source Files