cryptostore

package module
v0.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 12, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

README

cryptostore

Уніфіковані сховища для шифрованих даних Webitel поверх cryptobox. Ядро описує, як значення перетворюється на самопозначений шифр (Codec), де він фізично лежить (Store + реєстр DSN-схем) і як його мігрувати при ротації ключів (Migration) — без розкриття ключів у запитах до сховища.

Шифрування/дешифрування повністю в коді — матеріал ключів ніколи не потрапляє у SQL.

Конкретні сховища й record-орієнтований шар — окремі пакети/модулі:

Пакет Призначення
cryptostore (це ядро) Codec (шифр значень + inline-кадр + blind-index токени), Store/Migration, реєстр DSN-схем + OpenStore, env-default Default()
cryptostore/schema record-орієнтований кодек: декларативна схема полів (unit → field → policy) над Codec; шифрує іменовані поля записів і заповнює __cs_idx
cryptostore/postgres Store над pgx: Usage/Migrate через SQL-функції cryptostore_*; DSN-схема postgres
cryptostore/blobstore (+ /gocloud) об'єктне сховище: envelope-DEK + потоковий шифр; локальна тека (blobfs) + хмара (s3/gs/file/mem)
cmd/cryptoctl CLI-агент над сховищами (status/usage/migrate/can-retire + crypto-команди)

Встановлення

go get github.com/webitel/crypto/cryptostore

Модель: inline, self-marking ciphertext

Codec віддає шифр, який сам себе позначає як шифр, тож шифровані й «легасі» відкриті значення співіснують в одній колонці/атрибуті, а міграція — поступова (без окремої companion-колонки). Маркер — лише дешевий фільтр; арбітр — AEAD-автентифікація.

Один cryptobox-блоб (несе версію ключа, KID) у двох транспортних кодуваннях:

  • bytea-колонкаbinTag(\x00 \x01) ‖ blob. Провідний NUL не трапляється у значенні з text-колонки (текст не містить NUL), тож маркер однозначний; 2-й байт — версія формату кадру (НЕ алгоритм — він імпліцитний у cryptobox Box).
  • json-атрибут / text"cbox:" ‖ base64(blob), JSON-безпечний рядок. Тег опаковий, без in-band версії: зміна розкладки реєструє новий тег, а не бампає цифру.

На читанні: framed-bytea, що не відкривається, — це помилка (пошкодження: провідний NUL не міг бути в колишньому тексті); framed-text, що не відкривається, — відкат до легасі-plaintext (тег cbox:, на відміну від NUL, міг би випадково збігтися з валідним рядком). Розкладка детально — docs/inline-encryption.md.

Codec

Codec — storage-agnostic, value-oriented ядро: шифрує/дешифрує окремі значення (з inline-кадром) і дає детерміновані blind-index токени. Він нічого не знає про таблиці, колонки чи об'єкти — це будують шари вище.

import (
    "github.com/webitel/crypto/cryptobox"
    "github.com/webitel/crypto/cryptostore"
)

// cipher — cryptobox.Box над keyring (значення шифруються ним; KID — у заголовку blob).
ring, _ := cryptobox.ParseKeyring("correct-horse-battery-staple-2025!")
box := cryptobox.NewBox(cryptobox.ChaCha20Poly1305, ring)

// index — НЕЗАЛЕЖНИЙ keyring для blind-index (щоб ротація cipher-ключів не ламала
// токени пошуку). nil → пошук вимкнено, SearchToken повертає ErrIndexKey.
indexRing, _ := cryptobox.ParseKeyring(os.Getenv("WBTL_CRYPTO_SEARCH_KEYRING"))

codec, _ := cryptostore.NewCodec(box, indexRing)
  • Encrypt/Decrypt — bytea-кадр (self-marking); EncryptText/DecryptText/RecryptText — json-кадр.
  • Cipher() — сирий, безкадровий cryptobox (напр. загорнути per-object DEK).
  • SearchToken(ctx, domain, data) — детермінований токен, scoped до domain (різні домени не колізять).
  • PrimaryKID(ctx) — поточна primary-версія ключа (та, якою шифрує Encrypt); CanIndex() — чи є index-keyring.

Record-орієнтоване API (шифрування іменованих полів записів + __cs_idx) будується над Codec у cryptostore/schema.

env-default (без ручного складання keyring'ів)

Щоб сервіс не збирав ланцюг keyring → Box → Codec вручну, ключі беремо з середовища:

codec, err := cryptostore.Default() // процес-глобальний *Codec, меморизований sync.Once
Змінна Призначення
WBTL_CRYPTO_CIPHER_{KEYRING,KEYFILE} keyring для шифрування значень (обов'язково)
WBTL_CRYPTO_SEARCH_{KEYRING,KEYFILE} keyring для blind-index (опційно; відсутній → пошук вимкнено)
WBTL_CRYPTO_DIR системна тека (дефолт /var/lib/webitel/crypto)

_KEYRING містить ключі inline (розділювач — новий рядок або кома), _KEYFILE — шлях до файлу. Системний фолбек: якщо не задано ні env, ні _KEYFILE, читаються добре відомі файли <WBTL_CRYPTO_DIR>/.cipher і /.search — тож devops наповнює теку, і сервіс працює без жодних змінних. Порядок: env → системний файл. (.cipher-фолбек живе у cryptobox.Default, тож будь-який сервіс на cryptobox його підхоплює; .search — тут, через cryptostore.SearchKeys.) Для нестандартних ключів — cryptobox.NewKeyring + cryptostore.NewCodec.

Store і Migration

Store — інтерфейс конкретного сховища (де фізично лежить шифр). Реалізації — у підмодулях (postgres, blobstore). Спільне для всіх — метрики по ключах і міграція під єдиним Migration:

mig := cryptostore.NewMigration(store)
usage, _ := mig.Usage(ctx)                 // map[KID]KeyUsage{Count} — частка по ключу
mig.Run(ctx, cryptostore.MigrateOptions{BatchSize: 500, RatePerSec: 2000})
ok, _ := mig.CanRetire(ctx, 1)             // true → ключ безпечно вивести (cryptobox ring.Retire(1))

Ротація opportunistic: Encrypt завжди бере primary, тож будь-який природний UPDATE/INSERT перешифровує запис безкоштовно; Migration.Run доганяє рештки — батчами, резюмабельно, з CAS по старій версії ключа. Без contract-фази — оригінали не дропимо, plaintext витісняється шифром на місці. CanRetire(kid) каже, чи на ключі ще лишились дані.

Реєстр сховищ (DSN-схема → драйвер)

OpenStore відкриває одне сховище за схемою його DSN. DSN має вигляд <scheme>[:opaque] (postgres://…, blobfs:/var/lib/x, s3://bucket чи голий postgres). Схема (частина до першого :, через cryptostore.Scheme) обирає драйвер; opaque — рядок драйвера (ядро його не валідує). Драйвер реєструється у своєму init(), тож його підключають імпортом-для-side-effect:

import (
    "github.com/webitel/crypto/cryptostore"
    _ "github.com/webitel/crypto/cryptostore/postgres"          // схема "postgres"
    _ "github.com/webitel/crypto/cryptostore/blobstore/gocloud" // схеми s3/gs/file/mem
)

codec, _ := cryptostore.Default() // ключі з env
st, _ := cryptostore.OpenStore(ctx, cryptostore.StoreOptions{
    DSN:       "postgres://user:pass@localhost:5432/db",
    Name:      "pg:main",
    Codec:     codec,
    SchemaRaw: []byte(`{"version":1,"units":{"directory.users":{"fields":{"email":{"search":true}}}}}`),
})

StoreOptions несе *Codec, повний DSN, сирий конфіг-блок (ConfigRaw — параметри поза DSN), сиру схему (SchemaRaw), verbose-прапорець і BaseDir (тека конфіг-файлу). Драйвер прив'язує відносний шлях через opts.Resolve(path) — відносно конфіг-файлу, не CWD. Сховища з однаковим DSN шерять одне підключення (див. postgres — один пул на БД). Багатосховищний агент, що збирає все з конфіг-файлів і веде за єдиною Migration, — окремий CLI cmd/cryptoctl.

Помилки

errors.Is(err, cryptostore.ErrIndexKey) // index keyring відсутній / матеріал закороткий

Помилки схеми полів (schema.ErrUnmanaged, ErrNotSearchable, …) — у schema.

Тести

go test -race ./...   # ядро + schema

Postgres-інтеграційні тести запускаються лише за наявності DSN — див. postgres.

Documentation

Index

Constants

This section is empty.

Variables

View Source
var ErrIndexKey = errors.New("cryptostore: invalid blind index key")

ErrIndexKey signals invalid or missing blind-index root key material.

Functions

func AggregateUsage

func AggregateUsage(stores []StoreUsage) map[cryptobox.KID]KeyUsage

AggregateUsage folds per-store usage into a combined per-key total, keyed by KID. It performs no I/O — pass it the result of UsageByStore to avoid a second scan.

func Environment

func Environment(namespace ...string) env.Options

Base module environment to deal with default configuration

func FrameBinary

func FrameBinary(blob []byte) []byte

FrameBinary wraps blob for storage in a bytea column.

func FrameText

func FrameText(blob []byte) string

FrameText wraps blob as a JSON-safe string (tagged base64).

func Register

func Register(scheme string, driver DriverFunc)

Register makes a DriverFunc available under a DSN scheme (the part before the first ":", e.g. "postgres", "blobfs", "s3"). Implementations call it from an init(), so importing the impl package (e.g. _ "github.com/webitel/crypto/cryptostore/postgres") enables that scheme; a driver may register several schemes. It panics on empty or duplicate registration.

The driver is assumed record-oriented — it consumes a field schema (schema.Config) and fills the search-index column. A driver that encrypts whole objects and ignores the field schema (blob/object storage) registers via RegisterSchemaless instead.

func RegisterSchemaless added in v0.2.0

func RegisterSchemaless(scheme string, driver DriverFunc)

RegisterSchemaless is Register for a driver that encrypts whole objects and does NOT consume a field schema (blob/object storage). It marks the scheme schemaless (reported by Schemaless), so an agent can skip merging a field-schema baseline into such stores. Everything else — the init()-time call, multiple schemes per driver, the empty/duplicate panics — matches Register.

func Schemaless added in v0.2.0

func Schemaless(scheme string) bool

Schemaless reports whether scheme's driver was registered via RegisterSchemaless — i.e. it ignores the field schema (blob/object storage). It is false for a normal (record-oriented) driver AND for an unknown scheme, so an agent that gates a field schema on !Schemaless is fail-closed: a scheme it does not recognize is still treated as record-oriented (e.g. it still receives a pinned baseline).

func Scheme

func Scheme(dsn string) string

Scheme returns the DSN scheme: everything before the first ":", or the whole string when there is no ":". The opaque remainder (if any) is the driver's to interpret — core only uses the scheme to pick the opener.

func Schemes

func Schemes() []string

Schemes returns the registered DSN schemes, sorted — useful for diagnostics.

func SearchKeys

func SearchKeys() (*cryptobox.Keyring, error)

SearchKeys returns the blind-index keyring, resolved from the environment exactly once (mirrors cryptobox.DefaultKeys, minus the Box): WBTL_CRYPTO_SEARCH_KEYRING / _KEYFILE, and when neither is set, the well-known system file <SystemDir>/.search. Search is OPTIONAL: with nothing configured anywhere it returns (nil, nil) and search stays disabled. The result is memoized for the life of the process.

func SystemDir

func SystemDir() string

SystemDir is the well-known directory for system-level crypto configuration: WBTL_CRYPTO_DIR, or /var/lib/webitel/crypto by default. DevOps populates it (.cipher, .search, and — for cryptoctl — store.d/, schema.d/) so a deployment can run with no environment variables at all. It delegates to cryptobox.SystemDir, the single source of truth for the directory.

func UnframeBinary

func UnframeBinary(v []byte) ([]byte, bool)

UnframeBinary returns the wrapped blob and true if v carries the binary tag.

func UnframeText

func UnframeText(s string) ([]byte, bool)

UnframeText returns the wrapped blob and true if s carries the text tag and decodes as base64.

Types

type Codec

type Codec struct {
	// contains filtered or unexported fields
}

Codec is the storage-agnostic, value-oriented cryptographic core: it encrypts and decrypts individual values — with self-marking inline framing (see frame.go) — and derives blind-index search tokens. It knows nothing about tables, columns, objects or any storage shape; higher layers build on it (cryptostore/schema is the record-oriented codec for row stores, blob stores wrap objects, …).

Codec.Encrypt/Decrypt add the inline frame; for raw, unframed cryptobox operations (e.g. wrapping a DEK) use Cipher() directly. A Codec is safe for concurrent use.

As in cryptobox, data denotes open, unencrypted bytes (the cleartext in/out of the codec) — not necessarily human-readable text; blob/value denote ciphertext.

func Default

func Default() (*Codec, error)

Default returns the process-wide value codec assembled from the environment: cryptobox.Default (WBTL_CRYPTO_CIPHER_{KEYRING,KEYFILE}, or the well-known <SystemDir>/.cipher) for value encryption, and the optional SearchKeys (WBTL_CRYPTO_SEARCH_{KEYRING,KEYFILE}, or <SystemDir>/.search) for the blind index. It is built once, on the first call, and the same codec (or the same error) is returned thereafter — so a service obtains a ready *Codec without wiring keyrings:

base, err := cryptostore.Default()
codec := schema.NewCodec(base, baseline)

The cipher keyring is required (Default errors if it is unset everywhere); the search keyring is optional — when absent, search is disabled and SearchToken returns ErrIndexKey. For keys supplied inline in code, or a second independent codec, build one with cryptobox.NewKeyring + NewCodec instead.

func NewCodec

func NewCodec(cipher cryptobox.Cipher, index *cryptobox.Keyring) (*Codec, error)

NewCodec builds the value codec from a cryptobox cipher (value encryption; the key version is carried in the blob header) and an optional blind-index keyring. Pass a nil index when no searchable data is used; SearchToken then returns ErrIndexKey. Keep the index keyring independent of the cipher keyring so data-key rotation does not invalidate search tokens.

func (*Codec) CanIndex

func (c *Codec) CanIndex() bool

CanIndex reports whether a blind-index keyring was configured, i.e. whether SearchToken works. Stores use it to fail fast when a schema declares searchable fields but no search keyring is available.

func (*Codec) Cipher

func (c *Codec) Cipher() cryptobox.Cipher

Cipher returns the underlying cryptobox cipher for raw, unframed encryption — e.g. wrapping a per-object DEK, or a value that carries its own envelope. Codec.Encrypt/Decrypt add the inline self-marking frame; Cipher().Encrypt/Decrypt do not (and never migrate legacy data).

func (*Codec) Decrypt

func (c *Codec) Decrypt(ctx context.Context, v []byte) (data []byte, encrypted bool, err error)

Decrypt decrypts a binary-framed value. If v is not framed it is treated as legacy plaintext and returned unchanged with encrypted=false. A framed value that fails authentication is an error: the binary tag (leading NUL) cannot occur in former text, so a failure means corruption, not a coincidence. For raw, unframed opening use Cipher().Decrypt.

func (*Codec) DecryptText

func (c *Codec) DecryptText(ctx context.Context, s string) (data []byte, encrypted bool, err error)

DecryptText decrypts a text-framed value. If s is not tagged it is returned as plaintext bytes with encrypted=false. A tagged value that fails authentication is also treated as plaintext (the text tag could, unlike the binary tag, coincide with a legitimate plaintext string).

func (*Codec) Encrypt

func (c *Codec) Encrypt(ctx context.Context, data []byte) ([]byte, error)

Encrypt seals data and binary-frames it for a bytea column (self-marking ciphertext). For raw, unframed sealing use Cipher().Encrypt.

func (*Codec) EncryptText

func (c *Codec) EncryptText(ctx context.Context, data []byte) (string, error)

EncryptText seals data and text-frames it for a json attribute.

func (*Codec) PrimaryKID

func (c *Codec) PrimaryKID(ctx context.Context) (cryptobox.KID, error)

PrimaryKID reports the key version Encrypt currently uses, by sealing an empty probe and reading its header. It needs no access to the keyring, so key material stays inside cryptobox.

func (*Codec) RecryptText

func (c *Codec) RecryptText(ctx context.Context, s string) (out string, recrypted bool, err error)

RecryptText re-encrypts a text-framed value under the current primary key, returning the new framed string. A value that is not tagged (or fails to open) is returned unchanged with recrypted=false — used to migrate json attributes.

func (*Codec) SearchToken

func (c *Codec) SearchToken(ctx context.Context, domain string, data []byte) ([]byte, error)

SearchToken returns the deterministic blind-index token for data, scoped to domain (e.g. "schema.table.column"). Equal data under the same domain yields an equal token, enabling exact-match lookups without revealing the key; different domains never collide. Returns ErrIndexKey if no index keyring was configured. ctx is threaded to the keyring's KeyMaterial source (file, KMS, …).

type DriverFunc

type DriverFunc func(ctx context.Context, opts StoreOptions) (Store, error)

DriverFunc builds a Store for a registered DSN scheme — the factory a driver package registers for its scheme(s). A Store that owns resources (e.g. a connection pool) may also implement io.Closer; Open's caller is responsible for closing such stores on shutdown.

type FieldRef

type FieldRef struct {
	Table  string
	Column string
}

FieldRef identifies one encrypted location ("schema.table"."column") within a store, for usage reporting and migration scoping.

type KeyUsage

type KeyUsage struct {
	KID   cryptobox.KID
	Count int64 // number of encrypted values sealed under this key
}

KeyUsage is the count of encrypted values a store holds under one key version. It answers "what share of data sits under each key?" — the signal that drives migration planning and the "is this key safe to retire?" check.

type MigrateMode

type MigrateMode int

MigrateMode selects how re-encryption to a new primary key proceeds after a key rotation.

const (
	// MigrateLazy performs no active re-encryption. Data moves to the new key
	// opportunistically: because EncodeRows always seals with the primary key,
	// any natural write of a row migrates that row for free. Stragglers remain
	// readable (every non-retired key stays in the keyring). This is the safe
	// default — it adds zero extra writes and never turns a read into a write.
	MigrateLazy MigrateMode = iota

	// MigrateEager actively re-encrypts all data under non-primary keys now, via a
	// throttled, resumable background pass over each store. Use it before retiring
	// an old key, or when policy requires prompt re-encryption.
	MigrateEager
)

func (MigrateMode) String

func (m MigrateMode) String() string

type MigrateOptions

type MigrateOptions struct {
	// DryRun counts the work without writing.
	DryRun bool
	// BatchSize is the number of values re-encrypted per transaction. 0 picks a
	// store-defined default.
	BatchSize int
	// RatePerSec throttles re-encryption to at most this many values per second.
	// 0 means unlimited.
	RatePerSec int
	// OnProgress, if set, is called periodically (typically per batch) during a
	// long operation with the cumulative progress so far. It must not block.
	OnProgress func(Progress)
}

MigrateOptions tunes an eager migration pass.

type MigrateResult

type MigrateResult struct {
	Store     string
	Scanned   int64
	Recrypted int64
	Remaining int64 // values still under a non-primary key after this pass
}

MigrateResult summarizes one Migrate call.

type Migration

type Migration struct {
	// contains filtered or unexported fields
}

Migration coordinates usage reporting and eager re-encryption across a set of stores. It is the engine behind the standalone agent: it answers "what share of data is under each key?", drives eager migration, and reports when an old key is safe to retire.

func NewMigration

func NewMigration(stores ...Store) *Migration

NewMigration builds a controller over the given stores.

func (*Migration) CanRetire

func (m *Migration) CanRetire(ctx context.Context, kid cryptobox.KID) (bool, error)

CanRetire reports whether no data remains under kid across all stores, i.e. whether the key can be retired in the keyring without orphaning ciphertext (which would otherwise surface as cryptobox.ErrKeyRetired on read).

func (*Migration) Run

Run performs an eager migration pass over every store, returning each store's result. Stores are migrated sequentially so the global rate limit and load are predictable. Call repeatedly until every result reports Remaining == 0.

func (*Migration) Usage

func (m *Migration) Usage(ctx context.Context) (map[cryptobox.KID]KeyUsage, error)

Usage aggregates per-key usage across every store, keyed by KID.

func (*Migration) UsageByStore

func (m *Migration) UsageByStore(ctx context.Context) ([]StoreUsage, error)

UsageByStore reports each store's per-key usage in a single scan per store, preserving store order. Callers that want both the per-store breakdown and the combined total (e.g. the status/usage commands) should call this once and fold the result with AggregateUsage — rather than calling Usage and then re-scanning each store.

type Progress

type Progress struct {
	Store string // store name
	Unit  string // unit/table, for field-level operations (else empty)
	Field string // field/column, for field-level operations (else empty)
	Done  int64  // items processed so far
	Total int64  // total items to process, or -1 when unknown
}

Progress reports the cumulative state of a long-running migration or backfill.

type Store

type Store interface {
	// Name is a short identifier used in status output (e.g. "postgres:main").
	Name() string

	// Usage reports per-key counts across every field the store manages, grouped by
	// the key version recorded for each value.
	Usage(ctx context.Context) ([]KeyUsage, error)

	// Migrate re-encrypts values not yet under the primary key and reports
	// progress. It must be resumable (safe to call repeatedly until Remaining is
	// 0) and safe under concurrent writes: a value a natural write already moved
	// to the primary key must not be clobbered (guard the write with the old key
	// version). With opt.DryRun it only counts what would be migrated.
	Migrate(ctx context.Context, opt MigrateOptions) (MigrateResult, error)
}

Store is a physical backing store of encrypted data (a Postgres database, a blob bucket, …). It exposes only the two operations that need to enumerate stored ciphertext: per-key usage and key migration. The codec itself stays in the codec — a Store re-encrypts values through its schema.Codec (Decode → Encode).

Implementations live in their own modules (cryptostore/postgres, …) because each carries its own driver dependencies.

func OpenStore

func OpenStore(ctx context.Context, opts StoreOptions) (Store, error)

OpenStore opens a single store via the driver registered for opts.DSN's scheme. It is the core per-store primitive; assembling many stores from a config file (with a shared Codec and a Migration over them) is the agent's job — see cryptoctl.

type StoreOptions

type StoreOptions struct {
	DSN       string          // [D]ata [S]ource [N]ame ; scheme selects the driver, opaque is the driver's connection options
	Name      string          // store friendly / display name
	Codec     *Codec          // value-oriented encryption codec
	ConfigRaw json.RawMessage // store impl-specific extras not expressible in the DSN
	SchemaRaw json.RawMessage // field schema for record-oriented stores (may be empty)
	Verbose   bool            // enable store-level debug logging (e.g. SQL query tracing)
	// BaseDir is the directory of the config source that declared the store (empty
	// when the config did not come from a file). Drivers anchor relative filesystem
	// paths — a blobfs directory, a sqlite catalog — to it via Resolve, so those
	// paths are relative to the config file, not the process working directory.
	BaseDir string
}

StoreOptions carries everything a store DriverFunc needs to build a Store from configuration: the shared codec core, the store's data source name, and its raw config and schema blocks. Core does not parse the DSN opaque part or the config/schema blocks — the driver does — so core stays agnostic of impl specifics.

func (*StoreOptions) Resolve

func (x *StoreOptions) Resolve(path string) string

Resolve interprets a driver's filesystem path relative to the config source that declared the store: an absolute path (or empty) is returned unchanged; a relative one is joined with BaseDir. When BaseDir is empty (config not from a file), the path is returned as-is, i.e. left to resolve against the process working directory — the historical behavior. Drivers whose DSN opaque or config block names a local path should route it through Resolve.

type StoreUsage

type StoreUsage struct {
	Store string
	Keys  []KeyUsage
}

StoreUsage is one store's per-key usage, as returned by UsageByStore.

Directories

Path Synopsis
blobstore module
gocloud module
postgres module
Package schema is the field-oriented encryption layer: it encrypts named fields within records (a SQL row, a NoSQL document) according to a declarative Config, on top of the storage-agnostic cryptostore.Codec core.
Package schema is the field-oriented encryption layer: it encrypts named fields within records (a SQL row, a NoSQL document) according to a declarative Config, on top of the storage-agnostic cryptostore.Codec core.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL