scrinium

README

Scrinium

A content-addressable storage engine in Go, with a high-level API and reference applications (FUSE, WebDAV, HTML browser).

Status

In development. The on-disk format and public API may change.

Layout

Single Go module (scrinium.dev):

scrinium/
├── go.mod                   # scrinium.dev
├── *.go                     # high-level wrapper API: scrinium.Open / scrinium.Init
│
├── engine/                  # the engine itself
│   ├── core/                # Store implementation
│   ├── domain/              # types (Manifest, Artifact, ...)
│   ├── driver/              # storage backends (localfs, ...)
│   ├── index/               # metadata index backends (sqlite, ...)
│   ├── plugin/              # encoders/decoders (zstd, aes-gcm, ...)
│   ├── projection/          # read-side: View, FSOps, fsmeta
│   ├── agent/, curator/, maintenance/   # workers
│   ├── errs/, event/        # cross-cutting types
│   └── internal/            # engine-private helpers
│
├── cmd/                     # reference binaries
│   ├── scrinium-fuse/       # FUSE mount (build tag: fuse)
│   ├── scrinium-webdav/     # WebDAV server
│   └── scrinium-webview/    # HTML browser
│
└── examples/                # example programs
    ├── hello/               # smallest open + put + get
    ├── ingest/              # batch ingest from a directory tree
    └── browse/              # read-only inspector

Quick start

The smallest program — open or create a store, put one artifact, read it back:

package main

import (
  "bytes"
  "context"
  "io"
  "log"

  scrinium "scrinium.dev"
  "scrinium.dev/engine/domain"
)

func main() {
  ctx := context.Background()

  cfg := scrinium.DefaultConfig()
  cfg.Store = "file:///tmp/my-store"

  // OpenOrInit opens the store if it exists, initialises it
  // otherwise — and surfaces real errors (bad URI, permissions)
  // instead of silently reinitialising on top.
  s, _, _, err := scrinium.OpenOrInit(ctx, cfg)
  if err != nil {
    log.Fatal(err)
  }
  defer s.Close()

  id, err := s.Store.Put(ctx,
    domain.Artifact{Payload: bytes.NewReader([]byte("hello"))},
    domain.PutOptions{Namespace: "demo"},
  )
  if err != nil {
    log.Fatal(err)
  }

  rh, _ := s.Store.Get(ctx, id, domain.GetOptions{})
  defer rh.Close()
  body, _ := io.ReadAll(rh)
  log.Printf("read back: %s", body)
}

See examples/ for runnable variations (go run ./hello, ./ingest, ./browse).

Building

go build ./...                  # build everything
go test ./...                   # test everything (FUSE included on Linux/macOS)
make ci                         # fmt + vet + test + fuzz-smoke

To run a single example or binary directly:

go run ./examples/hello
go run ./cmd/scrinium-webdav --store=/tmp/store --listen=:8080

Reference binaries

Pre-built CLI applications under cmd/ demonstrate three integrations:

• scrinium-fuse — POSIX filesystem on Linux/macOS via FUSE.
• scrinium-webdav — cross-platform WebDAV server.
• scrinium-webview — read-only HTML browser for inspecting a store.

Install from source:

go install scrinium.dev/cmd/scrinium-webdav@latest

Embedding

For applications that want to host Scrinium directly, use the top-level scrinium package:

• scrinium.OpenOrInit(ctx, cfg) — open or create. Convenient for examples and single-binary tools; returns a created flag so the host knows when a recovery kit has been produced.
• scrinium.Open(ctx, cfg) — open an existing store. Returns errs.ErrStoreNotFound (which bridges to fs.ErrNotExist) if no store has been initialised at the location.
• scrinium.Init(ctx, cfg) — create a fresh store. Returns the recovery kit on encrypted stores; the host MUST persist it.
• (*Scrinium).Close() — wipe secrets, release resources. Idempotent.

Production daemons typically separate "init" and "serve" subcommands so an operator can audit when a brand-new store is being created.

For full control over wiring, compose engine/core, engine/projection, and friends directly. The top-level package is a convenience over them.

License

Apache License 2.0 — see LICENSE and NOTICE.

Contributions are accepted under the same license. Every commit must include a Developer Certificate of Origin sign-off — see CONTRIBUTING.md for details.

Documentation

Overview

Package scrinium is the high-level entry point for embedding Scrinium into an application. It bundles the bootstrap pipeline every reference binary uses (driver dial, index dial, store open, view backfill, fsops assembly) into two functions:

scrinium.Init  — create a fresh store on disk
scrinium.Open  — open an existing store and return a runtime

The returned *Scrinium owns the Store, the StoreIndex, the projection View, the FSOps facade, and the boot-unique MountSession. Surfaces (FUSE, WebDAV, HTTP, gRPC, custom protocols) consume those fields directly; this package does not impose a surface-specific shape.

Why a top-level package

Scrinium has many supporting packages (core, domain, driver, index, plugin, projection, agent, curator, ...). Each is reusable on its own; together they form the engine. The top-level scrinium package is a thin opinion layer over them: it picks reasonable defaults (sqlite index next to file:// stores, sha256 hashing, fsmeta path resolution, fsindex extension), wires fail-safe error paths (deferred cleanup of partially-opened resources), and zeroizes secrets on shutdown.

Hosts that want full control over wiring should compose core.OpenStore and friends directly; this package is a convenience.

Lifecycle

1. Build a scrinium.Config (use scrinium.DefaultConfig and edit the fields you care about).
2. Call scrinium.Open(ctx, cfg) — opens store, index, view, fsops.
3. Use the returned *Scrinium until shutdown.
4. Call (*Scrinium).Close — wipes secrets, closes resources.

First-time use

scrinium.Init creates a new store at a fresh location, writes the descriptor and system.config, and returns a *Scrinium that is already open and ready to use. For encrypted stores it produces a recovery kit that the host MUST persist (out of band) before the first Close — it is the only path back into the store if the passphrase is lost.

Examples

See the examples/ module in the Scrinium repository for full runnable programs (hello, ingest, browse).

Index

• type Config
  • func DefaultConfig() Config
  • func (c Config) Validate() error
• type Scrinium
  • func Init(ctx context.Context, cfg Config) (_ *Scrinium, recoveryKit []byte, retErr error)
  • func Open(ctx context.Context, cfg Config) (_ *Scrinium, retErr error)
  • func OpenOrInit(ctx context.Context, cfg Config) (*Scrinium, []byte, bool, error)
  • func (s *Scrinium) Close() error
  • func (s *Scrinium) ListExtensions() []index.ExtensionInfo
  • func (s *Scrinium) RoutingConfig() projection.RoutingConfig
  • func (s *Scrinium) StatsProvider(ctx context.Context, startedAt time.Time, capacityTimeout time.Duration) func() []byte

Types

type Config

type Config struct {
	// Store URI of the artifact store.
	Store string `yaml:"store"`

	// Index URI of the metadata index. Empty defaults to
	// sqlite at <storeDir>/index.db when Store is a file://
	// URI; for any other store scheme the Index must be set
	// explicitly.
	Index string `yaml:"index"`

	// PassphraseFile points at a file holding the store's
	// encryption passphrase. Empty means unencrypted store
	// (Plain DEK). The file is read at Open time; it is
	// expected to contain the passphrase as raw bytes (one
	// line, no trailing newline is required but tolerated —
	// trailing whitespace is stripped).
	//
	// The host is responsible for ensuring file permissions
	// on the passphrase file are appropriately restrictive
	// (e.g. 0600 owned by the daemon user).
	PassphraseFile string `yaml:"passphraseFile"`

	// Namespace constrains writes/visibility to a single
	// namespace. Empty = global.
	Namespace string `yaml:"namespace"`

	// ScratchDir is the staging directory for in-flight
	// writes. Empty = <storeDir>/.scratch when the store is
	// local; required for non-local stores.
	ScratchDir string `yaml:"scratchDir"`

	// ScratchQuota is the maximum bytes the scratch directory
	// may hold across all in-flight writes. 0 = unlimited.
	ScratchQuota int64 `yaml:"scratchQuota"`

	// ReadOnly disables writes through the FSOps facade.
	// Stores opened read-only never publish events for
	// modifications (there are none to publish).
	ReadOnly bool `yaml:"readOnly"`

	// Editing controls draft/transit semantics. Values:
	//   "off"    — reject all in-place edits (strict CAS).
	//   "on"     — allow all editing operations.
	//   "custom" — consult AllowRename/AllowSetattr/...
	//              individually. Each flag defaults to false
	//              (a nil pointer is treated as off).
	// Empty string is treated as "off".
	Editing string `yaml:"editing"`

	// Custom editing flags. Only consulted when Editing is
	// "custom"; unused otherwise. Pointers so the YAML
	// loader can distinguish "unset" from "explicit false"
	// when assembling a policy from layered sources (file +
	// env + CLI).
	AllowRename   *bool `yaml:"allowRename,omitempty"`
	AllowSetattr  *bool `yaml:"allowSetattr,omitempty"`
	AllowTruncate *bool `yaml:"allowTruncate,omitempty"`
	AllowAppend   *bool `yaml:"allowAppend,omitempty"`

	// DefaultMode/UID/GID fill in fsmeta defaults for
	// artifacts written without explicit POSIX bits.
	DefaultMode uint32 `yaml:"defaultMode"`
	DefaultUID  uint32 `yaml:"defaultUid"`
	DefaultGID  uint32 `yaml:"defaultGid"`

	// Routing — service prefix and which trees are visible.
	ServicePrefix   string              `yaml:"servicePrefix"`
	RootView        projection.RootView `yaml:"rootView"`
	ByPathFallback  string              `yaml:"byPathFallback"` // "orphaned" | "drop"
	ShowStats       bool                `yaml:"showStats"`
	ShowByArtifact  bool                `yaml:"showByArtifact"`
	ShowOrphaned    bool                `yaml:"showOrphaned"`
	ShowByDate      bool                `yaml:"showByDate"`
	ShowBySession   bool                `yaml:"showBySession"`
	ShowByNamespace bool                `yaml:"showByNamespace"`
	ShowRaw         bool                `yaml:"showRaw"`
}

Config is the shared configuration consumed by every Scrinium-backed application. Surface-specific configs (webdav listen address, fuse mount point, etc.) live in the respective host packages and reference this Config.

Config is meant to be loaded from YAML or built up programmatically; CLI flag binding is the host's job.

Two URIs identify the backend:

• Store points at the artifact storage (file://, s3://...).
• Index points at the metadata index (sqlite://, postgres://).

Bare paths in Store are accepted for backward compatibility (treated as file://). Index always requires an explicit scheme.

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns a Config with the recommended defaults: Plain-DEK store, ServicePrefix "_scrinium", service trees enabled, root view = byPath, fallback = orphaned, editing off, default mode 0644.

Callers customise from here rather than building from scratch — the field set keeps growing and zero-valued configs accumulate footguns.

func (Config) Validate

func (c Config) Validate() error

Validate reports configuration mistakes that would otherwise surface at Open time as confusing wrappers around lower-level errors. Lightweight checks only — full validation happens inside Open against the actual filesystem and index.

type Scrinium

type Scrinium struct {
	// Config is the validated config Scrinium was opened with.
	// Surfaces consult routing/policy fields here when they
	// need them past Open.
	Config Config

	// Store is the high-level CAS store. Surfaces use it for
	// Put/Get and for capacity queries (stats endpoints).
	Store core.Store

	// Index is the metadata index. Surfaces rarely touch this
	// directly; it's exposed for diagnostics like the
	// extension list rendered in stats.
	Index core.StoreIndex

	// View is the read-side projection of the store: trees by
	// path, by date, etc. Both FUSE and WebDAV adapters route
	// reads through it.
	View *projection.View

	// FSOps is the read/write filesystem facade — the layer
	// FUSE and WebDAV adapters wrap. Carries the mount session
	// and editing policy resolved from Config.
	FSOps *projection.FSOps

	// FSIndex is the filesystem-projection index extension
	// kept in scope so it can be referenced after Open
	// (e.g. ListExtensions for stats).
	FSIndex *fsindex.Extension

	// MountSession is the boot-unique identifier this Scrinium
	// instance presents in stats and uses as a tiebreaker.
	// Generated at Open or Init time.
	MountSession string
	// contains filtered or unexported fields
}

Scrinium holds the long-lived resources every Scrinium-backed application shares: an open Store, a StoreIndex, a projection View, an FSOps facade, plus a boot-unique MountSession used in stats and as a tiebreaker. Construct with Open or Init; shut down with Close.

Hosts consume the fields directly. Surfaces (FUSE, WebDAV, HTTP, gRPC, etc.) read from View and FSOps; admin tooling reaches into Store and Index for management operations.

The struct is intentionally a plain bag of pointers — no internal state, no methods beyond the lifecycle ones. That leaves hosts free to wrap or extend it without inheritance gymnastics.

func Init

func Init(ctx context.Context, cfg Config) (_ *Scrinium, recoveryKit []byte, retErr error)

Init creates a fresh Scrinium store at the location described by cfg.Store, then opens it and returns a ready-to-use *Scrinium runtime. The store directory is created if it does not exist.

The flow:

1. Resolve and create the store directory (file:// only).
2. Open driver via DialDriver.
3. Resolve and open index (sqlite by default for file:// stores).
4. Register fsindex extension.
5. core.InitStore — writes the descriptor + system.config. If cfg.PassphraseFile is set, the store is created encrypted and a recovery kit is produced.
6. Build View + FSOps to deliver a ready-to-use runtime.

The recoveryKit return value is non-nil only for encrypted stores. The host MUST persist it (out of band — print, save to a vault, etc.) before the first Close. It is the only path back into the store if the passphrase is lost.

On error, any partially-opened resources are cleaned up. Init does not delete the store directory on error: a half- initialised tree is recoverable; a deleted one is not.

func Open

func Open(ctx context.Context, cfg Config) (_ *Scrinium, retErr error)

Open builds a Scrinium runtime from a Config: parses URIs, opens store and index, registers fsindex, builds the View, generates the mount session, prepares the scratch directory, and assembles FSOps.

The order of operations:

1. Validate config.
2. Open driver via DialDriver(cfg.Store).
3. Resolve and open index (default sqlite under store dir when cfg.Index is empty and store is file://).
4. Register fsindex extension. Must precede OpenStore so the very first IndexManifest call dispatches into it.
5. core.OpenStore with hash registry. If cfg.PassphraseFile is set, the store is opened with a passphrase provider that reads the file; this transitions an encrypted Store from Locked to Unlocked.
6. Build View with fsmeta resolver and fsindex backing.
7. Generate MountSession.
8. Prepare scratch directory.
9. Build FSOps with policy from Config.

On any failure the partial state is unwound: opened resources are closed, errors are wrapped with their stage for fast diagnosis. Cleanup errors are logged to stderr; they cannot be returned alongside the primary error.

func OpenOrInit

func OpenOrInit(ctx context.Context, cfg Config) (*Scrinium, []byte, bool, error)

OpenOrInit opens the store if it already exists, or initialises a fresh one if it does not. Convenient for examples, single- binary tools, and tests; production daemons typically separate "init" and "serve" subcommands so an operator can audit when a brand-new store is being created.

Return values:

• *Scrinium — ready-to-use runtime, identical in shape whether it came from Open or Init.
• []byte — recovery kit. Non-nil ONLY when this call just initialised an encrypted store (i.e. created == true AND cfg.PassphraseFile is set). Host MUST persist this out of band.
• bool created — true if this call initialised the store, false if it opened an existing one.
• error — any failure other than "store does not exist". A genuine error (bad URI, no permission, disk fault) does NOT silently fall through to Init.

The "store not found" detection uses errors.Is against errs.ErrStoreNotFound, which bridges to fs.ErrNotExist — so the branch fires only on the specific not-initialised case, not on every Open failure.

Example:

s, kit, created, err := scrinium.OpenOrInit(ctx, cfg)
if err != nil {
    return err
}
defer s.Close()
if created && kit != nil {
    persistRecoveryKit(kit)  // host responsibility
}

func (*Scrinium) Close

func (s *Scrinium) Close() error

Close releases the runtime's resources in reverse order of Open. Errors are accumulated; every resource is given a chance to close before we return. Multiple errors are joined via errors.Join so callers can errors.Is/As any of them.

Close is idempotent: the first call performs the shutdown, subsequent calls return the same error without doing anything. This matches the io.Closer convention used across the stdlib (*os.File, *sql.DB) — a `defer s.Close()` paired with an explicit s.Close() in the main path is safe.

Close is also tolerant of partially-constructed Scrinium values: any nil resource is skipped. This allows tests and supervisors that abort partway through Open to call Close without a nil-pointer panic.

Order matters for the resources that ARE present:

• View first: stops backfill goroutines so they aren't reading from a Store that's about to be closed.
• FSIndex next: clears its reference to the StoreIndex so any stray post-close GetByID calls fail cleanly ("not registered") instead of touching a closed handle.
• Store next: wipes secrets (DEK, capability token, default StaticKeyResolver). We want secrets gone before the index is torn down (sqlite teardown can flush dirty pages — minor, but the principle is "secrets first").
• Index last: releases the sqlite handle (or future PG connection pool). Leaked DB handles keep the file locked on Windows and waste fds elsewhere.

The Driver (localfs) holds nothing closable.

func (*Scrinium) ListExtensions

func (s *Scrinium) ListExtensions() []index.ExtensionInfo

ListExtensions returns the registered index extensions. Empty slice on backends that don't support extension listing — the surface presents "no extensions" rather than failing.

The lookup is cheap (in-memory map on the backend) so we don't cache; surfaces call this on every stats render.

func (*Scrinium) RoutingConfig

func (s *Scrinium) RoutingConfig() projection.RoutingConfig

RoutingConfig is a snapshot of the visibility/policy fields from the runtime's Config in the shape projection.View and the surface routers consume. Surfaces typically build it once at startup and pass it down to their tree builders.

Use this helper instead of constructing projection.RoutingConfig manually — it tracks Scrinium's preferred defaults and means a new Show* flag added later only needs to be wired here.

func (*Scrinium) StatsProvider

func (s *Scrinium) StatsProvider(ctx context.Context, startedAt time.Time, capacityTimeout time.Duration) func() []byte

StatsProvider returns a function that renders the runtime's stats snapshot as bytes — the format consumed by the _scrinium/stats endpoint in every reference surface.

The returned closure captures startedAt (the time the surface began serving) and reaches into Scrinium for the rest. Surfaces wire it into their stats route or pseudo-file handler.

The capacityTimeout caps the call to Store.Capacity — a slow driver should never hang `cat _scrinium/stats`. Pass a reasonable value like 2*time.Second; on timeout or other error capacity is omitted from the output (rendered as "n/a" fields rather than failing the whole stats read).

Use this helper instead of building the snapshot manually — it tracks the format projection.RenderStats expects, and surfaces inherit improvements (new fields, format tweaks) without each having to update.