index

Documentation

Overview

Package index contains implementations of core.StoreIndex and curator.MultistoreIndex.

Subpackages:

• index/sqlite — the primary StoreIndex implementation for the embedded mode. (M1.2)
• index/postgres — a shared StoreIndex for multi-host deployments. Lands when needed.
• index/memory — an in-memory implementation for tests.
• index/multistore — a curator.MultistoreIndex implementation. (M4.1)

In M0 this package contains only a stub doc and the shared types/options used across implementations.

DAG: index imports core (the StoreIndex contract), driver (for access to Capabilities when picking optimisations), and event (for emitting metric events).

Index

• Constants
• Variables
• func DialIndex(ctx context.Context, uri string, opts ...IndexOption) (core.StoreIndex, error)
• func RegisterDialer(scheme string, d Dialer)
• func RegisteredSchemes() []string
• type Dialer
• type EventArgs
• type EventKind
  • func (k EventKind) String() string
• type ExtensionHost
• type ExtensionInfo
• type ExtensionLister
• type ExtensionRegistry
• type ExtensionStore
• type IndexContentionErrorPayload
• type IndexExtension
• type IndexOption
  • func WithIndexPublisher(p core.Publisher) IndexOption
• type IndexOptions
• type IndexSizePayload
• type IndexWriteLatencyPayload

Constants

const (
	// EventIndexWriteLatency — latency of one mutating method
	// (IndexManifest, DeleteManifest, RebindBlob, ...). Emitted on
	// every successful and failing call so dashboards can compare
	// success/failure latency distributions.
	EventIndexWriteLatency = "index.write_latency"

	// EventIndexContentionError — contention condition observed
	// (SQLite SQLITE_BUSY past busy_timeout; equivalent in other
	// backends). The operation may still have succeeded after
	// retry; subscribers correlate with WriteLatency to tell the
	// difference.
	EventIndexContentionError = "index.contention_error"

	// EventIndexSize — periodic snapshot of index size. Emitted on
	// a backend-configurable interval; not a per-write event.
	EventIndexSize = "index.size"
)

Metric events emitted by StoreIndex and MultistoreIndex implementations. Routed through the Publisher passed via WithIndexPublisher; without a publisher they are silently dropped — emission is a diagnostic surface, not a correctness requirement.

All four event-type prefixes ("core.", "agent.", "curator.", "index.") are reserved per docs/2. Internals/01 §1.7. The "index." namespace covers any backend (sqlite, postgres, in-memory) — payload shapes are backend-agnostic.

Variables

var (
	// ErrStopScan is returned by an ExtensionStore.Scan callback
	// to stop the iteration without surfacing it as an error.
	ErrStopScan = errors.New("scrinium/index: stop scan")

	// ErrExtensionExists is returned by Register when an
	// extension with the same Name() is already registered.
	ErrExtensionExists = errors.New("scrinium/index: extension already registered")

	// ErrSchemaRegression is returned by Register when the
	// extension's SchemaVersion() is less than the persisted
	// value. Backends do not auto-downgrade.
	ErrSchemaRegression = errors.New("scrinium/index: extension schema version regressed")

	// ErrBackendMismatch is returned by extension Setup when
	// the backend does not implement an interface the extension
	// requires (typically a backend escape hatch like SQLBackend).
	ErrBackendMismatch = errors.New("scrinium/index: extension incompatible with backend")

	// ErrEmptyPrefix is returned by DeletePrefix when called
	// with an empty prefix. "Delete all rows of a table" is an
	// explicit operation; callers must Scan and Delete to get it.
	ErrEmptyPrefix = errors.New("scrinium/index: empty prefix in DeletePrefix")
)

Sentinel errors. Wrapped by backends with %w so callers can errors.Is against them.

Functions

func DialIndex

func DialIndex(ctx context.Context, uri string, opts ...IndexOption) (core.StoreIndex, error)

DialIndex opens a StoreIndex by URI. The scheme selects the backend (registered via RegisterDialer); the rest of the URI is forwarded to the dialer.

Unlike DialDriver, there is no bare-path fallback: index URIs are new from day one (no legacy "indexPath" config to honour), and a bare path is ambiguous between sqlite, file store, etc. The scheme is mandatory.

func RegisterDialer

func RegisterDialer(scheme string, d Dialer)

RegisterDialer attaches a Dialer to a URI scheme. Called from package init() in index implementations:

// index/sqlite/register.go
func init() { index.RegisterDialer("sqlite", openSQLiteURI) }

Re-registering the same scheme panics — collisions are programming errors that should fail at startup, not produce silent overrides.

func RegisteredSchemes

func RegisteredSchemes() []string

RegisteredSchemes returns the schemes currently registered. Sorted; useful for error messages and --help output.

Types

type Dialer

type Dialer func(ctx context.Context, u *url.URL, opts ...IndexOption) (core.StoreIndex, error)

Dialer opens a StoreIndex from a parsed URI. Implementations live in scheme-specific packages (index/sqlite, eventually index/postgres) and register themselves via RegisterDialer in their package init().

IndexOption values are forwarded so callers can pass cross- backend tunables (e.g. a Publisher) while leaving backend- specific tuning to URI query parameters or backend-specific options.

type EventArgs

type EventArgs struct {
	Manifest   domain.Manifest
	ArtifactID domain.ArtifactID
	BlobRefs   []string
}

EventArgs carries operation-specific arguments for Apply. Not every field is populated for every kind — see the EventKind constants for per-kind documentation.

type EventKind

type EventKind uint8

EventKind names an index-level operation an extension can observe. The set is closed (defined here, exhaustive switch in implementations).

const (
	// EventKindManifestIndexed corresponds to StoreIndex.IndexManifest.
	// EventArgs.Manifest carries the full manifest just written;
	// EventArgs.ArtifactID is a duplicate of Manifest.ArtifactID.
	EventKindManifestIndexed EventKind = iota

	// EventKindManifestDeleted corresponds to StoreIndex.DeleteManifest.
	// EventArgs.ArtifactID is the id of the deleted artifact;
	// EventArgs.BlobRefs lists the blobs the deletion decremented;
	// EventArgs.Manifest is zero.
	EventKindManifestDeleted

	// EventKindBlobRebound corresponds to StoreIndex.RebindBlob.
	// EventArgs.BlobRefs[0] is the rebound blob ref; the rest of
	// EventArgs is zero.
	EventKindBlobRebound
)

func (EventKind) String

func (k EventKind) String() string

String returns a human-readable name for the kind. Used in error messages and diagnostic events.

type ExtensionHost

type ExtensionHost interface {
	// Extensions returns the registry through which IndexExtension
	// implementations are attached to this backend.
	Extensions() ExtensionRegistry
}

ExtensionHost is the optional capability a StoreIndex backend exposes when it supports registering host-side extensions.

Backends that support extensions implement this; the rest are transparently skipped by callers that type-assert it. Lives here (not on core.StoreIndex) so the core package needs no import of engine/index — the assertion happens at the wiring layer instead.

type ExtensionInfo

type ExtensionInfo struct {
	// Name is the extension's stable identifier
	// (IndexExtension.Name()).
	Name string

	// SchemaVersion is the persisted schema version for this
	// extension on this backend, after the most recent successful
	// Setup. Used to surface migration state.
	SchemaVersion int
}

ExtensionInfo is the public, backend-agnostic descriptor of a registered index extension. Surfaces (stats endpoints, debug pages, examples) consume this rather than reaching into a concrete backend type.

Backends that report registered extensions return slices of this type via ExtensionLister. The type is intentionally flat — no behaviour, no pointers — so it can travel through any layer without dragging dependencies.

type ExtensionLister

type ExtensionLister interface {
	ListExtensions() []ExtensionInfo
}

ExtensionLister is the optional capability a StoreIndex backend exposes when it can enumerate currently-registered extensions.

Distinct from ExtensionHost (the registration-side capability) because read and write surfaces are conceptually independent — a future read-only proxy backend might list without registering, or a constrained backend might register without listing. In practice today's sqlite backend implements both.

Returns an empty slice (never nil) when no extensions are registered. Order is unspecified — callers that need stable listings sort by Name.

type ExtensionRegistry

type ExtensionRegistry interface {
	// Register attaches an extension to the index. Setup runs
	// in a single transaction; failure rolls back any work the
	// extension did during Setup, persisted schema_version is
	// not bumped, and no subscriptions are recorded.
	//
	// Returns ErrExtensionExists if Name() collides with an
	// already-registered extension.
	// Returns ErrSchemaRegression if SchemaVersion() is less
	// than the persisted value for this Name().
	Register(ctx context.Context, ext IndexExtension) error
}

ExtensionRegistry is the surface returned by StoreIndex.Extensions. The only mutation is Register; backends expose nothing else through this interface.

type ExtensionStore

type ExtensionStore interface {
	// Put writes (or overwrites) a value for the given key.
	// Idempotent.
	Put(table, key string, value []byte) error

	// Get retrieves the value for a key.
	// (value, true, nil) — key present.
	// (nil, false, nil) — key absent.
	// Errors are reserved for backend-infrastructure failures.
	Get(table, key string) ([]byte, bool, error)

	// Delete removes a key. No-op if the key is absent (no
	// "not found" error).
	Delete(table, key string) error

	// DeletePrefix removes every key whose lexicographic value
	// starts with prefix. An empty prefix is rejected to make
	// "delete all" an explicit, deliberate operation (callers
	// who really want it must Scan and Delete one by one).
	DeletePrefix(table, prefix string) error

	// Scan iterates entries with the given prefix in
	// lexicographic key order. cb returning ErrStopScan stops
	// the walk without an error; any other error is propagated.
	Scan(table, prefix string, cb func(key string, value []byte) error) error

	// Inc atomically adds delta to the int64 value (encoded as
	// big-endian 8 bytes). Creates the key with delta if absent.
	// Returns the new value.
	Inc(table, key string, delta int64) (int64, error)
}

ExtensionStore is the backend-agnostic data plane an extension uses for its own state. All mutations are scoped to the surrounding transaction; reads see committed state.

Tables are auto-namespaced by extension Name; the same `table` argument from two extensions does not collide.

type IndexContentionErrorPayload

type IndexContentionErrorPayload struct {
	Operation string
	WaitedFor time.Duration
}

IndexContentionErrorPayload is a write-contention event (for example, SQLITE_BUSY past busy_timeout). WaitedFor is how long the call waited before failing or succeeding.

type IndexExtension

type IndexExtension interface {
	// Name is the stable identifier for this extension. Used
	// as the namespace prefix in ExtensionStore. Must be
	// unique among extensions registered to the same backend.
	// Recommended: dotted, lower-case, like "scrinium.fsindex".
	Name() string

	// SchemaVersion is the current data layout version. The
	// backend persists the version of the most-recent successful
	// Setup; on a later registration with a different version,
	// Setup is called with the stored value as oldVersion and
	// the extension migrates.
	SchemaVersion() int

	// Subscribe returns the event kinds this extension wants to
	// observe. An empty slice means "no subscriptions" — useful
	// for read-only extensions that initialise data in Setup
	// and never react to mutations. Called once at Register;
	// the result is cached.
	Subscribe() []EventKind

	// Setup runs once per registration, inside the registration
	// transaction. oldVersion is 0 on the first registration; a
	// positive value is the persisted SchemaVersion from a
	// previous run. Implementations migrate from oldVersion to
	// SchemaVersion().
	Setup(ctx context.Context, store ExtensionStore, oldVersion int) error

	// Apply is invoked for each subscribed event, inside the
	// backend's transaction. Mutations through store land in
	// the same transaction as the main index write; an error
	// returned here aborts both.
	Apply(ctx context.Context, store ExtensionStore, kind EventKind, args EventArgs) error

	// Close releases extension-side resources. Backend storage
	// remains owned by the StoreIndex.
	Close() error
}

IndexExtension is the contract host-side index extensions satisfy. An extension lives inside a StoreIndex backend, shares its transactions, and exposes its own read API to the host.

Two-paragraph mental model:

(1) Subscriptions. Extensions declare which mutations they care about via Subscribe. The backend dispatches matching events into Apply WITHIN the same transaction as the main index write — so an extension cannot drift from the main index state. A failure in Apply rolls the whole transaction back, including the main write.

(2) Storage. Extensions own no SQL, no DB handles, no migration code: they put bytes into a backend-agnostic ExtensionStore keyed by (table, key). The backend translates to its own substrate. Tables are namespace-prefixed by extension Name to prevent collisions between extensions.

Contract spec: 3. Contracts/06 Index Extensions.md. Behaviour and sqlite implementation: 4. API Reference/16.

type IndexOption

type IndexOption func(*IndexOptions)

IndexOption is an option for the constructor of a StoreIndex implementation. It applies to sqlite.NewStore, postgres.New, NewMultistore, etc.

func WithIndexPublisher

func WithIndexPublisher(p core.Publisher) IndexOption

WithIndexPublisher provides a Publisher for emitting metric events (write_latency, contention_error, size). When omitted, events are silently dropped — the index's behaviour does not change.

type IndexOptions

type IndexOptions struct {
	// Publisher is the event bus to which the implementation
	// emits index.* metric events. nil disables emission.
	Publisher core.Publisher
}

IndexOptions is the resolved option aggregate that index implementations apply at construction time. Exported because concrete backends (sqlite, postgres) live in subpackages and must read the resolved values to wire them into their own state.

Add fields here only when they are common to every backend. Backend-specific knobs (busy_timeout for SQLite, pool size for Postgres) stay private to the implementing package.

type IndexSizePayload

type IndexSizePayload struct {
	DBBytes       int64
	WALBytes      int64
	BlobCount     int64
	ManifestCount int64
}

IndexSizePayload is a periodic snapshot of the index size. Emitted by the implementation at a configurable interval.

type IndexWriteLatencyPayload

type IndexWriteLatencyPayload struct {
	Operation string
	Duration  time.Duration
}

IndexWriteLatencyPayload is the latency of a single mutating operation. Operation is the method name (for example, "IndexManifest", "DeleteManifest").