federated

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 25, 2026 License: MIT Imports: 27 Imported by: 0

Documentation

Index

Constants

View Source
const NotePartialParquetExclusion = "partial parquet scan: excluded corrupt objects"

NotePartialParquetExclusion prefixes the execution-plan note recording a partial parquet scan (#251): verification-confirmed corrupt objects were excluded and the query answered from the readable remainder. Notes never cross the HTTP boundary (toExecutionPlan drops them, #301/#306); embedders and the e2e harness assert on this exported prefix.

Variables

View Source
var (
	ErrNoParquetPaths         = forma.ErrNoParquetPaths
	ErrManifestSchemaMismatch = forma.ErrManifestSchemaMismatch
	// ErrParquetSetInconsistent joined them for #301: internal/httpapi
	// classifies it to redact the object keys it carries out of public response
	// bodies, and cannot import this package without pulling DuckDB CGO into a
	// pure-Go test build. Redaction is gated on sentinel evidence rather than on
	// the status, so this holds on any status the error is classified as, not
	// only 5xx.
	ErrParquetSetInconsistent = forma.ErrParquetSetInconsistent
)

ErrNoParquetPaths and ErrManifestSchemaMismatch are defined in the public root package and re-exported here for internal call sites. They must be matchable by embedders that reach the engine through factory.NewEntityManager* and cannot import an internal package — the whole point of #299 was a discriminator callers can act on, which a package-private sentinel is not. These are aliases, not copies: errors.Is/errors.As behave identically whether a caller reaches for the forma.* or federated.* name.

View Source
var ErrDuckDBUnavailable = errors.New("duckdb unavailable")

ErrDuckDBUnavailable marks queries rejected before reaching DuckDB: the client is not configured (or closed), or the circuit breaker is open. Transient infrastructure — degradable under AllowPartialDegradedMode.

View Source
var ErrFederatedReadFailed = errors.New("federated read failed")

ErrFederatedReadFailed marks a DuckDB federated read that failed at execution or while streaming rows: unreadable parquet (corrupt bytes, truncation, schema mismatch), storage-layer rejections (credentials), or a mid-stream fault. The referenced objects exist — a manifest-listed object missing from storage classifies as ErrParquetSetInconsistent instead. Transient/object-scoped — degradable under AllowPartialDegradedMode.

View Source
var ErrKeysetUnsupportedOnPostgres = errors.New("keyset cursor unsupported on the postgres-only path")

ErrKeysetUnsupportedOnPostgres marks a request that carries a keyset cursor and routed to the Postgres-only path. That path builds a model.PersistentRecordQuery, which has no cursor field, so it can neither apply nor reject the cursor itself: pre-#354 it silently answered an unfiltered first page and pagination never advanced. Not degradable — the degraded fallback IS the Postgres-only path.

View Source
var ErrPostgresReadFailed = errors.New("postgres read failed")

ErrPostgresReadFailed marks the Postgres side of a federated read failing: the dirty-ID consistency fetch or a Postgres-source page read. A Postgres outage is not degradable in practice — the degraded fallback is itself Postgres-only — so this classification is what #187 scenario 9 asserts on all its probes; degradability is still decided at the engine seam, not here.

View Source
var ErrSchemaMetadataCacheRequired = errors.New("schema metadata cache required but not loaded")

ErrSchemaMetadataCacheRequired marks a federated query that cannot build a correct entity_main projection because the schema's metadata cache is not loaded. It is a configuration / data-contract error, not a transient infrastructure failure, so the public Query path must not absorb it under AllowPartialDegradedMode — degrading would silently return a Postgres-only partial result and hide the missing-cache problem #151 makes loud.

Functions

func DuckDBPostgresConnStringFromPool

func DuckDBPostgresConnStringFromPool(pool *pgxpool.Pool) string

DuckDBPostgresConnStringFromPool derives the libpq-style connection string DuckDB's postgres_scanner needs from an existing pgx pool. It returns "" for a nil pool or pool config.

Every string value is quoted via pgdsn.Quote, the same rule internal/cdc's BuildPGDSN has used since #290. The unquoted form this replaced was two bugs:

  • A password (or host, user, dbname) containing a space produced a DSN libpq could not parse, so postgres_scan failed to attach at all.
  • The credential scrubber cannot tell where an unquoted value ends. It terminates on whitespace, so an attach failure — whose prose quotes this whole string back — logged the tail of the password past the placeholder, which is the exposure #301 exists to close.

All four string fields are quoted, not just the password: libpq accepts quoted values for every keyword, a host may be a socket path, and user and database names may legally contain spaces. Quoting uniformly means no field can be the one that breaks parsing. sslmode is deliberately not emitted here — it is absent from this DSN today, and emitting it with an empty quoted value would be rejected by libpq.

The result is embedded in a single-quoted DuckDB SQL literal (postgres_scan('{{.PG_CONN}}', …)), so the renderer escapes it with sqlutil.EscapeLiteral (internal/sqlutil/literal.go), called from the DuckDB renderer.

func EvaluateRoutingPolicy

EvaluateRoutingPolicy makes a routing decision based on config, query hints and options.

func MergePersistentRecordsByTier

func MergePersistentRecordsByTier(inputs map[model.DataTier][]*model.PersistentRecord, preferHot bool) ([]*model.PersistentRecord, error)

MergePersistentRecordsByTier performs a merge-on-read across multiple data tiers. Inputs are provided as a map from model.DataTier -> slice of *model.PersistentRecord. Last-write-wins semantics are applied using model.PersistentRecord.UpdatedAt and ChangeLog flushed state.

Behavior:

  • Records are deduplicated by (SchemaID, RowID).
  • For each key, the record with the highest UpdatedAt is chosen. If UpdatedAt is equal, a tombstone (DeletedAt != nil && *DeletedAt != 0) beats a live copy; DeletedAt = 0 is the cold-tier live encoding (#274), never a tombstone. Remaining ties use deterministic tier priority (Hot > Warm > Cold).
  • If a record originates from the ChangeLog buffer (flushed_at == 0) it is considered the authoritative hot source and wins ties regardless of UpdatedAt.
  • The chosen record is returned with OtherAttributes merged across all source tiers for that (SchemaID, RowID) with attribute-level deduplication.
  • Attributes are deduplicated by (AttrID, ArrayIndices).
  • For an attribute present in multiple source records, the attribute from the record with the latest UpdatedAt is chosen. Ties are resolved using deterministic tier ordering (Hot > Warm > Cold).
  • Deleted records (DeletedAt != nil && DeletedAt != 0) are excluded from results.
  • Result slice is sorted by SchemaID then RowID for deterministic output.

The preferHot parameter is deprecated and ignored; tier priority is always deterministic.

func ValidateDuckDBConfig

func ValidateDuckDBConfig(cfg forma.DuckDBConfig) error

ValidateDuckDBConfig performs basic sanity checks on user-provided DuckDB configuration.

Types

type CircuitBreaker

type CircuitBreaker struct {
	// contains filtered or unexported fields
}

CircuitBreaker is a lightweight in-memory circuit breaker with strict single-probe half-open recovery (#246, supersedes the #185 immediate-forgiveness design).

States (all transitions guarded by mu):

  • closed: no open period recorded; every caller is admitted.
  • open (now < openUntil): every caller is rejected.
  • half-open (openUntil elapsed): Allow admits exactly one caller as the probe and rejects the rest until the probe resolves. RecordSuccess closes the breaker (failure history cleared); RecordFailure re-opens it for a fresh openDuration without threshold re-accumulation.

Probe abandonment: a probe that never reports (its query was cancelled between Allow and Record*) lapses openDuration after admission, and the next Allow reclaims the slot. A lost probe therefore costs at most one extra openDuration of rejections. The flip side: a probe still legitimately running past openDuration is indistinguishable from an abandoned one, so a slow dependency can see more than one concurrent probe — "exactly one" holds only for probes that resolve within openDuration.

Stale callers: RecordSuccess closes the breaker from any state — a query admitted before the breaker opened that completes afterwards is real evidence the dependency is healthy. A stale RecordFailure landing while a probe is in flight re-opens the breaker: conservative, and indistinguishable from a probe failure without per-caller tokens. ReleaseProbe, by contrast, IS token-scoped (#349 review R2-2): it can run long after admission (the #251 post-verification path), by which time the probe slot may belong to a newer caller — an unscoped release would free a reservation its caller never held and admit a second concurrent probe.

func NewCircuitBreaker

func NewCircuitBreaker(threshold int, window, openDuration time.Duration) *CircuitBreaker

NewCircuitBreaker creates a configured circuit breaker.

func (*CircuitBreaker) Allow

func (cb *CircuitBreaker) Allow() (admitted bool, probe ProbeToken)

Allow reports whether the caller may proceed. In half-open state it atomically reserves the single probe slot: the one caller that receives admitted=true with a non-zero token while others are rejected MUST resolve the probe via RecordSuccess or RecordFailure (or let the reservation lapse after openDuration). Callers admitted while the breaker is closed receive the zero token: they hold no reservation, and their ReleaseProbe is a no-op.

func (*CircuitBreaker) IsOpen

func (cb *CircuitBreaker) IsOpen() bool

IsOpen reports whether the timed open period is currently active. Observation only (tests, telemetry): admission control — including the half-open probe reservation — lives in Allow.

func (*CircuitBreaker) RecordFailure

func (cb *CircuitBreaker) RecordFailure()

RecordFailure records a failure occurrence. A probe failure re-opens the breaker directly; otherwise failures accumulate in the sliding window and open the breaker at threshold.

func (*CircuitBreaker) RecordSuccess

func (cb *CircuitBreaker) RecordSuccess()

RecordSuccess closes the breaker and clears the failure history. It resolves an in-flight probe, and also closes from open state when a pre-open in-flight query completes (see the type doc on stale callers).

func (*CircuitBreaker) ReleaseProbe

func (cb *CircuitBreaker) ReleaseProbe(probe ProbeToken)

ReleaseProbe relinquishes a half-open probe reservation without recording evidence either way, freeing the slot for the next caller. It is for callers admitted by Allow that then failed BEFORE touching the dependency — a misconfiguration caught during path resolution, invalid caller input — so they learned nothing about its health and must not consume the probe. It is also for the one post-execution caller whose failure indicts a specific object rather than the dependency: the #251 corrupt-confirmed path, where per-file verification has just drained every OTHER object through this same engine and store — live proof of health that makes RecordFailure dishonest, while the query as a whole still failed, making RecordSuccess equally so.

Without this, such a caller abandons the reservation: the slot stays occupied until it lapses (openDuration), and every request in that window is rejected with ErrDuckDBUnavailable. That matters because ErrDuckDBUnavailable IS degradable while the pre-execution errors are deliberately not, so a misconfiguration that must always be loud would be answered from Postgres alone on the very next request (#299 review P1).

Neutral by design: the breaker stays open, so a real outage still gates traffic; only the probe slot returns. Recording success here would close the breaker on no evidence, and recording failure would extend the outage for a dependency that was never consulted.

Release is scoped to the caller's own reservation (#349 review R2-2): only the token Allow handed out frees the slot, so a caller admitted while the breaker was closed (zero token) — or one whose lapsed reservation was already reclaimed by a newer probe — cannot clear the probe another caller is running.

type DBFederatedQueryEngine

type DBFederatedQueryEngine struct {
	// contains filtered or unexported fields
}

DBFederatedQueryEngine executes federated queries across the Postgres hot tier and the DuckDB/Parquet warm+cold tiers: routing policy, dirty-set exclusion, DuckDB execution, merge, and pagination. It is the sole implementation of the FederatedQueryEngine interface.

func NewDBFederatedQueryEngine

func NewDBFederatedQueryEngine(pgSource PostgresFederatedSource, dirtyIDFetcher DirtyIDFetcher, duck DuckDBQueryExecutor, breaker *CircuitBreaker, cfg forma.DuckDBConfig, metadataCache *schemameta.MetadataCache, pgConnString string, opts ...EngineOption) *DBFederatedQueryEngine

NewDBFederatedQueryEngine assembles the engine from its injected seams. duck and breaker may be nil: a nil duck marks DuckDB as unavailable and a nil breaker disables circuit breaking.

func (*DBFederatedQueryEngine) ExecuteDuckDBFederatedQuery

func (e *DBFederatedQueryEngine) ExecuteDuckDBFederatedQuery(
	ctx context.Context,
	tables model.StorageTables,
	q *model.FederatedAttributeQuery,
	limit, offset int,
	attributeOrders []model.AttributeOrder,
	opts *model.FederatedQueryOptions,
) ([]*model.PersistentRecord, int64, error)

ExecuteDuckDBFederatedQuery runs the DuckDB optimized query template using the provided model.FederatedAttributeQuery. It fetches dirty IDs from the Postgres change_log (if available), injects exclusions into the DuckDB WHERE clause, executes the query against the global DuckDB client, and returns matched PersistentRecords along with the total record count.

Note: This implementation performs a best-effort scan of columns produced by the optimized query template. It mirrors the column ordering used by the Postgres template:

  • main table projection (entity_main columns, order defined by model.EntityMainColumnDescriptors)
  • attributes_json (TEXT)
  • total_records (bigint)
  • total_pages (bigint)
  • current_page (int)

A read failure attributed to specific corrupt parquet objects is retried exactly once against the readable remainder (#251), so one call can issue two scans plus a per-object verification drain — worth knowing when sizing the caller's deadline. The execution plan describes only the pass that produced the returned page.

func (*DBFederatedQueryEngine) ExecuteFederatedPaginatedQuery

func (e *DBFederatedQueryEngine) ExecuteFederatedPaginatedQuery(
	ctx context.Context,
	tables model.StorageTables,
	fq *model.FederatedAttributeQuery,
	limit, offset int,
	attributeOrders []model.AttributeOrder,
	opts *model.FederatedQueryOptions,
) ([]*model.PersistentRecord, int64, error)

ExecuteFederatedPaginatedQuery performs a federated fetch across Postgres (hot) and DuckDB (cold/warm), merges results with last-write-wins semantics, and returns the requested page plus an accurate total deduplicated across sources.

Notes: - This is an MVP coordinator: it caps per-source fetches (opts.MaxRows or default) to avoid OOM. - For very large result sets a keys-only two-phase approach should be implemented later.

func (*DBFederatedQueryEngine) Query

Query implements FederatedQueryEngine. Hot-only requests delegate directly to Postgres; otherwise the routing policy decides between Postgres and the DuckDB federated path, falling back to Postgres on DuckDB failure when opts.AllowPartialDegradedMode is set.

func (*DBFederatedQueryEngine) StreamDuckDBFederatedQuery

func (e *DBFederatedQueryEngine) StreamDuckDBFederatedQuery(
	ctx context.Context,
	tables model.StorageTables,
	q *model.FederatedAttributeQuery,
	limit, offset int,
	attributeOrders []model.AttributeOrder,
	opts *model.FederatedQueryOptions,
	rowHandler func(context.Context, *model.PersistentRecord) error,
) (int64, error)

StreamDuckDBFederatedQuery streams DuckDB federated query results using a rowHandler callback. It reuses the same rowHandler semantics as Postgres' StreamOptimizedQuery to avoid loading the entire result set into memory.

type DirtyIDFetcher

type DirtyIDFetcher interface {
	FetchDirtyRowIDs(ctx context.Context, changeLogTable string, schemaID int16) ([]uuid.UUID, error)
}

DirtyIDFetcher retrieves row IDs from the change log that are newer than the flushed Parquet tiers and must be excluded from DuckDB results.

type DirtyIDPool

type DirtyIDPool = dirtyIDPool

type DuckDBClient

type DuckDBClient struct {
	DB *sql.DB
	// contains filtered or unexported fields
}

DuckDBClient wraps a database/sql DB opened with the DuckDB driver.

func NewDuckDBClient

func NewDuckDBClient(cfg forma.DuckDBConfig) (*DuckDBClient, error)

NewDuckDBClient creates and configures a DuckDB client according to the provided config. It attempts to load common extensions (httpfs/parquet) and configure S3 access via PRAGMA when requested.

func NewDuckDBClientContext

func NewDuckDBClientContext(ctx context.Context, cfg forma.DuckDBConfig) (*DuckDBClient, error)

NewDuckDBClientContext creates and configures a DuckDB client while honoring the caller-provided context during bootstrap.

func (*DuckDBClient) Close

func (c *DuckDBClient) Close() error

Close closes the underlying DuckDB DB.

func (*DuckDBClient) HealthCheck

func (c *DuckDBClient) HealthCheck(ctx context.Context) error

HealthCheck performs a simple query to validate the DuckDB connection and basic runtime pragmas.

type DuckDBClientQueryExecutor

type DuckDBClientQueryExecutor struct {
	// contains filtered or unexported fields
}

DuckDBClientQueryExecutor adapts a live *DuckDBClient to the DuckDBQueryExecutor seam.

func (*DuckDBClientQueryExecutor) Query

func (e *DuckDBClientQueryExecutor) Query(ctx context.Context, sql string, args ...any) (duckDBRowsIterator, error)

Query executes sql against the wrapped DuckDB client.

type DuckDBQueryExecutor

type DuckDBQueryExecutor interface {
	Query(ctx context.Context, sql string, args ...any) (duckDBRowsIterator, error)
}

DuckDBQueryExecutor executes SQL against DuckDB. A nil executor means DuckDB is unavailable; the engine then degrades per model.FederatedQueryOptions.

func NewDuckDBClientQueryExecutor

func NewDuckDBClientQueryExecutor(client *DuckDBClient) DuckDBQueryExecutor

NewDuckDBClientQueryExecutor wraps client as a DuckDBQueryExecutor. It returns a nil interface when client (or its DB) is nil so that the engine's duck==nil unavailability guard fires early — before dirty-set fetching and without recording circuit-breaker failures — matching the pre-extraction repository semantics.

type EngineOption

type EngineOption func(*DBFederatedQueryEngine)

EngineOption customizes optional engine collaborators.

func WithCorruptPathRetention

func WithCorruptPathRetention(d time.Duration) EngineOption

WithCorruptPathRetention overrides how long a verification-confirmed corrupt parquet object stays excluded from path resolution (#251). The entry always expires — a terminal verdict must never be memoized forever (#326): repair, compaction, or manifest reconcile self-heal only through re-verification. A non-positive d effectively disables exclusion — entries expire the moment they are added — so misconfigured callers fail open to today's all-or-nothing scan; production callers should keep the default.

func WithFlushVisibilityGrace

func WithFlushVisibilityGrace(d time.Duration) EngineOption

WithFlushVisibilityGrace overrides the #252 clock-skew margin subtracted from the query's path-resolution timestamp when computing the dirty-barrier cutoff. d == 0 is the exact anchor (the default); d > 0 hardens against cross-host clock skew (flushed_at is stamped on the CDC host, the cutoff on the query host) at the cost of hot-serving rows flushed up to d before the query; d < 0 disables the widening entirely (the pre-#252 barrier).

func WithLogger

func WithLogger(l *zap.Logger) EngineOption

WithLogger gives the engine a logger; the default is zap.NewNop(). The engine reports itself through returned errors and the execution plan, so this stays narrow — two outlets whose observations have nowhere else to go: the pre-read validator's stamp-versus-footer cross-check (#256), invisible because the read it observes SUCCEEDS (a manifest entry whose column stamp contradicts the object's real footer is an operator's problem no caller's result would ever mention), and the scan-guard violation identification (#351), invisible under AllowPartialDegradedMode because the degraded fallback absorbs the error and toExecutionPlan drops plan Notes.

func WithParquetSource

func WithParquetSource(src ParquetSource) EngineOption

WithParquetSource injects the manifest-driven parquet path resolver. A nil source keeps the legacy behavior: paths come only from the query's render hints (caller-supplied glob or explicit list).

func WithPlanCache

func WithPlanCache(c *queryplan.Cache) EngineOption

WithPlanCache injects a shared compiled-plan cache (#142).

type ManifestSchemaMismatchError

type ManifestSchemaMismatchError = forma.ManifestSchemaMismatchError

type NoParquetPathsError

type NoParquetPathsError = forma.NoParquetPathsError

type ParquetGuardViolationError

type ParquetGuardViolationError struct {
	// SchemaID is the schema the failed federated read was addressed to.
	SchemaID int16
	// Paths are the full storage URIs of the objects whose guarded
	// single-file drain failed deterministically while their bare drain read
	// clean. Operator detail; safe internally because httpapi redacts
	// non-published error text and toExecutionPlan drops Notes (#301/#306),
	// the same boundary contract corruptParquetRetryError relies on.
	Paths []string
	// contains filtered or unexported fields
}

ParquetGuardViolationError decorates a read failure that neither the missing-object classification (#187) nor the corruption confirmation (#251) claimed, with the objects identified by the guarded per-file drain (#351). Deliberate wording: the paths FAIL the guarded single-file scan — an invariant statement, not a causation claim, because a single-file scan is strictly stricter than the set scan (a file missing only deleted_at fails alone but is tolerated in a set where a sibling carries the column). Unwrap keeps the original classification chain (ErrFederatedReadFailed): identification must not change degradability, retry, or breaker behavior.

func (*ParquetGuardViolationError) Error

func (*ParquetGuardViolationError) Unwrap

func (e *ParquetGuardViolationError) Unwrap() error

type ParquetSetInconsistentError

type ParquetSetInconsistentError = forma.ParquetSetInconsistentError

type ParquetSource

type ParquetSource interface {
	// Paths returns the schema's parquet objects as full s3:// URIs (or a
	// fallback glob for schemas with no manifest yet). Returning empty fails
	// the read with ErrNoParquetPaths (#299): there is nothing to scan, and
	// every query reaching the DuckDB engine wants warm and/or cold data, so
	// an empty set cannot be answered honestly. A schema with no data yet
	// should yield its fallback glob rather than nothing.
	//
	// stamps carries each stamped entry's write-time footer columns keyed by
	// its returned path; nil/absent keys mean unstamped — the validator falls
	// back to probing (#256).
	Paths(ctx context.Context, schemaID int16) (paths []string, stamps map[string]map[string]string, err error)
	// MissingIn probes the given scanned path set (full s3:// URIs; glob
	// and foreign-bucket entries are skipped as unprovable) and returns the
	// bucket-relative keys absent from storage. It is consulted only on the
	// read-error path — zero happy-path probes — and only over the exact
	// set the failed scan used: re-resolving the manifest here would
	// classify against a newer snapshot than the one that failed, so a
	// concurrent flush/compaction could hide the lost key or surface an
	// unrelated one (#249 review).
	MissingIn(ctx context.Context, scanned []string) ([]string, error)
}

ParquetSource resolves the authoritative parquet object set of a schema — typically from the CDC manifest — so federated reads scan exactly the listed objects instead of expanding a storage glob. The distinction is what makes cold-tier loss detectable (#187 scenario 2): a glob silently shrinks to whatever objects survive, while a listed object missing from storage fails the scan and classifies via MissingIn.

type PostgresDirtyIDFetcher

type PostgresDirtyIDFetcher struct {
	// contains filtered or unexported fields
}

PostgresDirtyIDFetcher reads unflushed row IDs from the change_log table, satisfying the DirtyIDFetcher seam with a live Postgres pool.

func NewPostgresDirtyIDFetcher

func NewPostgresDirtyIDFetcher(pool dirtyIDPool) *PostgresDirtyIDFetcher

NewPostgresDirtyIDFetcher wraps pool as a DirtyIDFetcher.

func (*PostgresDirtyIDFetcher) FetchDirtyRowIDs

func (f *PostgresDirtyIDFetcher) FetchDirtyRowIDs(ctx context.Context, changeLogTable string, schemaID int16) ([]uuid.UUID, error)

FetchDirtyRowIDs returns the row IDs in changeLogTable for schemaID that have not been flushed to Parquet yet.

type PostgresFederatedSource

type PostgresFederatedSource interface {
	QueryPersistentRecords(ctx context.Context, query *model.PersistentRecordQuery) (*model.PersistentRecordPage, error)
	RunOptimizedQuery(ctx context.Context, tables model.StorageTables, schemaID int16, clause string, args []any, limit, offset int, attributeOrders []model.AttributeOrder, useMainTableAsAnchor bool) ([]*model.PersistentRecord, int64, error)
	BuildHybridConditions(tables model.StorageTables, fq *model.FederatedAttributeQuery) (string, []any, error)
}

PostgresFederatedSource is the Postgres-side seam the federated engine queries for hot-tier records. It is intentionally wider than one method: federated pagination needs the optimized clause/args path and hybrid condition building, which QueryPersistentRecords cannot substitute.

type ProbeToken

type ProbeToken uint64

ProbeToken identifies one half-open probe reservation. The zero token means "no probe held" — callers admitted while the breaker was closed carry it — and releasing it is always a no-op.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL