Documentation
¶
Index ¶
- Constants
- Variables
- func DuckDBPostgresConnStringFromPool(pool *pgxpool.Pool) string
- func EvaluateRoutingPolicy(cfg forma.DuckDBConfig, fq *model.FederatedAttributeQuery, ...) model.RoutingDecision
- func MergePersistentRecordsByTier(inputs map[model.DataTier][]*model.PersistentRecord, preferHot bool) ([]*model.PersistentRecord, error)
- func ValidateDuckDBConfig(cfg forma.DuckDBConfig) error
- type CircuitBreaker
- type DBFederatedQueryEngine
- func (e *DBFederatedQueryEngine) ExecuteDuckDBFederatedQuery(ctx context.Context, tables model.StorageTables, ...) ([]*model.PersistentRecord, int64, error)
- func (e *DBFederatedQueryEngine) ExecuteFederatedPaginatedQuery(ctx context.Context, tables model.StorageTables, ...) ([]*model.PersistentRecord, int64, error)
- func (e *DBFederatedQueryEngine) Query(ctx context.Context, tables model.StorageTables, ...) (*model.PersistentRecordPage, error)
- func (e *DBFederatedQueryEngine) StreamDuckDBFederatedQuery(ctx context.Context, tables model.StorageTables, ...) (int64, error)
- type DirtyIDFetcher
- type DirtyIDPool
- type DuckDBClient
- type DuckDBClientQueryExecutor
- type DuckDBQueryExecutor
- type EngineOption
- type ManifestSchemaMismatchError
- type NoParquetPathsError
- type ParquetGuardViolationError
- type ParquetSetInconsistentError
- type ParquetSource
- type PostgresDirtyIDFetcher
- type PostgresFederatedSource
- type ProbeToken
Constants ¶
const NotePartialParquetExclusion = "partial parquet scan: excluded corrupt objects"
NotePartialParquetExclusion prefixes the execution-plan note recording a partial parquet scan (#251): verification-confirmed corrupt objects were excluded and the query answered from the readable remainder. Notes never cross the HTTP boundary (toExecutionPlan drops them, #301/#306); embedders and the e2e harness assert on this exported prefix.
Variables ¶
var ( ErrNoParquetPaths = forma.ErrNoParquetPaths ErrManifestSchemaMismatch = forma.ErrManifestSchemaMismatch // ErrParquetSetInconsistent joined them for #301: internal/httpapi // classifies it to redact the object keys it carries out of public response // bodies, and cannot import this package without pulling DuckDB CGO into a // pure-Go test build. Redaction is gated on sentinel evidence rather than on // the status, so this holds on any status the error is classified as, not // only 5xx. ErrParquetSetInconsistent = forma.ErrParquetSetInconsistent )
ErrNoParquetPaths and ErrManifestSchemaMismatch are defined in the public root package and re-exported here for internal call sites. They must be matchable by embedders that reach the engine through factory.NewEntityManager* and cannot import an internal package — the whole point of #299 was a discriminator callers can act on, which a package-private sentinel is not. These are aliases, not copies: errors.Is/errors.As behave identically whether a caller reaches for the forma.* or federated.* name.
ErrDuckDBUnavailable marks queries rejected before reaching DuckDB: the client is not configured (or closed), or the circuit breaker is open. Transient infrastructure — degradable under AllowPartialDegradedMode.
var ErrFederatedReadFailed = errors.New("federated read failed")
ErrFederatedReadFailed marks a DuckDB federated read that failed at execution or while streaming rows: unreadable parquet (corrupt bytes, truncation, schema mismatch), storage-layer rejections (credentials), or a mid-stream fault. The referenced objects exist — a manifest-listed object missing from storage classifies as ErrParquetSetInconsistent instead. Transient/object-scoped — degradable under AllowPartialDegradedMode.
var ErrKeysetUnsupportedOnPostgres = errors.New("keyset cursor unsupported on the postgres-only path")
ErrKeysetUnsupportedOnPostgres marks a request that carries a keyset cursor and routed to the Postgres-only path. That path builds a model.PersistentRecordQuery, which has no cursor field, so it can neither apply nor reject the cursor itself: pre-#354 it silently answered an unfiltered first page and pagination never advanced. Not degradable — the degraded fallback IS the Postgres-only path.
var ErrPostgresReadFailed = errors.New("postgres read failed")
ErrPostgresReadFailed marks the Postgres side of a federated read failing: the dirty-ID consistency fetch or a Postgres-source page read. A Postgres outage is not degradable in practice — the degraded fallback is itself Postgres-only — so this classification is what #187 scenario 9 asserts on all its probes; degradability is still decided at the engine seam, not here.
var ErrSchemaMetadataCacheRequired = errors.New("schema metadata cache required but not loaded")
ErrSchemaMetadataCacheRequired marks a federated query that cannot build a correct entity_main projection because the schema's metadata cache is not loaded. It is a configuration / data-contract error, not a transient infrastructure failure, so the public Query path must not absorb it under AllowPartialDegradedMode — degrading would silently return a Postgres-only partial result and hide the missing-cache problem #151 makes loud.
Functions ¶
func DuckDBPostgresConnStringFromPool ¶
DuckDBPostgresConnStringFromPool derives the libpq-style connection string DuckDB's postgres_scanner needs from an existing pgx pool. It returns "" for a nil pool or pool config.
Every string value is quoted via pgdsn.Quote, the same rule internal/cdc's BuildPGDSN has used since #290. The unquoted form this replaced was two bugs:
- A password (or host, user, dbname) containing a space produced a DSN libpq could not parse, so postgres_scan failed to attach at all.
- The credential scrubber cannot tell where an unquoted value ends. It terminates on whitespace, so an attach failure — whose prose quotes this whole string back — logged the tail of the password past the placeholder, which is the exposure #301 exists to close.
All four string fields are quoted, not just the password: libpq accepts quoted values for every keyword, a host may be a socket path, and user and database names may legally contain spaces. Quoting uniformly means no field can be the one that breaks parsing. sslmode is deliberately not emitted here — it is absent from this DSN today, and emitting it with an empty quoted value would be rejected by libpq.
The result is embedded in a single-quoted DuckDB SQL literal (postgres_scan('{{.PG_CONN}}', …)), so the renderer escapes it with sqlutil.EscapeLiteral (internal/sqlutil/literal.go), called from the DuckDB renderer.
func EvaluateRoutingPolicy ¶
func EvaluateRoutingPolicy(cfg forma.DuckDBConfig, fq *model.FederatedAttributeQuery, opts *model.FederatedQueryOptions) model.RoutingDecision
EvaluateRoutingPolicy makes a routing decision based on config, query hints and options.
func MergePersistentRecordsByTier ¶
func MergePersistentRecordsByTier(inputs map[model.DataTier][]*model.PersistentRecord, preferHot bool) ([]*model.PersistentRecord, error)
MergePersistentRecordsByTier performs a merge-on-read across multiple data tiers. Inputs are provided as a map from model.DataTier -> slice of *model.PersistentRecord. Last-write-wins semantics are applied using model.PersistentRecord.UpdatedAt and ChangeLog flushed state.
Behavior:
- Records are deduplicated by (SchemaID, RowID).
- For each key, the record with the highest UpdatedAt is chosen. If UpdatedAt is equal, a tombstone (DeletedAt != nil && *DeletedAt != 0) beats a live copy; DeletedAt = 0 is the cold-tier live encoding (#274), never a tombstone. Remaining ties use deterministic tier priority (Hot > Warm > Cold).
- If a record originates from the ChangeLog buffer (flushed_at == 0) it is considered the authoritative hot source and wins ties regardless of UpdatedAt.
- The chosen record is returned with OtherAttributes merged across all source tiers for that (SchemaID, RowID) with attribute-level deduplication.
- Attributes are deduplicated by (AttrID, ArrayIndices).
- For an attribute present in multiple source records, the attribute from the record with the latest UpdatedAt is chosen. Ties are resolved using deterministic tier ordering (Hot > Warm > Cold).
- Deleted records (DeletedAt != nil && DeletedAt != 0) are excluded from results.
- Result slice is sorted by SchemaID then RowID for deterministic output.
The preferHot parameter is deprecated and ignored; tier priority is always deterministic.
func ValidateDuckDBConfig ¶
func ValidateDuckDBConfig(cfg forma.DuckDBConfig) error
ValidateDuckDBConfig performs basic sanity checks on user-provided DuckDB configuration.
Types ¶
type CircuitBreaker ¶
type CircuitBreaker struct {
// contains filtered or unexported fields
}
CircuitBreaker is a lightweight in-memory circuit breaker with strict single-probe half-open recovery (#246, supersedes the #185 immediate-forgiveness design).
States (all transitions guarded by mu):
- closed: no open period recorded; every caller is admitted.
- open (now < openUntil): every caller is rejected.
- half-open (openUntil elapsed): Allow admits exactly one caller as the probe and rejects the rest until the probe resolves. RecordSuccess closes the breaker (failure history cleared); RecordFailure re-opens it for a fresh openDuration without threshold re-accumulation.
Probe abandonment: a probe that never reports (its query was cancelled between Allow and Record*) lapses openDuration after admission, and the next Allow reclaims the slot. A lost probe therefore costs at most one extra openDuration of rejections. The flip side: a probe still legitimately running past openDuration is indistinguishable from an abandoned one, so a slow dependency can see more than one concurrent probe — "exactly one" holds only for probes that resolve within openDuration.
Stale callers: RecordSuccess closes the breaker from any state — a query admitted before the breaker opened that completes afterwards is real evidence the dependency is healthy. A stale RecordFailure landing while a probe is in flight re-opens the breaker: conservative, and indistinguishable from a probe failure without per-caller tokens. ReleaseProbe, by contrast, IS token-scoped (#349 review R2-2): it can run long after admission (the #251 post-verification path), by which time the probe slot may belong to a newer caller — an unscoped release would free a reservation its caller never held and admit a second concurrent probe.
func NewCircuitBreaker ¶
func NewCircuitBreaker(threshold int, window, openDuration time.Duration) *CircuitBreaker
NewCircuitBreaker creates a configured circuit breaker.
func (*CircuitBreaker) Allow ¶
func (cb *CircuitBreaker) Allow() (admitted bool, probe ProbeToken)
Allow reports whether the caller may proceed. In half-open state it atomically reserves the single probe slot: the one caller that receives admitted=true with a non-zero token while others are rejected MUST resolve the probe via RecordSuccess or RecordFailure (or let the reservation lapse after openDuration). Callers admitted while the breaker is closed receive the zero token: they hold no reservation, and their ReleaseProbe is a no-op.
func (*CircuitBreaker) IsOpen ¶
func (cb *CircuitBreaker) IsOpen() bool
IsOpen reports whether the timed open period is currently active. Observation only (tests, telemetry): admission control — including the half-open probe reservation — lives in Allow.
func (*CircuitBreaker) RecordFailure ¶
func (cb *CircuitBreaker) RecordFailure()
RecordFailure records a failure occurrence. A probe failure re-opens the breaker directly; otherwise failures accumulate in the sliding window and open the breaker at threshold.
func (*CircuitBreaker) RecordSuccess ¶
func (cb *CircuitBreaker) RecordSuccess()
RecordSuccess closes the breaker and clears the failure history. It resolves an in-flight probe, and also closes from open state when a pre-open in-flight query completes (see the type doc on stale callers).
func (*CircuitBreaker) ReleaseProbe ¶
func (cb *CircuitBreaker) ReleaseProbe(probe ProbeToken)
ReleaseProbe relinquishes a half-open probe reservation without recording evidence either way, freeing the slot for the next caller. It is for callers admitted by Allow that then failed BEFORE touching the dependency — a misconfiguration caught during path resolution, invalid caller input — so they learned nothing about its health and must not consume the probe. It is also for the one post-execution caller whose failure indicts a specific object rather than the dependency: the #251 corrupt-confirmed path, where per-file verification has just drained every OTHER object through this same engine and store — live proof of health that makes RecordFailure dishonest, while the query as a whole still failed, making RecordSuccess equally so.
Without this, such a caller abandons the reservation: the slot stays occupied until it lapses (openDuration), and every request in that window is rejected with ErrDuckDBUnavailable. That matters because ErrDuckDBUnavailable IS degradable while the pre-execution errors are deliberately not, so a misconfiguration that must always be loud would be answered from Postgres alone on the very next request (#299 review P1).
Neutral by design: the breaker stays open, so a real outage still gates traffic; only the probe slot returns. Recording success here would close the breaker on no evidence, and recording failure would extend the outage for a dependency that was never consulted.
Release is scoped to the caller's own reservation (#349 review R2-2): only the token Allow handed out frees the slot, so a caller admitted while the breaker was closed (zero token) — or one whose lapsed reservation was already reclaimed by a newer probe — cannot clear the probe another caller is running.
type DBFederatedQueryEngine ¶
type DBFederatedQueryEngine struct {
// contains filtered or unexported fields
}
DBFederatedQueryEngine executes federated queries across the Postgres hot tier and the DuckDB/Parquet warm+cold tiers: routing policy, dirty-set exclusion, DuckDB execution, merge, and pagination. It is the sole implementation of the FederatedQueryEngine interface.
func NewDBFederatedQueryEngine ¶
func NewDBFederatedQueryEngine(pgSource PostgresFederatedSource, dirtyIDFetcher DirtyIDFetcher, duck DuckDBQueryExecutor, breaker *CircuitBreaker, cfg forma.DuckDBConfig, metadataCache *schemameta.MetadataCache, pgConnString string, opts ...EngineOption) *DBFederatedQueryEngine
NewDBFederatedQueryEngine assembles the engine from its injected seams. duck and breaker may be nil: a nil duck marks DuckDB as unavailable and a nil breaker disables circuit breaking.
func (*DBFederatedQueryEngine) ExecuteDuckDBFederatedQuery ¶
func (e *DBFederatedQueryEngine) ExecuteDuckDBFederatedQuery( ctx context.Context, tables model.StorageTables, q *model.FederatedAttributeQuery, limit, offset int, attributeOrders []model.AttributeOrder, opts *model.FederatedQueryOptions, ) ([]*model.PersistentRecord, int64, error)
ExecuteDuckDBFederatedQuery runs the DuckDB optimized query template using the provided model.FederatedAttributeQuery. It fetches dirty IDs from the Postgres change_log (if available), injects exclusions into the DuckDB WHERE clause, executes the query against the global DuckDB client, and returns matched PersistentRecords along with the total record count.
Note: This implementation performs a best-effort scan of columns produced by the optimized query template. It mirrors the column ordering used by the Postgres template:
- main table projection (entity_main columns, order defined by model.EntityMainColumnDescriptors)
- attributes_json (TEXT)
- total_records (bigint)
- total_pages (bigint)
- current_page (int)
A read failure attributed to specific corrupt parquet objects is retried exactly once against the readable remainder (#251), so one call can issue two scans plus a per-object verification drain — worth knowing when sizing the caller's deadline. The execution plan describes only the pass that produced the returned page.
func (*DBFederatedQueryEngine) ExecuteFederatedPaginatedQuery ¶
func (e *DBFederatedQueryEngine) ExecuteFederatedPaginatedQuery( ctx context.Context, tables model.StorageTables, fq *model.FederatedAttributeQuery, limit, offset int, attributeOrders []model.AttributeOrder, opts *model.FederatedQueryOptions, ) ([]*model.PersistentRecord, int64, error)
ExecuteFederatedPaginatedQuery performs a federated fetch across Postgres (hot) and DuckDB (cold/warm), merges results with last-write-wins semantics, and returns the requested page plus an accurate total deduplicated across sources.
Notes: - This is an MVP coordinator: it caps per-source fetches (opts.MaxRows or default) to avoid OOM. - For very large result sets a keys-only two-phase approach should be implemented later.
func (*DBFederatedQueryEngine) Query ¶
func (e *DBFederatedQueryEngine) Query(ctx context.Context, tables model.StorageTables, fq *model.FederatedAttributeQuery, opts *model.FederatedQueryOptions) (*model.PersistentRecordPage, error)
Query implements FederatedQueryEngine. Hot-only requests delegate directly to Postgres; otherwise the routing policy decides between Postgres and the DuckDB federated path, falling back to Postgres on DuckDB failure when opts.AllowPartialDegradedMode is set.
func (*DBFederatedQueryEngine) StreamDuckDBFederatedQuery ¶
func (e *DBFederatedQueryEngine) StreamDuckDBFederatedQuery( ctx context.Context, tables model.StorageTables, q *model.FederatedAttributeQuery, limit, offset int, attributeOrders []model.AttributeOrder, opts *model.FederatedQueryOptions, rowHandler func(context.Context, *model.PersistentRecord) error, ) (int64, error)
StreamDuckDBFederatedQuery streams DuckDB federated query results using a rowHandler callback. It reuses the same rowHandler semantics as Postgres' StreamOptimizedQuery to avoid loading the entire result set into memory.
type DirtyIDFetcher ¶
type DirtyIDFetcher interface {
FetchDirtyRowIDs(ctx context.Context, changeLogTable string, schemaID int16) ([]uuid.UUID, error)
}
DirtyIDFetcher retrieves row IDs from the change log that are newer than the flushed Parquet tiers and must be excluded from DuckDB results.
type DirtyIDPool ¶
type DirtyIDPool = dirtyIDPool
type DuckDBClient ¶
DuckDBClient wraps a database/sql DB opened with the DuckDB driver.
func NewDuckDBClient ¶
func NewDuckDBClient(cfg forma.DuckDBConfig) (*DuckDBClient, error)
NewDuckDBClient creates and configures a DuckDB client according to the provided config. It attempts to load common extensions (httpfs/parquet) and configure S3 access via PRAGMA when requested.
func NewDuckDBClientContext ¶
func NewDuckDBClientContext(ctx context.Context, cfg forma.DuckDBConfig) (*DuckDBClient, error)
NewDuckDBClientContext creates and configures a DuckDB client while honoring the caller-provided context during bootstrap.
func (*DuckDBClient) Close ¶
func (c *DuckDBClient) Close() error
Close closes the underlying DuckDB DB.
func (*DuckDBClient) HealthCheck ¶
func (c *DuckDBClient) HealthCheck(ctx context.Context) error
HealthCheck performs a simple query to validate the DuckDB connection and basic runtime pragmas.
type DuckDBClientQueryExecutor ¶
type DuckDBClientQueryExecutor struct {
// contains filtered or unexported fields
}
DuckDBClientQueryExecutor adapts a live *DuckDBClient to the DuckDBQueryExecutor seam.
type DuckDBQueryExecutor ¶
type DuckDBQueryExecutor interface {
Query(ctx context.Context, sql string, args ...any) (duckDBRowsIterator, error)
}
DuckDBQueryExecutor executes SQL against DuckDB. A nil executor means DuckDB is unavailable; the engine then degrades per model.FederatedQueryOptions.
func NewDuckDBClientQueryExecutor ¶
func NewDuckDBClientQueryExecutor(client *DuckDBClient) DuckDBQueryExecutor
NewDuckDBClientQueryExecutor wraps client as a DuckDBQueryExecutor. It returns a nil interface when client (or its DB) is nil so that the engine's duck==nil unavailability guard fires early — before dirty-set fetching and without recording circuit-breaker failures — matching the pre-extraction repository semantics.
type EngineOption ¶
type EngineOption func(*DBFederatedQueryEngine)
EngineOption customizes optional engine collaborators.
func WithCorruptPathRetention ¶
func WithCorruptPathRetention(d time.Duration) EngineOption
WithCorruptPathRetention overrides how long a verification-confirmed corrupt parquet object stays excluded from path resolution (#251). The entry always expires — a terminal verdict must never be memoized forever (#326): repair, compaction, or manifest reconcile self-heal only through re-verification. A non-positive d effectively disables exclusion — entries expire the moment they are added — so misconfigured callers fail open to today's all-or-nothing scan; production callers should keep the default.
func WithFlushVisibilityGrace ¶
func WithFlushVisibilityGrace(d time.Duration) EngineOption
WithFlushVisibilityGrace overrides the #252 clock-skew margin subtracted from the query's path-resolution timestamp when computing the dirty-barrier cutoff. d == 0 is the exact anchor (the default); d > 0 hardens against cross-host clock skew (flushed_at is stamped on the CDC host, the cutoff on the query host) at the cost of hot-serving rows flushed up to d before the query; d < 0 disables the widening entirely (the pre-#252 barrier).
func WithLogger ¶
func WithLogger(l *zap.Logger) EngineOption
WithLogger gives the engine a logger; the default is zap.NewNop(). The engine reports itself through returned errors and the execution plan, so this stays narrow — two outlets whose observations have nowhere else to go: the pre-read validator's stamp-versus-footer cross-check (#256), invisible because the read it observes SUCCEEDS (a manifest entry whose column stamp contradicts the object's real footer is an operator's problem no caller's result would ever mention), and the scan-guard violation identification (#351), invisible under AllowPartialDegradedMode because the degraded fallback absorbs the error and toExecutionPlan drops plan Notes.
func WithParquetSource ¶
func WithParquetSource(src ParquetSource) EngineOption
WithParquetSource injects the manifest-driven parquet path resolver. A nil source keeps the legacy behavior: paths come only from the query's render hints (caller-supplied glob or explicit list).
func WithPlanCache ¶
func WithPlanCache(c *queryplan.Cache) EngineOption
WithPlanCache injects a shared compiled-plan cache (#142).
type ManifestSchemaMismatchError ¶
type ManifestSchemaMismatchError = forma.ManifestSchemaMismatchError
type NoParquetPathsError ¶
type NoParquetPathsError = forma.NoParquetPathsError
type ParquetGuardViolationError ¶
type ParquetGuardViolationError struct {
// SchemaID is the schema the failed federated read was addressed to.
SchemaID int16
// Paths are the full storage URIs of the objects whose guarded
// single-file drain failed deterministically while their bare drain read
// clean. Operator detail; safe internally because httpapi redacts
// non-published error text and toExecutionPlan drops Notes (#301/#306),
// the same boundary contract corruptParquetRetryError relies on.
Paths []string
// contains filtered or unexported fields
}
ParquetGuardViolationError decorates a read failure that neither the missing-object classification (#187) nor the corruption confirmation (#251) claimed, with the objects identified by the guarded per-file drain (#351). Deliberate wording: the paths FAIL the guarded single-file scan — an invariant statement, not a causation claim, because a single-file scan is strictly stricter than the set scan (a file missing only deleted_at fails alone but is tolerated in a set where a sibling carries the column). Unwrap keeps the original classification chain (ErrFederatedReadFailed): identification must not change degradability, retry, or breaker behavior.
func (*ParquetGuardViolationError) Error ¶
func (e *ParquetGuardViolationError) Error() string
func (*ParquetGuardViolationError) Unwrap ¶
func (e *ParquetGuardViolationError) Unwrap() error
type ParquetSetInconsistentError ¶
type ParquetSetInconsistentError = forma.ParquetSetInconsistentError
type ParquetSource ¶
type ParquetSource interface {
// Paths returns the schema's parquet objects as full s3:// URIs (or a
// fallback glob for schemas with no manifest yet). Returning empty fails
// the read with ErrNoParquetPaths (#299): there is nothing to scan, and
// every query reaching the DuckDB engine wants warm and/or cold data, so
// an empty set cannot be answered honestly. A schema with no data yet
// should yield its fallback glob rather than nothing.
//
// stamps carries each stamped entry's write-time footer columns keyed by
// its returned path; nil/absent keys mean unstamped — the validator falls
// back to probing (#256).
Paths(ctx context.Context, schemaID int16) (paths []string, stamps map[string]map[string]string, err error)
// MissingIn probes the given scanned path set (full s3:// URIs; glob
// and foreign-bucket entries are skipped as unprovable) and returns the
// bucket-relative keys absent from storage. It is consulted only on the
// read-error path — zero happy-path probes — and only over the exact
// set the failed scan used: re-resolving the manifest here would
// classify against a newer snapshot than the one that failed, so a
// concurrent flush/compaction could hide the lost key or surface an
// unrelated one (#249 review).
MissingIn(ctx context.Context, scanned []string) ([]string, error)
}
ParquetSource resolves the authoritative parquet object set of a schema — typically from the CDC manifest — so federated reads scan exactly the listed objects instead of expanding a storage glob. The distinction is what makes cold-tier loss detectable (#187 scenario 2): a glob silently shrinks to whatever objects survive, while a listed object missing from storage fails the scan and classifies via MissingIn.
type PostgresDirtyIDFetcher ¶
type PostgresDirtyIDFetcher struct {
// contains filtered or unexported fields
}
PostgresDirtyIDFetcher reads unflushed row IDs from the change_log table, satisfying the DirtyIDFetcher seam with a live Postgres pool.
func NewPostgresDirtyIDFetcher ¶
func NewPostgresDirtyIDFetcher(pool dirtyIDPool) *PostgresDirtyIDFetcher
NewPostgresDirtyIDFetcher wraps pool as a DirtyIDFetcher.
func (*PostgresDirtyIDFetcher) FetchDirtyRowIDs ¶
func (f *PostgresDirtyIDFetcher) FetchDirtyRowIDs(ctx context.Context, changeLogTable string, schemaID int16) ([]uuid.UUID, error)
FetchDirtyRowIDs returns the row IDs in changeLogTable for schemaID that have not been flushed to Parquet yet.
type PostgresFederatedSource ¶
type PostgresFederatedSource interface {
QueryPersistentRecords(ctx context.Context, query *model.PersistentRecordQuery) (*model.PersistentRecordPage, error)
RunOptimizedQuery(ctx context.Context, tables model.StorageTables, schemaID int16, clause string, args []any, limit, offset int, attributeOrders []model.AttributeOrder, useMainTableAsAnchor bool) ([]*model.PersistentRecord, int64, error)
BuildHybridConditions(tables model.StorageTables, fq *model.FederatedAttributeQuery) (string, []any, error)
}
PostgresFederatedSource is the Postgres-side seam the federated engine queries for hot-tier records. It is intentionally wider than one method: federated pagination needs the optimized clause/args path and hybrid condition building, which QueryPersistentRecords cannot substitute.
type ProbeToken ¶
type ProbeToken uint64
ProbeToken identifies one half-open probe reservation. The zero token means "no probe held" — callers admitted while the breaker was closed carry it — and releasing it is always a no-op.
Source Files
¶
- circuit_breaker.go
- cold_columns.go
- connstring.go
- corrupt_paths.go
- duckdb_conn.go
- duckdb_query.go
- duckdb_query_build.go
- duckdb_query_execute.go
- duckdb_query_helpers.go
- duckdb_query_resolve.go
- engine.go
- engine_options.go
- errors.go
- keyset.go
- merge.go
- pagination.go
- parquet_guard_identify.go
- parquet_schema_validation.go
- parquet_source.go
- parquet_verify.go
- routing.go