Documentation
¶
Index ¶
- Constants
- func DropWindow(s Settings, effective func() Effective) (time.Duration, bool)
- func HiveArchivePath(archiveDir, bintrailID, partitionName string) (string, error)
- func StartLoop(ctx context.Context, settings func() Settings, targets func() []RotateTarget, ...) <-chan struct{}
- type Effective
- type Guard
- type Options
- type Result
- type RetainSource
- type RotateTarget
- type Settings
- type Window
Constants ¶
const LegacyRetain = "30d"
LegacyRetain is what an index keeps while its operator sets no retention and the index itself records none (#1709): the built-in default every index ran under before the record existed. It is a fact about the past, so it never follows indexer.DefaultRotateRetain — an upgrade must not silently shorten the window an operator has been relying on.
Variables ¶
This section is empty.
Functions ¶
func DropWindow ¶ added in v0.99.0
DropWindow is the window the loop drops ONE index's partitions on under s, and whether that is known. It is for a caller that has to bound something against what rotation WILL drop rather than report it, so every doubt answers not known:
- rotation off: this loop drops nothing, and an operator who turned it off usually rotates on a schedule of their own that nothing here can read;
- an explicit window: every index drops on it;
- no explicit window: the index's own (effective is ResolveEffective for it, asked only in this case). A record that could not be read is not known, although the loop then keeps the legacy window: the loop's own read, a cycle later, may succeed and drop on a shorter one.
The loop can drop LESS than this says (the upgrade guard, a partition not archived yet), never more.
func HiveArchivePath ¶
HiveArchivePath returns the Hive-partitioned path for a binlog_events partition archive. The layout is:
<archiveDir>/bintrail_id=<uuid>/event_date=<YYYY-MM-DD>/event_hour=<HH>/events.parquet
Each hourly partition maps to exactly one file. The event_hour= directory level enables DuckDB Hive partition pruning on hour-scoped queries.
func StartLoop ¶
func StartLoop(ctx context.Context, settings func() Settings, targets func() []RotateTarget, onCycle ...func(failed bool, deferred int)) <-chan struct{}
StartLoop announces and launches the built-in rotation loop: one cycle immediately, then one every interval, each cycle rotating every target the provider returns (the boot index plus the control plane's per-source databases). Rotation is the secondary job — failures are logged, never fatal to the stream. Returns immediately; the loop stops when ctx is cancelled, and the returned channel closes when it has fully exited (used by tests for deterministic shutdown; production callers may ignore it).
settings is a PROVIDER, read fresh each cycle (mirroring the targets provider): the console can edit the global rotation policy at runtime, and an edit applies on the next tick — retain/add-future immediately, and a changed interval re-tunes the ticker. The enabled/disabled decision and the startup banner are taken once from the initial read: a daemon started with rotation off runs no loop (re-enabling needs a restart). onCycle callbacks (optional) observe each cycle's health — failed reports a rotation error, deferred counts the steps the cycle left undone: partitions it declined to drop, and an add of future partitions it skipped (Result.Deferred). They run inside the cycle's recover guard, so a panicking callback cannot take down the loop.
Types ¶
type Effective ¶ added in v0.85.0
type Effective struct {
Retain time.Duration
Raw string
Source RetainSource
// Err is set when Source is RetainUnreadable, for the log line; the window
// is usable either way.
Err error
// RecordedAt is when the record was written (the index's creation), set
// only for RetainRecorded. The upgrade guard exempts history that
// accumulated at or after it — see UpgradeGuard.
RecordedAt time.Time
}
Effective is the window one index rotates on while the operator sets none, and where it came from. It is what every SURFACE should report (doctor's projection, the console's capacity card and rotation panel, status), so a screen cannot name a window the loop does not use.
func ResolveEffective ¶ added in v0.85.0
ResolveEffective answers for one index database, exactly as the rotation loop does. It never fails: an index it cannot ask keeps the legacy window, which is the same direction the loop takes.
func (Effective) DescribeSource ¶ added in v0.85.0
DescribeSource is the one sentence a surface puts beside the window, so the three of them cannot drift into three wordings.
func (Effective) StatusInfo ¶ added in v0.85.0
func (e Effective) StatusInfo() *status.RetentionInfo
StatusInfo renders this answer for the status report, which says which window is in effect and where it came from. It lives here so the screen and the loop cannot drift: one decision, two surfaces (#1709). nil when there is no usable window to report.
type Guard ¶ added in v0.104.0
type Guard struct {
Blocked bool
// Oldest is the oldest named hourly partition, for the sentence that says
// how far back the history the loop will not touch reaches.
Oldest time.Time
// HistoryPredatesRecord is true when the index carries a record and yet
// holds history older than it: created empty and then filled (a restore,
// `bintrail index` over old binlog files). False for an index with no
// record, the pre-record deployment the guard was written for.
HistoryPredatesRecord bool
}
Guard is the upgrade guard's answer for one index under an implicit window: whether the loop is refusing to drop (#2323). Blocked means every cycle tops up future partitions and drops NOTHING until the operator sets --rotate-retain, so a surface that reports the window must not present it as the size the index settles at — the index is growing unbounded.
func UpgradeGuard ¶ added in v0.104.0
UpgradeGuard evaluates the guard for one index exactly as the loop does on its next cycle, for a surface that reports e (ResolveEffective's answer). It is only meaningful under an implicit window: with Settings.Explicit the loop never consults the guard, and a caller that knows the window was set explicitly must not call this. The error is the partition listing failing; the loop then skips drops for that one cycle and retries, so a surface treats it as "not known to be blocked", never as blocked.
type Options ¶
type Options struct {
// RetainDur drops partitions older than this. Zero disables drops (the
// cycle only tops up future partitions).
RetainDur time.Duration
// RetainRaw is the operator-facing retain string (e.g. "7d"), used only in
// the "no partitions older than %s" message.
RetainRaw string
// AddFuture is the declarative future-headroom target.
AddFuture int
// NoReplace suppresses one-for-one replacement of dropped partitions.
NoReplace bool
// ArchiveDir, when set, archives each partition to Parquet before dropping.
ArchiveDir string
ArchiveCompression string
BintrailID string
// ArchiveS3, when set (requires ArchiveDir), uploads archives to S3.
ArchiveS3 string
ArchiveS3Region string
// Retry skips partitions whose Parquet already exists and S3 uploads that
// already succeeded.
Retry bool
// Format "json" suppresses the per-partition stdout chatter (callers that
// emit their own JSON, or the built-in loop, pass "json").
Format string
// ProtectUnarchived: when the index has any archiving history, the
// no-archive drop path only drops already-archived partitions — the
// built-in rotation must not be the first to destroy data an archiving flow
// would preserve. The explicit rotate command leaves this false.
ProtectUnarchived bool
// QuietEmpty marks a cycle over an index that holds no events and has no
// writer (a source-less daemon's boot index, #1715): its drops and
// top-ups are housekeeping over empty partitions, so the per-partition
// lines go to Debug and the completion line says index_empty=true. The
// work itself is unchanged: the layout must stay current for the day a
// stream starts writing it, and it must not grow toward MySQL's
// partition cap.
QuietEmpty bool
// PruneLocalAfterUpload removes the local staging Parquet once it has been
// uploaded to S3 and the partition dropped. The unattended built-in loop
// sets this so a container's staging dir doesn't grow without bound; the
// read side falls back to S3 when the local copy is gone. Requires
// ArchiveS3 (a local-only archive IS the durable copy — never pruned). The
// explicit rotate command leaves this false (operator keeps both copies).
PruneLocalAfterUpload bool
}
Options configures one rotation cycle. The fields replace the package-level rot* flag globals the engine used to read directly, so the explicit `rotate` command and `up`'s built-in rotation loop can both drive Perform with their own settings.
type Result ¶
type Result struct {
Dropped, Added, Deferred int
// AddSkipped is how many future partitions the cycle meant to add and
// did not, because binlog_events was in use (#2280); AddFutureTarget is
// the --add-future value that adds them in a later run. Both are zero
// when nothing was skipped.
AddSkipped, AddFutureTarget int
}
Result is one rotation cycle's outcome. Deferred counts partitions past retention that this cycle did NOT drop to avoid data loss: the ProtectUnarchived guard refusing an unarchived partition, an S3 upload that failed in this run, an upload still pending (in either branch, whichever source it belongs to), a partition that changed after it was archived, or one whose drop found binlog_events in use (#2280). An add of future partitions skipped for that reason counts as one more, so Deferred is the steps a cycle left undone, not strictly a number of partitions. The built-in loop sums it across targets to drive escalation. The explicit `rotate` command surfaces only Dropped/Added, but can still produce Deferred>0 for the last three. A named struct, not a positional tuple: three same-typed ints invite silent misordering at call sites.
type RetainSource ¶ added in v0.85.0
type RetainSource string
RetainSource says where an implicit window came from, for a surface that reports it to an operator.
const ( // RetainRecorded: the index carries the retention it was created under. RetainRecorded RetainSource = "recorded" // RetainLegacy: the index carries no record, so it keeps the window every // such index ran under before the record existed. RetainLegacy RetainSource = "legacy" // RetainUnreadable: the record could not be read, so the legacy window is // used — same window as RetainLegacy, different reason, and a surface must // not present a failed read as a fact about the index. RetainUnreadable RetainSource = "unreadable" // RetainConfigured: the operator set the window themselves // (--rotate-retain or BINTRAIL_ROTATE_RETAIN; Settings.Explicit), so // every index rotates on it and the record is not consulted. Never // answered by ResolveEffective, which exists for the implicit case: a // surface reports it when the loop's settings say Explicit (#2315), so the // screen cannot project over a record the loop is not using. RetainConfigured RetainSource = "configured" )
type RotateTarget ¶ added in v0.13.0
type RotateTarget struct {
DSN string
// NoWriter: as far as this daemon knows, nothing streams into this
// index (the boot index of a source-less `watch`, #1715). The daemon
// cannot verify that — another process may point at the same DSN — which
// is why the mark alone decides nothing: a per-cycle probe of the events
// table does. While it holds no events its rotation is
// housekeeping over empty partitions — still done, so the layout stays
// current for the day a stream starts writing it and never grows toward
// the partition cap — but logged as such (Options.QuietEmpty) instead of
// as freed space. A per-source index always has a writer and never sets
// this. Once such an index holds events (a source-ful era left them) it
// is rotated and logged like any other.
NoWriter bool
ArchiveDir string
ArchiveS3 string
ArchiveS3Region string
BintrailID string
ArchiveCompression string
}
RotateTarget is one index database the loop rotates this cycle, plus its optional per-source archive config. ArchiveS3 == "" means drop-only (the historical behavior). When ArchiveS3 is set, the cycle archives each expired partition to ArchiveDir (a local staging dir) under bintrail_id=BintrailID, uploads it to ArchiveS3, then drops it — and prunes the local copy. The provider (cmd/bintrail-console) is responsible for only setting the archive fields once BintrailID is known (resolved from the source's stream_state).
type Settings ¶
type Settings struct {
Enabled bool
Retain time.Duration
RetainRaw string
Interval time.Duration
AddFuture int
// Explicit records whether the operator configured --rotate-retain
// themselves (flag or env — bindCommandEnv marks env-set flags Changed).
// When false (running on the built-in default), the upgrade guard
// refuses to drop pre-existing deep history: an operator who never chose
// a retention must not lose months of forensic record to a binary
// upgrade.
Explicit bool
}
Settings is the parsed configuration of `up`'s built-in rotation — the loop that keeps an unattended index from growing until the volume fills and takes the forensic record with it (#420).
func ParseSettings ¶
ParseSettings validates the --rotate-* flag values and is the sole author of every Settings field. retain accepts the rotate command's Nd/Nh forms, plus "off", "0", or "" to disable the built-in rotation entirely. explicit is whether the operator set --rotate-retain themselves (cmd.Flags().Changed — true for flag and env alike).
type Window ¶ added in v0.104.0
type Window struct {
// Retain is the window; zero means rotation is off (the doctor grades
// that as unbounded growth).
Retain time.Duration
// Note is the phrase for an implicit window's source (DescribeSource);
// empty for an explicit one, which the settings alone cannot attribute to
// a flag or to a policy saved in the web interface.
Note string
// Guard is the upgrade guard's verdict; never Blocked under an explicit
// window or with rotation off.
Guard Guard
}
Window is what the loop does with ONE index under settings s, for a doctor run that grades that index: the window it drops on, where an implicit one came from, and whether the upgrade guard is refusing to drop on it (#2328).
func ResolveWindow ¶ added in v0.104.0
ResolveWindow mirrors rotateOneIndex branch for branch: rotation off drops nothing; an explicit window (the daemon's flag or env, or a policy saved in the web interface, which the settings provider parses as explicit) is what every index drops on; otherwise the index's own record, behind the upgrade guard. A guard that cannot be evaluated is "not known to be blocked".
func ResolveWindowDSN ¶ added in v0.104.0
ResolveWindowDSN is ResolveWindow for a caller holding only the index DSN (the daemons' preflights and the web interface's Checks report). It is best effort by design: these runs sit on the boot path, where the index database may not exist yet, so an index that cannot be opened answers the settings' own window with no note and no guard — exactly what those runs assumed before they asked the index — and never an error.
Source Files