migration

Documentation

Overview

Package migration copies raw time series data out of one SquirrelDB backend (the source, Config.Backend) into another (the target, Config.ExtraWriteBackend) — the Cassandra→ClickHouse migration. It is a plain export→import loop over the source's export.RawExporter, checkpointing its resumable position into the target's state store so it can resume after an interruption. Live writes are meant to be dual-written concurrently (the daemon in dual-write mode); this tool only backfills history.

Index

• func Run(ctx context.Context, shutdownCtx context.Context, sq *core.SquirrelDB, ...) error
• func RunMigrate(args []string) int
• type Options
• type StateStoreBackend

Functions

func Run

func Run(
	ctx context.Context,
	shutdownCtx context.Context,
	sq *core.SquirrelDB,
	opts Options,
	logger zerolog.Logger,
) error

Run performs the migration described by opts, reading from sq.Backend and writing to sq.MigrateToBackend. It returns nil on completion or graceful interruption (ctx cancelled), with the checkpoint persisted.

func RunMigrate

func RunMigrate(args []string) int

RunMigrate implements the `squirreldb migrate` subcommand: it backfills raw data from the source backend (config `backend`) into the target (`extra_write_backend`). It returns a process exit code.

Types

type Options

type Options struct {
	// RepairFrom and RepairTo bound the migrated point timestamps (inclusive) for a
	// REPAIR run — e.g. re-copying a range that a best-effort dual-write dropped.
	// Setting either makes this a repair run: it starts fresh over that window and
	// does not read or write the full-migration checkpoint or the dual-write
	// boundary. Both zero = a normal (resumable) backfill.
	RepairFrom time.Time
	RepairTo   time.Time
	// DualWriteStart is the backfill's upper bound: only points strictly before it
	// are migrated, because the daemon's dual-write owns everything at or after it
	// (set it to when the last mono-write instance became dual-write). On a fresh
	// migration, zero means "now()"; the chosen value is persisted and reused on
	// every resume — including of the current, still-growing shard — so the whole
	// backfill uses one consistent boundary. Ignored on a resume.
	DualWriteStart time.Time
	// ImportMutableLabels, when set, copies the mutable labels (and the telemetry
	// cluster ID) from the source into the target at the end of a graceful run,
	// overwriting the target's — see importMutableLabels.
	ImportMutableLabels bool
	// BatchPoints is the number of points accumulated (across chunks) before one
	// store insert. Bigger = fewer, larger inserts (ClickHouse makes one part per
	// insert and dislikes many small ones) at the cost of more memory in this
	// process. It is a point count (not a chunk count) so memory stays bounded
	// regardless of how large individual series are.
	BatchPoints int
	// ThrottleWrites caps the rate of store inserts per second (0 = unlimited).
	// The relevant limit for the ClickHouse target is inserts/s (it must keep up
	// merging parts), not points/s — so the operator throttles writes and sizes
	// throughput with BatchPoints.
	ThrottleWrites float64
}

Options configures a migration run.

type StateStoreBackend

type StateStoreBackend interface {
	StateStore(ctx context.Context) (types.State, error)
}