chmigrate

Documentation

Overview

Package chmigrate copies historical data from a single-node ClickHouse instance into a Distributed-CH cluster managed by Coremetry. Day-partitioned via ClickHouse's remote() function so each step is bounded, resumable, and observable.

Index

• type Migrator
  • func (m *Migrator) Run(ctx context.Context, plans []Plan) error
• type Plan
• type SourceConfig

Types

type Migrator

type Migrator struct {
	Conn   driver.Conn
	Source SourceConfig
	// LocalTable resolves a logical name to the actual local
	// flavour. In cluster mode this returns "<name>_local"; in
	// single-node mode it's identity. Wired by the caller so the
	// migrator stays agnostic of cluster mode internals.
	LocalTable func(name string) string
	// Progress is invoked after each day finishes (success or
	// skip). Non-nil-only; safe to leave nil for quiet runs.
	Progress func(day time.Time, plan Plan, copied uint64, skipped bool)
}

Migrator runs a slice of Plans serially, day by day, with progress logging and per-day idempotency. Re-running with the same range is safe: a day whose row count already matches the remote source is skipped.

func (*Migrator) Run

func (m *Migrator) Run(ctx context.Context, plans []Plan) error

Run executes every plan, day by day, in chronological order. Returns on first error so the caller can retry the same range after fixing the underlying issue (network, credentials, etc.).

type Plan

type Plan struct {
	Table string // "spans" | "logs" | "metric_points" | "profiles"
	// exemplars/span_links are EXCLUDED by decision (exemplar audit v0.8.431,
	// açık soru 4): their TTL rides retention.spans (days, not weeks) — by the
	// time a single-node→cluster migration runs, the copyable window is a
	// rounding error and the pivot degrades gracefully without it.
	TimeCol  string    // "time" (default) | "start_time" for profiles
	From, To time.Time // date range (inclusive on `From`, exclusive on `To`)
}

Plan is a single-table migration job. Produces a sequence of per-day INSERT SELECTs from remote → local. The table name is the un-suffixed logical name; Migrate() resolves the local shard name (`<name>_local` in cluster mode, `<name>` otherwise).

type SourceConfig

type SourceConfig struct {
	Addr     string // e.g. "old-ch:9000"
	Database string // e.g. "coremetry"
	Username string
	Password string
}

SourceConfig points the migration at a remote single-node CH. Username / Password feed straight into the remote() function; the migration runs entirely server-side on the cluster (no data flows through this process), so credentials only need network read access from the cluster's CH nodes.