Documentation
¶
Overview ¶
Package store owns the database layer: migrations, partition maintenance and the sqlc-generated queries.
Index ¶
- Constants
- Variables
- func DefaultPartitionCounts(ctx context.Context, pool *pgxpool.Pool) (map[string]int64, error)
- func Down(ctx context.Context, dsn string) error
- func DropExpiredPartitions(ctx context.Context, pool *pgxpool.Pool, policy RetentionPolicy, now time.Time) ([]string, error)
- func EnsurePartitionRange(ctx context.Context, pool *pgxpool.Pool, from, to time.Time) (int, error)
- func EnsurePartitions(ctx context.Context, pool *pgxpool.Pool, ahead int) (int, error)
- func Migrate(ctx context.Context, dsn string, log *slog.Logger) error
- func PartitionName(table string, at time.Time) string
- func PartitionedTableBytes(ctx context.Context, pool *pgxpool.Pool, table string) (int64, error)
- func Status(ctx context.Context, dsn string) ([]string, error)
- type RetentionPolicy
Constants ¶
const AuditTable = "audit_logs"
AuditTable is the audit log, retained under its own window.
const PartitionLookahead = 2
PartitionLookahead is how many months of partitions to create beyond the current one.
Variables ¶
var AnalyticsTables = []string{"click_events", "visitors"}
AnalyticsTables are the partitioned tables the analytics retention window applies to.
audit_logs is partitioned identically and is deliberately not here. Audit retention is a different policy from analytics retention — the reason to keep an audit trail is that someone may need to ask what happened a long time afterwards — and quietly deleting it on the analytics setting would be a surprise of exactly the wrong kind. It has its own window; see RetentionPolicy.
var Migrations = func() fs.FS { sub, err := fs.Sub(migrationsFS, "migrations") if err != nil { panic("store: cannot open embedded migrations: " + err.Error()) } return sub }()
Migrations is the embedded set rooted at the migration files themselves. goose scans the root of the FS it is given, so the "migrations/" prefix has to be stripped or it finds nothing.
var PartitionedTables = []string{"click_events", "visitors", "audit_logs"}
PartitionedTables are the RANGE-partitioned tables. All are keyed on a timestamptz and partitioned by month.
Maintaining partitions for a table nothing writes to yet is deliberate rather than an oversight, and it paid off here. audit_logs was maintained through the whole of Phase 1 with no writer; when M21 gave it one, partitions already existed for every month and no backfill was needed. `visitors` is still in that position. The cost is one to_regclass check per table per month — the partition already exists on all but one run an hour. The alternative fails in the direction that matters: rows landing in the default partition, which retention never drops, so a dormant table would quietly become the one place data is kept forever.
Functions ¶
func DefaultPartitionCounts ¶
DefaultPartitionCounts reports how many rows sit in each default partition.
A non-zero count is an operational alert, not a curiosity: it means rows arrived outside every explicit range, and attaching the partition that should have held them will now fail until they are moved out.
func DropExpiredPartitions ¶
func DropExpiredPartitions(ctx context.Context, pool *pgxpool.Pool, policy RetentionPolicy, now time.Time) ([]string, error)
DropExpiredPartitions drops monthly partitions whose entire range is older than their table's retention window, and reports what it dropped.
A window of zero or less keeps that table forever, matching the configuration contract that 0 means "forever". A table absent from the policy is never touched at all, which is what keeps a partitioned table added later from silently inheriting somebody else's window.
Retention is enforced at month granularity, and only when the newest row a partition could hold is already outside the window. The alternative — deleting rows older than exactly N days — would mean a DELETE across the largest table in the system, then a VACUUM to reclaim the space, on a schedule. Dropping a partition is instant, reclaims the space immediately, and cannot half-finish. The cost is that data survives up to a month past the nominal window, which is the right way to be wrong: keeping data slightly too long is recoverable, and deleting it slightly too early is not.
Daily rollups live in their own unpartitioned tables and are untouched, so historical charts keep working after the raw events are gone.
func EnsurePartitionRange ¶
EnsurePartitionRange creates monthly partitions covering every month from `from` to `to` inclusive, plus a default partition per table.
Separate from EnsurePartitions because the months that need to exist are not always the ones around today: restoring a backup and seeding a load-test dataset both write into the past, and an insert with no matching partition lands in the default one, where it silently blocks attaching the partition that should have held it.
func EnsurePartitions ¶
EnsurePartitions creates monthly partitions for the current month and the next `ahead` months, plus a default partition per table. It reports how many it created and is safe to call repeatedly.
Two things here are load-bearing.
The session timezone is pinned to UTC for the DDL. Bounds on a timestamptz column resolve against the session timezone at DDL time, so the identical bound literal produces a different absolute range under a different timezone, leaving either a gap that silently routes rows to the default partition or an overlap that makes attaching fail. Demonstrated in docs/adr/0001-partitioning-and-sqlc.md.
It looks more than one month ahead. Creating next month's partition on the last day of this one is a single point of failure with a hard deadline; two months of headroom turns a missed run into a warning rather than an outage.
func Migrate ¶
Migrate applies all pending migrations, then ensures partitions exist.
Runs in-process at boot, before the listener opens. An init container would need either a shell (distroless has none) or a second image, plus depends_on wiring that confuses a first-time operator; in-process means `docker compose up` on an empty volume produces a working app with no extra concepts.
A Postgres session lock serializes replicas racing at startup, so a rolling deploy cannot run the same migration twice.
func PartitionName ¶
PartitionName returns the partition a timestamp belongs to.
func PartitionedTableBytes ¶ added in v0.2.0
PartitionedTableBytes reports the on-disk size of a partitioned table: every partition, including indexes and TOAST.
This exists because the audit log's retention default is "keep forever", and that default is only defensible if the growth it permits is visible. An operator who never sets AUDIT_RETENTION_DAYS has chosen unbounded growth, and they should find that out from a graph rather than from a full disk.
Summed over the partitions rather than read from the parent: a partitioned table has no storage of its own, so pg_total_relation_size on the parent answers 0 no matter how much data is underneath it.
Catalogue and free-space-map arithmetic only — no scan of the table — so this stays cheap on the table it is most needed for.
Types ¶
type RetentionPolicy ¶ added in v0.2.0
RetentionPolicy maps a partitioned table to the number of days its data is kept. Zero or less keeps that table forever, matching the configuration contract that 0 means "forever".
A map rather than one number and a list of tables, because the two policies have different defaults and answer to different settings: 395 days from ANALYTICS_RETENTION_DAYS, and forever from AUDIT_RETENTION_DAYS. Expressing that as a single window over a table list was how audit_logs came to be exempt-by-omission, which worked only for as long as there was exactly one window.
func NewRetentionPolicy ¶ added in v0.2.0
func NewRetentionPolicy(analyticsDays, auditDays int) RetentionPolicy
NewRetentionPolicy builds the policy from the two configured windows.
Source Files
Directories