store

package
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 4, 2026 License: MIT Imports: 17 Imported by: 0

Documentation

Overview

Agent activity: the second append-only ledger. usage_events answers "what did this cost"; activity_events answers "what did the agent actually do" — which tool was called, which skill was invoked, which hook fired, how many times.

The two are joined, never merged. An activity row stores no token counts at all: it names the usage_events row whose provider record contained the call (usage_dedup_key), and the read path below derives tokens and cost from the ledger by dividing that row's total between the calls that share it. See model.ActivityEvent for why that shape is the honest one, and activityDivisorSQL for why the divisor is counted in the table rather than read from the calls_in_turn each adapter stamped.

Schema versioning and the migration runner. Open never writes before reading the recorded version: fresh databases are created directly at SchemaVersion, older ones run the ordered additive migrations, and a newer database refuses to open so an older binary can never stamp a version backwards.

Derived rollup of usage_events (issue #59). The ledger stays the only history; this table is a summary that exists so time-bucketed reporting stops scanning 360k rows to return 24 numbers. Every row is reproducible from usage_events, nothing reads it as a source of truth, and it may be dropped and rebuilt at any time.

Two rules keep it honest:

  • It is keyed by the UTC 15-MINUTE bucket an event falls in, never by local time. Rolling up by local time would bake the writing machine's calendar into stored data; the local fold happens on READ, in SQL, exactly the way query.go folds event_time_unix. The width is 15 minutes and not an hour because every real-world UTC offset is a whole number of quarter hours, while half-hour zones (Asia/Kolkata at +05:30 among them) split an hour bucket across two local buckets - an hourly key would silently move the first half hour of every local day into the previous one. Resolution BELOW the bucket width requires ledger events. Exact ending buckets are read separately; unaligned starts use the ledger for the whole query.
  • Its deltas are written inside the transaction that appends the events (insertEventsTx), so a crash cannot land events without the matching delta. The watermark below catches the one case that discipline cannot: a rollup created empty by the migration, or left behind by a write that predates it.

SQLite-backed implementation of the two handles. Pure Go via modernc.org/sqlite (CGO_ENABLED=0). The append-only guarantee is enforced by schema.sql (UNIQUE(dedup_key) + no-UPDATE/no-DELETE triggers); this file only ever appends to usage_events (INSERT .. ON CONFLICT(dedup_key) DO NOTHING) and upserts mutable accumulator and code-change state.

Package store is the append-only usage ledger and the query surface over it.

The database is one SQLite file driven by modernc.org/sqlite (pure Go, CGO_ENABLED=0). It holds the token ledger, the activity ledger, the turn attribution table, a derived rollup and the collector's own working state, and Open / OpenReadOnly are the only ways in. schema.sql always describes the full latest schema; older files run the ordered steps in migrate.go.

HISTORY IS APPENDED BY DEFAULT

usage_events records what a harness spent, and two of the tables beside it are on the same terms: activity_events (one row per tool, skill or hook invocation) and usage_turn_context (what a turn ran under). Each carries BEFORE UPDATE and BEFORE DELETE triggers that RAISE(ABORT), so a mutation aborts in the database rather than being caught by discipline in this package. Rows arrive through INSERT .. ON CONFLICT(dedup_key) DO NOTHING - deliberately not INSERT OR IGNORE, which would also swallow CHECK violations - which is what makes a re-read of an unchanged source a no-op rather than a double count. The default correction mechanism is a NEW row with kind='adjustment', preserving the earlier observation. Explicit exceptions are SyncUnpriced (filling unknown costs) and the opt-in ReconcileClaudeBatch (growing Claude usage). Neither changes the insertion semantics of ApplyBatch, ApplyEvents or InsertEvents.

aggregate_state, source_checkpoints, usage_rollup and activity_usage_counts are mutable working tables (schema_meta holds the version stamp and the rollup watermark). They are working state, not history: losing any of them costs a re-read or a rebuild, never a fact.

Two consequences a caller feels. First, a write can partially succeed: a row that fails its own insert (CHECK violation, empty dedup key) is skipped and reported in the returned error while the rest of the batch commits, so the returned counts stay meaningful when the error is non-nil - one poison row must not abort a batch that is re-read every cycle. Second, one read of one source commits as ONE transaction (ApplyBatch): events, activity, turn contexts and the checkpoint together, because a checkpoint that outran its data would skip that data forever.

TWO HANDLES, AND THE ABSENCE THAT SEPARATES THEM

Open returns a *Ledger: the full handle. It creates the file if absent, applies WAL, synchronous=FULL, busy_timeout=5000 and foreign_keys=ON, migrates an older schema, refuses a newer one (an older binary must never stamp a version backwards), and chmods the database and its WAL/SHM sidecars to 0600 because the raw column can hold transcript content. It is what the collector holds.

OpenReadOnly returns a *Reader: an EXISTING database, opened mode=ro plus query_only(1), with no schema creation, no migration and no file mode touched. Its schema version must equal this binary's EXACTLY and is refused in either direction - migrating would be a write, and a reader that quietly serves a schema it does not understand is worse than one that will not start.

A Reader HAS NO WRITE METHOD. That is the whole design (issue #72, decisions 2 and 8): the append-only guarantee used to be defended by a flag this package checked at the top of each write, on a handle whose type still advertised InsertEvents to whoever held it, so "a serving process cannot write the ledger" was a promise kept at runtime. It is now a property of the type - a program that calls a write through a read handle does not compile, and there is no test to write about it. Ledger EMBEDS *Reader, so the collector holds one handle carrying both halves and hands l.Reader to anything that only queries. EnsureRollup and RebuildRollup are writes and live on Ledger; RollupStale is a read and lives on Reader, since a serving process still has to know that the summary it would answer from covers nothing.

This package exports NO fat store interface. A consumer that wants a fake declares its own interface over the methods it actually calls (collect.Store and tui.DataSource are the two in this repository), which keeps the seam beside the code that depends on it and lets a method be added here without breaking an implementation nobody in this module wrote.

TWO ERRORS WORTH BRANCHING ON, AND NO MORE

A sentinel is a promise: once it exists, code outside this module tests for it and it can never be retired quietly. So this package exports one where a caller can do something different because of it, and nothing where it cannot (issue #72, decision 6). Everything else is a plain wrapped error carrying a message a person reads.

ErrSchemaNewer, matched with errors.Is, is the database written by a NEWER build than the binary opening it. Both handles refuse it - Open will not migrate a schema it does not understand, OpenReadOnly will not serve one - and the answer is the same either way: upgrade. An OLDER database is deliberately not this error; Open migrates it, and through the read handle it asks for the opposite action.

SkippedRowsError, matched with errors.As, is the partial success described above: a non-nil error WHOSE COUNTS ARE STILL TRUE. It names the table, how many rows were offered, and every row that was refused with its own dedup key and cause, so a caller can log what was rejected instead of parsing a message. Unwrap reaches the first row's cause, so errors.Is still finds the CHECK violation or driver error underneath. A caller that treats a non-nil error from a batch write as "nothing happened" under-reports a pass that mostly worked.

A READ HANDLE IS SAFE AGAINST A LIVE COLLECTOR

This is a promise of the API, not an accident of the implementation, and it is not withdrawn without a major version. A read handle may be open, and queried, while a collector writes the same file: WAL means readers do not block the writer and the writer does not block readers, so a report never waits on a collection pass and never sees half of one - each source's pass commits atomically.

The honest sentence: a read can still fail with SQLITE_BUSY, because WAL has moments (crash recovery, a checkpoint that restarts the log) where a reader needs a lock a writer holds, and the wait before it gives up is the busy_timeout - fixed at 5000ms in both DSNs, not configurable. A read is idempotent, so a retry is always safe. One case sits outside the promise: the schema version is checked once, at open, so a NEWER collector migrating the file underneath a live read handle is not covered; reopen it.

ONE DIMENSION PER QUERY, ALWAYS

Cost is partitioned six ways: the five turn-context dimensions (model.TurnDimensions - agent, skill, mcp_tool, mcp_server, plugin) plus activity_events' per-call attribution. Each is honest alone and meaningless summed, the way cost-by-region and cost-by-product are two views of one budget. A turn commonly carries three or four contexts at once and EVERY row names the turn's full cost, so a dimension-blind join counts the same tokens once per context the turn held: measured on a real ledger, 6.213bn tokens and $6,023.52 reported for turns that cost 4.984bn and $4,700.90, a 28.1% overstatement from one missing predicate.

So the dimension is a REQUIRED ARGUMENT of SummarizeTurnContext and TopTurnContext, never a filter field a caller could leave unset. It is validated against the closed vocabulary before any SQL exists, and the WHERE builder writes the dimension predicate unconditionally, so no argument and no combination of empty filters produces a statement without it. Grouping by "dimension" is refused by name (it is exactly the operation that concatenates two partitions), as is grouping by any dimension other than the one queried; ActivityFilter's Kinds and Names are refused because honouring them means joining activity_events, where a turn with two matching calls joins twice. No query here reads two dimensions, or both this table and activity_events, in one statement - and a caller must not add two results together either.

Within one dimension the join is 1:1 (PRIMARY KEY (usage_dedup_key, dimension) against a UNIQUE dedup_key), so a bucket's cost is a plain SUM with no divisor. The activity queries are the opposite case and DO divide: one assistant turn commonly emits several calls against a single usage object, so each call takes the turn's counts divided by the number of rows naming that turn - counted in the table, integer, non-negative, which bounds the shares by the turn's real total by construction. Calls whose usage row is absent are reported as unattributed: unknown, never free. Rows with no stamped cost are likewise unpriced and never $0 - a bucket carries UnpricedEvents so a caller knows its CostMicroUSD is an understatement until those are display-priced.

WithRaw IS AN AUDIT PAYLOAD, NOT A DATA SOURCE

usage_events.raw exists to answer "what did the provider actually say", and adapters now build it from an explicit allow-list of usage/model/identity fields. Rows appended BEFORE that allow-list landed can hold whole transcript lines, and append-only means they are never rewritten - that history is exactly as it was stored. The default ListEvents projection therefore names its columns and excludes raw; WithRaw is the explicit opt-in, and the CLI passes it for export --include-raw and nothing else. The gate that matters is upstream of every reader: privacy.no_raw drops the column at collection, and what was never stored cannot be projected.

Turn-context cost attribution: what did a skill, an agent, an MCP server or a plugin actually cost.

usage_events answers "what did this turn cost"; activity_events answers "what did the agent call". Neither answers "what did X cost", and the reason is worth stating because it looks like it should.

An activity row of kind=skill records the turn that INVOKED a skill — one call, one row. The work the skill then goes on to do is thousands of further turns, and none of them is a Skill call. Locally that gap is 44 invocation rows against 8,039 records of actual work: asking activity_events which skill is expensive returns the cost of pressing the button, not the cost of the thing the button started. For subagents the gap is not a gap but a chasm — 79,816 usage-bearing records ran as a subagent, 77.6% of all token-bearing turns on this machine, described by a couple of hundred `Agent` call rows.

The missing facts live on the source record. Claude Code stamps every assistant record with up to FIVE top-level attribution strings — attributionAgent, attributionSkill, attributionMcpTool, attributionMcpServer, attributionPlugin — naming what the turn was running under along each axis. Those are properties of the TURN, not extra calls within it, and the whole design follows from that one observation.

ONE TABLE, FIVE DIMENSIONS, AND WHY THAT IS NOT FOUR TABLES

Each of the five could have had its own table on the model usage_skill_context set. It should not, and the giveaway is that the second one would have been a verbatim copy of the first with a column renamed: same key, same 1:1 join to the ledger, same absent divisor, same append-only triggers, same three indexes. A table per dimension makes the SIXTH attribution axis a schema migration, five near-identical query builders that can drift apart, and five places to remember the partition rule. usage_turn_context carries the axis as a COLUMN, so a new dimension is a new CHECK value and nothing else.

WHY THIS CANNOT DOUBLE-COUNT WITHIN A DIMENSION

A usage row carries at most one value per dimension — every one of the five source fields is a scalar string, verified across 99,894 occurrences in the local corpus, 100% of them JSON strings — and this table makes that a database constraint rather than a hope: (usage_dedup_key, dimension) is the PRIMARY KEY. usage_events.dedup_key is likewise UNIQUE. Once a query is pinned to ONE dimension the join below is therefore 1:1 in both directions and cannot multiply a row, so SUM(u.cost_micro_usd) over it adds each turn's cost AT MOST ONCE. There is no divisor here and nothing to share, because nothing competes: unlike the tool-call split, which divides one turn's cost among the calls that shared it, a turn context IS the turn.

WHY IT CAN DOUBLE-COUNT ACROSS DIMENSIONS, AND WHAT STOPS IT

That "pinned to one dimension" is load-bearing and is the price of the single table. A turn commonly carries three or four contexts at once — measured: 3,816 records carry agent+mcp_tool+mcp_server, 2,201 carry agent+skill+plugin, 9 carry all five — and EVERY one of those rows names the turn's full cost, because each is a complete answer to a different question. A query that forgets the dimension predicate joins such a turn once per context and reports up to five times the real spend.

So the dimension is not a filter field that a caller might leave unset. It is a REQUIRED ARGUMENT of every read below, validated against the closed vocabulary before any SQL is built, and stitched into the WHERE clause by the builder rather than by the caller. Grouping by "dimension" — the one grouping that would put two partitions in one result set — is refused by name, as is grouping by any dimension OTHER than the one being queried. There is no code path in this package that reads two dimensions in one statement.

These five plus the tool-call attribution in activity_events are SIX PARTITIONS OF THE SAME DOLLARS, in the way cost-by-region and cost-by-product are two views of one budget: each honest alone, meaningless added together. No query in this package reads usage_turn_context and activity_events at once, for the same reason.

WHY NOT A COLUMN ON activity_events

Because most of the fact would vanish. A turn can run under a skill or as a subagent and call no tool at all — thinking, planning, reading its own output — and it emits no activity row to hang a column on. Measured: 3,361 of 8,039 skill-context records carry zero tool_use blocks (41.8%), and the agent dimension covers call-less turns far more often still. Keying on the usage row instead of on a call captures those for free, because the usage row is what exists in every case.

WHAT THIS DELIBERATELY DOES NOT DO

It does not compose with the activity ledger's Kinds/Names filters, and the queries refuse them rather than ignoring them. "Skill cost among turns that called Bash" sounds reasonable and is a trap: reaching it means joining activity_events, at which point a turn with two Bash calls joins twice and its cost doubles. The refusal is the guard rail, not tidiness.

Index

Constants

View Source 
const (
	BackupPhaseCopying    = "copying"
	BackupPhaseVerifying  = "verifying"
	BackupPhasePublishing = "publishing"
)
View Source 
const SchemaVersion = 9

SchemaVersion is the schema version this binary creates fresh databases at and can open. Bump when the schema changes, keep schema.sql describing the full latest schema, and add the matching step to migrations (migrate.go).

Variables

View Source 
var ErrSchemaNewer = errors.New("store: database schema is newer than this binary")

ErrSchemaNewer reports a database written by a NEWER build of aiusage than the one trying to open it. It is the one refusal at open that a caller can act on - upgrade the binary, or point at a different file - as against a corrupt file, a missing directory or a permission problem, which are all "the open failed" and are handled the same way.

The refusal is absolute and is the same one in both directions of the split: Open will not migrate a schema it does not understand (an older binary stamping a version backwards would silently strip whatever the newer one added), and OpenReadOnly will not serve one. The wrapped message names both versions, which is what a person reading the error needs; errors.Is is what a program needs.

It does NOT cover the opposite case. A database OLDER than the binary is not an error at all through Open - it migrates - and through OpenReadOnly it is a plain error, because "open it read-write once" is the same instruction whatever produced the mismatch.

Functions

func RecordedSchemaVersion 

func RecordedSchemaVersion(ctx context.Context, path string) (int, error)

RecordedSchemaVersion reads only the version stamp from an existing database. It returns 0 for an unversioned or unrecognized layout and never creates or migrates the file.

Types

type ActivityBucket 

type ActivityBucket struct {
	// Keys maps each GroupBy dimension to its value for this bucket
	// (e.g. {"name":"Bash","tool":"claude-code"}). Ordered via OrderedKeys.
	Keys        map[string]string
	OrderedKeys []string

	// Calls counts invocations in the bucket. It is the frequency answer and is
	// never affected by attribution: a call with no joinable usage row is still
	// a call that happened.
	Calls int64
	// Sessions is the distinct non-empty session count within the group.
	// Distinct counts do not add across buckets.
	Sessions int64

	AttributedInput  int64
	AttributedOutput int64
	AttributedTotal  int64
	// AttributedCostMicroUSD sums the shares of costs STAMPED on the joined
	// usage rows. Calls whose usage row carries no stamped cost contribute
	// nothing and are counted in UnpricedCalls instead, so a bucket with
	// UnpricedCalls > 0 is an understatement until those are display-priced.
	AttributedCostMicroUSD int64

	// UnattributedCalls counts calls with NO joinable usage row: the source
	// records its calls and its token counts in unrelated records (codex), the
	// call carries no usage at all (hooks), or the partner row predates
	// activity collection. Their tokens are not missing, they are unknowable
	// from this table — reading a zero share as "free" would be the lie this
	// count exists to prevent.
	UnattributedCalls int64
	// UnpricedCalls counts calls that DID join a usage row which carries no
	// stamped cost (collected before v3, or a model the price ladder could not
	// price). Their tokens are attributed; their cost is not.
	UnpricedCalls int64
	// ComputedCostCalls counts calls whose joined usage row was priced from a
	// public rate card rather than by the harness (model.PriceProvenance). Zero
	// means every dollar in AttributedCostMicroUSD traces to a vendor's own
	// number. It counts CALLS, not usage rows: several calls sharing one turn
	// each carry that turn's provenance, which is the level the figure they
	// qualify is summed at.
	ComputedCostCalls int64
}

ActivityBucket is one grouped row of summarised activity.

The Attributed* fields are this call's SHARE of the tokens its turn actually cost, taken from usage_events and divided between the calls that name it (see activityDivisorSQL). The division is integer, so a turn's shares sum to at most the turn's real total — the split can only ever UNDERSTATE, never inflate, which is the direction an attribution guess is allowed to be wrong in.

type ActivityFilter 

type ActivityFilter struct {
	Since time.Time // inclusive lower bound on event_time (zero = open)
	Until time.Time // exclusive upper bound on event_time (zero = open)

	Tools    []string // restrict to these agent CLIs (empty = all)
	Kinds    []string // restrict to these kinds: tool|skill|hook (empty = all)
	Names    []string // restrict to these tool/skill/hook names (empty = all)
	Projects []string // restrict to these projects (empty = all)
	Sessions []string // restrict to these sessions (empty = all)
	Models   []string // restrict to these models (empty = all)
	// Providers restricts to linked usage with these recorded providers. Empty
	// means all; [""] selects linked usage with an unknown provider. Unlinked
	// activity and turn contexts are excluded whenever this filter is set.
	Providers []string
	// Values restricts the TURN-CONTEXT queries to these context values (empty =
	// all) — agent types, skill names, MCP server or tool names, plugin names,
	// whichever dimension the query named. It is ignored by
	// SummarizeActivity/TopActivity/ListActivity, which read activity_events and
	// have no turn-context column: a turn context is a property of the turn,
	// recorded in usage_turn_context. See SummarizeTurnContext and the
	// turncontext.go package comment.
	Values []string
	// Skills is the pre-generalisation spelling of Values, kept for callers that
	// predate the other four dimensions. It restricts the SKILL dimension and is
	// REFUSED on any other, rather than quietly filtering agent names against a
	// list of skills and returning an empty result that reads as "that agent
	// cost nothing".
	Skills []string

	// GroupBy lists grouping dimensions, applied in order. Valid values:
	// "hour","day","week","month","tool","kind","name","project","session",
	// "model". Empty means a single grand-total bucket.
	GroupBy []string
}

ActivityFilter selects and groups agent activity for reporting. It mirrors Filter's vocabulary so a surface can carry one set of crumbs across both ledgers, plus the two activity introduces (kind, name). Provider filters use the linked usage row because activity has no recorded provider of its own.

type ActivityOrder 

type ActivityOrder string

ActivityOrder names the metric TopActivity ranks by. 

const (
	// ActivityByCalls ranks by invocation count — "what do I call most".
	ActivityByCalls ActivityOrder = "calls"
	// ActivityByCost ranks by attributed cost — "which skill is expensive".
	ActivityByCost ActivityOrder = "cost"
	// ActivityByTokens ranks by attributed total tokens, the answer to the same
	// question on a ledger whose rows were never priced.
	ActivityByTokens ActivityOrder = "tokens"
)

type ActivitySummary 

type ActivitySummary struct {
	GroupBy []string
	Buckets []ActivityBucket
	Totals  ActivityBucket
}

ActivitySummary is the result of SummarizeActivity: grouped buckets plus a grand total.

type Applied 

type Applied struct {
	Events   int
	Activity int
	// TurnContexts counts new usage_turn_context rows: (turn, dimension) pairs
	// newly recorded as having run under something. ONE turn can contribute up
	// to five of them, one per dimension, so this counts rows and not turns. A
	// pair already carrying a context counts here no more than a duplicate event
	// counts in Events.
	TurnContexts int
	// CodeChanges counts inserted or updated source snapshots. Identical
	// repeats and older source versions do not count.
	CodeChanges int
}

Applied reports committed new ledger/context rows and changed code snapshots. Duplicate ledger/context keys and unchanged code snapshots do not count.

type BackupProgress 

type BackupProgress struct {
	Phase   string `json:"phase"`
	Batches int64  `json:"batches"`
}

BackupProgress reports coarse phases and completed Online Backup batches. Callers may omit the callback when they do not need interactive progress.

type BackupResult 

type BackupResult struct {
	Path          string            `json:"path"`
	SizeBytes     int64             `json:"size_bytes"`
	SHA256        string            `json:"sha256"`
	SchemaVersion int               `json:"schema_version"`
	Compatible    bool              `json:"compatible"`
	State         VerificationState `json:"state"`
	RowCounts     map[string]int64  `json:"row_counts,omitempty"`
	Verification  Verification      `json:"verification"`
}

BackupResult describes a verified, atomically published online backup.

func Backup 

func Backup(ctx context.Context, source, destination string) (BackupResult, error)

Backup creates a complete SQLite snapshot through the driver's Online Backup API, verifies it, then publishes it without overwriting an existing path.

func BackupWithProgress 

func BackupWithProgress(ctx context.Context, source, destination string, report func(BackupProgress)) (BackupResult, error)

BackupWithProgress is Backup with coarse progress notifications for an interactive command. The callback runs synchronously and must return promptly.

type Bucket 

type Bucket struct {
	// Keys maps each GroupBy dimension to its value for this bucket
	// (e.g. {"day":"2026-05-29","tool":"codex"}). Ordered via OrderedKeys.
	Keys        map[string]string
	OrderedKeys []string // dimension names in GroupBy order
	Events      int64
	// Sessions is the distinct non-empty session_id count within the group,
	// computed store-level (COUNT DISTINCT) so callers never have to
	// materialize one bucket per session just to count them. Distinct counts
	// do not add across buckets.
	Sessions      int64
	Input         int64
	Output        int64
	CacheCreation int64
	CacheRead     int64
	Reasoning     int64
	Total         int64
	// CostMicroUSD sums the costs stamped at collect time (millionths of USD).
	// Rows with no stamped cost contribute nothing to it — they are counted in
	// UnpricedEvents instead and must be display-priced separately, so a bucket
	// with UnpricedEvents > 0 is an UNDERSTATEMENT until that is folded in.
	CostMicroUSD int64
	// UnpricedEvents counts rows in the bucket whose cost_micro_usd is NULL
	// (collected before v3, or a model the pricing ladder could not price).
	UnpricedEvents int64
	// ComputedCostEvents counts the PRICED rows of the bucket whose cost this
	// project derived from a public rate card rather than reading off the
	// harness (model.PriceProvenance). It is what lets a surface say the sum is
	// an estimate: zero means every dollar in CostMicroUSD is one a vendor
	// reported.
	ComputedCostEvents int64
}

Bucket is one grouped row of summarised usage.

type CodeChangeSummary 

type CodeChangeSummary struct {
	LinesAdded     int64
	LinesRemoved   int64
	KnownChanges   int64
	UnknownChanges int64
}

CodeChangeSummary totals the latest reported snapshots for one session. UnknownChanges counts snapshots whose line counts are unavailable.

type DBStats 

type DBStats struct {
	Path          string
	Events        int64
	Snapshots     int64
	DistinctTools int64
	DistinctModel int64
	SizeBytes     int64
	EarliestEvent time.Time
	LatestEvent   time.Time
	SchemaVersion int // version recorded in the database, not the binary's
}

DBStats describes the database as a whole for the `doctor` command.

type Filter 

type Filter struct {
	Since time.Time // inclusive lower bound on event_time (zero = open)
	Until time.Time // exclusive upper bound on event_time (zero = open)

	Tools     []string // restrict to these tools (empty = all)
	Models    []string // restrict to these models (empty = all)
	Providers []string // restrict to recorded providers (empty = all; [""] = unknown)
	Projects  []string // restrict to these projects (empty = all)
	Sessions  []string // restrict to these sessions (empty = all)

	// GroupBy lists grouping dimensions, applied in order. Valid values:
	// "hour","day","week","month","tool","model","provider","project",
	// "session". A provider bucket keyed by the empty string is the one
	// holding the rows whose source never named a billing provider.
	// Empty means a single grand-total bucket.
	GroupBy []string
}

Filter selects and groups usage for reporting.

type Ledger 

type Ledger struct{ *Reader }

Ledger is the FULL handle: a Reader plus the appends. It is what Open returns and what the collector holds, and it is the only type in this package that can add a row to any of the three append-only tables, upsert accumulator state or code-change snapshots, or rebuild the derived rollup.

The Reader is embedded by pointer so a caller that only reads can be handed l.Reader by name rather than by an interface conversion, and so the two handles are never two connections to one file.

func Open 

func Open(path string) (*Ledger, error)

Open opens (creating if absent) the database at path with WAL and busy_timeout=5000 pragmas, then reads the recorded schema version before touching anything: same version opens as-is, older versions run the ordered migrations (migrate.go), and a newer version is refused so an older binary can never stamp it backwards. The handle is read/write because the collector appends to it; all reporting paths only issue SELECTs.

func (*Ledger) ApplyBatch 

func (l *Ledger) ApplyBatch(ctx context.Context, b ObservationBatch) (Applied, error)

ApplyBatch is ApplyObservation plus turn contexts and code-change snapshots. It uses the same transaction and ordering (events first, so the rows naming them by dedup key find them present), same idempotence: a turn context conflicts on (usage dedup key, dimension) and does nothing, which is what stops a re-read serving a turn's cost twice.

func (*Ledger) ApplyEvents 

func (l *Ledger) ApplyEvents(ctx context.Context, events []model.UsageEvent, cp *model.SourceCheckpoint) (int, error)

ApplyEvents appends usage events (same idempotent semantics as InsertEvents) and upserts the source checkpoint in ONE transaction. A checkpoint persisted outside the event transaction could outrun the events it claims — a crash between the two commits would then skip data forever. A nil cp degrades to a plain event insert; empty events with a non-nil cp writes just the checkpoint.

It is the events-only shorthand for ApplyBatch. Per-row skips (poison rows) still commit alongside the checkpoint: they are permanent CHECK violations a re-read cannot fix, so holding the checkpoint back would only re-parse them forever.

func (*Ledger) ApplyObservation 

func (l *Ledger) ApplyObservation(ctx context.Context, events []model.UsageEvent, activity []model.ActivityEvent, cp *model.SourceCheckpoint) (Applied, error)

ApplyObservation appends usage events, appends agent activity rows, and upserts the source checkpoint — all in ONE transaction. It is ApplyBatch without the turn contexts.

The single transaction is the point. Activity rows reference usage rows by dedup key, and the checkpoint gates the re-read of both: splitting them across transactions would let a crash advance the checkpoint past activity that never landed, losing it permanently, or leave a call pointing at a usage row that rolled back. Activity is appended AFTER the events so the row it names already exists.

Activity has the same idempotence as events (ON CONFLICT(dedup_key) DO NOTHING) and the same per-row skip behaviour, so the returned counts stay meaningful when the error is non-nil.

func (*Ledger) ApplySnapshot 

func (l *Ledger) ApplySnapshot(ctx context.Context, events []model.UsageEvent, state model.AggregateSnapshot, cp *model.SourceCheckpoint) (int, error)

ApplySnapshot atomically appends an aggregate cell's delta events and records its new accumulator state in ONE transaction, so a crash can never persist the events without the state (the next cycle would re-derive the same delta under a fresh dedup key — a permanent double count).

When events is non-empty but every dedup key already exists, the state write is SKIPPED: an unchanged baseline lets the next poll re-derive the colliding delta instead of dropping it. A non-nil cp is upserted under the same condition and in the same transaction, so a checkpoint can never claim data whose state write was skipped or rolled back — a collided delta stays re-derivable only while neither baseline nor checkpoint advances. Returns the number of events actually inserted.

func (*Ledger) EnsureRollup 

func (l *Ledger) EnsureRollup(ctx context.Context) (bool, error)

EnsureRollup brings the usage rollup and activity usage counts back in step with their ledgers, and reports whether either derived table was rebuilt.

The usage rollup has two checks for different failures. The watermark (highest ledger id folded in) catches the empty rollup a v4 migration leaves behind and any write that skipped the delta. The event count catches the rollup that tracked the newest ids but lost older ones - a rollup filled from a partial ledger would otherwise pass the watermark check forever. Activity counts are compared by key and count in both directions, so moving counts between keys cannot conceal drift behind an unchanged total.

func (*Ledger) InsertEvents 

func (l *Ledger) InsertEvents(ctx context.Context, events []model.UsageEvent) (int, error)

InsertEvents appends events idempotently in a single transaction. Returns the count of rows actually inserted (new dedup keys). Existing dedup keys are ignored; rows are never updated or deleted. A row that fails its own insert (CHECK violation, empty dedup key) is skipped and reported in the returned error while the rest of the batch still commits — one poison row must not abort a batch that is re-read and retried every cycle.

func (*Ledger) RebuildRollup 

func (l *Ledger) RebuildRollup(ctx context.Context) error

RebuildRollup drops and recreates the whole rollup from usage_events in one transaction, so a reader never sees a half-built summary. It is the definition of the table's contents: any disagreement between the rollup and the ledger is resolved by running it, never by correcting the ledger.

func (*Ledger) ReconcileClaudeBatch added in v0.1.1 

func (l *Ledger) ReconcileClaudeBatch(ctx context.Context, b ObservationBatch) (Applied, error)

ReconcileClaudeBatch explicitly opts out of immutable accounting for growing Claude usage records. New observations have ApplyBatch's insertion semantics. For an existing Claude usage identity, a strictly larger total may replace token counters and cost, provided every counter is nondecreasing and the model, provider, tier, session, project and request/message identities agree. Older/equal totals are ignored. Conflicting growth rolls back the whole batch. IDs, timestamps, raw payloads and other recorded metadata are preserved.

The supplied cost must price the complete replacement observation. A nil cost makes the revised row unpriced; an earlier partial cost is never carried over. Existing vendor costs cannot be replaced with computed costs. Usage, rollup, activity, turn contexts and checkpoint commit together. Returned counts only count inserts, not revised usage. Replays do not insert extra turns or calls.

The rollup must be current; call EnsureRollup before reconciliation. Generic ApplyBatch/ApplyEvents/InsertEvents remain append-only. Single-writer ownership is required as for collection. This does not repair deleted source records, non-monotonic revisions or previously stamped unrelated pricing errors.

func (*Ledger) SyncUnpriced 

func (l *Ledger) SyncUnpriced(ctx context.Context, revision string,
	price func(model.UsageEvent) (int64, string, bool)) (int, error)

SyncUnpriced fills at most 512 NULL costs from one revision of the pricing tables. Priced rows are never selected or overwritten. The persisted cursor avoids rescanning old unknown models until the revision changes; later IDs remain eligible. The bound covers candidates and writes, not SQLite's scan through intervening priced rows. Raw and transient cache-TTL data are absent. The callback must account for that absence; pricing.Engine.PriceStoredEvent refuses costs that depend on the missing cache-lifetime split.

This is a bounded historical update exception. The strict update trigger is removed and restored inside the same write transaction as the guarded price updates, derived rollup changes and cursor. Failure rolls all of them back.

func (*Ledger) UpsertState 

func (l *Ledger) UpsertState(ctx context.Context, st model.AggregateSnapshot) error

UpsertState records the latest observed counters for (tool, key), replacing any previous value. This is mutable accumulator state, not history.

type ListOption 

type ListOption func(*listOptions)

ListOption tunes what ListEvents projects. It exists so the expensive column is opt-in at the call site rather than a default every caller pays for.

func WithEventTimeKeyset 

func WithEventTimeKeyset(after time.Time, afterID int64, limit int) ListOption

WithEventTimeKeyset turns ListEvents into one page of its existing total event-time order. Rows after (after, afterID) are returned, ordered by (event_time_unix, id); afterID 0 starts the walk and ignores after. Export uses this cursor so a late-observed old event keeps the same position the unpaged public output has always used. Existing WithKeyset remains id-ordered.

func WithKeyset 

func WithKeyset(afterID int64, limit int) ListOption

WithKeyset turns ListEvents into one page of a keyset walk: at most limit rows with a row id ABOVE afterID, ordered by id. Pass afterID 0 for the first page and the last id of a page for the next one.

The order changes with the option, and it has to: ids are AUTOINCREMENT, so ordering by id is a total order that a cursor can resume from exactly, while the default (event_time, id) order interleaves later-ingested rows with earlier event times and would make an id cursor skip them. A caller that wants event-time order must page some other way, or not page at all.

A limit of 0 or less means unlimited, which is the point of a cap living at the caller: the store enforces the walk, not the policy.

func WithRaw 

func WithRaw() ListOption

WithRaw restores the raw audit payload to a ListEvents projection. ONLY the export --include-raw path may pass it: raw can carry full transcript content for rows appended before the usage-object allow-list landed, and every other consumer has no use for it. Without it, raw never leaves the database.

type ObservationBatch 

type ObservationBatch struct {
	Events   []model.UsageEvent
	Activity []model.ActivityEvent
	// TurnContexts records what each usage event ran UNDER: its subagent, its
	// skill, its MCP tool and server, its plugin. At most one value per (event,
	// dimension), keyed by the event's dedup key — see model.TurnContext.
	TurnContexts []model.TurnContext
	// Checkpoint, when non-nil, is upserted in the same transaction.
	Checkpoint *model.SourceCheckpoint
	// CodeChanges replaces mutable per-change snapshots when their source
	// version is not older than the stored version.
	CodeChanges []model.CodeChange
}

ObservationBatch is everything one read of one source commits together. It is a struct rather than a parameter list because the set grows: activity joined events in v5, skill contexts in v6, all five turn-context dimensions in v7, and each addition must land in the SAME transaction as the checkpoint that gates its re-read, not in a second call that a crash could separate from the first.

type PreparedRestore 

type PreparedRestore struct {
	SourcePath   string       `json:"source_path"`
	StagePath    string       `json:"stage_path"`
	SnapshotTime time.Time    `json:"snapshot_time"`
	Verification Verification `json:"verification"`
}

PreparedRestore is a verified current-schema staging database. The input backup remains untouched; Cleanup removes only the staging file.

func PrepareRestore 

func PrepareRestore(ctx context.Context, backupPath, targetPath string) (*PreparedRestore, error)

PrepareRestore copies the input into a staging database beside the target, verifies it, migrates only the staging copy when needed, and repairs only its derived rollup. It does not touch the live target.

func (*PreparedRestore) Cleanup 

func (p *PreparedRestore) Cleanup()

Cleanup removes the staging database and any transient sidecars.

type QuarantineResult 

type QuarantineResult struct {
	TargetPath                    string       `json:"target_path"`
	QuarantinePath                string       `json:"quarantine_path"`
	Verification                  Verification `json:"verification"`
	TargetUsable                  bool         `json:"target_usable"`
	HistoricalCompletenessUnknown bool         `json:"historical_completeness_unknown"`
}

QuarantineResult describes an explicit reset that preserved the old SQLite bundle before creating a fresh verified database.

func Quarantine 

func Quarantine(ctx context.Context, targetPath string) (QuarantineResult, error)

Quarantine moves the database, WAL, and SHM as one recoverable bundle, then creates a fresh current database. The caller must stop collection and hold the collection lock.

type Reader 

type Reader struct {
	// contains filtered or unexported fields
}

Reader is the READ handle: every query, summary, listing and ranking this package answers, and NOT ONE METHOD THAT WRITES. That absence is the point. The append-only ledger used to be defended at three depths - the schema's no-UPDATE/no-DELETE triggers, the mode=ro connection, and a flag this package checked at the top of each write method - all three of them at RUNTIME, on a handle whose type advertised InsertEvents to anyone holding it. A consumer given one of these cannot call a write, cannot compile a program that calls a write, and needs no test to prove it did not (issue #72, decision 2 and 8).

It is returned by OpenReadOnly and embedded in Ledger, so the collector holds one handle carrying both halves while every reporting surface can be handed the read half alone. Consumers that want a fake declare their own narrow interface over the methods they actually call - this package exports no fat one to implement.

func OpenReadOnly 

func OpenReadOnly(path string) (*Reader, error)

OpenReadOnly opens an EXISTING database for reading only (issue #60): the connection is mode=ro, no schema is created, no migration is run, and no file mode is touched. It returns a Reader, which HAS NO WRITE METHOD, so "this process cannot write the ledger" is a property of the type its caller holds rather than a promise checked somewhere further down.

The connection is still opened mode=ro plus query_only(1) - defence in depth, the same DSN every adapter uses over an agent's own database. The compile-time absence covers this package's own API; the pragmas cover a future statement issued from inside this package, which is where a write would have to come from now that no caller can ask for one.

A schema version that differs from this binary's in EITHER direction is refused. Migrating would be a write, and a reader that quietly serves a schema it does not understand is worse than one that will not start.

func (*Reader) Checkpoint 

func (s *Reader) Checkpoint(ctx context.Context, tool, sourcePath string) (*model.SourceCheckpoint, error)

Checkpoint returns the stored incremental state for (tool, sourcePath), or nil when none has been recorded.

func (*Reader) Close 

func (s *Reader) Close() error

Close releases the database handle.

func (*Reader) IngestWatermark 

func (s *Reader) IngestWatermark(ctx context.Context) (time.Time, error)

IngestWatermark returns the observed time of the NEWEST row in the ledger, or the zero time when the ledger is empty. It is the "how far has collection got" token a serving process publishes: it moves when a cycle appends and stands still when one appends nothing, which is exactly what a client invalidating its queries needs to know.

It reads the last row by id rather than taking MAX(observed_time_unix). id is an INTEGER PRIMARY KEY, so the newest row is one index step, while the max over an unindexed column is a full scan of a 300k-row ledger on every meta poll. The two agree because observed time is stamped at insert and inserts are id-ordered.

This is deliberately NOT on the Store interface: it serves the read-only web surface, and the collector has no use for it.

func (*Reader) LastEventTimes 

func (s *Reader) LastEventTimes(ctx context.Context) (map[string]time.Time, error)

SourceStats returns per-tool stored stats for the `sources` command. LastEventTimes reports the newest event_time per tool over the whole ledger. It is the freshness question ("when did codex last write?") answered without SourceStats' full-table aggregate, which on a 370k-row ledger costs seconds. The recursive CTE is SQLite's loose index scan: each step seeks the next distinct tool through idx_events_tool_time, and each MAX is one seek to the end of that tool's range in the same index, so the cost is a handful of B-tree probes however many rows the ledger holds.

func (*Reader) LastState 

func (s *Reader) LastState(ctx context.Context, tool, key string) (*model.AggregateSnapshot, error)

LastState returns the latest observed counters for the (tool, key) accumulator cell, or nil if none has been recorded.

func (*Reader) ListActivity 

func (s *Reader) ListActivity(ctx context.Context, f ActivityFilter) ([]model.ActivityEvent, error)

ListActivity returns activity rows matching the filter, ordered by event time then row id. It exists for tests and for a future export; the reporting surfaces group instead of listing.

func (*Reader) ListEvents 

func (s *Reader) ListEvents(ctx context.Context, f Filter, opts ...ListOption) ([]model.UsageEvent, error)

ListEvents returns events matching Filter, ordered by event_time then row id (a total order, which keyset pagination will need). The projection names its columns and EXCLUDES raw; pass WithRaw to include it (export --include-raw only). WithKeyset switches it to one page of an id-ordered walk; WithEventTimeKeyset pages without changing the default total order.

func (*Reader) RollupStale 

func (s *Reader) RollupStale(ctx context.Context) (bool, error)

RollupStale reports whether the usage rollup disagrees with the usage ledger, the question EnsureRollup asks before rebuilding that table. It cannot repair the table. It is exported for the READ-ONLY serving path: a process that cannot write still has to know that the summary it would answer from covers nothing, so it can go to the ledger instead of serving the zeros of a rollup a migration created empty.

Cheap in the common case and cheapest when the answer is yes: a watermark that disagrees with MAX(id) returns before the two aggregate queries run. The caller is still expected to cache the verdict rather than ask per request.

func (*Reader) SessionCodeChanges 

func (s *Reader) SessionCodeChanges(ctx context.Context, tool, sessionID, project string) (CodeChangeSummary, error)

SessionCodeChanges requires an exact tool and session. An empty project includes every project in that session; a nonempty project is matched exactly.

func (*Reader) SourceStats 

func (s *Reader) SourceStats(ctx context.Context) ([]SourceStat, error)

func (*Reader) Stats 

func (s *Reader) Stats(ctx context.Context) (DBStats, error)

Stats returns whole-database diagnostics for the `doctor` command.

func (*Reader) Summarize 

func (s *Reader) Summarize(ctx context.Context, f Filter) (*Summary, error)

Summarize aggregates usage matching Filter, grouped per Filter.GroupBy. Time dimensions (hour/day/week/month) are bucketed in the local timezone so "today" matches the wall clock; categorical dimensions group by their stored value.

func (*Reader) SummarizeActivity 

func (s *Reader) SummarizeActivity(ctx context.Context, f ActivityFilter) (*ActivitySummary, error)

SummarizeActivity aggregates agent activity (tool calls, skill invocations, hook firings) matching ActivityFilter, grouped per its GroupBy, with tokens and cost ATTRIBUTED from the ledger: each call takes its joined usage row's counts divided by the number of calls that shared that turn. The division is integer and every operand non-negative, so the attributed total over any window is at most the same window's usage_events total — the split can understate, never inflate.

func (*Reader) SummarizeRollup 

func (s *Reader) SummarizeRollup(ctx context.Context, f Filter) (*RollupSummary, error)

SummarizeRollup answers a bucket query from the derived rollup instead of the ledger. It is the fast path behind time-bucketed reporting: identical inputs must yield identical numbers to Summarize over the same range, which TestRollupMatchesLedger pins through store queries on both sides. Callers must first check RollupStale and use Summarize when it is true, or rebuild through EnsureRollup on a writable handle. This method trusts the derived contents and does not validate out-of-band edits.

The public contract remains the one version 4 exposed: Since/Until are snapped OUTWARD to whole UTC buckets (15 minutes), because 

a bucket is the finest thing the table knows. The snapped bounds come back
in the result so a caller can label what it actually got instead of
implying it asked for it.

func (*Reader) SummarizeSkillCost 

func (s *Reader) SummarizeSkillCost(ctx context.Context, f ActivityFilter) (*SkillCostSummary, error)

SummarizeSkillCost is SummarizeTurnContext pinned to the skill dimension. It is a delegation and not a second implementation: the skill partition is not a special case of anything, it is one of five, and giving it its own SQL would be the two-tables mistake wearing a function signature.

func (*Reader) SummarizeTurnContext 

func (s *Reader) SummarizeTurnContext(ctx context.Context, dim model.TurnDimension, f ActivityFilter) (*TurnContextSummary, error)

SummarizeTurnContext aggregates what the turns that ran UNDER each value of ONE dimension actually cost, grouped per ActivityFilter.GroupBy (which accepts "value" and the queried dimension's own name, and refuses the call-level "kind"/"name"). This is the real answer to "which skill/agent is expensive": TopActivity ranks the turn that INVOKED a skill, which is one call, while the skill's own work is every turn that followed under it, and for a subagent there is no invoking call in the ledger at all.

Unlike the activity queries it does NOT divide. Within one dimension each usage event has at most one context and the join is 1:1, so a bucket's cost is the full ledger cost of its turns and the buckets partition the window without overlap. It also counts turns that called no tool at all, which the activity ledger has no row for.

THE DIMENSION IS A REQUIRED ARGUMENT, NOT A FILTER. The five dimensions plus activity_events' tool-call attribution are SIX PARTITIONS OF THE SAME DOLLARS: a turn commonly carries three or four contexts at once, each naming its full cost, so a query spanning two of them reports the same tokens twice. An unknown dimension is refused, grouping by "dimension" is refused, and grouping by a dimension OTHER than the queried one is refused — the mixing is unexpressible rather than discouraged.

func (*Reader) TopActivity 

func (s *Reader) TopActivity(ctx context.Context, f ActivityFilter, by ActivityOrder, limit int) ([]ActivityBucket, error)

TopActivity ranks SummarizeActivity's grouped buckets by one metric and returns at most limit of them (0 = uncapped) — the "which skill is expensive" query, ordered and capped in SQL so a caller never materialises the whole vocabulary to show ten rows. It needs at least one GroupBy dimension: ranking a single grand-total bucket is not a ranking.

func (*Reader) TopSkillCost 

func (s *Reader) TopSkillCost(ctx context.Context, f ActivityFilter, by ActivityOrder, limit int) ([]SkillCostBucket, error)

TopSkillCost is TopTurnContext pinned to the skill dimension. See SummarizeSkillCost.

func (*Reader) TopTurnContext 

func (s *Reader) TopTurnContext(ctx context.Context, dim model.TurnDimension, f ActivityFilter, by ActivityOrder, limit int) ([]TurnContextBucket, error)

TopTurnContext ranks SummarizeTurnContext's buckets by one metric and caps them at limit (0 = uncapped), ordering and limiting in SQL — the real "which skill/agent/server is expensive" query. It needs at least one GroupBy dimension, takes the dimension on the same required-argument terms SummarizeTurnContext does, and ActivityByCalls ranks by turns here.

func (*Reader) UnpricedGroups 

func (s *Reader) UnpricedGroups(ctx context.Context, f Filter) ([]UnpricedGroup, error)

UnpricedGroups returns the token totals of the matching rows that have NO stamped cost, grouped by Filter.GroupBy plus (tool, model, provider, service_tier). It exists so cost surfaces can value historical and unpriceable rows from the CURRENT price table without materialising one bucket per event; an empty result means every matching row is stamped.

The grouping columns are appended AFTER the caller's dimensions so the returned Keys align one-to-one with Summarize's buckets.

type RestoreResult 

type RestoreResult struct {
	SourcePath            string       `json:"source_path"`
	TargetPath            string       `json:"target_path"`
	SafetyBackupPath      string       `json:"safety_backup_path,omitempty"`
	CorruptQuarantinePath string       `json:"corrupt_quarantine_path,omitempty"`
	SnapshotTime          time.Time    `json:"snapshot_time"`
	Verification          Verification `json:"verification"`
	TargetUsable          bool         `json:"target_usable"`
	RolledBack            bool         `json:"rolled_back"`
	EventsMayBeAbsent     bool         `json:"events_may_be_absent"`
}

RestoreResult reports the result of applying a prepared full snapshot.

func ApplyRestore 

func ApplyRestore(ctx context.Context, plan *PreparedRestore, targetPath string, replace bool) (RestoreResult, error)

ApplyRestore replaces the target through SQLite's Online Backup API. The caller must stop collection and hold the collection lock for the whole call.

type RollupSummary 

type RollupSummary struct {
	GroupBy []string
	Buckets []Bucket
	Totals  Bucket
	// Since and Until are the requested bounds snapped OUTWARD to whole UTC
	// 15-minute buckets - the rollup's resolution - so the buckets can cover
	// slightly more than was asked for. Zero means that bound was open. A
	// caller labelling a range must label these, not the ones it passed in.
	Since time.Time
	Until time.Time
}

RollupSummary is the result of SummarizeRollup: the same buckets Summarize produces, from the derived rollup instead of the ledger, plus the range those buckets actually cover.

Schema 8 makes the bucket measures complete, including Sessions and ComputedCostEvents. The distinct type remains for source compatibility and because this method still reports its outward-snapped range explicitly.

type SkillCostBucket 

type SkillCostBucket = TurnContextBucket

SkillCostBucket and SkillCostSummary are the skill-flavoured names for the general types. They are ALIASES, not copies: there is exactly one struct, one query builder and one table behind them, so the two spellings cannot drift into two answers to the same question.

type SkillCostSummary 

type SkillCostSummary = TurnContextSummary

SkillCostBucket and SkillCostSummary are the skill-flavoured names for the general types. They are ALIASES, not copies: there is exactly one struct, one query builder and one table behind them, so the two spellings cannot drift into two answers to the same question.

type SkippedRow 

type SkippedRow struct {
	DedupKey string
	Err      error
}

SkippedRow is one row a batch insert refused, and the reason. DedupKey is the row's own key, which is what makes the report actionable: it names the row in the source the adapter derived it from, and it is empty exactly when the missing key IS the reason.

type SkippedRowsError 

type SkippedRowsError struct {
	// Table is the SQL table the rows were bound for: usage_events,
	// activity_events or usage_turn_context. One batch write can produce one of
	// these per table, and ApplyBatch returns them in ledger order - the usage
	// skip first, since it is the authoritative half - so a caller that reports
	// one line sees the one that matters.
	Table string
	// Total is how many rows were OFFERED, not how many failed; len(Rows) is
	// the failures. The ratio is the useful thing: 1 of 3000 is a poison row, and
	// 3000 of 3000 is an adapter that has started emitting nonsense.
	Total int
	// Rows lists the skipped rows in the order they were offered.
	Rows []SkippedRow
}

SkippedRowsError is the PARTIAL SUCCESS this package's batch writes can return: a non-nil error accompanied by counts that are still true (issue #72, decision 6). It is the one genuinely unusual contract here and so it is the one shape with a name.

Why a batch does not fail whole: a row that cannot be inserted (a CHECK violation, an empty dedup key) is a PERMANENT property of that row, and the source it came from is re-read every cycle. Aborting the transaction would discard the good rows beside it and then re-derive exactly the same poison row on the next pass, forever - so the bad rows are skipped, the rest commits, and the checkpoint advances past all of it. Only these known row violations may skip. Every other insert or commit error rolls the batch back and returns zero counts. ApplySnapshot also rolls back rejected delta rows and never returns SkippedRowsError.

The consequence for a caller is the part worth stating: WHEN THIS ERROR IS RETURNED, THE COUNTS RETURNED WITH IT ARE REAL. InsertEvents' int and ApplyBatch's Applied describe rows that actually landed, and a caller that treats a non-nil error as "nothing happened" will under-report a pass it should be reporting in full. Read it with errors.As: 

applied, err := st.ApplyBatch(ctx, batch)
var skipped *store.SkippedRowsError
if errors.As(err, &skipped) {
	log.Printf("%d of %d %s rejected; %d events landed",
		skipped.Skipped(), skipped.Total, skipped.Table, applied.Events)
}

Unwrap returns the FIRST row's error, so errors.Is against whatever the driver produced still works, on the same row the message names. The rest are in Rows.

func (*SkippedRowsError) Error 

func (e *SkippedRowsError) Error() string

func (*SkippedRowsError) Skipped 

func (e *SkippedRowsError) Skipped() int

Skipped is len(Rows), named because "how many were rejected" reads better at a call site than a length.

func (*SkippedRowsError) Unwrap 

func (e *SkippedRowsError) Unwrap() error

Unwrap exposes the first skipped row's cause, so errors.Is reaches the driver error or the CHECK violation behind it.

type SourceStat 

type SourceStat struct {
	Tool         string
	Models       []string
	Sessions     int64
	Events       int64
	Total        int64
	FirstEvent   time.Time
	LastEvent    time.Time
	LastObserved time.Time
}

SourceStat summarises stored usage per tool for the `sources` command.

type Summary 

type Summary struct {
	GroupBy []string
	Buckets []Bucket
	Totals  Bucket
}

Summary is the result of Summarize: grouped buckets plus a grand total.

type TurnContextBucket 

type TurnContextBucket struct {
	// Keys maps each GroupBy dimension to its value for this bucket
	// (e.g. {"value":"adhd"}). Ordered via OrderedKeys.
	Keys        map[string]string
	OrderedKeys []string

	// Turns counts usage rows that ran under this context — one per provider
	// record, not per tool call.
	Turns int64
	// Sessions is the distinct non-empty session count within the group.
	// Distinct counts do not add across buckets.
	Sessions int64

	InputTokens  int64
	OutputTokens int64
	TotalTokens  int64
	// CostMicroUSD sums the costs STAMPED on the joined usage rows. Turns whose
	// usage row carries no stamped cost contribute nothing and are counted in
	// UnpricedTurns instead, so a bucket with UnpricedTurns > 0 is an
	// understatement until those are display-priced.
	CostMicroUSD int64

	// UnjoinedTurns counts context rows whose usage row is absent from the
	// ledger — a poison row the insert skipped, or a ledger pruned under the
	// context. It should be zero in practice, since a context is only ever
	// emitted alongside an accepted usage event, and it is reported rather than
	// assumed because a silent zero would be indistinguishable from free.
	UnjoinedTurns int64
	// UnpricedTurns counts turns that DID join a usage row carrying no stamped
	// cost (collected before v3, or a model the price ladder could not price).
	// Their tokens are counted; their cost is not.
	UnpricedTurns int64
	// ComputedCostTurns counts turns whose joined usage row was priced from a
	// public rate card rather than by the harness (model.PriceProvenance). Zero
	// means every dollar in CostMicroUSD traces to a vendor's own number.
	ComputedCostTurns int64
}

TurnContextBucket is one grouped row of turn-context cost along ONE dimension.

The token and cost fields carry NO "Attributed" prefix, and the omission is deliberate: unlike ActivityBucket's divided shares, these are the joined usage rows' FULL counts. Within the queried dimension each turn belongs to exactly one bucket, so no share is taken and none is lost. A bucket's cost is the real ledger cost of the turns that ran under that value, not an estimate of it.

type TurnContextSummary 

type TurnContextSummary struct {
	// Dimension is the axis these buckets partition. It is echoed back because
	// a summary is meaningless without it: two summaries on different
	// dimensions cover the same dollars and must never be concatenated.
	Dimension model.TurnDimension
	GroupBy   []string
	Buckets   []TurnContextBucket
	Totals    TurnContextBucket
}

TurnContextSummary is the result of SummarizeTurnContext: grouped buckets plus a grand total, both scoped to the single dimension the query named.

type UnpricedGroup 

type UnpricedGroup struct {
	Keys        map[string]string
	OrderedKeys []string
	Tool        string
	Model       string
	Provider    string
	ServiceTier string

	Events        int64
	Input         int64
	Output        int64
	CacheCreation int64
	CacheRead     int64
	Reasoning     int64
}

UnpricedGroup aggregates the rows of one bucket that carry NO stamped cost, split by the attributes a price lookup needs (tool for the reasoning-billing rule, model/provider for the rate, service tier for the tier rate). Keys repeats the Filter.GroupBy dimension values so a caller can fold the display-priced result back into the matching Summarize bucket.

type Verification 

type Verification struct {
	Path                string            `json:"path"`
	State               VerificationState `json:"state"`
	Reason              string            `json:"reason,omitempty"`
	SchemaVersion       int               `json:"schema_version"`
	Compatible          bool              `json:"compatible"`
	IntegrityOK         bool              `json:"integrity_ok"`
	ForeignKeysOK       bool              `json:"foreign_keys_ok"`
	ApplicationSchemaOK bool              `json:"application_schema_ok"`
	// RollupChecked covers both derived usage rollup and activity counts.
	RollupChecked bool `json:"rollup_checked"`
	// RollupStale is true when either derived table disagrees with its ledger.
	RollupStale      bool             `json:"rollup_stale"`
	IntegrityErrors  []string         `json:"integrity_errors,omitempty"`
	ForeignKeyErrors []string         `json:"foreign_key_errors,omitempty"`
	SchemaErrors     []string         `json:"schema_errors,omitempty"`
	RowCounts        map[string]int64 `json:"row_counts,omitempty"`
}

Verification describes a complete read-only check of one database.

func Verify 

func Verify(ctx context.Context, path string) (Verification, error)

Verify checks an existing database without creating, migrating, repairing, checkpointing, or changing permissions.

func (Verification) StateError 

func (v Verification) StateError() error

StateError turns a completed non-ok verification into an error suitable for a command exit. Newer schemas preserve ErrSchemaNewer for callers that branch on it.

type VerificationState 

type VerificationState string

VerificationState is the stable result of a read-only database check. 

const (
	VerificationOK           VerificationState = "ok"
	VerificationRepairable   VerificationState = "repairable"
	VerificationIncompatible VerificationState = "incompatible"
	VerificationCorrupt      VerificationState = "corrupt"
)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.