logvault

package
v0.44.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 18, 2026 License: Apache-2.0 Imports: 22 Imported by: 0

Documentation

Overview

Package logvault captures fak's durable logs — the guard decision journals, the harness session stores, the dispatch/dos/loop ledgers — into one central vault directory, incrementally and tamper-evidently.

The vault layout is by-source first, original relative paths beneath, so a restore is a copy back: 

<vault>/vault-manifest.jsonl                 — this package's own hash-chained ledger
<vault>/by-source/<source-id>/<rel-path>     — current mirror of each captured file
<vault>/by-source/<source-id>/.history/...   — superseded versions of rewritten files

Every capture appends chained rows to the manifest; the manifest replay is the incremental state (no separate state file), and Verify re-derives both the chain and the mirror hashes. The chain construction mirrors internal/journal (sha256 over 0x1f-delimited fields in declaration order); the row schema is logvault's own because journal.Row's pre-image is frozen to guard semantics.

Index

Constants

View Source 
const (
	OpFull    = "capture-full"
	OpAppend  = "capture-append"
	OpRewrite = "capture-rewrite"
	OpTouch   = "capture-touch" // content verified unchanged, mtime advanced
	OpSkip    = "skip-error"
)

Op names for manifest rows and plan lines.

View Source 
const (
	OpSyncCopy = "sync-copy" // one scrubbed copy of a source-vault mirror
	OpSyncMark = "sync-mark" // end-of-pass row binding the destination chain to the source chain head
)

Op names for receiving-manifest rows written by SyncTo.

View Source 
const AnchorName = "vault-head.json"

AnchorName is the out-of-band chain-head sidecar: {seq, hash} of the last completed capture, rewritten atomically after each one. Verify cross-checks it so a truncated or deleted manifest cannot silently verify clean.

View Source 
const ColdParkSourceID = "cold-park"

ColdParkSourceID is the synthetic source id under which cold-park archives are banked (by-source/cold-park/<name>). It is not a capture source — nothing re-walks it; the archive is a one-shot mirror.

View Source 
const DefaultScratchpadCapBytes = 256 << 20

DefaultScratchpadCapBytes bounds the opt-in scratchpad tier. The %TEMP%/claude tree is multi-GB (2.7 GB / 77k files witnessed 2026-07-03), new-dir-per-session and mostly cold within a day, so the opt-in defaults to an aggressive 256 MiB cap — enough to bank a handful of recent sessions' kept artifacts, never the whole re-derivable tree.

View Source 
const DrillLogName = "drill-log.jsonl"

DrillLogName is the vault-root ledger every drill run appends its row to — the durable "the restore path was exercised on <date> and passed/failed" record a cadence runner leaves behind.

View Source 
const DrillSchema = "fak-logvault-drill/1"

DrillSchema stamps every drill-log row.

View Source 
const LockName = "vault.lock"

LockName is the vault's single-writer capture lock file.

View Source 
const ManifestName = "vault-manifest.jsonl"

ManifestName is the manifest's file name inside the vault root.

View Source 
const OpColdAdopt = "cold-adopt"

OpColdAdopt is the manifest op stamped when a tree is adopted as a cold-park archive. It carries the archive's own sha, so Verify re-hashes the banked archive exactly like a captured mirror.

View Source 
const OpGCPrune = "gc-prune"

OpGCPrune is the manifest op stamped when a retention pass reclaims one superseded .history/ version. Like skip-error it carries no capture SHA, so replayStates never folds it into mirror state — the chain simply witnesses its own pruning, and Verify keeps re-deriving cleanly across it.

Variables

This section is empty.

Functions

func NewestCaptureUnixNano 

func NewestCaptureUnixNano(fps []SourceFootprint) int64

NewestCaptureUnixNano returns the newest SUCCESSFUL capture time across every source in the footprint (0 when nothing has ever been captured) — the single "last successful capture" anchor a scalar surface (a /metrics gauge) reports for the whole vault. Skip-error-only sources contribute 0 and never move it.

func ReadAnchor 

func ReadAnchor(vaultDir string) (seq uint64, hash string, ok bool, err error)

ReadAnchor is the exported form of the chain-head sidecar read: the (seq, hash) an off-box witness (e.g. a Slack digest) can quote as tamper-evidence for a vault it never reads the content of. ok is false when the vault has never completed a capture (no anchor written yet).

func RenderLogvaultGauges 

func RenderLogvaultGauges(nowUnixNano, lastCaptureUnixNano, vaultBytes int64, verifyMismatches int, chainBroken bool) string

RenderLogvaultGauges renders the three fak_logvault_* gauges in Prometheus/ OpenMetrics text form from already-computed observability values. It is PURE (no I/O, no clock): callers supply the footprint fold (NewestCaptureUnixNano / TotalBytes), the verify outcome, and the scrape clock. Per the conflation law each gauge's HELP text declares the value WITNESSED — fak computed it from its own hash-chained manifest and a mirror re-hash, never a self-reported counter — so a scrape can never mistake it for an unverified live number.

func TotalBytes 

func TotalBytes(fps []SourceFootprint) int64

TotalBytes sums the current tracked footprint across every source — the vault's logical current-mirror size (excluding retired .history/ versions).

func VerifyManifest 

func VerifyManifest(path string) (int, error)

VerifyManifest re-derives the chain and returns the row count, or an error naming the first broken link.

func WriteAnchor 

func WriteAnchor(vaultDir string, seq uint64, hash string) error

WriteAnchor atomically records the chain head sidecar for vaultDir.

Types

type AdoptReport 

type AdoptReport struct {
	SrcDir      string // the tree that was adopted
	ArchiveRel  string // vault-relative path of the banked archive, forward slash
	ArchivePath string // absolute on-disk path of the banked archive
	SHA256      string // sha256 of the archive bytes (content address)
	Bytes       int64  // archive size
	Files       int    // regular files packed
	Deduped     bool   // an identical archive was already banked (no re-write, no new row)
	DeleteCmd   string // the command an OPERATOR may run to delete the original — never run by the tool
}

AdoptReport is the outcome of a cold-park adoption.

type DrillRow 

type DrillRow struct {
	Schema          string `json:"schema"`
	TSUnixNano      int64  `json:"ts_unix_nano"`
	Vault           string `json:"vault"`
	Source          string `json:"source"`
	HeadSeq         uint64 `json:"head_seq"`
	Files           int    `json:"files"`
	Bytes           int64  `json:"bytes"`
	FromHistory     int    `json:"from_history"`
	Mismatches      int    `json:"mismatches"`
	JournalsChecked int    `json:"journals_checked"`
	JournalsFailed  int    `json:"journals_failed"`
	Pass            bool   `json:"pass"`
	Err             string `json:"err,omitempty"` // a restore that refused outright (vs. one that ran and found problems)
}

DrillRow is one drill run's durable record.

type GCCandidate 

type GCCandidate struct {
	Source   string // source id (the by-source/<id> directory name)
	RelPath  string // the ORIGINAL file's forward-slash rel path (not the .history slot name)
	HistFile string // vault-relative path of the .history/ slot file, forward slash
	SHA16    string // the 16-hex content slot in the slot file name
	Bytes    int64  // bytes the prune reclaims
}

GCCandidate is one .history/ version a retention pass would prune (propose) or did prune (live).

type GCPolicy 

type GCPolicy struct {
	// HistoryDepth is the maximum number of superseded versions to KEEP per
	// (source, rel-path) in .history/. Zero (or negative) means unlimited —
	// nothing is ever proposed for prune (the fail-safe default: keep everything).
	HistoryDepth int
}

GCPolicy is the retention policy a pass enforces. It is intentionally small: bounded .history/ depth is the whole lever.

type GCReport 

type GCReport struct {
	Policy        GCPolicy      // the policy this pass applied
	Candidates    []GCCandidate // proposed (or, when Applied, the pruned set), deterministic order
	ReclaimBytes  int64         // total bytes across Candidates
	SkipErrorRows int           // advisory: skip-error rows in the manifest (noise — never deleted; the chain is append-only)
	Applied       bool          // true only under an explicit --live grant
}

GCReport is the outcome of a retention pass.

type JournalCheck 

type JournalCheck struct {
	RelPath string
	Kind    string // "decision-journal" (internal/journal) | "usage-log" (internal/usagelog)
	Rows    int    // sound rows the verifier counted
	Err     string // "" = chain intact end-to-end
}

JournalCheck is the end-to-end verifier verdict for one restored chained journal: its own verifier re-run against the restored copy.

type Manifest 

type Manifest struct {
	// contains filtered or unexported fields
}

Manifest is the vault's append-only chained ledger.

func OpenManifest 

func OpenManifest(vaultDir string) (*Manifest, error)

OpenManifest opens (creating if absent) the vault manifest in append mode, recovering the chain head from existing rows so a new capture CONTINUES the chain. A torn final line (crash mid-append) is TRUNCATED before reopening — appending after partial bytes would merge the next row into one unparseable line and permanently fail verification on an untampered vault.

func (*Manifest) Append 

func (m *Manifest) Append(row ManifestRow) (ManifestRow, error)

Append stamps the order anchor + chain hash and commits the row.

func (*Manifest) Close 

func (m *Manifest) Close() error

Close flushes, fsyncs, and closes the underlying file.

func (*Manifest) Head 

func (m *Manifest) Head() (uint64, string)

Head returns the current chain head (last committed seq + hash).

type ManifestRow 

type ManifestRow struct {
	Seq        uint64 `json:"seq"`             // monotonic 1-based order anchor
	TSUnixNano int64  `json:"ts_unix_nano"`    // wall-clock time anchor
	Op         string `json:"op"`              // capture-full | capture-append | capture-rewrite | capture-touch | skip-error
	Source     string `json:"source"`          // registry source id
	RelPath    string `json:"rel_path"`        // forward-slash path relative to the source root
	Bytes      int64  `json:"bytes"`           // bytes written to the vault by this op
	SizeAfter  int64  `json:"size_after"`      // source byte position the hash covers at capture time
	MTimeNano  int64  `json:"mtime_unix_nano"` // source file mtime at capture time (nanoseconds — second granularity misses same-second truncate-replaces)
	SHA256     string `json:"sha256"`          // full-content hash of the source at capture ("" on skip-error)
	Note       string `json:"note,omitempty"`
	PrevHash   string `json:"prev_hash"` // hash of the previous row ("" at genesis)
	Hash       string `json:"hash"`      // manifestChainHash(PrevHash, this row)
}

ManifestRow is one durable capture record. Field order up to Note is the hash-chain pre-image order — do not reorder without bumping the chain.

func ReadManifestRows 

func ReadManifestRows(path string) ([]ManifestRow, error)

ReadManifestRows reads every well-formed row; a missing file yields an empty history and a torn/corrupt final line is skipped (the tolerant consumer read).

type RestoreOptions 

type RestoreOptions struct {
	Source string // source id (a by-source/<id> subtree) — required
	To     string // target directory — required (callers default it to a FRESH dir)
	At     uint64 // reconstruct the state as of this manifest seq (0 = current head)
	Force  bool   // allow restoring into a non-empty existing directory (never the default)
}

RestoreOptions selects what to restore and where.

type RestoreProblem 

type RestoreProblem struct {
	RelPath string
	Reason  string
}

RestoreProblem is one file the restore could not prove: a hash mismatch, an unrestorable historical state, or a refused path. The acceptance bar for a sound restore is an EMPTY problem list.

type RestoreReport 

type RestoreReport struct {
	Source      string
	To          string
	HeadSeq     uint64 // the manifest seq the restore replayed to (At, clamped to the head)
	Files       int    // files restored AND re-hash-verified against the chain
	Bytes       int64  // bytes written to the target
	FromHistory int    // files reconstructed from .history/ retires rather than the current mirror
	Problems    []RestoreProblem
	Journals    []JournalCheck
}

RestoreReport folds one restore's outcome.

func (RestoreReport) JournalFailures 

func (r RestoreReport) JournalFailures() int

JournalFailures counts restored chained journals whose own verifier refused.

func (RestoreReport) OK 

func (r RestoreReport) OK() bool

OK reports the acceptance condition: zero hash mismatches/problems and every restored chained journal verifying clean.

type Source 

type Source struct {
	ID       string   // vault subdirectory name, e.g. "guard-audit"
	Root     string   // absolute root on this box
	Includes []string // when non-empty, ONLY matching files are captured (same syntax as Excludes)
	Excludes []string // rel-path prefixes ("tmp/") or base-name globs ("*.exe") to skip
	MaxBytes int64    // when >0, an aggressive cap: the walk stops admitting files once cumulative admitted size reaches it (bounds an opt-in tier like the scratchpad)
	Note     string   // why this source matters (shown by plan/du)
}

Source is one root the vault captures. Every source is optional: a missing root is a valid empty source, never an error (the `fak audit usage` posture — a box that never ran the dispatcher simply has no .dispatch-runs).

func AllProjectSources 

func AllProjectSources(repoRoot, home string) []Source

AllProjectSources returns one harness-store-<slug> source per Claude Code project directory under ~/.claude/projects, EXCLUDING this repo's own store — DefaultSources already captures that as the canonical "harness-store" source, and a second source over the same directory would duplicate the fak project store into the vault. This is the `--all-projects` expansion: every other repo the harness touched (transcripts + auto-memory) rides along, each under its own by-source/harness-store-<slug>/ subtree.

A missing ~/.claude/projects (or an empty home) is the valid-empty posture: no extra sources, never an error.

func DefaultSources 

func DefaultSources(repoRoot, home string) []Source

DefaultSources is the registry of every durable log store discovered on a fak box (writer inventory 2026-07-03): the hash-chained guard/usage/loop journals, the DOS trust-kernel state, dispatch run logs, the Claude Code harness store, and the per-user fak state dir. Scratch trees that dominate the on-disk size but are re-derivable (.fak/tmp checkouts, .dos/_dos_park) are excluded.

func ScratchpadSource 

func ScratchpadSource(tempDir string, capBytes int64) Source

ScratchpadSource returns the opt-in scratchpad tier: the harness's per-session working dirs under %TEMP%/claude. It is DEFAULT-EXCLUDED (only layered in when the caller opts in) because the tree is re-derivable working files, and it is bounded by an aggressive cumulative-size cap (capBytes, or DefaultScratchpadCapBytes when non-positive) so the opt-in can never pull the whole multi-GB tree into the vault.

type SourceFootprint 

type SourceFootprint struct {
	Source              string // registry source id
	Files               int    // distinct files currently tracked in the vault (skip-error rows do not count)
	ManifestRows        int    // total manifest rows naming this source (capture ops + skip-errors)
	Bytes               int64  // current vault footprint: sum of the latest witnessed SizeAfter per tracked file (current mirrors, excluding .history/)
	Errors              int    // skip-error rows for this source (advisory: a file that could not be read this or a prior cycle)
	LastCaptureUnixNano int64  // newest TSUnixNano over this source's SUCCESSFUL capture rows (0 = never successfully captured)
}

SourceFootprint is one source's observability rollup, folded PURELY from the vault manifest. Every field is a WITNESSED value fak computed from its own hash-chained record — the sizes were stat'd and the hashes computed at capture time and recorded in the chain — never a self-reported live counter. This is the "is my backup current?" answer surfaced by `fak logvault du` and the audit-usage vault section (#2455).

func Footprint 

func Footprint(rows []ManifestRow) []SourceFootprint

Footprint folds manifest rows into a per-source observability rollup, sorted by source id for a stable render. It is PURE: no I/O and no clock read — the caller supplies the already-read rows (ReadManifestRows) and applies its own clock for the capture-age subtraction, matching the audit-usage honesty fence (every disk/clock read stays in the CLI shell).

LastCaptureUnixNano tracks only SUCCESSFUL ops, so a source whose most recent cycle only recorded skip-errors keeps the timestamp of its last good capture — the honest "when was the last SUCCESSFUL capture" answer the acceptance asks for. Bytes and Files reflect the current tracked mirrors (skip-errors advance neither); Errors counts every skip-error row so a silent read failure surfaces.

type SourceStats 

type SourceStats struct {
	Source    string
	Files     int // files examined (after excludes)
	Unchanged int
	Full      int
	Append    int
	Rewrite   int
	Errors    int
	CopyBytes int64 // bytes a capture would write / did write to the vault
	Missing   bool  // source root absent on this box (valid-empty)
}

SourceStats folds one source's outcome for a plan or capture pass.

type SyncStats 

type SyncStats struct {
	Files         int   // source mirrors considered (the manifest-replayed current state)
	Copied        int   // scrubbed copies written this pass
	Unchanged     int   // source versions the destination already held
	Errors        int   // mirrors refused/unreadable (recorded as skip-error rows, never shipped)
	Redacted      int   // scrub spans redacted across the shipped bytes
	CopyBytes     int64 // scrubbed bytes written to the destination
	VerifyRows    int   // receiving-side manifest chain rows verified on arrival
	VerifyChecked int   // receiving-side mirrors re-hashed on arrival
}

SyncStats folds one sync pass's outcome, including the receiving-side verify that SyncTo always runs (the fail-closed arrival check).

type Vault 

type Vault struct {
	Dir     string
	Sources []Source
}

Vault is a capture session against one vault directory.

func (*Vault) AdoptCold 

func (v *Vault) AdoptCold(srcDir string) (AdoptReport, error)

AdoptCold packs srcDir into one deterministic archive banked in the vault and witnessed by a manifest row. It reads srcDir but never mutates or deletes it. Re-adopting byte-identical content resolves to the same content address and dedups: the existing archive is kept and no duplicate row is appended.

func (*Vault) Capture 

func (v *Vault) Capture() ([]SourceStats, error)

Capture copies every new/changed source file into the vault and appends one chained manifest row per operation. Sources are read-only: files are opened for read and never locked; a file that cannot be read (e.g. a Windows sharing violation) is recorded as a skip-error row and retried next capture. The vault itself is single-writer: a cross-process lock serializes captures so two runs cannot interleave manifest rows and fork the chain.

func (*Vault) Drill 

func (v *Vault) Drill(source, ledgerPath string) (DrillRow, RestoreReport, error)

Drill restores one source into a fresh temp directory, verifies it (re-hash against the chain + chained-journal verifiers), appends one DrillRow to the vault's drill-log (and to ledgerPath when non-empty — e.g. a committed repo ledger), and removes the temp tree. source == "" picks the smallest captured source, so a cadence run stays cheap by default. A failed restore still journals its row — the drill's whole point is that a rotten restore path becomes a recorded red, not a silent skip.

func (*Vault) GC 

func (v *Vault) GC(pol GCPolicy, live bool) (GCReport, error)

GC runs a retention pass. With live=false (the default) it PROPOSES: it walks the vault, computes what would be pruned, and returns the report without touching a single byte. With live=true it deletes the proposed slot files and appends one OpGCPrune manifest row per deletion, all under the vault's single-writer lock. Fail-closed: an empty candidate set never takes the lock.

func (*Vault) MetricsText 

func (v *Vault) MetricsText(verifySample int, nowUnixNano int64) (string, error)

MetricsText is the /metrics provider core: it reads the manifest, folds the footprint (Footprint), runs a bounded Verify (verifySample mirrors re-hashed, 0 = all — the same knob `fak logvault verify -sample` uses), and renders the three #2455 gauges. nowUnixNano anchors the last-capture age; pass the scrape-time clock.

A missing or empty vault renders the family with a -1 "never captured" age and zero footprint/mismatches — the valid-empty posture, never an error. A broken manifest chain renders as chainBroken (>=1 mismatch), NOT an error: the whole point is to make "this vault is NOT intact" scrapeable. Only a manifest that exists but cannot be read returns an error.

func (*Vault) Plan 

func (v *Vault) Plan() ([]SourceStats, error)

Plan diffs the live sources against the manifest replay without copying or hashing. Grown files are counted as appends and shrunk/touched files as rewrites optimistically; Capture makes the real (hash-checked) decision.

func (*Vault) Restore 

func (v *Vault) Restore(opts RestoreOptions) (RestoreReport, error)

Restore copies one source's replayed state out of the vault into opts.To, re-hashing every restored byte against the manifest chain and re-running the chained-journal verifiers over restored journals. Fail-closed: a vault whose chain does not verify refuses before any byte is copied, a target that overlaps the vault is always refused, and a non-empty target needs an explicit Force grant. The vault is read-only throughout.

func (*Vault) SyncTo 

func (v *Vault) SyncTo(dstDir string, sample int) (SyncStats, []VerifyProblem, error)

SyncTo replicates the vault's replayed current state into dstDir (a second vault directory — another disk first, a remote box via a file transport later), gating every outbound byte through the redaction scrub and proving chain integrity on arrival. sample bounds the receiving-side mirror re-hash exactly as Verify does (0 = all).

The returned problems are the receiving-side verify findings: a green sync returns (stats, nil, nil). Fail-closed contract: a source vault whose chain does not verify, or a destination that overlaps the source, refuses before any byte is copied.

func (*Vault) Verify 

func (v *Vault) Verify(sample int) (chainRows int, checked int, problems []VerifyProblem, err error)

Verify re-derives the manifest chain, cross-checks it against the head anchor (so a truncated or deleted manifest cannot verify clean), then re-hashes mirror files against the replayed state. sample bounds how many mirrors are re-hashed (0 = all), chosen by deterministic stride so repeated runs cover the same set.

Honesty note: the chain + anchor catch corruption, truncation, and casual edits. They are not proof against an adversary with full write access to the vault, who can recompute hashes and rewrite the anchor — that requires an off-vault anchor (the off-box replication rung).

func (*Vault) WitnessedFiles added in v0.42.0 

func (v *Vault) WitnessedFiles(sourceID string) (map[string]string, error)

WitnessedFiles returns the current, independently re-hashed mirror state for one source. The manifest chain and head anchor are verified before any state is returned; every returned digest was then re-derived from the mirror bytes. It shares the vault writer lock so capture/GC cannot race the read-back.

type VerifyProblem 

type VerifyProblem struct {
	Source  string
	RelPath string
	Reason  string
}

VerifyProblem is one mirror that fails re-verification against the manifest.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.