gitcommons

Documentation

Overview

Package gitcommons materializes the public abstract tier of the knowledge commons as a git-native tree (ADR-0001, quarry-commons): content-addressed artifact abstracts, a prefix-sharded behavioral-key index, and a Bloom-filter root digest for cheap "novel or prior-art?" existence checks before any fetch.

git is transport + dedup + distribution; the in-Envelope ed25519 signature stays the root of trust; retrieval is never authoritative (a hit is a candidate the client re-verifies locally, a miss degrades to "assume novel").

Index

• type Bloom
  • func NewBloom(n int, p float64) *Bloom
  • func UnmarshalBloom(data []byte) (*Bloom, error)
  • func (b *Bloom) Add(key string)
  • func (b *Bloom) Marshal() []byte
  • func (b *Bloom) Test(key string) bool
• type Manifest
• type Source
  • func Open(dir string) (*Source, error)
  • func (s *Source) Lookup(_ context.Context, keys []string) ([]channels.PriorArt, error)
• type Stats
  • func Add(dir string, envs []*artifact.Envelope) (Stats, error)
  • func Generate(dir string, envs []*artifact.Envelope) (Stats, error)
• type VerifyReport
  • func Verify(dir string) (VerifyReport, error)
  • func (r VerifyReport) OK() bool

Types

type Bloom

type Bloom struct {
	// contains filtered or unexported fields
}

Bloom is the root digest: a consumer pulls only this (kB) and answers existence locally, sparse-fetching a key shard only on a probable hit. No false negatives (a miss is a definitive "novel"); the false-positive rate is bounded at build.

func NewBloom

func NewBloom(n int, p float64) *Bloom

NewBloom sizes a filter for n keys at false-positive rate p (0<p<1) and returns an empty filter ready for Add.

func UnmarshalBloom

func UnmarshalBloom(data []byte) (*Bloom, error)

UnmarshalBloom parses a serialized digest.

func (*Bloom) Add

func (b *Bloom) Add(key string)

Add inserts a key.

func (*Bloom) Marshal

func (b *Bloom) Marshal() []byte

Marshal serializes the filter: magic + n + m + k (big-endian u32) + bitset.

func (*Bloom) Test

func (b *Bloom) Test(key string) bool

Test reports whether a key MAY be present. false is definitive (never present); true is probabilistic (bounded false-positive rate).

type Manifest

type Manifest struct {
	Schema    string  `json:"schema"`    // "quarry-commons/v1"
	Prefix    int     `json:"prefix"`    // shard prefix bytes
	Keys      int     `json:"keys"`      // distinct behavioral keys
	Artifacts int     `json:"artifacts"` // content-addressed abstracts
	BloomFP   float64 `json:"bloom_fp"`  // digest false-positive target
	DigestB   int     `json:"digest_bytes"`
}

Manifest is the tree's self-description (deterministic, no timestamp) so a reader learns the shard prefix without probing the layout. Written to commons.json at the tree root.

type Source

type Source struct {
	// contains filtered or unexported fields
}

Source is the git-native down-query channel: it reads a pulled quarry-commons tree and answers behavioral-key lookups locally — no network, no service. It implements channels.PatternSource and is the loop's only commons source now that the Worker query client is retired (ADR-0001).

The query is: Bloom digest as the existence pre-filter (a miss is a definitive "novel"), then resolve each probable-hit key to its artifact ids via the ONE shard it lives in, then read the content-addressed abstracts. Retrieval is not authoritative (ADR-0001): every hit is a candidate the caller re-grounds against its own oracle-confirmed PoV.

CHECKOUT ASSUMPTION: this reader operates over a checked-out tree on the local filesystem — Open reads commons.json + the digest, and resolve() reads shard and artifact files directly. The thin-client promise (pull only the digest, then sparse-fetch the ONE shard a probable hit needs) requires the shard to already be present: either a full checkout, a sparse-checkout pre-sliced by the shard paths Lookup will touch, or a fetch hook the caller wires in front of resolve(). If a shard is absent, resolve() returns no ids and the key reads as "novel" — the safe direction (never a false merge), but for a partial checkout that is a false negative, so a thin client MUST ensure the shard is fetched before trusting a "novel" result. Wiring an on-demand fetch hook is a deliberate TODO, not done here.

func Open

func Open(dir string) (*Source, error)

Open loads a quarry-commons tree at dir: its manifest (for the shard prefix) and the Bloom root digest. Shards are read lazily, on a probable hit.

func (*Source) Lookup

func (s *Source) Lookup(_ context.Context, keys []string) ([]channels.PriorArt, error)

Lookup returns the prior art matching the union of a crash's multi-resolution keys. Bloom-negative keys are skipped without touching the tree.

type Stats

type Stats struct {
	Artifacts   int
	Keys        int // distinct behavioral keys
	Entries     int // (key, artifact_id) rows across all shards
	Shards      int
	Prefix      int // shard prefix length in bytes
	Views       int // relevance-view files (by-class + by-project)
	DigestBytes int
	TreeBytes   int
}

Stats reports what a Generate wrote — the numbers that make the tree auditable.

func Add

func Add(dir string, envs []*artifact.Envelope) (Stats, error)

Add incrementally merges new public abstract envelopes into an existing tree at dir: it writes only new (content-addressed) artifacts, merges new rows into the affected shards, unions new ids into the affected relevance views, and rebuilds the root digest — no full re-materialize, so it touches O(new) files, not O(tree).

FP-resize / reshard caveat: the Bloom is always rebuilt from the merged key set, so the digest stays byte-identical to what Generate/Verify expect and its size tracks the new total (no stale false-positive rate). If the addition would cross a shardPrefix boundary — the whole index must re-shard — Add transparently falls back to a full Generate over the union of on-disk + new envelopes.

Add is additive only: it never removes artifacts. Use Generate to reflect removals.

func Generate

func Generate(dir string, envs []*artifact.Envelope) (Stats, error)

Generate materializes the git-native commons tree at dir from public abstract envelopes: content-addressed artifacts, a prefix-sharded behavioral-key index (sorted + deterministic so commits are reproducible and merges are conflict-free), and the Bloom root digest. It does NOT git-init/commit — the caller decides.

type VerifyReport

type VerifyReport struct {
	Artifacts int
	Keys      int
	Failures  []string
}

VerifyReport is the outcome of auditing a quarry-commons tree.

func Verify

func Verify(dir string) (VerifyReport, error)

Verify audits a quarry-commons tree — the anti-poisoning gate the CI Action runs on every PR. The gate is at least as strong as the consumers that trust its output: nothing is believed just because it is in the tree.

• Every artifact is content-addressed (id == hash(content)), the file path is the content-addressed path for that id, it is abstracts-only (no specimen, no reproducer), it is Public-tier, and it satisfies Envelope.Verify (signed-iff- non-reproducing, signature valid).
• The key index is bound to crash content: every shard row's key is actually one of the referenced artifact's behavioral keys (artifact.CrashKeys), and it sits in the shard the manifest prefix routes it to — so a row can neither be mapped to an unrelated artifact (index poisoning) nor hidden in the wrong shard.
• The manifest (commons.json) agrees with the tree: its prefix is the canonical prefix for the key count, and its Keys/Artifacts counts match reality — the consumer trusts manifest.Prefix for every shard resolution.
• The root digest is not merely a superset: it is byte-identical to the Bloom deterministically regenerated from the verified key set, so a forged all-bits digest (which would defeat "a miss is definitively novel") cannot pass.

It returns a report; a non-empty Failures list means the tree must not merge.

func (VerifyReport) OK

func (r VerifyReport) OK() bool

OK reports whether the tree passed with no failures.