evidencegcs

README

providers/evidence-gcs/ — GCS-backed WORM evidence Store

Trust tier: reference provider implementation (runs in the Tier-1 control plane; holds no key at rest).

Reference durable backing for the Console7 evidence log on Google Cloud Storage. It implements control-plane/evidence.Store — the narrow, append-only persistence seam the real EvidenceSink commits hash-chained, sink-signed records through (ARCHITECTURE.md §5; DESIGN.md §6). In-tree reference implementation; community providers live out-of-tree against the published SDK.

Note on the seam. control-plane/evidence already is the real EvidenceSink (append-only, hash-chained, checkpoint-signing, fail-closed). It owns the integrity layer and persists every sealed Entry through a Store. This package is that Store — the durable WORM backing — not a second EvidenceSink.

What it upholds

• Append-only / WORM — at two trust levels. Each committed Entry is one GCS object at <prefix>/<zero-padded-sequence>.
  • Against the append identity (the workload SA: create/get/list, no delete): GCS requires storage.objects.delete to overwrite as well as to delete an object, so the append path can neither overwrite nor remove a committed record. The DoesNotExist precondition and the no-delete-path in the package are in-band defence-in-depth on top.
  • Against a privileged actor (the deploy identity can delete objects and remove an unlocked retention policy): the authoritative control is the bucket's retention policy + lock (deploy/gcp/modules/evidence) — the boundary control of record (GOAL.md tenet 3; tenet 7). The lock is off by default, so the shipped default is tamper-evident (the Sink's signed hash-chain detects mutation/truncation), not tamper-resistant against a privileged actor; production must set is_locked=true.
• Integrity in both directions. Every write carries a CRC32C the server verifies; every read re-verifies the bytes against the stored CRC32C — a corrupted record can neither be silently persisted nor silently returned.
• Chain-hash-faithful codec. Records are encoded with the two timestamps as explicit int64 UnixNano (exactly what chainHash consumes), so a rehydrated log re-derives byte-identical hashes and VerifyChain passes.
• Separate from the operational DB and in the adopter's tenancy — evidence is never egressed to the maintainer (GOAL.md tenet 1).

Architecture

EvidenceSink (integrity: hash-chain + signed checkpoints)   control-plane/evidence
  └─ evidence.Store  ← THIS package
        └─ objectIO port  ──► gcsObjectIO (cloud.google.com/go/storage)   gcs_gcp.go
                          └─► InMemoryObjectIO (fake, credential-free)     fakes.go

The GCS SDK is confined to gcs_gcp.go behind the objectIO port; all Store logic (sequence→name, no-overwrite/no-gap, codec, count) runs against the fake in go test ./... with no bucket and no credentials.

Wiring

store, err := evidencegcs.New(ctx, evidencegcs.Config{Bucket: "console7-evidence"})
if err != nil { /* ... */ }
defer store.Close()
sink := evidence.New(store, sinkSigner, caRoot, ckptEvery) // control-plane/evidence

New is the context-taking, erroring constructor a fallible durable backing needs: it pre-flights connectivity and hydration so a GCS fault surfaces before the Store backs a Sink.

Real vs deferred

Property	This PR	Deferred / residual
Durable append-only record store	real (no-overwrite precondition, no-gap, CRC32C, hydration, fail-closed)	—
WORM vs the append identity	real (workload SA has no delete; GCS overwrite needs delete)	—
WORM vs a privileged actor (bucket-lock immutability)	retention policy authored; lock optional (is_locked, default off → tamper-evident only)	production sets the lock deliberately (irreversible)
Deploy-identity least-privilege	APPLY uses project-wide roles/storage.admin for bucket create	custom bucket-mgmt role (excl. objects.*) is a future tightening (already self-grant-capable via projectIamAdmin)
Durable checkpoint persistence	—	Sink checkpoints stay an in-memory parallel log; persisting them (closing tail-truncation durably) is a later hardening on this Store
SIEM forward (EvidenceSink.Stream)	—	a real adopter-SIEM webhook is a separate port in a later PR

Tests

• go test ./providers/evidence-gcs/... — white-box invariants (no-overwrite, no-gap, fail-closed, codec→chain-hash equivalence, hydration) + the end-to-end VerifyChain over the real Sink backed by this Store.
• conformance/evidence_gcs_test.go — the EvidenceSink contracts run against the real Sink backed by this Store, credential-free.
• go test -tags evidence_gcs_integration ./providers/evidence-gcs/... (opt-in, env-gated, never in CI) — live GCS create-if-absent / read / count + no-overwrite.

Deploy

deploy/gcp/modules/evidence provisions the bucket (uniform access, public-access-prevention, versioning, retention policy, optional lock) and a least-privilege custom role (storage.objects.create/get/list only — no delete, no overwrite) bound to the workload SA at bucket scope. It is instantiated from deploy/gcp/main.tf; the bucket name is surfaced as the root output evidence_bucket_name.

Documentation

Overview

Package evidencegcs is the Console7 reference durable backing for the WORM evidence log on Google Cloud Storage (ARCHITECTURE.md §5; DESIGN.md §6). It implements the control-plane/evidence.Store seam — the narrow, append-only persistence port the real EvidenceSink (control-plane/evidence) commits hash-chained, sink-signed records through. It is an in-tree reference implementation; community providers live out-of-tree against the published SDK.

Where it sits

The EvidenceSink (control-plane/evidence) already owns the integrity layer: it stamps the authoritative AppendedAt, hash-chains each record, signs checkpoints through the keybroker, and fails closed. It holds NO durability of its own — it persists every sealed Entry through a control-plane/evidence.Store. This package is the production Store: a GCS bucket whose immutability is enforced at the storage layer by a retention policy + (optionally) a bucket lock. The hash-chain is integrity ON TOP; the bucket-lock is durability/immutability UNDER it.

Sink (integrity: chain + checkpoints + signing)  — control-plane/evidence
  └─ Store (durable, append-only WORM)            — THIS package, over GCS

Sequence → object mapping (the append-only contract over GCS)

Each committed Entry is ONE GCS object named "<prefix>/<zero-padded-sequence>" (fixed width, so lexical listing order equals numeric order). The contract Store.Append requires — commit at exactly the next slot, never a gap, never a rewrite — is realised structurally:

• NO REWRITE: the object is written with a DoesNotExist precondition (ifGenerationMatch=0), so a write to an already-occupied slot fails atomically server-side, independent of any in-memory count. This is the WORM property that matters: committed history cannot be overwritten even by a buggy or racing writer.
• NO GAP: Append requires the immediate predecessor object to exist (for sequence>0). With the no-rewrite precondition this yields exactly next-slot semantics (sequence == count) without an O(n) listing on the append path.

There is deliberately NO delete and NO overwrite path anywhere in this package, mirroring the memStore the seam ships with.

Who can mutate the log — the two trust levels (read this for the real WORM posture)

"WORM" here is a layered guarantee, and what holds depends on WHO the adversary is:

• The APPEND identity (the workload SA) is granted object create/get/list ONLY. GCS requires storage.objects.delete to OVERWRITE an existing object as well as to delete one, so the append path can neither overwrite nor remove a committed record — append-only WORM holds against the writer by IAM, plus the DoesNotExist precondition as in-band defence-in-depth.
• A PRIVILEGED actor — notably the deploy/Terraform identity, which holds bucket-admin and can delete objects and REMOVE an unlocked retention policy — is held back ONLY by the bucket's retention policy + LOCK (deploy/gcp/modules/evidence). That lock is the authoritative boundary control (GOAL.md tenet 3; the immutable-evidence success criterion, tenet 7). It is OFF BY DEFAULT (so dev/dogfood buckets stay destroyable), which means the SHIPPED DEFAULT posture is tamper-EVIDENT — the Sink's signed hash-chain detects any mutation or truncation — but NOT tamper-RESISTANT against a privileged actor. Production MUST set is_locked=true to make the WORM guarantee authoritative (and irreversible).

The on-disk codec preserves the chain hash

control-plane/evidence.chainHash derives a record's tamper-evidence link from its sequence, the two timestamps via UnixNano (Location-independent), and the string/[]byte fields. The codec here encodes the two times as explicit int64 UnixNano (not RFC3339), so the round-trip Entry → object bytes → Entry reproduces a byte-identical chain hash and VerifyChain passes over a rehydrated GCS log. This is asserted directly (provider_test.go) and end-to-end by the conformance run (the real Sink backed by this Store).

The GCS SDK is confined behind a port

The Store logic (store.go: sequence→name, the precondition writes, the codec, the count) depends only on the objectIO port (ports.go); the cloud.google.com/go/storage client is confined to the adapter (gcs_gcp.go) wired by New (new.go). Tests and the conformance harness wire the in-memory fake (fakes.go) instead, so the contract logic runs under `go test ./...` with no GCS bucket and no credentials — the same logic-vs-fake split the other GCP providers use. The exported fake also lets out-of-tree providers conformance-test themselves.

Real vs deferred in this PR

• REAL: durable append-only record store — sequence→object mapping, atomic no-overwrite (DoesNotExist precondition), no-gap enforcement, CRC32C integrity on every write, hydration of an existing log, fail-closed on any GCS durability fault, append-only WORM against the workload identity by IAM (create/get/list, no delete). A distinct bucket from the operational database (GOAL.md tenet 1: evidence stays in the adopter's tenancy, never egressed to the maintainer). NOTE: tamper-RESISTANCE against a privileged actor is the retention LOCK's job, which is off by default — see the two-trust-levels section above.
• DEFERRED — durable checkpoint persistence: the Sink's signed checkpoints remain its in-memory parallel log this phase (control-plane/evidence Checkpoint doc). Persisting them to GCS (so a resumed Sink continues, rather than restarts, the checkpoint chain — closing the tail-truncation residual durably) is a later hardening on this same Store.
• DEFERRED — SIEM forward: EvidenceSink.Stream is the Sink's existing fail-closed ref check; a real adopter-SIEM webhook (to the ADOPTER's SIEM only, never the maintainer — GOAL.md tenet 1) is a separate port in a later PR.
• RESIDUAL — the bucket lock (default off): the deploy module always sets a retention policy but leaves the lock OPTIONAL (is_locked, default off) so dev/dogfood buckets stay destroyable. The lock is the AUTHORITATIVE WORM control against a privileged actor (GOAL.md tenet 3: the boundary wins; tenet 7: evidence recorded immutably); with it off the default posture is tamper-evident, not tamper-resistant (above). Production sets it deliberately (irreversible). This package's no-overwrite precondition and the Sink's hash-chain are in-band defence-in-depth on top of it.
• RESIDUAL — deploy-identity privilege: deploy/gcp/bootstrap grants the APPLY identity project-wide roles/storage.admin (for bucket create + retention/lock). That identity already holds resourcemanager.projectIamAdmin (secrets module), i.e. it can self-grant any role, so storage.admin does not raise its ceiling — but a least-privilege custom role (buckets.create/get/update/setRetentionPolicy/lockRetentionPolicy, excluding objects.*) is a tracked future tightening (GOAL.md tenet 5).

Index

• Constants
• type Config
• type InMemoryObjectIO
  • func NewInMemoryObjectIO() *InMemoryObjectIO
  • func (m *InMemoryObjectIO) Count(ctx context.Context, prefix string) (uint64, error)
  • func (m *InMemoryObjectIO) Exists(ctx context.Context, name string) (bool, error)
  • func (m *InMemoryObjectIO) Get(ctx context.Context, name string) ([]byte, bool, error)
  • func (m *InMemoryObjectIO) PutIfAbsent(ctx context.Context, name string, data []byte) error
  • func (m *InMemoryObjectIO) SetFailGet(fail bool)
  • func (m *InMemoryObjectIO) SetFailPut(fail bool)
• type Store
  • func New(ctx context.Context, cfg Config, opts ...option.ClientOption) (*Store, error)
  • func NewWithObjectIO(obj objectIO, prefix string) *Store
  • func (s *Store) Append(ctx context.Context, entry evidence.Entry) error
  • func (s *Store) At(ctx context.Context, seq uint64) (evidence.Entry, bool, error)
  • func (s *Store) Close() error
  • func (s *Store) Len(ctx context.Context) (uint64, error)

Constants

const DefaultObjectPrefix = "records"

DefaultObjectPrefix is the object-name prefix used when Config.ObjectPrefix is empty. Records are stored at "<prefix>/<zero-padded-sequence>", so the prefix namespaces the evidence log within its bucket and keeps Count/list scoped to it.

Types

type Config

type Config struct {
	// Bucket is the GCS bucket that holds the evidence log. It MUST be a distinct bucket from
	// the operational database (control-plane/evidence.Store SECURITY note; DESIGN.md §6), with
	// a retention policy (and, in production, a bucket lock) enforcing immutability.
	Bucket string
	// ObjectPrefix prefixes every record object. Defaults to DefaultObjectPrefix. Two Sinks
	// sharing one bucket MUST use distinct prefixes — each prefix is an independent append-only
	// log, and Count/hydration are scoped to it.
	ObjectPrefix string
}

Config configures the production Store (New). The bucket and the workload identity that may write to it are provisioned by deploy/gcp/modules/evidence; Bucket is that module's bucket_name output.

type InMemoryObjectIO

type InMemoryObjectIO struct {
	// contains filtered or unexported fields
}

InMemoryObjectIO is a fake objectIO backed by a map. It stores object BYTES (so the Store's codec is exercised, not bypassed), enforces create-if-absent, and can be told to fail to exercise the Store's fail-closed durability paths.

func NewInMemoryObjectIO

func NewInMemoryObjectIO() *InMemoryObjectIO

NewInMemoryObjectIO returns an empty fake object store.

func (*InMemoryObjectIO) Count

func (m *InMemoryObjectIO) Count(ctx context.Context, prefix string) (uint64, error)

Count returns the number of objects whose name starts with prefix.

func (*InMemoryObjectIO) Exists

func (m *InMemoryObjectIO) Exists(ctx context.Context, name string) (bool, error)

Exists reports whether name is present.

func (*InMemoryObjectIO) Get

func (m *InMemoryObjectIO) Get(ctx context.Context, name string) ([]byte, bool, error)

Get returns a copy of the object's bytes; a missing object is (nil,false,nil).

func (*InMemoryObjectIO) PutIfAbsent

func (m *InMemoryObjectIO) PutIfAbsent(ctx context.Context, name string, data []byte) error

PutIfAbsent writes iff name is absent; a re-write returns errSlotOccupied (modelling the GCS DoesNotExist precondition).

func (*InMemoryObjectIO) SetFailGet

func (m *InMemoryObjectIO) SetFailGet(fail bool)

SetFailGet makes Get return an error, to exercise read-fault handling.

func (*InMemoryObjectIO) SetFailPut

func (m *InMemoryObjectIO) SetFailPut(fail bool)

SetFailPut makes PutIfAbsent return a (non-occupied) durability error, to exercise the Sink's fail-closed Append path.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store implements control-plane/evidence.Store over GCS via the objectIO port. It owns the sequence→object-name mapping, the no-overwrite/no-gap enforcement, and the chain-hash- preserving codec; the cloud.google.com/go/storage client is confined to gcs_gcp.go. Build it with New (production) or NewWithObjectIO (tests/conformance), then hand it to control-plane/evidence.New as the Sink's durable backing.

func New

func New(ctx context.Context, cfg Config, opts ...option.ClientOption) (*Store, error)

New constructs the production Store, dialing GCS. Credentials resolve from Application Default Credentials (GKE Workload Identity in deployment) — no key file. Pass option.ClientOptions for tests/integration (e.g. an emulator endpoint); production passes none.

New is the context-taking, erroring constructor control-plane/evidence.Store anticipates for a fallible durable backing: it PRE-FLIGHTS connectivity and hydration with the caller's context (a Len, and an At over the current tail) so a GCS fault surfaces HERE — before the Store is handed to control-plane/evidence.New, whose own hydration is best-effort over a background context. A backing that cannot be read at startup must not masquerade as empty (which would collide with the next-sequence-only Append at sequence 0).

Call Close at shutdown to release the client.

func NewWithObjectIO

func NewWithObjectIO(obj objectIO, prefix string) *Store

NewWithObjectIO wires a Store over an explicit objectIO (the in-memory fake in tests and conformance, or an out-of-tree adapter). It performs no I/O and has no client to close.

func (*Store) Append

func (s *Store) Append(ctx context.Context, entry evidence.Entry) error

Append durably commits entry at exactly entry.Ref.Sequence. It is fail-closed and preserves the append-only, contiguous-run shape:

• no rewrite: the object is written with a DoesNotExist precondition, so a write to an occupied slot fails atomically server-side (mapped from errSlotOccupied), independent of any in-memory count.
• no gap: for sequence>0 the immediate predecessor must already exist. Because the store has no delete path, the committed objects are always a contiguous 0..n-1 run (induction from empty); given that invariant, predecessor-exists + the DoesNotExist precondition imply exactly next-slot semantics (sequence == count) WITHOUT an O(n) listing on the append path, and the result stays contiguous. (The reference memStore checks sequence == len directly; this is the equivalent under the no-delete invariant, traded for cheaper writes.)

Any GCS fault surfaces as an error so the Sink fails the Append closed and never advances its chain over a record that did not durably commit. Object immutability against a privileged actor is the bucket retention lock's job (doc.go), not this in-band guard's.

func (*Store) At

func (s *Store) At(ctx context.Context, seq uint64) (evidence.Entry, bool, error)

At returns the committed entry at seq. An absent slot is (Entry{},false,nil), not an error.

func (*Store) Close

func (s *Store) Close() error

Close releases the GCS client New opened. It is safe to call on a fake-backed Store (NewWithObjectIO), where it is a no-op.

func (*Store) Len

func (s *Store) Len(ctx context.Context) (uint64, error)

Len returns the number of committed entries (the next free sequence), counting objects under the prefix. The Sink calls it at hydration and on Verify/Seal, not on the append path.