Documentation
¶
Overview ¶
Package index parses, freezes, and (eventually) verifies the signed connector registry index — the security-critical foundation `install`, `audit`, and the publish Action all trust. It implements docs/design-documents/20260714-connector-registry-index-schema.md (R-1, frozen) and plan-v2 §2 ("shared foundations").
What this package is public, not internal ¶
index-CI's own independent re-verification tooling (a separate repo) must run the IDENTICAL verification code the client runs — otherwise "index-CI re-verifies everything" is a comment, not a fact. Index-CI pins an exact github.com/conduitio/conduit module version and bumps it deliberately (never @latest), per plan-v2 §12 item 3's API-stability commitment: once a tagged release ships this package, its exported surface is a versioned API with the same announce → warn → remove discipline as config/protocol.
What ships now vs. later (PR-0 vs PR-2) ¶
This package's PR-0 scope is everything that does NOT require a signing key to exist yet: envelope parsing, duplicate-key rejection with a nesting-depth cap (P0-2), JCS (RFC 8785) canonicalization, the typed schema structs (schema_v1.go, golden-round-tripped against registry-index/sample-index.json), rollback/staleness comparisons (freeze.go, pure functions over already-parsed values), the persisted high-water-mark state file (state.go), and a bounded index fetch (fetch.go). ParseUnverified performs a shape/schema check ONLY — it makes no claim about the signatures envelope and must never be mistaken for a trust decision.
Verify — the real, cryptographically-verifying counterpart that checks signatures against build-time-fixed TrustAnchors — lands in PR-2 (Tier 1, human sign-off required), once the root/freshness key material exists (plan-v2 §9's bootstrap ceremony). Nothing in this build calls Verify or treats a ParseUnverified result as trusted for any security decision; pkg/registry.FailClosedVerifier enforces that structurally.
Invariants ¶
- Invariant 6 (schema handling never silently mangles data): a schemaVersion newer than this build understands refuses (CodeSchemaTooNew) rather than guessing at an unknown shape; a duplicate JSON key at any nesting level refuses rather than resolving silently to "last key wins" (a real producer/verifier parser differential — see duplicatekey.go).
- Everything a client trusts for a security decision must live inside the signed payload; the outer envelope's signatures array is the only thing outside it, and a signature cannot cover itself.
Index ¶
- Constants
- Variables
- func Canonicalize(payload []byte) ([]byte, error)
- func CheckNoDuplicateKeys(raw []byte) (err error)
- func CheckRollback(fetchedVersion, highWaterMark int64) error
- func CheckStaleness(timestamp, now time.Time, maxStaleness time.Duration) error
- func Fetch(ctx context.Context, url string) ([]byte, error)
- func FetchFile(path string) ([]byte, error)
- func HashConnectors(connectors []Connector) (string, error)
- func KeyID(pub ed25519.PublicKey) (string, error)
- func SaveState(path string, s State) error
- type Artifact
- type Connector
- type ConnectorVersion
- type IndexMeta
- type Payload
- type ProvenanceRef
- type Publisher
- type Revocation
- type Signature
- type SignatureRef
- type State
- type TrustAnchors
- type VerifiedIndex
- type YankReason
Constants ¶
const DefaultMaxStaleness = 7 * 24 * time.Hour
DefaultMaxStaleness is the default freshness window (R-1 §b, OQ3 resolution): a client refuses an index whose payload.index.timestamp is older than now-minus-this. Operator-overridable, similar in spirit to install.allowUnsigned. The nightly freshness re-sign (well inside this window) keeps a quiet-period index fresh without false refusals.
const MaxIndexBytes int64 = 8 * 1024 * 1024 // 8 MiB
MaxIndexBytes caps a fetched index document's size (P0-2, plan-v2 §2.4 item 1) — generous for a catalog of dozens-to-low-hundreds of connectors; tune before ship.
const MaxSupportedSchemaVersion = 1
MaxSupportedSchemaVersion is the highest payload.schemaVersion this build understands. ParseUnverified refuses (CodeSchemaTooNew) anything higher, "upgrade Conduit", per R-1 §a's schema-confusion-downgrade rationale for keeping schemaVersion inside the signed payload.
Variables ¶
var ( // CodeSchemaTooNew is raised when payload.schemaVersion exceeds the // highest this build was compiled to understand. CodeSchemaTooNew = conduiterr.Register("registry.schema_too_new", codes.FailedPrecondition) // CodeIndexUnreachable is raised when fetching the index fails at the // network/HTTP layer (distinct from a fetch that succeeds but is too // large, stale, or rolled back). CodeIndexUnreachable = conduiterr.Register("registry.index_unreachable", codes.Unavailable) // CodeIndexTooLarge is raised when a fetched index exceeds the P0-2 size // cap (plan-v2 §2.4 item 1). ResourceExhausted already carries a defined // pkg/conduit/exitcode bucket (Environment) — see fetch.go's doc comment // for the footnote this resolves. CodeIndexTooLarge = conduiterr.Register("registry.index_too_large", codes.ResourceExhausted) // CodeIndexNestingTooDeep is raised when the duplicate-key walker's // recursion cap is hit (P0-2 item 2) — refuse, never stack-overflow. CodeIndexNestingTooDeep = conduiterr.Register("registry.index_nesting_too_deep", codes.FailedPrecondition) // CodeIndexIntegrity is raised when a recognized keyId's cryptographic // verification fails (tampering/corruption) — and, per this package's // duplicate-key walker, also when parse-time duplicate-key rejection // fires: R-1 frames duplicate-key resolution ambiguity as itself a // "signature-bypass primitive" (a producer/verifier parser // differential), which is the same integrity concern this code names, // not a separate condition. See duplicatekey.go's doc comment. CodeIndexIntegrity = conduiterr.Register("registry.index_integrity", codes.DataLoss) // CodeTrustAnchorExpired is raised when no keyId in signatures[] matches // any of this build's compiled-in trust anchors at all — the "upgrade // Conduit" case, distinct from CodeIndexIntegrity's "recognized key, // verification failed". CodeTrustAnchorExpired = conduiterr.Register("registry.trust_anchor_expired", codes.FailedPrecondition) // CodeIndexStale is raised when index.timestamp is older than // maxStaleness — distinct from CodeIndexUnreachable and // CodeIndexRollback. CodeIndexStale = conduiterr.Register("registry.index_stale", codes.FailedPrecondition) // CodeIndexRollback is raised when index.version is below the locally // persisted high-water mark. CodeIndexRollback = conduiterr.Register("registry.index_rollback", codes.FailedPrecondition) // CodeVersionYanked is raised when a resolved/pinned version carries // `yanked`. Registered here (owning package "index" per plan-v2 §4, // which lists it as shared with "registry") rather than in // pkg/registry, specifically so this package need not import // pkg/registry (which imports this package) — audit's REVOKED_PUBLISHER // finding (PR-4) reuses this code verbatim rather than minting a new // connector.*-prefixed one. CodeVersionYanked = conduiterr.Register("registry.version_yanked", codes.FailedPrecondition) )
Index error codes — the "index"-owned rows of the canonical registry error table (plan-v2 §4). Registered here (not in pkg/registry) so pkg/registry can import this package without a cycle; pkg/registry and pkg/registry/trust reference these directly rather than re-registering them under a different reason string.
Functions ¶
func Canonicalize ¶
Canonicalize returns the RFC 8785 (JCS) canonical byte form of a JSON value: object keys sorted (by UTF-16 code unit), numbers in a fixed representation, minimal escaping, no insignificant whitespace.
This is the exact byte sequence a signature is computed over and verified against (R-1 §a) — both the signing side (index build tooling) and every verifying side (this package's future Verify, index-CI's independent re-verification) must run the identical implementation, per R-1's OQ5 resolution ("whatever is chosen must be the SAME implementation ... on both the index-build side and index-CI's re-verification side"). That is why this is a single, shared, well-tested wrapper rather than an inline call at each site — a canonicalization mismatch between producer and verifier would reintroduce exactly the class of bug JCS exists to avoid.
Canonicalize has no cryptographic dependency and makes no trust decision of its own: it is a pure byte transform, safe to run before any signature has been checked (and, indeed, is a required INPUT to checking one).
func CheckNoDuplicateKeys ¶
CheckNoDuplicateKeys walks raw as generic JSON and refuses any object containing a duplicate key at any nesting level, before and independent of JCS canonicalization. This is required, not a style preference: JCS fixes serialization ambiguity, not parse-time duplicate-key resolution — a last-key-wins parse is a producer/verifier differential that JCS alone does not close, and per R-1 §a is itself "a signature-bypass primitive" (two parsers could legitimately disagree on which value a duplicated key held, while both verify the same signature over the same bytes).
It is deliberately narrow: a structural walk only, checked before ParseUnverified attempts to interpret schemaVersion or unmarshal the typed schema — it does not itself validate that raw otherwise matches the schema (a duplicate-free document can still fail ParseUnverified's later typed unmarshal).
A duplicate key found at any depth reports CodeIndexIntegrity (see that code's doc comment for why duplicate-key rejection is filed under the same code as signature-verification failure, not a separate one); nesting deeper than maxNestingDepth reports the distinct CodeIndexNestingTooDeep, so an operator/agent can tell "this index is too deeply nested to be legitimate" from "this index's content was tampered with".
CheckNoDuplicateKeys recovers from a panic in the underlying decoder and reports it as CodeIndexIntegrity rather than crashing the process. This is not defensive-programming theater: the P0-2 fuzz corpus (FuzzDuplicateKeyWalk) found goccy/go-json's streaming Decoder.Token() panics (an internal slice index out of range), not merely errors, on certain malformed multi-byte-adjacent-invalid-UTF-8 input — exactly the "attacker-controlled bytes parsed before any crypto check" scenario P0-2 exists to harden. The failing seed is preserved under testdata/fuzz/FuzzDuplicateKeyWalk/ as a permanent regression case.
func CheckRollback ¶
CheckRollback refuses (CodeIndexRollback) a fetched index whose version is lower than the highest this client has previously observed and verified. This is deliberately a separate, independently-triggerable check from CheckStaleness: rollback protection alone doesn't stop a frozen-but-never-rolled-back index (the highest version an attacker has ever served, just old); staleness alone doesn't stop a rollback to a recent-enough-to-pass-freshness older version. Both are required (R-1 §b item 3).
highWaterMark is the caller's persisted value (see State/LoadState); it must be updated ONLY after a fetch passes every check (signature, schema, rollback, staleness) — never on a rejected fetch, so an attacker can't ratchet the client's trusted floor forward with garbage. That update discipline is the caller's responsibility (this function only compares).
func CheckStaleness ¶
CheckStaleness refuses (CodeIndexStale) an index whose timestamp is older than now-maxStaleness. Distinct from CodeIndexUnreachable (fetch-layer failure) and CodeIndexRollback (monotonic-counter check) — see CheckRollback's doc comment for why both checks are required together.
func Fetch ¶
Fetch retrieves the raw index bytes from url over HTTP(S), enforcing MaxIndexBytes. It performs no parsing and no trust decision — see ParseUnverified for the next step. A response exceeding the cap fails with CodeIndexTooLarge, distinct from any other fetch failure (CodeIndexUnreachable), so a caller can tell "the origin sent too much" from "the origin was unreachable".
func FetchFile ¶
FetchFile reads the raw index bytes from a local path (offline/bundle mode), enforcing the same MaxIndexBytes cap as Fetch.
func HashConnectors ¶
HashConnectors returns "sha256:<hex>" over the JCS-canonicalized connectors[] array — the value persisted in State.LastVerifiedConnectorsHash and compared by Verify's freshness-only acceptance path (R-1 §a.2.c): a freshness signature may only extend index.timestamp/index.version over BYTE-IDENTICAL connectors[] content, never authorize different content on its own. Hashing (rather than comparing raw canonical bytes directly) keeps State small and gives a stable, loggable value for diagnostics.
The hash is computed over the canonicalized connectors array alone (not the full payload, which also carries index.version/timestamp that legitimately change on every heartbeat re-sign) — hashing the whole payload would make every nightly freshness re-sign look like new content and defeat the byte-identical check this function exists to support.
func KeyID ¶
KeyID returns the anchor keyId this package uses for an ed25519 public key: "sha256:<hex fingerprint of the SPKI-encoded public key>" (see TrustAnchors' doc comment). Both the index-build/signing tooling (a sibling repo, per plan-v2 §7) and this package's tests must derive keyIds identically — this is that single shared derivation, exported so test fixtures and (via the pinned module import, plan-v2 §2.5) index-CI tooling never hand-roll their own and risk drifting from what Verify actually expects.
func SaveState ¶
SaveState persists s to path atomically (temp file + rename in the same directory), so a crash mid-write can never leave a torn state file (Invariant 5) and can never corrupt the high-water mark into a value an attacker could exploit to widen the rollback window. Callers must only call SaveState after a fetch has passed every verification/freshness/ rollback check (see CheckRollback's doc comment) — this function itself performs no such check; it is a pure persistence primitive.
Types ¶
type Artifact ¶
type Artifact struct {
OS string `json:"os"`
Arch string `json:"arch"`
Kind string `json:"kind"`
URL string `json:"url"`
SHA256 string `json:"sha256"`
Size int64 `json:"size"`
Signature SignatureRef `json:"signature"`
SLSAProvenance *ProvenanceRef `json:"slsaProvenance,omitempty"`
}
Artifact is one (os, arch) build for a version. Signature is per-artifact (not per-version) because a cosign blob signature is inherently over one specific artifact's digest — R-1's OPEN QUESTIONS documents the divergence from the design doc's literal version-level field grouping.
type Connector ¶
type Connector struct {
Name string `json:"name"`
DisplayName string `json:"displayName,omitempty"`
Description string `json:"description,omitempty"`
Repository string `json:"repository,omitempty"`
Publisher Publisher `json:"publisher"`
Versions []ConnectorVersion `json:"versions"`
}
Connector is one registered connector name's entry.
type ConnectorVersion ¶
type ConnectorVersion struct {
Version string `json:"version"`
ReleasedAt *time.Time `json:"releasedAt,omitempty"`
MinConduitVersion string `json:"minConduitVersion"`
MinProtocolVersion string `json:"minProtocolVersion"`
Artifacts []Artifact `json:"artifacts"`
SLSAProvenance *ProvenanceRef `json:"slsaProvenance,omitempty"`
// Deprecated has no omitempty: the schema's documented default is false,
// and the frozen sample index writes it explicitly even when false —
// dropping it on marshal would fail the golden round-trip test against
// that fixture.
Deprecated bool `json:"deprecated"`
Yanked *YankReason `json:"yanked,omitempty"`
}
ConnectorVersion is one published release. Entries are append-only once published: index-CI rejects a PR that mutates any field of an already-present version other than Deprecated, Yanked (R-1 §d item 4).
type IndexMeta ¶
IndexMeta is the freeze/rollback-protection metadata (R-1 §b): Version is a monotonically increasing counter bumped on every signed rebuild (content OR heartbeat re-sign); Timestamp is the RFC 3339 build time checked against maxStaleness.
type Payload ¶
type Payload struct {
SchemaVersion int `json:"schemaVersion"`
Index IndexMeta `json:"index"`
Connectors []Connector `json:"connectors"`
}
Payload is the typed schemaVersion-1 shape of the signed payload object — the ONLY types index/trust/registry construct a security decision from, field-for-field matching docs/design-documents/registry-index/index-schema.json. Every field a client trusts (urls, sha256, expectedIdentity*, versions, revocations, the monotonic counter, the timestamp) lives in here, per R-1 §a.
These types are defined exactly once in the module — pkg/registry and pkg/registry/trust reference this package's types rather than declaring their own parallel shapes, which was itself one of plan-v2's coherence fixes over the original step-plans (§0 item 2).
func ParseUnverified ¶
ParseUnverified parses raw index bytes into a typed Payload WITHOUT checking any signature — it is a shape/schema check only, in bold: it makes NO claim about the signatures envelope, and its result must never be used for a security decision. Concretely, in the order R-1 §a requires:
- Reject any duplicate JSON key at any nesting level (CheckNoDuplicateKeys, P0-2) — before and independent of interpreting any field.
- Extract payload/signatures generically (schemaVersion is not yet trusted at this point, so nothing schema-version-specific has run).
- Peek payload.schemaVersion and refuse (CodeSchemaTooNew) anything newer than MaxSupportedSchemaVersion, "upgrade Conduit" — before attempting the typed unmarshal, so an unrecognized future shape never reaches struct decoding.
- Unmarshal payload into the typed schemaVersion-1 Payload struct.
The real, cryptographically-verifying counterpart (Verify, checking signatures against build-time-fixed TrustAnchors per R-1 §a step 2c) is implemented below — nothing in this build may treat ParseUnverified's result as trusted for any security decision; see pkg/registry.FailClosedVerifier.
type ProvenanceRef ¶
type ProvenanceRef struct {
BundleURL string `json:"bundleURL"`
PredicateType string `json:"predicateType"`
}
ProvenanceRef points at a SLSA provenance attestation bundle, verifiable offline the same way as SignatureRef.
type Publisher ¶
type Publisher struct {
ExpectedOIDCIssuer string `json:"expectedOIDCIssuer"`
ExpectedIdentityPattern string `json:"expectedIdentityPattern"`
Revoked *Revocation `json:"revoked,omitempty"`
}
Publisher is the per-name identity pinning — the actual root-of-trust decision for this connector name (R-1 §c). Changing ExpectedOIDCIssuer or ExpectedIdentityPattern for an already-registered name requires the same human-reviewed path as first registration (R-1 §d).
type Revocation ¶
type Revocation struct {
Reason string `json:"reason"`
RevokedAt *time.Time `json:"revokedAt,omitempty"`
RevokedBy string `json:"revokedBy,omitempty"`
}
Revocation invalidates every version under a connector name regardless of individual Yanked status — the compromise is at the identity level, not the artifact level (R-1 §e).
type Signature ¶
type Signature struct {
Role string `json:"role"`
KeyID string `json:"keyId"`
Algorithm string `json:"algorithm"`
Signature string `json:"signature"`
}
Signature is one detached signature over the JCS-canonicalized bytes of the sibling payload (R-1 §a). role distinguishes the reviewer-gated root key (authorizes content) from the unattended freshness key (may only extend index.timestamp/bump index.version over byte-identical connectors[]) — see freeze.go and R-1's OQ3 resolution.
type SignatureRef ¶
type SignatureRef struct {
BundleURL string `json:"bundleURL"`
RekorLogIndex *int64 `json:"rekorLogIndex,omitempty"`
}
SignatureRef points at the Sigstore bundle covering one artifact's own digest — offline-verifiable (cert chain + Rekor inclusion proof/SET embedded), no live Fulcio/Rekor query required at install time.
type State ¶
type State struct {
Version int64 `json:"version"`
// LastVerifiedConnectorsHash is "sha256:<hex>" over the JCS-canonicalized
// connectors[] array from the last root-verified index, or "" if no
// index has ever been root-verified yet (see HashConnectors). Additive
// field: a State persisted by PR-0/PR-1 (before this field existed)
// unmarshals with this empty, matching "no root-verified content on
// record" — Verify's freshness path then correctly requires root.
LastVerifiedConnectorsHash string `json:"lastVerifiedConnectorsHash,omitempty"`
}
State is the locally persisted rollback high-water mark (R-1 §b item 1): the highest payload.index.version this client has successfully verified, plus the sha256 of the JCS-canonicalized connectors[] array from the last ROOT-verified (not freshness-only) index (PR-2, R-1 §a.2.c) — the value Verify's freshness-only acceptance path compares against, so a freshness signature can only ever extend timestamp/version over byte-identical content, never authorize new content on its own.
func LoadState ¶
LoadState reads the persisted high-water mark from path. A missing file is not an error: it returns the zero State (Version: 0), matching "no index has ever been verified yet" — R-1 §b's documented gap that a client with no prior state has no rollback protection on its very first fetch (it falls back to the staleness check alone).
type TrustAnchors ¶
type TrustAnchors struct {
// Roots holds every currently-trusted root public key, keyed by keyId
// (e.g. "sha256:<hex fingerprint of the SPKI-encoded public key>").
// During a rotation window both the outgoing and incoming root key are
// present simultaneously (R-1 OQ2) for a retention window tracking
// Conduit's supported-version policy.
Roots map[string]ed25519.PublicKey
// Freshness holds the currently-trusted freshness public key(s), keyed
// by keyId. A freshness signature may only extend index.timestamp/bump
// index.version over byte-identical connectors[]; it never authorizes
// content on its own (R-1 OQ3).
Freshness map[string]ed25519.PublicKey
}
TrustAnchors holds the build-time-fixed root and freshness public keys compiled into the conduit binary (R-1 OQ1/OQ2 resolution: trust anchors are fixed at build time and are NEVER updated at runtime — there is no runtime-fetched rotation statement, which would let a compromised old key forge a rotation to an attacker key).
The production key material was generated by the bootstrap ceremony (plan-v2 §9) and is go:embed'd into the binary at cmd/conduit/root/connectors/trustanchors/{root,freshness}.pub.pem; a released build verifies the served signed index at https://registry.conduitdata.io/index.json against it. This type itself is generic — a build with no embedded anchors (a custom or stripped build) leaves it empty, and every real index then fails closed with CodeTrustAnchorExpired (never fail-open).
type VerifiedIndex ¶
type VerifiedIndex struct {
Payload Payload
Verified bool
// RootVerified is true only when acceptance came from a role:"root"
// signature — as opposed to a role:"freshness" signature accepted
// because connectors[] was byte-identical to the last root-verified
// content (R-1 §a.2.c). Callers persisting State.LastVerifiedConnectorsHash
// (via HashConnectors) should only update it when RootVerified is true:
// a freshness-only acceptance by construction already matches the
// existing persisted hash, so re-deriving it is a no-op at best and,
// if ever wrong, must never widen what freshness alone can authorize.
RootVerified bool
}
VerifiedIndex wraps a parsed Payload together with an explicit Verified flag, so a caller can never mistake a ParseUnverified result (schema-valid bytes, cryptographically unchecked) for a trusted one by accident. pkg/registry.FailClosedVerifier always sets Verified: false; the install pipeline asserts Verified == true before using any field for a security decision — a second, structural belt-and-suspenders check on top of pkg/registry.ArtifactVerifier unconditionally refusing until a real verifier is wired in (plan-v2 §2.2).
func Verify ¶
func Verify(raw []byte, anchors TrustAnchors, lastVerifiedConnectorsHash string) (*VerifiedIndex, error)
Verify is the cryptographically-verifying counterpart to ParseUnverified, implementing R-1 §a steps 2(a)-(d) in full:
- Duplicate-key rejection (shared with ParseUnverified, via decodeEnvelope).
- JCS-canonicalize the raw payload bytes.
- For each signatures[] entry, resolve keyId against the compiled-in anchors for that entry's declared role ("root" or "freshness") and verify the ed25519 signature over the canonical bytes.
- Accept if any role:"root" signature verifies. Otherwise, accept if a role:"freshness" signature verifies AND the payload's connectors[] (canonicalized, hashed via HashConnectors) is byte-identical to lastVerifiedConnectorsHash — the last root-verified content on record (persisted in State, R-1 §a.2.c: "a freshness signature may extend freshness, never authorize content").
- Only once accepted, peek schemaVersion and unmarshal into the typed Payload (shared with ParseUnverified, via decodeTypedPayload).
Distinguishes two failure modes as DISTINCT errors, never conflated (R-1 §a.2.c):
- ErrTrustAnchorExpired (CodeTrustAnchorExpired): no keyId in signatures[] matches ANY compiled-in anchor (root or freshness) at all — "upgrade Conduit", fail closed, never fall back to a stale cache.
- ErrIndexIntegrity (CodeIndexIntegrity): at least one keyId WAS recognized, but no recognized signature both verified cryptographically AND was sufficient to authorize this content (a root signature that fails crypto; a freshness signature that verifies crypto but doesn't match lastVerifiedConnectorsHash; a duplicate key; malformed signature bytes) — tampering/corruption, refuse and report loudly.
lastVerifiedConnectorsHash is the caller's persisted State.LastVerifiedConnectorsHash (empty string if no index has ever been root-verified on this machine — R-1 §b's documented first-fetch gap: a freshness-only index can never be the FIRST index a client ever accepts, by construction, since an empty hash can never equal a real one).