urlcatfeed

Documentation

Overview

Package urlcatfeed is the producer + trust kernel for the public, signed SaaS URL-category feed distributed from feeds.culvertlabs.com (roadmap/FEEDS-DISTRIBUTION-F0-DESIGN.md). It ships two topology-independent halves:

• F1 — a DETERMINISTIC generator: it takes an approved source dataset and emits the exact, canonical, normalized signed-artifact bytes and the signed-manifest payload bytes. Given semantically identical input it emits byte-identical output regardless of map order, source ordering, locale, timezone, or repeated execution (generate.go, normalize.go).
• F2 — a Sigstore KEYLESS trust kernel: it verifies a self-contained signed manifest ENVELOPE and the immutable artifact bundle OFFLINE against a shared Sigstore trusted root plus a PINNED, feed-specific workflow identity (verify.go, identity.go). Verify-before-parse: no manifest field is ever returned before the signature over its exact bytes has passed.

This package holds NO private key, embeds NO trusted root (the root is supplied by the caller so the SAME baked public-good root the release catalog uses can be reused by value), reaches NO network, and is independent of the CP/DP topology, the on-disk activation model, migration, and the client downloader — all of which are later, separately-gated slices (F3+). It NEVER imports or alters the release-catalog trust path; the feed identity is its own pin.

Index

• Constants
• Variables
• func AssembleEnvelope(manifestBytes, bundleJSON []byte) ([]byte, error)
• func CanonicalCategoryName(raw string) (string, error)
• func NormalizeHost(raw string) (string, error)
• type ArtifactCategory
• type ArtifactPayload
• type Envelope
• type GenerateInput
• type GenerateResult
  • func Generate(in GenerateInput) (*GenerateResult, error)
• type Identity
  • func OfficialIdentity() Identity
• type ManifestPayload
• type ReadinessConflict
• type ReadinessReport
  • func EvaluateReadiness(ds SourceDataset) *ReadinessReport
• type SourceCategory
• type SourceDataset
• type Verifier
  • func NewVerifierFromJSON(trustedRootJSON []byte, id Identity) (*Verifier, error)
  • func NewVerifierFromMaterial(tm root.TrustedMaterial, id Identity) (*Verifier, error)
  • func (v *Verifier) VerifyArtifact(artifactBytes, bundleJSON []byte, manifest *ManifestPayload) (*ArtifactPayload, error)
  • func (v *Verifier) VerifyEnvelope(envelopeBytes []byte) (*ManifestPayload, error)

Constants

const (
	// OfficialIssuer is the EXACT GitHub Actions OIDC issuer (no regex).
	OfficialIssuer = "https://token.actions.githubusercontent.com"

	// OfficialSANRegex anchors the SAN to a TAGGED release run of THIS repo's
	// dedicated feed signing workflow (publish-feeds.yml) on a feeds-vX.Y.Z
	// SemVer tag — exact repo + exact workflow file + a strict SemVer tag ref, no
	// wildcard tail (Finding 6). Only that run can mint a feed-valid identity;
	// arbitrary tag names after "feeds-v" (e.g. feeds-vgarbage) are rejected.
	// Prerelease/build-metadata tags are intentionally NOT accepted; adding them
	// requires an explicit grammar update here + feeds_identity.env + the SSOT
	// test, together.
	OfficialSANRegex = `^https://github\.com/KidCarmi/Culvert/\.github/workflows/publish-feeds\.yml@refs/tags/feeds-v[0-9]+\.[0-9]+\.[0-9]+$`
)

F2 — pinned feed signing identity (SSOT).

The feed has its OWN keyless signing identity, DISTINCT from the release catalog's ci.yml identity (F0 §5): a dedicated signing workflow file (publish-feeds.yml) and a dedicated tag namespace (feeds-v*). This is a separate pinned IDENTITY, not a separate cryptographic root — verification reuses the shared Sigstore public-good trusted root by value (verify.go takes the root bytes; the caller passes the same baked root the catalog uses).

These constants are the single source of truth in the binary. The repo-root feeds_identity.env carries the SAME two strings for CI's cosign verify, and feeds_identity_test.go (package main) pins them byte-equal so CI and the binary can never drift.

const (
	// SchemaVersion is the only supported major schema. An unknown value is
	// rejected (fail-closed), never coerced.
	SchemaVersion = 1

	// Protocol is the ONLY supported feed protocol. There is no unsigned mode
	// and no fallback (F0 §13) — this string is asserted on both sides.
	Protocol = "signed_manifest_v1"

	// FeedID is the canonical identifier for this feed. A manifest/artifact
	// whose "feed" differs is rejected (cross-feed substitution guard).
	FeedID = "url-categories/saas"
)

Wire constants. These are bound INSIDE the signed bytes and re-checked on verify, so a manifest/artifact for a different schema, protocol, or feed is rejected even if correctly signed.

const (
	MaxValidity     = 30 * 24 * time.Hour // 30 days
	MaxArtifactSize = 8 << 20             // 8 MiB — manifest.artifact_size ceiling
	MaxBundleBytes  = 1 << 20             // 1 MiB — cosign bundle ceiling
)

Protocol ceilings (Finding 3 / additional hardening). MaxValidity is the HARD producer + structural-verify ceiling on a signed manifest's validity window (expires_at - generated_at). It is defense-in-depth, distinct from the normal 14-day publisher value (F5) and from the client's current-time freshness / checkpoint enforcement (F3b). MaxArtifactBytes / MaxBundleBytes bound the in-memory inputs the verifier will accept; the F3 downloader MUST enforce the same read bounds on the wire as a precondition.

const (
	MaxCategoryNameCodePoints = 64
	MaxCategoryNameBytes      = 255
)

Category-name contract bounds (Finding 4).

Variables

var (
	ErrVersion     = errors.New("urlcatfeed: feed_version must be >= 1 and strictly greater than the previous version")
	ErrExpiry      = errors.New("urlcatfeed: expires_at must be after generated_at (whole-second UTC)")
	ErrNoCats      = errors.New("urlcatfeed: dataset has no categories with hosts")
	ErrEnvelope    = errors.New("urlcatfeed: envelope assembly failed")
	ErrZeroTime    = errors.New("urlcatfeed: generated_at/expires_at must be non-zero")
	ErrMaxValidity = errors.New("urlcatfeed: validity window exceeds the protocol maximum (30d)")
)

Generator error sentinels — tests assert the exact class via errors.Is.

var (
	// ErrJSONTrailing marks input with more than one JSON value.
	ErrJSONTrailing = errors.New("urlcatfeed: trailing data after JSON value")
	// ErrJSONDuplicateKey marks a duplicate object key at any nesting depth.
	ErrJSONDuplicateKey = errors.New("urlcatfeed: duplicate JSON object key")
	// ErrNoncanonical marks a signed document whose bytes are not the exact
	// canonical serialization of their decoded value.
	ErrNoncanonical = errors.New("urlcatfeed: document is not in canonical form")
)

var (
	ErrEmptyHost         = errors.New("urlcatfeed: empty host")
	ErrBadChars          = errors.New("urlcatfeed: host contains scheme/port/path/userinfo or illegal characters")
	ErrWildcard          = errors.New("urlcatfeed: wildcard host not allowed")
	ErrIPLiteral         = errors.New("urlcatfeed: IP literal not allowed (DNS names only)")
	ErrEmptyLabel        = errors.New("urlcatfeed: empty DNS label")
	ErrLabelLength       = errors.New("urlcatfeed: DNS label length out of range (1..63)")
	ErrHostLength        = errors.New("urlcatfeed: host length exceeds 253 octets")
	ErrIDNA              = errors.New("urlcatfeed: host failed IDNA/UTS-46 normalization")
	ErrPublicSuffix      = errors.New("urlcatfeed: bare public-suffix host not allowed")
	ErrEmptyCategory     = errors.New("urlcatfeed: empty category name")
	ErrCategoryCase      = errors.New("urlcatfeed: category names collide case-insensitively")
	ErrMultiCategory     = errors.New("urlcatfeed: host assigned to more than one category")
	ErrSuffixConflict    = errors.New("urlcatfeed: ancestor/descendant hosts in different categories")
	ErrCategoryUTF8      = errors.New("urlcatfeed: category name is not valid UTF-8")
	ErrCategoryChar      = errors.New("urlcatfeed: category name has control/format/separator characters")
	ErrCategoryLength    = errors.New("urlcatfeed: category name exceeds length bound")
	ErrEmptyCategoryHost = errors.New("urlcatfeed: category has no hosts")
)

Rejection sentinels — tests assert the exact class via errors.Is.

var (
	ErrConfig    = errors.New("urlcatfeed: verifier config")
	ErrMalformed = errors.New("urlcatfeed: malformed envelope/bundle")
	ErrOversize  = errors.New("urlcatfeed: input exceeds size bound")
	ErrVerify    = errors.New("urlcatfeed: signature verification failed")
	ErrPayload   = errors.New("urlcatfeed: payload failed validation")
	ErrBinding   = errors.New("urlcatfeed: artifact does not match manifest binding")
)

Distinct error kinds so callers/tests/alerting can tell a config error apart from a malformed artifact apart from an authenticity failure.

Functions

func AssembleEnvelope

func AssembleEnvelope(manifestBytes, bundleJSON []byte) ([]byte, error)

AssembleEnvelope builds the single self-contained manifest envelope object from the EXACT manifest-payload bytes and the cosign bundle produced over those same bytes (F0 §4.2). payload_b64 is standard base64 of manifestBytes, so the verifier decodes back to the exact bytes the bundle signed.

Hardening: the payload must be a CANONICAL, structurally-valid manifest (not merely non-empty), and the bundle must parse as a real cosign bundle (not any valid JSON) — so a producer defect can never wrap a malformed manifest or a non-bundle blob into a shippable envelope.

func CanonicalCategoryName

func CanonicalCategoryName(raw string) (string, error)

CanonicalCategoryName returns the canonical category name for raw, or an error if raw cannot be a valid category name (Finding 4). Canonical form is: valid UTF-8, NFC-normalized, surrounding whitespace trimmed, no control/format/line- separator/paragraph-separator characters, non-empty, and within the length bounds. The producer applies this to source names; the verifier requires an artifact's name to ALREADY equal its canonical form.

func NormalizeHost

func NormalizeHost(raw string) (string, error)

NormalizeHost reduces raw to a canonical DNS A-label host or rejects it. The result is parity-equal in spirit to the policy engine's hostutil.NormalizeHost (lowercase, trailing-dot stripped) but STRICTER: it additionally IDNA-folds to an A-label and rejects wildcards, IP literals, ports/schemes/paths, empty labels, over-length names, and bare public suffixes.

Types

type ArtifactCategory

type ArtifactCategory struct {
	Name  string   `json:"name"`
	Hosts []string `json:"hosts"`
}

ArtifactCategory is one category in the generated, normalized artifact: hosts are normalized, deduplicated, and sorted.

type ArtifactPayload

type ArtifactPayload struct {
	SchemaVersion int                `json:"schema_version"`
	Protocol      string             `json:"protocol"`
	Feed          string             `json:"feed"`
	FeedVersion   int64              `json:"feed_version"`
	GeneratedAt   string             `json:"generated_at"`
	Categories    []ArtifactCategory `json:"categories"`
}

ArtifactPayload is the immutable, deterministically-generated artifact. Its bytes are digest-bound by the manifest and signed by their own bundle.

type Envelope

type Envelope struct {
	PayloadB64 string          `json:"payload_b64"`
	Bundle     json.RawMessage `json:"bundle"`
}

Envelope is the SINGLE self-contained mutable object served at manifest.sigstore.json (F0 §4.2, B6): the exact manifest-payload bytes (base64) plus the cosign keyless bundle over those exact bytes. The client fetches ONE object, so there is no manifest/bundle observation race. The outer wrapper is UNTRUSTED — only PayloadB64's decoded bytes, once their bundle verifies, become trusted.

type GenerateInput

type GenerateInput struct {
	Source          SourceDataset
	FeedVersion     int64     // monotonic; must be >= 1
	PrevFeedVersion int64     // 0 = none; FeedVersion must be strictly greater
	GeneratedAt     time.Time // formatted RFC3339 UTC
	ExpiresAt       time.Time // must be after GeneratedAt
}

GenerateInput is the full, explicit input to Generate. Timestamps are supplied by the caller (never read from the clock here) so generation stays pure and deterministic.

type GenerateResult

type GenerateResult struct {
	ArtifactPath    string
	ArtifactSigPath string
	ArtifactBytes   []byte // canonical artifact JSON (signed by its own bundle)
	ArtifactSHA256  string // hex digest of ArtifactBytes
	ManifestBytes   []byte // canonical manifest-payload JSON (signed by the envelope bundle)
	Manifest        ManifestPayload
	HostCount       int
	CategoryCount   int
}

GenerateResult carries the produced bytes and their bindings.

func Generate

func Generate(in GenerateInput) (*GenerateResult, error)

Generate deterministically builds the normalized artifact and the manifest payload from an approved source dataset. It rejects the WHOLE input on any normalization or integrity violation (§7.4/§7.5) — it never emits a partial or ambiguous feed.

type Identity

type Identity struct {
	Issuer   string
	SANRegex string
}

Identity is a pinned certificate identity policy: an exact OIDC issuer and an anchored SAN regex. Both are required.

func OfficialIdentity

func OfficialIdentity() Identity

OfficialIdentity returns the baked default feed identity policy.

type ManifestPayload

type ManifestPayload struct {
	SchemaVersion   int    `json:"schema_version"`
	Protocol        string `json:"protocol"`
	Feed            string `json:"feed"`
	FeedVersion     int64  `json:"feed_version"`
	GeneratedAt     string `json:"generated_at"`
	ExpiresAt       string `json:"expires_at"`
	ArtifactPath    string `json:"artifact_path"`
	ArtifactSHA256  string `json:"artifact_sha256"`
	ArtifactSize    int64  `json:"artifact_size"`
	ArtifactSigPath string `json:"artifact_sig_path"`
	CategoryCount   int    `json:"category_count"`
	HostCount       int    `json:"host_count"`
}

ManifestPayload is the small pointer document. Its exact bytes are what the manifest envelope's Sigstore bundle signs; it binds the artifact by digest.

type ReadinessConflict

type ReadinessConflict struct {
	Kind   string // "invalid_host" | "multi_category" | "suffix_conflict" | "category_name"
	Host   string // normalized host (or raw for invalid_host / "" for category_name)
	Detail string // human-readable specifics (sorted, deterministic)
}

ReadinessConflict is one deterministic reason the dataset is not publishable.

type ReadinessReport

type ReadinessReport struct {
	Ready            bool
	TotalRawHosts    int
	UniqueHosts      int
	InvalidHosts     []ReadinessConflict
	MultiCategory    []ReadinessConflict
	SuffixConflict   []ReadinessConflict
	CategoryName     []ReadinessConflict
	StructuralIssues []ReadinessConflict // generator-parity invariants (empty dataset, zero-host category, case collision)
}

ReadinessReport is the complete, deterministically-ordered evaluation result.

func EvaluateReadiness

func EvaluateReadiness(ds SourceDataset) *ReadinessReport

EvaluateReadiness produces the full inventory. It is pure and deterministic: every list is sorted, so the same dataset always yields identical output.

type SourceCategory

type SourceCategory struct {
	Name  string   `json:"name"`
	Hosts []string `json:"hosts"`
}

SourceCategory is one category of the APPROVED input dataset: a display name and its raw (un-normalized) host list.

type SourceDataset

type SourceDataset struct {
	Categories []SourceCategory `json:"categories"`
}

SourceDataset is the approved input to the generator.

type Verifier

type Verifier struct {
	// contains filtered or unexported fields
}

Verifier verifies feed signatures against fixed trust material and a pinned identity. Immutable after construction; safe for concurrent use.

func NewVerifierFromJSON

func NewVerifierFromJSON(trustedRootJSON []byte, id Identity) (*Verifier, error)

NewVerifierFromJSON builds a verifier from a Sigstore trusted-root JSON snapshot (the shared public-good root, supplied by value) and a pinned identity.

func NewVerifierFromMaterial

func NewVerifierFromMaterial(tm root.TrustedMaterial, id Identity) (*Verifier, error)

NewVerifierFromMaterial is the test-friendly core: production passes a parsed trusted root; tests pass a VirtualSigstore. Configured for OFFLINE keyless verification (transparency log + integrated timestamps, never wall-clock).

func (*Verifier) VerifyArtifact

func (v *Verifier) VerifyArtifact(artifactBytes, bundleJSON []byte, manifest *ManifestPayload) (*ArtifactPayload, error)

VerifyArtifact verifies the immutable artifact against its own bundle AND binds it to an already-verified manifest (size + digest + feed_version + counts). It independently re-runs the integrity rejection rules (§7.4) so a signed-but- colliding artifact is rejected as a whole candidate. Returns the trusted artifact payload or no object.

func (*Verifier) VerifyEnvelope

func (v *Verifier) VerifyEnvelope(envelopeBytes []byte) (*ManifestPayload, error)

VerifyEnvelope verifies the single self-contained manifest envelope and returns the trusted manifest payload. Verify-before-parse: it extracts only the two UNTRUSTED wrapper fields, verifies the bundle over the exact decoded payload bytes, and ONLY THEN parses/validates the manifest. A forged or unsigned envelope yields no manifest object (and, for the caller, drives zero artifact fetches).