telemetry

Documentation

Overview

Package telemetry provides the OpenTelemetry-based metrics exporter for GitOps Reverser. It configures Prometheus-compatible metrics collection for monitoring controller operations.

Every instrument declared here MUST have at least one production recording site. A metric that is defined but never recorded is a contract the code does not honor; document it in docs/interpreting-metrics.md only once it actually emits.

Index

• Variables
• func CollectHistogramCount(reader *sdkmetric.ManualReader, metricName string, match map[string]string) (uint64, bool)
• func CollectInt64Sum(reader *sdkmetric.ManualReader, metricName string, match map[string]string) (int64, bool)
• func InitOTLPExporter(_ context.Context) (func(context.Context) error, error)
• func InitTestExporter() (*sdkmetric.ManualReader, error)

Variables

var (

	// GitOperationsTotal counts git operations performed by branch workers.
	GitOperationsTotal metric.Int64Counter
	// ObjectsWrittenTotal counts objects that resulted in file writes.
	ObjectsWrittenTotal metric.Int64Counter
	// CommitsTotal counts commit batches pushed to git, labelled by the recording
	// BranchWorker's {provider_namespace, provider_name, branch, author_kind} identity.
	// Both the per-event and backfill-resync commit paths feed this one counter.
	CommitsTotal metric.Int64Counter
	// ResyncSweepDeletesTotal counts managed documents deleted by mark-and-sweep
	// resyncs, labelled by the swept resource {group, version, resource}.
	ResyncSweepDeletesTotal metric.Int64Counter
	// PruneRetainedDocumentsTotal counts managed documents a GitTarget's spec.prune.mode
	// KEPT that a mark-and-sweep would otherwise have deleted, labelled by
	// {prune_mode, gittarget_namespace, gittarget_name}. It is the retention twin of
	// ResyncSweepDeletesTotal and the only numeric trace a suppressed drop leaves: such a
	// drop produces no plan action, no commit, and no ResyncStats entry. A non-zero value
	// is the configured behaviour, never a fault.
	PruneRetainedDocumentsTotal metric.Int64Counter

	// PlacementsTotal counts new-file placements resolved for a resource with no document in
	// Git yet — the only case placement runs for — labelled by {source, disposition,
	// gittarget_namespace, gittarget_name, group, version, resource}. source is which
	// mechanism chose the path (declared / kustomize_root / canonical) and disposition is what
	// it did with it (new_file / appended).
	//
	// It exists because sibling inference was deleted (docs/design/open-asks-priority.md): a
	// repository with a hand-authored layout now needs a placement.byType line, and
	// `source="canonical"` is how its operator learns which type in which target is missing
	// one, WITHOUT reading the folder. The (GitTarget, type) labels are the whole point — a
	// bare "a fall-back happened somewhere" counter is not actionable, which is why the
	// design doc argued against leading with one. Cardinality is bounded by targets ×
	// watched types, and placement fires only for a type/name the target has never written.
	//
	// Every increment is a resource that WAS mirrored; a resource the writer refused is
	// PlacementRefusalsTotal instead, never a value of source here.
	PlacementsTotal metric.Int64Counter
	// PlacementRefusalsTotal counts resources the writer declined to place, labelled by
	// {reason, gittarget_namespace, gittarget_name, group, version, resource}. Every
	// increment is a resource NOT in the mirror: a declared template that escapes spec.path
	// or is not identity-complete, a sensitive resource whose path is already taken, a
	// plaintext resource routed at an encrypted file, or two resources of mixed sensitivity
	// racing onto one brand-new file. The write is retried on the next event or resync, so a
	// steady non-zero rate means a policy that needs fixing rather than a transient.
	//
	// This is the counter that did not exist: a refusal left a log line at the skip site and,
	// on the resync path only, ResyncStats.PlacementSkipped — a field in a summary, not a
	// series anything can alert on.
	PlacementRefusalsTotal metric.Int64Counter
	// PlacementKustomizationEntriesTotal counts attempts to add a new file to the
	// resources: list of the kustomization that governs it, labelled by {outcome,
	// gittarget_namespace, gittarget_name}. outcome is added, no_change, or failed.
	//
	// `failed` is the one to watch, and it is otherwise invisible: the document is committed
	// and the entry is not, so kustomize never builds the file. The object is in Git, looks
	// mirrored, and is not applied by anything.
	PlacementKustomizationEntriesTotal metric.Int64Counter

	// TargetReconcileCompletedTotal counts completed watch recovery passes per
	// GitTarget: each increment marks either a streaming-snapshot resync applied on
	// the branch worker or a cursor-backed watch resume (see Manager.recordTargetReconcileCompleted).
	// Labelled by {gittarget_namespace,
	// gittarget_name, trigger} where trigger is `rule_change` (the GVR/rule reconcile
	// path). A counter, not a
	// latched gauge, on purpose: a counter resets to 0 on a fresh pod, so a
	// per-pod `{pod="<new>"} > 0` check after a rollout proves the new pod did
	// its own reconcile — robust to the old pod's stale series that a Prometheus
	// pod scrape may still be holding during the rollout, which a latched gauge
	// (or a cross-pod sum-over-baseline) cannot distinguish.
	// The label keys avoid the reserved `namespace`/`name`: a pod scrape with
	// honor_labels=false would overwrite a metric's `namespace` attribute with the
	// scraped pod's own namespace, making a per-GitTarget `namespace` selector
	// silently match nothing. Load-bearing for the restart-reconcile e2e spec and
	// useful long-term for spotting excessive reconciles via increase(...[5m]);
	// treat the name/labels as a public observability contract.
	TargetReconcileCompletedTotal metric.Int64Counter
	// BranchWorkerQueueDepth gauges pending work for a single branch worker:
	// accepted-but-not-yet-handled items (queued or actively being processed)
	// plus any committed-but-not-yet-pushed work the worker is still holding. It
	// reads 0 only when the worker has fully drained (every accepted item handled
	// and nothing retained for replay), so it never reports drained while a
	// commit/push is still in flight. Labelled by {provider_namespace,
	// provider_name, branch}; the namespace/name keys are prefixed to avoid the
	// reserved Prometheus pod-scrape target labels (see
	// TargetReconcileCompletedTotal). Load-bearing for the restart-reconcile e2e
	// spec's drain wait; treat the name/labels as a public observability contract.
	BranchWorkerQueueDepth metric.Int64Gauge

	// ResyncBackgroundFailuresTotal counts rule-change resyncs whose apply failed or
	// timed out at the worker AFTER being enqueued. Delivery is marked on enqueue (the
	// resync is fire-and-forget to avoid an unbounded re-gather loop — see
	// Manager.recordTargetReconcileCompleted), so a failed background apply is otherwise
	// only logged. This counter makes those failures observable/alertable without
	// triggering an immediate re-gather. Labelled by {gittarget_namespace,
	// gittarget_name}; a sustained increase means snapshots are not committing and the
	// folder is relying on steady-state events to catch up.
	ResyncBackgroundFailuresTotal metric.Int64Counter

	// AuditEventsTotal is the single per-event census: every successfully decoded, converted, and
	// validated audit event increments it exactly once, labelled by {outcome, category, group,
	// version, resource, verb}. Audit is attribution-only — it names the author of a watch-observed
	// change; it never carries object state. Liveness = sum(...) > 0; the e2e invariant gates on
	// category="error" == 0.
	AuditEventsTotal metric.Int64Counter
	// AuditEventListsTotal counts inbound audit EventList requests at the webhook boundary,
	// labelled by bounded outcome (processed/empty/decode_error/process_error).
	AuditEventListsTotal metric.Int64Counter
	// AuditEventListEventsTotal counts decoded audit event items delivered in EventLists,
	// labelled by the same bounded outcome.
	AuditEventListEventsTotal metric.Int64Counter
	// AuditEventListDurationSeconds records how long the webhook takes to answer an
	// EventList request, labelled by outcome.
	AuditEventListDurationSeconds metric.Float64Histogram
	// AttributionResolutionsTotal counts watch-event attribution resolver outcomes, labelled by
	// {tier, actor_kind, group, version, resource}. tier names WHICH evidence answered
	// (delete_sticky/exact/deletecollection_body_uid/latest/name/deletecollection_scope/
	// resource_version/absent) and actor_kind
	// names WHO it named (user/serviceaccount/none) — two orthogonal questions, so they are two
	// labels. Match coverage is tier!="absent"; anything narrower reads the collection and name
	// tiers as misses.
	AttributionResolutionsTotal metric.Int64Counter
	// AttributionFactsTotal counts attribution fact lifecycle events, labelled by bounded op:
	// "written" is one fact appended to the fact log, "matched" is one joined by a watch event.
	// Together they say how much of what is published is ever used. They are NOT subtractable:
	// written counts every type, matched only the streams this process follows.
	AttributionFactsTotal metric.Int64Counter
	// AttributionResolutionWaitSeconds records resolver wait time by {tier, event_kind, group,
	// version, resource}. event_kind is write or removal, and the split is load-bearing: a removal
	// holds a fallback and keeps waiting where a write does not, so the removal wait is the number
	// --author-attribution-grace is tuned from.
	AttributionResolutionWaitSeconds metric.Float64Histogram
	// AttributionFactIndexEntries gauges the entries the in-memory fact index currently holds across
	// every scope and match structure. Read against the eviction counter it says whether the caps
	// are binding.
	AttributionFactIndexEntries metric.Int64Gauge
	// AttributionFactIndexEvictionsTotal counts facts dropped from the in-memory fact index because
	// it was full, labelled by bounded reason (per_type/total). An attribution lost to a full index
	// has to look different from one that was never published, or a burst is silently absorbed.
	AttributionFactIndexEvictionsTotal metric.Int64Counter
	// AttributionCollectionWithoutUIDSetTotal counts collection facts published without the uid set
	// the precise join would have used, labelled by bounded reason (uid_cap/no_uids). The scope
	// fallback is already correct, so this says how often the precise path was available — not that
	// anything broke.
	AttributionCollectionWithoutUIDSetTotal metric.Int64Counter
	// AttributionFactStreamGapsTotal counts occasions a fact stream was trimmed past this process's
	// follower, labelled by stream. Every gap is facts lost for good, and it is the one loss a log
	// transport can see at all.
	AttributionFactStreamGapsTotal metric.Int64Counter
	// AttributionFactStreamDecodeErrorsTotal counts fact-stream entries the follower could not
	// decode, labelled by transport. Such an entry is skipped and its position passed, so the facts
	// it carried are lost — and unlike a trim gap the loss leaves no other trace, which is why this
	// is the loss path that most needed a counter.
	AttributionFactStreamDecodeErrorsTotal metric.Int64Counter
	// AttributionFactFollowerErrorsTotal counts fact-follower read failures, labelled by transport.
	// The follower retries with a backoff rather than returning, so the errors are otherwise only a
	// log line.
	AttributionFactFollowerErrorsTotal metric.Int64Counter
	// AttributionFactFollowerLastSuccessTimestampSeconds gauges the Unix time of the follower's last
	// successful read, idle rounds included. It matters more than the error counter: only it
	// separates "erroring occasionally while making progress" from "has read nothing in ten
	// minutes", and only the second is an outage. Read it as time() - <gauge>.
	AttributionFactFollowerLastSuccessTimestampSeconds metric.Int64Gauge
	// AttributionTransportInfo is an info gauge, always 1, labelled by the fact transport in force
	// (redis/memory). It is interpretive metadata rather than a signal: a burst of unresolved
	// commits after a restart is EXPECTED under the in-process transport, which drops every fact
	// with the process, and a bug under Redis.
	AttributionTransportInfo metric.Int64Gauge

	// APICatalogResources gauges the count of served top-level resources in the catalog,
	// split by the default-watch-policy allowed/excluded state.
	APICatalogResources metric.Int64Gauge
	// APICatalogGroupVersions gauges discovered group/versions, split into trusted vs degraded.
	APICatalogGroupVersions metric.Int64Gauge
	// APICatalogRefreshTotal counts API resource catalog refreshes by outcome.
	APICatalogRefreshTotal metric.Int64Counter
	// APICatalogRefreshDurationSeconds records the wall time of one catalog refresh.
	APICatalogRefreshDurationSeconds metric.Float64Histogram
	// APICatalogGeneration gauges the current APIResourceCatalog generation.
	APICatalogGeneration metric.Int64Gauge
	// WatchedTypes gauges the number of watched types per GitTarget, labelled by
	// gittarget_namespace and gittarget_name.
	WatchedTypes metric.Int64Gauge

	// SecretEncryptionAttemptsTotal counts total Secret encryption attempts.
	SecretEncryptionAttemptsTotal metric.Int64Counter
	// SecretEncryptionSuccessTotal counts successful Secret encryptions.
	SecretEncryptionSuccessTotal metric.Int64Counter
	// SecretEncryptionFailuresTotal counts failed Secret encryptions.
	SecretEncryptionFailuresTotal metric.Int64Counter
	// SecretEncryptionCacheHitsTotal counts cache hits for encrypted Secret content.
	SecretEncryptionCacheHitsTotal metric.Int64Counter
	// SecretEncryptionMarkerSkipsTotal counts marker-based skips that reused cached Secret content.
	SecretEncryptionMarkerSkipsTotal metric.Int64Counter
)

Functions

func CollectHistogramCount

func CollectHistogramCount(
	reader *sdkmetric.ManualReader,
	metricName string,
	match map[string]string,
) (uint64, bool)

CollectHistogramCount returns the total sample count of the named float histogram data points whose attributes are a superset of match. ok is false when no matching data point exists.

func CollectInt64Sum

func CollectInt64Sum(
	reader *sdkmetric.ManualReader,
	metricName string,
	match map[string]string,
) (int64, bool)

CollectInt64Sum returns the summed value of the named Int64 counter or gauge data points whose attributes are a superset of match. ok is false when no matching data point exists.

func InitOTLPExporter

func InitOTLPExporter(_ context.Context) (func(context.Context) error, error)

InitOTLPExporter initializes the OTLP-to-Prometheus bridge.

func InitTestExporter

func InitTestExporter() (*sdkmetric.ManualReader, error)

InitTestExporter wires the global instruments to a meter provider backed by a manual reader, so unit tests can collect and assert recorded metric values. It returns the reader to collect from.