vmetrics

Documentation

Overview

Package vmetrics implements a read backend for an external VictoriaMetrics deployment over its Prometheus-compatible HTTP API (v0.9.1150, Faz 1).

Why: operators who already run VictoriaMetrics as their metrics store do not want a second copy of the same series inside Coremetry's ClickHouse. When this backend is enabled, the metric DISCOVERY and QUERY surfaces (catalogue + picker, Explore, dashboard metric panels, MCP query_metric, label values, attribute keys) read from VM instead.

The query dialect is MetricsQL (a PromQL superset) — see promql.go for the two VictoriaMetrics-specific behaviours the translation relies on.

Scope discipline — Faz 1 deliberately leaves in ClickHouse:

• everything SPAN-derived (services, operations, topology, traces, exceptions, DB/messaging surfaces). Those are not metrics.
• fixed-name INTERNAL readers (hosts, infra, JVM panels, db capacity). They are wired to specific metric names + CH columns and each needs its own translation; a partial rewrite would make some panels read VM and others CH on the same page.

Faz 2 (v0.9.1157) closed the last two operator-facing gaps, so the list above is now the WHOLE exclusion set:

• p50/p95/p99 translate to histogram_quantile over the `_bucket` series (promql.go),
• GET /api/metrics/histogram builds its heatmap from `sum by (le) (increase(…))` (histogram.go),
• GET /api/metrics/promql forwards the operator's query VERBATIM — MetricsQL extensions included, since pre-validating with Coremetry's PromQL-subset parser would reject queries VM runs happily.

There is NO silent fallback to ClickHouse. If the operator enabled VM and VM is unreachable, the endpoint fails with VM's error. A fallback would answer the operator's question with data from a store they did not ask about, and they would have no way to tell — the same honesty rule the Tempo fallback banner exists for, applied to a case where a banner is not enough because the NUMBERS would differ.

Index

• Constants
• Variables
• func ValidRateWindowFloor(v int) bool
• type Service
  • func New() *Service
  • func (s *Service) Available() bool
  • func (s *Service) Configure(cfg Settings)
  • func (s *Service) Configured() bool
  • func (s *Service) CurrentSettings() Settings
  • func (s *Service) ListMetricNames(ctx context.Context, service, pattern string, limit, offset int) ([]chstore.MetricInfo, int, error)
  • func (s *Service) LoadPersisted(ctx context.Context, store settingsStore) error
  • func (s *Service) MetricAttrKeys(ctx context.Context, metric, service string, since time.Duration) ([]string, error)
  • func (s *Service) MetricLabelValues(ctx context.Context, metric, key string, since time.Duration) ([]string, error)
  • func (s *Service) QueryMetric(ctx context.Context, f chstore.MetricQueryFilter) ([]chstore.SpanMetricSeries, error)
  • func (s *Service) QueryMetricHistogram(ctx context.Context, f chstore.MetricQueryFilter) (*chstore.HistogramSeries, error)
  • func (s *Service) QueryMetricNoted(ctx context.Context, f chstore.MetricQueryFilter) ([]chstore.SpanMetricSeries, string, error)
  • func (s *Service) QueryPromQLRange(ctx context.Context, query string, from, to time.Time, ...) ([]chstore.SpanMetricSeries, error)
  • func (s *Service) ResolvedRateWindowFloor() int
  • func (s *Service) SavePersisted(ctx context.Context, store settingsStore, cfg Settings) error
  • func (s *Service) Snapshot() Snapshot
  • func (s *Service) StartConfigRefresh(ctx context.Context, store settingsStore, interval time.Duration)
  • func (s *Service) Test(ctx context.Context, cfg Settings) error
• type Settings
• type Snapshot

Constants

const (
	MinRateWindowFloorSec = 10
	MaxRateWindowFloorSec = 3600
)

The bounds an OPERATOR-SET floor must satisfy (v0.9.1164). The setting exists because 5m is a substitute for a number VM will not tell us (see above), and a substitute is exactly the kind of constant an install with a known export interval should be able to correct: a 10s-scrape Prometheus federation wants ~30s here, not 300s, and the 5m default is currently smoothing five minutes of detail out of every rate chart on that install.

10s floor  — below the finest OTLP export interval anyone runs, and a
             window under ~10s makes rate() dependent on scrape jitter.
3600s ceiling — an hour is already wider than any panel's step at the
             ranges Coremetry offers; past it the "floor" would BE the
             window on every chart, which is a different feature (and
             one the caller-supplied RateWindowSec already provides
             per-query).

0 stays the sentinel for "unset — use the default". A separate `null` spelling would only add a second way to say the same thing to the JSON blob, and the frontend already renders 0 as an empty box (the aiTuning contract). Exported because the PUT validator lives in internal/api and must reject exactly what the reader would ignore. The bounds having ONE spelling is the point: a second copy in the handler is how a value gets accepted by the form and then silently dropped by the query — the operator sees their number saved in Settings and the old window in the chart, with nothing on screen connecting the two.

Variables

var ErrUnfilteredBuckets = errors.New("unfiltered bucket-family query refused by the VictoriaMetrics guard")

ErrUnfilteredBuckets marks a query Coremetry CAN express but REFUSES to send, because on a large VM install it would scan the whole bucket family (v0.9.1164).

Deliberately a SECOND sentinel rather than a reuse of ErrUnsupported, and the difference is the operator's next move. ErrUnsupported means "this query has no MetricsQL form" — nothing they change in Settings will make it work, they have to ask a different question. This one means "this query is fine and this install has it switched off": the fix is either a filter or a checkbox, and the message names both. Folding the two into one sentinel would make the checkbox undiscoverable, since the refusal would read as a permanent limitation of the backend.

Both map to 400 (internal/api/metricsource.go, upstream) — a refusal is a statement about the REQUEST, and a 502 here would send the operator to check a VictoriaMetrics that is perfectly healthy.

var ErrUnsupported = errors.New("unsupported by the VictoriaMetrics backend")

ErrUnsupported marks a query Coremetry cannot EXPRESS against VM — a refused aggregation, a filter operator with no MetricsQL matcher, DB instance scoping. It is deliberately distinct from a transport failure: VM is healthy and reachable, the REQUEST is the thing that does not translate.

The API maps it to 400, not 502. Reporting it as 502 would tell the operator their VictoriaMetrics is broken and send them to check a cluster that is fine — a wrong diagnosis is worse than a blunt one.

Functions

func ValidRateWindowFloor

func ValidRateWindowFloor(v int) bool

ValidRateWindowFloor reports whether a persisted/submitted floor is acceptable: unset (0), or inside the bounds. Shared by the PUT validator (which 400s on false) and by resolveRateWindowFloor.

Types

type Service

type Service struct {
	// contains filtered or unexported fields
}

Service is the per-process VM client. Config is swapped under an RWMutex so an admin PUT takes effect without a restart, and every method is nil-receiver-safe.

func New

func New() *Service

func (*Service) Available

func (s *Service) Available() bool

Available reports whether VM CAN be read at all — a base URL exists, regardless of the Enabled toggle (v0.9.1151, deneme modu).

The two predicates answer DIFFERENT questions and the split is the whole feature:

Configured() → "VM is the default for every metric read" (the
               Settings toggle the operator flips for the install)
Available()  → "a single ?metricsrc=vm request can reach VM" (the
               per-request trial gate)

Trial mode exists because metric NAMES differ between the two backends (VM sanitises dots to underscores). An operator has to see one real chart from VM before committing the whole install to it, and flipping the global toggle to find out would move every panel, picker and dashboard of every user at once.

func (*Service) Configure

func (s *Service) Configure(cfg Settings)

Configure swaps the live config.

func (*Service) Configured

func (s *Service) Configured() bool

Configured reports whether VM should serve the metric read surfaces BY DEFAULT. This is the predicate the API's source selector reads for a request that expresses no preference — enabled with an empty URL is not configured, so a half-filled form cannot route reads at a backend that cannot answer.

func (*Service) CurrentSettings

func (s *Service) CurrentSettings() Settings

CurrentSettings returns the full config INCLUDING the token — only for the handler's stored-token round-trip. Never write its return value to a response.

func (*Service) ListMetricNames

func (s *Service) ListMetricNames(ctx context.Context, service, pattern string, limit, offset int) ([]chstore.MetricInfo, int, error)

ListMetricNames answers the catalogue + MetricNamePicker.

VM reports metric NAMES and nothing else: /api/v1/label/__name__/values has no description, unit, instrument type, or last-seen. Those fields come back ZERO, which the frontend renders as "—". That is the honest shape — VM genuinely does not know (its /api/v1/metadata is a Prometheus-only surface VM does not populate) — and the source badge on the catalogue tells the operator which backend answered, so an empty Unit column reads as "VM doesn't report it" rather than "the metric is broken".

Pattern matching and paging are client-side (pageNames): VM's label- values endpoint offers neither. The response is bounded by the metric NAME cardinality, which is small even on huge installs — thousands of names, not millions of series.

func (*Service) LoadPersisted

func (s *Service) LoadPersisted(ctx context.Context, store settingsStore) error

LoadPersisted hydrates the in-memory config from system_settings. Missing blob = empty config (Configured() false → every read stays on ClickHouse). Called once at boot from main().

func (*Service) MetricAttrKeys

func (s *Service) MetricAttrKeys(ctx context.Context, metric, service string, since time.Duration) ([]string, error)

MetricAttrKeys answers "what can I write inside {}?".

__name__ is dropped: it is not an attribute key, and the CH sibling (which reads the attr_keys array) never returns it. Leaving it in would offer the operator a filter key that duplicates the metric selector.

v0.9.1159 — same candidate alternation as MetricLabelValues, and the reason discoveryNameCandidates stands the histogram family up on `_count` rather than `_bucket` lands HERE: `le` is a label on the bucket series, so a `_bucket` scope would hand the operator the histogram's own internal dimension as if it were one of their attributes.

func (*Service) MetricLabelValues

func (s *Service) MetricLabelValues(ctx context.Context, metric, key string, since time.Duration) ([]string, error)

MetricLabelValues answers the filter-value suggestion list. Capped at the same 200 the CH sibling uses so the picker behaves identically on both backends.

v0.9.1159 — the `match[]` scope is a candidate ALTERNATION (discoveryNameCandidates). Label discovery had the same spelling bug as the query path, plus one that bites harder: a histogram metric has no base series in VM, so the suggestion list came back empty for exactly the metric the operator was trying to filter, and an empty picker asserts "no such values" rather than "wrong name".

func (*Service) QueryMetric

func (s *Service) QueryMetric(ctx context.Context, f chstore.MetricQueryFilter) ([]chstore.SpanMetricSeries, error)

QueryMetric runs the multi-series time-bucketed query behind Explore, the dashboard "metric" panels and MCP query_metric.

Signature-identical to chstore.Store's — the seam invariant. The NOTE variant below carries the extra half; this wrapper drops it so callers that cannot render a note (MCP, the evaluator) are unaffected.

func (*Service) QueryMetricHistogram

func (s *Service) QueryMetricHistogram(ctx context.Context, f chstore.MetricQueryFilter) (*chstore.HistogramSeries, error)

QueryMetricHistogram renders an explicit histogram as the time × bucket heatmap + per-bucket p50/p95/p99 that /api/metrics/histogram returns.

Signature-identical to chstore.Store's, which is what lets the API seam (internal/api/metricsource.go) hold both behind one interface and turn any future drift into a compile error.

The window is normalised HERE, the same way QueryMetric normalises it, so a from/to-less call cannot become an unbounded VM query. Note this is one place the two backends genuinely differ: the CH path leaves a zero window unbounded in the WHERE and returns early with bounds only, relying on its LIMIT + max_execution_time. Its behaviour is untouched; VM gets the bound because on this side there is no LIMIT to fall back on.

func (*Service) QueryMetricNoted

func (s *Service) QueryMetricNoted(ctx context.Context, f chstore.MetricQueryFilter) ([]chstore.SpanMetricSeries, string, error)

QueryMetricNoted is QueryMetric plus an OPERATOR-FACING NOTE explaining an empty result (v0.9.1157, Faz 2).

It is a separate method rather than a wider QueryMetric because QueryMetric's signature is load-bearing: the API seam is an interface both this and *chstore.Store satisfy, and matching them is what turns future drift into a compile error. So the note rides an OPTIONAL capability the HTTP layer type-asserts for (internal/api's metricNoteSource) and the ClickHouse source simply does not implement — it has nothing to explain, because its bucket layout is in the row rather than in a guessed name.

The note fires for exactly one case: a PERCENTILE that returned zero series. Scoped that tightly on purpose. An empty gauge query is honestly empty and we know nothing the operator does not; a percentile queried `<name>_bucket`, a series they never typed and cannot see, so "no data", "not a histogram" and "your write path names buckets differently" all render as one blank chart with three different fixes.

The window is normalized HERE the same way the CH path normalizes it (zero To → now, zero From → 24h back) so an unbounded call cannot become an unbounded VM query.

func (*Service) QueryPromQLRange

func (s *Service) QueryPromQLRange(ctx context.Context, query string, from, to time.Time, stepSeconds, maxDataPoints int) ([]chstore.SpanMetricSeries, error)

QueryPromQLRange proxies a RAW operator-written query to VM's query_range.

THERE IS NO PRE-VALIDATION PARSER ON THIS PATH, ON PURPOSE.

The ClickHouse sibling parses first (internal/promql) because it has to: its evaluator implements a subset and must reject what it cannot push into ClickHouse. Reusing that parser here would make Coremetry the arbiter of a dialect it does not implement — VM speaks MetricsQL, a PromQL SUPERSET, so `WITH(…)` templates, subqueries, `keep_metric_names`, `rollup_rate` and the rest are valid queries our parser rejects. An operator whose query works in vmui and 400s in Coremetry would reasonably conclude the endpoint is broken.

So the string travels verbatim and VM answers. A syntax error comes back as VM's own envelope error (status != success), which promapi surfaces with VM's message intact — a better diagnosis than ours would have been, since it comes from the engine that will actually run the query.

The BOUNDS that matter are still ours: the API caps the query length before this is reached, the window is normalised here, and the step goes through promStep — so the point count per series stays under VM's own ceiling instead of earning a 4xx on a wide window.

func (*Service) ResolvedRateWindowFloor

func (s *Service) ResolvedRateWindowFloor() int

ResolvedRateWindowFloor — cache anahtarı için ÇÖZÜLMÜŞ taban (v0.9.1165). Persisted değer değil: 0 ve 300 aynı davranıştır, aynı tag'i üretmeli ki gereksiz cache kaçırması olmasın. Neden anahtarda: taban sorgunun EMİTTED penceresini değiştirir ama istek bayt-aynıdır — ayar PUT'undan sonraki TTL boyunca eski tabanın gövdesi servis edilir (v0.5.187 sınıfının ayar-girdili hâli; 1164 canlı probu tam bunu "kablo kopuk" olarak gösterdi — kopuk olan kablo değil ANAHTARDI).

func (*Service) SavePersisted

func (s *Service) SavePersisted(ctx context.Context, store settingsStore, cfg Settings) error

SavePersisted writes the typed config to system_settings and swaps the live one. The handler merges the stored token in first so a partial update cannot blank it.

func (*Service) Snapshot

func (s *Service) Snapshot() Snapshot

Snapshot returns the public config view (no token).

func (*Service) StartConfigRefresh

func (s *Service) StartConfigRefresh(ctx context.Context, store settingsStore, interval time.Duration)

StartConfigRefresh keeps the in-memory config in sync with the shared persisted blob across pods (tempo/thanos precedent). interval ≤ 0 → 30s. The Redis config:victoria-metrics publish makes the common case sub-50ms; this poll is the backstop when Redis is absent.

func (*Service) Test

func (s *Service) Test(ctx context.Context, cfg Settings) error

Test probes a CANDIDATE config without saving or swapping it — the Settings tab's "Test" button. `up` is the query because it is the one series name every Prometheus-shaped store has an opinion about; an empty result is still a successful ANSWER (VM is reachable and speaking the API), so only transport / auth / envelope failures fail the probe.

type Settings

type Settings struct {
	Enabled bool   `json:"enabled"`
	BaseURL string `json:"baseUrl"`
	// AuthType — none | bearer. VM itself has no auth; a bearer token
	// covers the vmauth / ingress-with-JWT deployments operators actually
	// run. Basic auth is absent on purpose (no operator asked, and an
	// unused credential path is a liability).
	AuthType string `json:"authType,omitempty"`
	// Token holds the bearer token. Never echoed in Snapshot() — the UI
	// sees HasToken so the operator can tell one is configured.
	Token string `json:"token,omitempty"`
	// InsecureSkipVerify disables TLS chain verification. Named to match
	// the four sibling settings blobs (tempo / thanos / devops /
	// logstore) so the frontend form and the audit details read the same
	// across all of them.
	InsecureSkipVerify bool `json:"insecureSkipVerify,omitempty"`
	// RateWindowFloorS overrides the 300s rate/last lookbehind floor
	// (v0.9.1164). 0 = unset, use promLookbehindFloorSec; otherwise
	// [10, 3600] — validated on PUT, re-checked on read
	// (resolveRateWindowFloor) so a hand-edited blob cannot emit a window
	// nobody chose. The floor never reaches increase() or the heatmap; see
	// promRollupWindow.
	RateWindowFloorS int `json:"rateWindowFloorS,omitempty"`
	// AllowUnfilteredPercentiles lifts the bucket-scan guard
	// (v0.9.1164). The DEFAULT — false, the zero value — is the PROTECTED
	// state, which is the direction that matters: a fresh install, a
	// missing blob and a partially-written blob all land on "guarded", so
	// the protection can only be removed by an explicit admin decision
	// that lands in audit_log. See guardBucketScan for the decision table.
	AllowUnfilteredPercentiles bool `json:"allowUnfilteredPercentiles,omitempty"`
}

Settings is the persisted VictoriaMetrics read-backend config. One endpoint per install: vmselect already fans out across a cluster, so federation is VM's job, not ours.

type Snapshot

type Snapshot struct {
	Enabled            bool   `json:"enabled"`
	BaseURL            string `json:"baseUrl"`
	AuthType           string `json:"authType,omitempty"`
	HasToken           bool   `json:"hasToken"`
	InsecureSkipVerify bool   `json:"insecureSkipVerify,omitempty"`
	// Both v0.9.1164 knobs round-trip in full: neither is secret-adjacent,
	// and the form has to be able to show the operator the floor they set
	// (an input that cannot read its own stored value re-submits a blank on
	// every unrelated save — the aiTuning failure class).
	RateWindowFloorS           int  `json:"rateWindowFloorS,omitempty"`
	AllowUnfilteredPercentiles bool `json:"allowUnfilteredPercentiles,omitempty"`
}

Snapshot is what GET /api/settings/victoria-metrics returns: Settings with the token replaced by a presence bit.