Documentation
¶
Overview ¶
Package tenancy turns "who may read which telemetry" into the two shapes that can enforce it, from one input, so the two can never disagree.
The two shapes are:
- a vmauth configuration, where each principal is a user selected by a claim on its token and given the filters it is entitled to;
- the vm_access claim body an OIDC issuer mints into a token, which vmauth then templates into those same filters.
An estate uses one or the other depending on where it would rather hold the mapping — in the proxy's configuration, or in the issuer. Rendering both from one input is the point: a difference between them is a difference between what a token says a person may read and what the proxy lets them read, and that is not a difference anyone notices until it matters.
The scoping key is the CLUSTER and the NAMESPACE, under OpenTelemetry's names. A grant says "these namespaces on this cluster"; a project, a team or a product is a derivation from a name to a list of namespaces that lives with whoever writes the grants, never a label on the telemetry. The environment tier (`deployment.environment.name`) rides along on every signal and is deliberately not a key: two clusters can share a tier, so a filter on it would select both.
This package knows nothing about any estate. It takes principals and grants, validates them, and returns data structures. It reads no files and contacts nothing.
Index ¶
Constants ¶
const ( MetricsFilterPlaceholder = "{{.MetricsExtraFilters}}" LogsFilterPlaceholder = "{{.LogsExtraStreamFilters}}" )
The placeholders vmauth substitutes from the `vm_access` claim, as spelled in app/vmauth/jwt.go. Only the two this design uses are named; the rest are listed in supportedPlaceholders so that a typo is refused rather than forwarded as a literal.
const ( MetricsFilterArg = "extra_filters" LogsFilterArg = "extra_stream_filters" )
The query argument each store reads its enforced filter from.
They are different arguments on purpose. vmselect OR-s the `extra_filters` it receives; VictoriaLogs AND-s every `extra_stream_filters` into the query AND into every subquery inside it — which is what makes it an access-control mechanism rather than a convenience, because otherwise a subquery would escape it.
const ( // DefaultClusterLabel and DefaultNamespaceLabel are the METRICS label // keys, and what Config.ClusterLabel and Config.NamespaceLabel mean // when empty. DefaultClusterLabel = "k8s_cluster_name" DefaultNamespaceLabel = "k8s_namespace_name" // DefaultLogsClusterField and DefaultLogsNamespaceField are the LOG // stream fields, and what Config.LogsClusterField and // Config.LogsNamespaceField mean when empty. DefaultLogsClusterField = "k8s.cluster.name" DefaultLogsNamespaceField = "kubernetes.pod_namespace" // TracesClusterAttribute and TracesNamespaceAttribute are the span // resource attributes. Nothing in this package filters on them — the // trace store cannot be scoped, see AllowUnfilteredTraceReads — but // they are the third column of the same table, and a writer that // spells them differently is caught by the same test. TracesClusterAttribute = "k8s.cluster.name" TracesNamespaceAttribute = "k8s.namespace.name" // EnvironmentLabel and EnvironmentAttribute carry the environment // tier — `production`, `staging`, `development`, `test`, or whatever // the estate calls one. Descriptive only: it is never a key, because // two clusters can share a tier and a filter on it would select both. EnvironmentLabel = "deployment_environment_name" EnvironmentAttribute = "deployment.environment.name" )
The vocabulary: what every writer stamps and every filter selects on, per signal. These are OpenTelemetry's semantic-convention names, spelled the way each store can carry them.
The metrics spellings have underscores because a Prometheus label name cannot carry a dot. The log-path namespace field is the container-log agent's own native name rather than the convention's, because that agent cannot rename a field and the OTLP gateway can — so the gateway yields, and emits that spelling on its log pipeline beside the conventional one. charts/observability-emitters stamps exactly these; tests/agreement_test.go walks its rendered output and fails on any writer that spells one differently.
const AudienceClaim = "aud"
AudienceClaim is the claim the audience pin is rendered under.
It is fixed rather than an input. `aud` is the name OpenID Connect gives the claim, every issuer mints it under that name, and a second spelling of a spec-defined claim is not flexibility — it is a way to pin nothing at all while the configuration reads as though something were pinned.
Variables ¶
var ( MetricsRead = filteredRoute("metrics", MetricsFilterArg, MetricsFilterPlaceholder, "/prometheus/api/v1/query", "/prometheus/api/v1/query_range", "/prometheus/api/v1/series", "/prometheus/api/v1/labels", "/prometheus/api/v1/label/[^/]+/values", "/prometheus/api/v1/status/tsdb", "/prometheus/api/v1/status/buildinfo", "/prometheus/vmui.*", ) // MetricMetadataPath is the one endpoint in the paragraph above that // an estate may decide to admit, through // Config.AllowUnfilteredMetricMetadata. // // It returns every metric NAME in the store with its type and help // string, and no filter reaches it: measured against the store, // `extra_filters` naming a namespace that matches nothing returns // the same body as no filter at all. So admitting it tells every // principal which metrics exist, which on an install whose grants // are per-namespace is a list of the components and products another // tenant runs. // // It is separable from its three siblings because what it leaks is // bounded and the same for everyone: names, types and help. The // other three leak per-principal material — `/status/active_queries` // and `/status/top_queries` return other principals' query TEXT, // `/status/metric_names_stats` returns names with per-tenant counts // — and no flag here admits them. MetricMetadataPath = "/prometheus/api/v1/metadata" LogsRead = filteredRoute("logs", LogsFilterArg, LogsFilterPlaceholder, "/select/logsql/.*", "/select/vmui.*", ) // The trace store's read route, which carries NO filter, because // there is nothing to carry it in. // // VictoriaTraces routes `/select/jaeger/*` and `/select/tempo/*` away // from the LogsQL handler that reads `extra_filters` and into // handlers whose entire accepted input is // `tracecommon.GetCommonParams` — a tenant id from the // `AccountID`/`ProjectID` headers, `hidden_fields_filters`, and // `allow_partial_response`. `hidden_fields_filters` hides FIELDS from // a result, not rows, so it scopes nothing; the Jaeger query // parameters (`service`, `operation`, `tags`, ...) are the caller's // own and are overwritten rather than appended to, so a proxy cannot // narrow them. // // The headers are a real mechanism, but they are a different tenancy // model: they select one of the store's own tenant ids, and this // design scopes by LABEL precisely so that a fleet-wide question // stays answerable across tenants. Nothing writes per-tenant account // ids on the way in, so there would be nothing for them to select. // // So this route cannot be scoped through this proxy, and saying so is // the entire point of this declaration. See // Config.AllowUnfilteredTraceReads. TracesRead = unenforceableRoute("traces", "VictoriaTraces' Jaeger and Tempo select APIs accept no query argument that carries a server-side filter: "+ "their handlers take a tenant id from headers, `hidden_fields_filters` (which hides fields, not rows) "+ "and `allow_partial_response`, and nothing else. A principal given this route reads every tenant's spans", "/select/jaeger/.*", "/select/tempo/.*", ) )
MetricsRead, LogsRead and TracesRead are the routes a reader is given.
The PATHS are a list of named endpoints rather than a prefix. The tempting shapes are wrong in the same way: `/prometheus/.*` also matches `/prometheus/api/v1/write` and `/prometheus/api/v1/admin/tsdb/delete_series`; `/api/v1/.*` also matches `/api/v1/write` and `/api/v1/import`; `/.*` also matches `/internal/force_merge`, whose own authKey flag REPLACES the store's `-httpAuth.*` rather than adding to it. A reader's route that also accepts writes is not a reader's route, and nothing about it looks wrong until somebody uses it.
The FILTER is what makes the route a route for one principal rather than for everybody. See the comment at the top of this file.
charts/observability-stack renders these same routes into its VMUser objects, and its `tenancy` golden case exists to prove the two have not drifted.
var ( MetricsReadPaths = MetricsRead.Paths() LogsReadPaths = LogsRead.Paths() TracesReadPaths = TracesRead.Paths() )
MetricsReadPaths, LogsReadPaths and TracesReadPaths are the routes' paths, kept as plain lists for callers that only want to know what a reader may call.
They are derived rather than declared: a second copy of a route's paths is a second thing to forget to change.
Functions ¶
func CarriesFilter ¶
CarriesFilter reports whether a rendered url_prefix carries the given placeholder as the WHOLE value of the given query argument, which is the only form vmauth substitutes.
It is exported because the check belongs to whoever is holding a rendered route, and two of them are outside this package: the chart's agreement test, which walks the VMUser objects Helm produced, and anyone assembling a vmauth configuration from more than this library.
Types ¶
type Claim ¶
type Claim struct {
MetricsExtraFilters []string `json:"metrics_extra_filters,omitempty" yaml:"metrics_extra_filters,omitempty"`
LogsExtraStreamFilters []string `json:"logs_extra_stream_filters,omitempty" yaml:"logs_extra_stream_filters,omitempty"`
}
Claim is the vm_access claim body for one principal: what an OIDC issuer mints into a token so that vmauth can template the filters out of it.
Only the filter fields are set. Account and project ids are deliberately absent: this design scopes a query by label rather than by tenant id, because the log and trace stores cannot query across their own tenant ids and the fleet-wide question has to stay answerable.
The two lists are not the same list and are not the same length. MetricsExtraFilters carries one selector per grant, which vmselect OR-s. LogsExtraStreamFilters carries exactly one stream filter for the whole principal, because VictoriaLogs AND-s each one it is given — see logsFilter. They also name different fields, because the log store cannot carry the metrics label keys.
type Config ¶
type Config struct {
ClaimName string `json:"claimName" yaml:"claimName"`
// Audience is the OAuth client id this proxy's own tokens are minted
// for, pinned into every user's `match_claims` under AudienceClaim.
// It is required to render a vmauth configuration.
//
// It is required because vmauth validates a token's EXPIRY and, with
// OIDC discovery configured, its ISSUER, and nothing else. It has no
// audience option and never inspects `aud` on its own, so a token
// minted for any other client of the same issuer verifies here
// exactly as one minted for this proxy does, and is then selected by
// whichever principal its groups match. `match_claims` is the only
// place the check can be made, which is why the audience goes there
// rather than beside the issuer.
//
// It is whatever the issuer assigned — a UUID, a name, an
// `<id>@<project>` — and is escaped and anchored where it is
// rendered, so a client id carrying a dot or a `|` pins itself and
// nothing else. See claimMatch, which is also where the reason this
// value is escaped and a namespace name is refused is written down.
//
// It works whether the issuer mints `aud` as a string or as a list:
// vmauth tests a `match_claims` entry against an array claim element
// by element and matches if any one of them does, so the list form
// needs no special case here.
Audience string `json:"audience" yaml:"audience"`
MetricsBackend string `json:"metricsBackend,omitempty" yaml:"metricsBackend,omitempty"`
LogsBackend string `json:"logsBackend,omitempty" yaml:"logsBackend,omitempty"`
TracesBackend string `json:"tracesBackend,omitempty" yaml:"tracesBackend,omitempty"`
Principals []Principal `json:"principals" yaml:"principals"`
// ClusterLabel and NamespaceLabel are the label keys the collectors
// stamp on METRICS. Empty means DefaultClusterLabel and
// DefaultNamespaceLabel, which is what charts/observability-emitters
// stamps; they are inputs so that an estate whose collectors were not
// built here can state what its series actually carry.
ClusterLabel string `json:"clusterLabel,omitempty" yaml:"clusterLabel,omitempty"`
NamespaceLabel string `json:"namespaceLabel,omitempty" yaml:"namespaceLabel,omitempty"`
// LogsClusterField and LogsNamespaceField are the stream FIELD names
// the log store carries the same two dimensions under. Empty means
// DefaultLogsClusterField and DefaultLogsNamespaceField.
//
// They are separate inputs from the label keys, and defaulted
// separately, because the log path cannot carry the metrics names: a
// Prometheus label has no dots, and the container-log agent has no
// flag, header or ingest pipeline that renames a field. So the default
// on this path is what that agent natively writes — the namespace
// under `kubernetes.pod_namespace`, and the cluster under the
// conventional name it is given as a static extra field — and the
// OTLP gateway is configured to write the same spelling on its log
// pipeline. An earlier version of this package had no default here,
// because every default was right on one estate and wrong on the
// next; that was true while the field was derived from an estate's
// own namespace label, and it stopped being true when the key became
// the namespace's name, which every estate spells the same way.
//
// A filter naming a field the streams do not have matches nothing,
// and a stream filter that matches nothing is an empty result rather
// than an error. That is why these still exist as inputs: an estate
// whose log agent writes something else has to say so here.
LogsClusterField string `json:"logsClusterField,omitempty" yaml:"logsClusterField,omitempty"`
LogsNamespaceField string `json:"logsNamespaceField,omitempty" yaml:"logsNamespaceField,omitempty"`
// AllowUnfilteredTraceReads admits the trace store's read route even
// though nothing can scope it, and it has this name because that is
// what it does.
//
// vmauth applies a principal's filters by substituting them into the
// route, and VictoriaTraces' Jaeger and Tempo select APIs accept no
// query argument to substitute them into — see TracesRead. So a
// reader given those paths reads every namespace's spans on every
// cluster, whatever the claim beside the route says.
//
// The option exists because for some estates that is the right
// trade: one cluster, one team, traces that carry nothing a
// colleague may not see. It is off by default and has to be written
// down because the alternative — rendering the route anyway, since it
// looks like the other two — is the defect this package was fixed to
// remove, one level down.
//
// It admits exactly the trace route. It is not a general override,
// and no metrics or logs route can be rendered unfiltered by any
// value of it.
AllowUnfilteredTraceReads bool `json:"allowUnfilteredTraceReads,omitempty" yaml:"allowUnfilteredTraceReads,omitempty"`
// AllowUnfilteredMetricMetadata admits `/api/v1/metadata` on the
// metrics read route.
//
// The endpoint returns every metric name in the store with its type
// and help string, and no filter reaches it — so with it on, every
// principal learns which metrics exist, whatever their grant says.
// On an install whose grants are all `allNamespaces` that is nothing
// they could not already query; on one with per-namespace grants it
// is a list of what another tenant runs.
//
// It is off by default, and the cost of leaving it off is visible
// rather than hidden: Grafana's Prometheus-family datasources ask
// for this endpoint to put descriptions on metric names, and answer
// a 401 into their own logs when the proxy does not route it. The
// query builder keeps working — metric names come from
// `/label/__name__/values`, which IS filtered — and only the
// descriptions are missing.
AllowUnfilteredMetricMetadata bool `json:"allowUnfilteredMetricMetadata,omitempty" yaml:"allowUnfilteredMetricMetadata,omitempty"`
}
Config is the whole input.
ClaimName is the token claim carrying the principal's groups, e.g. "groups". MetricsBackend and LogsBackend are the addresses vmauth forwards to; they are the caller's, and this package neither defaults nor validates them beyond requiring them to be present when a vmauth configuration is rendered.
func (Config) RenderClaim ¶
RenderClaim returns the vm_access claim body for one principal.
The caller has usually validated the whole Config already; this validates the principal again rather than trusting that, because a claim minted from an unvalidated grant is a filter that may not mean what it says.
func (Config) RenderVMAuth ¶
func (c Config) RenderVMAuth(issuer string) (VMAuthConfig, error)
RenderVMAuth returns a vmauth configuration in which each principal is a user selected by its group AND by the audience its token was minted for, carrying its own filters, and reaching each store through a route that APPLIES them.
The routes themselves live in route.go. What happens here is the binding of each one to its backend, and the refusal of any binding that would forward a read the claim does not scope.
Backends are required here and not in Validate, because a Config is perfectly usable for RenderClaim without them — in the issuer-side shape, vmauth learns the backends from its own values file and the claim carries only the filters.
type Grant ¶
type Grant struct {
Cluster string `json:"cluster" yaml:"cluster"`
Namespaces []string `json:"namespaces,omitempty" yaml:"namespaces,omitempty"`
AllNamespaces bool `json:"allNamespaces,omitempty" yaml:"allNamespaces,omitempty"`
}
Grant is what one principal may read on one cluster.
Namespaces lists them by name. AllNamespaces means every namespace on that cluster, present or future.
The two are mutually exclusive and one is required. An empty Namespaces list does NOT mean "everything": a list that is empty because a derivation produced nothing is the most likely way to widen a grant by accident, so it is refused and breadth has to be asked for by name.
There is no project, team or product here on purpose. Those are derivations — a name that expands to a list of namespaces — and the expansion belongs to whoever writes the grants, where adding a namespace to a project is a visible act rather than a label somebody forgets.
type Principal ¶
type Principal struct {
Group string `json:"group,omitempty" yaml:"group,omitempty"`
Grants []Grant `json:"grants" yaml:"grants"`
// Groups is the alternative to Group for ONE population that arrives
// under several names: a token carrying ANY of them selects this
// principal, and the principal reads every grant it lists. Name is
// then required, because there is no single group to name the entry
// after.
//
// It exists for the case where the same role is granted the same
// reach on several clusters and each cluster spells the group
// differently (`cluster-a:role`, `cluster-b:role`). A token whose
// vmauth first match is one principal per cluster reads ONE cluster
// whichever it holds, because vmauth takes the first matching user
// and never a union of them; one principal listing every cluster in
// Grants and every spelling in Groups reads them all.
//
// Read the consequence before using it: a token holding ANY listed
// group is granted EVERY grant, so a token holding only the group for
// cluster-a reads cluster-b too. That is correct exactly when every
// holder of one spelling holds all of them, and a caller who cannot
// guarantee it should mint per-person `vm_access` claims (RenderClaim)
// instead. Each entry is escaped and anchored like Group.
Groups []string `json:"groups,omitempty" yaml:"groups,omitempty"`
// Name is the user's name when Groups is used. Ignored, and refused,
// beside Group.
Name string `json:"name,omitempty" yaml:"name,omitempty"`
}
Principal is a named population — a group in the issuer's vocabulary — and what it may read.
Group is matched against the token's claim; it is not a display name and not an address. It is whatever the identity provider calls that population, and is escaped and anchored where it is rendered — see claimMatch — so a group that reads as a regular expression matches only itself.
type ReadRoute ¶
type ReadRoute struct {
// contains filtered or unexported fields
}
ReadRoute is one row of a reader's url_map: the endpoints it may call, and the query argument that carries this principal's filter into the store behind them.
Its fields are unexported and it has no exported constructor, so the only ReadRoutes that exist are the ones declared in this file. That is deliberate: the defect this type prevents is a route somebody added without a filter, and a type whose value can be assembled field by field elsewhere prevents nothing.
func (ReadRoute) FilterArg ¶
FilterArg is the query argument the store reads the filter from, or "" on an unenforceable route.
func (ReadRoute) Placeholder ¶
Placeholder is the vmauth placeholder that carries the principal's filter into FilterArg, or "" on an unenforceable route.
func (ReadRoute) Unenforceable ¶
Unenforceable is why this route cannot be scoped, or "" when it can.
type VMAuthConfig ¶
type VMAuthConfig struct {
Users []VMAuthUser `json:"users" yaml:"users"`
}
VMAuthConfig is what vmauth loads.
type VMAuthJWT ¶
type VMAuthJWT struct {
OIDC string `json:"oidc,omitempty" yaml:"oidc,omitempty"`
MatchClaims map[string]string `json:"match_claims,omitempty" yaml:"match_claims,omitempty"`
DefaultVMAccess *Claim `json:"default_vm_access_claim,omitempty" yaml:"default_vm_access_claim,omitempty"`
}
VMAuthJWT selects a user by the claims on its token, and says what that token is taken to grant when it carries no `vm_access` claim of its own.
DefaultVMAccess belongs HERE and not beside url_map, which is where an earlier version of this file put it. vmauth's `UserInfo` has no such field, so a configuration with it one level higher does not merely lose the default — vmauth refuses to parse the file and exits:
cannot unmarshal AuthConfig data: yaml: unmarshal errors: line N: field default_vm_access_claim not found in type main.UserInfo
That is a fail-closed mistake rather than a leak, and the only reason it was caught is that somebody ran the rendered file against the binary. No golden can tell you that a field a program does not have is a field a program does not have.
type VMAuthURLMapRow ¶
type VMAuthURLMapRow struct {
SrcPaths []string `json:"src_paths" yaml:"src_paths"`
URLPrefix []string `json:"url_prefix" yaml:"url_prefix"`
}
VMAuthURLMapRow routes a set of paths to a backend.
type VMAuthUser ¶
type VMAuthUser struct {
Name string `json:"name,omitempty" yaml:"name,omitempty"`
JWT *VMAuthJWT `json:"jwt,omitempty" yaml:"jwt,omitempty"`
URLMap []VMAuthURLMapRow `json:"url_map,omitempty" yaml:"url_map,omitempty"`
RetryStatusCodes []int `json:"retry_status_codes,omitempty" yaml:"retry_status_codes,omitempty"`
LoadBalancingPol string `json:"load_balancing_policy,omitempty" yaml:"load_balancing_policy,omitempty"`
}
VMAuthUser is one entry in vmauth's `users` list.
Source Files