Thanos remote read adapter
- Long term storage: Using Thanos as long term storage for a Prometheus instance, in a very transparent way (no need for users to even see thanos-query). This could be part of a migration strategy to fully use Thanos for example.
- "ruler": Running rules outside of Thanos, to provide more predictable query reliability than thanos-ruler can offer.
- With Geras: Use Geras to serve OpenTSDB data on Thanos StoreAPI, then make that data available to Prometheus.
Use the Docker image.
Or build latest master:
go get github.com/G-Research/thanos-remote-read
This will give you
$(go env GOPATH)/bin/thanos-remote-read
This is a very simple proxy, so it accepts options of where to listen and where to forward to:
Usage of ./thanos-remote-read: -listen string [ip]:port to serve HTTP on (default ":10080") -store string Thanos Store API gRPC endpoint (default "localhost:10901") -ignore-warnings Ignore warnings from Thanos (default false)
./thanos-remote-read -store localhost:10901
See the use cases for more background,
-store can be pointed to anything
implementing Thanos StoreAPI, e.g.
thanos query, etc.
remote_read: - url: "http://localhost:10080/api/v1/read" # Potentially other options, see use cases docs and # https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_read
Ignoring certain selectors
In some situations you may have selectors that are present as external labels
on your Prometheus instance (e.g.
These will be passed to the remote read query. In general this is what you want for using Thanos as a way to make longer term storage visible to the Prometheus instance the data originated from. However in some cases you may have data aggregated in different ways in Thanos.
To remove specific labels from the query that is sent to Thanos do something like:
- url: "http://localhost:10080/api/v1/read?ignore=prometheus_replica"
Note that this will not do de-duplication across these labels, so you will need
to make sure if needed to either aggregate in Prometheus (e.g. something like
max(...) without(prometheus_replica)), or Thanos (via ruler).
Loading data from remote read may or may not be considered an error for the
whole query, depending on your exact use case. The default is to propagate the
error to Prometheus, as then you receive a warning when querying. This can be
changed with the
We welcome new contributors! We'll happily receive PRs for bug fixes or small changes. If you're contemplating something larger please get in touch first by opening a GitHub Issue describing the problem and how you propose to solve it.
Copyright 2020 G-Research
Licensed under the Apache License, Version 2.0 (the "License"); you may not use these files except in compliance with the License. You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Binary thanos-remote-read provides an adapter from Prometheus remote read protocol to Thanos StoreAPI.