oy-scrape-jitter
oy-scrape-jitter queries a Prometheus server to see how regular scrapes are.
Perfect scrape alignment happens when the distance between all the scrapes are
exactly the same. It enables Prometheus delta-of-delta
encoding to reduce significantly the size of the blocks.
Example usage
$ ./oy-scrape-jitter --prometheus.url=https://prometheus.demo.do.prometheus.io/ --log.results-only
ts=2022-04-26T12:11:34.659Z caller=main.go:117 level=info msg="overall results" aligned_targets=0 unaligned_targets=10 max_ms=25
This means that the maximum deviation seen in your scrape jobs is 25ms. You
could set --scrape.timestamp-tolerance=25ms
to reduce your disk usage over
time, by enabling Prometheus to correct timestamps up to 25ms.
Prometheus limits
Prometheus will only apply timestamp tolerance up to 1%. If your scrape interval
is 30s, you can only adjust timestamps up to 300ms. Setting a 500ms tolerance
will have no effects on jobs with a scrape interval lower than 500s, even if the
deviation is tiny.
Plotting the output
By using --plot.file=scrape.png
, you can generate a PNG file which shows the
scrape timestamps jitter with an histogram.