block

Documentation

Overview

Package block contains common functionality for interacting with TSDB blocks in the context of Thanos.

Index

• Constants
• func Delete(ctx context.Context, bucket objstore.Bucket, id ulid.ULID) error
• func Download(ctx context.Context, bucket objstore.Bucket, id ulid.ULID, dst string) error
• func IsBlockDir(path string) (id ulid.ULID, ok bool)
• func ReadIndexCache(fn string) (version int, symbols map[uint32]string, lvals map[string][]string, ...)
• func Repair(dir string, id ulid.ULID) (resid ulid.ULID, err error)
• func Upload(ctx context.Context, bkt objstore.Bucket, bdir string) error
• func VerifyIndex(fn string, minTime int64, maxTime int64) error
• func WriteIndexCache(fn string, r *index.Reader) error
• func WriteMetaFile(dir string, meta *Meta) error
• type IndexIssueStats
  • func GatherIndexIssueStats(fn string, minTime int64, maxTime int64) (stats IndexIssueStats, err error)
  • func (i IndexIssueStats) ErrSummary() error
• type Meta
  • func DownloadMeta(ctx context.Context, bkt objstore.Bucket, id ulid.ULID) (Meta, error)
  • func Finalize(bdir string, extLset map[string]string, resolution int64, ...) (*Meta, error)
  • func ReadMetaFile(dir string) (*Meta, error)
• type ThanosMeta

Constants

const (
	// MetaFilename is the known JSON filename for meta information.
	MetaFilename = "meta.json"
	// IndexFilename is the known index file for block index.
	IndexFilename = "index"
	// ChunksDirname is the known dir name for chunks with compressed samples.
	ChunksDirname = "chunks"

	// DebugMetas is a directory for debug meta files that happen in the past. Useful for debugging.
	DebugMetas = "debug/metas"
)

const IndexCacheFilename = "index.cache.json"

IndexCacheFilename is the canonical name for index cache files.

Functions

func Delete

func Delete(ctx context.Context, bucket objstore.Bucket, id ulid.ULID) error

Delete removes directory that is mean to be block directory. NOTE: Prefer this method instead of objstore.Delete to avoid deleting empty dir (whole bucket) by mistake.

func Download

func Download(ctx context.Context, bucket objstore.Bucket, id ulid.ULID, dst string) error

Download downloads directory that is mean to be block directory.

func IsBlockDir

func IsBlockDir(path string) (id ulid.ULID, ok bool)

func ReadIndexCache

func ReadIndexCache(fn string) (
	version int,
	symbols map[uint32]string,
	lvals map[string][]string,
	postings map[labels.Label]index.Range,
	err error,
)

ReadIndexCache reads an index cache file.

func Repair

func Repair(dir string, id ulid.ULID) (resid ulid.ULID, err error)

Repair open the block with given id in dir and creates a new one with the same data. It: - removes out of order duplicates - all "complete" outsiders (they will not accessed anyway) Fixable inconsistencies are resolved in the new block.

func Upload

func Upload(ctx context.Context, bkt objstore.Bucket, bdir string) error

Upload uploads block from given block dir that ends with block id. It makes sure cleanup is done on error to avoid partial block uploads. It also verifies basic features of Thanos block. TODO(bplotka): Ensure bucket operations have reasonable backoff retries.

func VerifyIndex

func VerifyIndex(fn string, minTime int64, maxTime int64) error

VerifyIndex does a full run over a block index and verifies that it fulfills the order invariants.

func WriteIndexCache

func WriteIndexCache(fn string, r *index.Reader) error

WriteIndexCache writes a cache file containing the first lookup stages for an index file.

func WriteMetaFile

func WriteMetaFile(dir string, meta *Meta) error

WriteMetaFile writes the given meta into <dir>/meta.json.

Types

type IndexIssueStats

type IndexIssueStats struct {
	Total int

	OutOfOrderCount int
	OutOfOrderSum   int
	ExactSum        int

	// Chunks that are before or after time range in meta.
	Outsiders int
	// Outsiders that will be never accessed. They are completely out of time range specified in block meta.
	CompleteOutsiders int
}

func GatherIndexIssueStats

func GatherIndexIssueStats(fn string, minTime int64, maxTime int64) (stats IndexIssueStats, err error)

GatherIndexIssueStats returns useful counters as well as outsider chunks (chunks outside of block time range) that helps to assess index health.

func (IndexIssueStats) ErrSummary

func (i IndexIssueStats) ErrSummary() error

type Meta

type Meta struct {
	Version int `json:"version"`

	tsdb.BlockMeta

	Thanos ThanosMeta `json:"thanos"`
}

Meta describes the a block's meta. It wraps the known TSDB meta structure and extends it by Thanos-specific fields.

func DownloadMeta

func DownloadMeta(ctx context.Context, bkt objstore.Bucket, id ulid.ULID) (Meta, error)

DownloadMeta downloads only meta file from bucket by block ID.

func Finalize

func Finalize(bdir string, extLset map[string]string, resolution int64, downsampledMeta *tsdb.BlockMeta) (*Meta, error)

Finalize sets Thanos meta to the block meta JSON and saves it to the disk. It also removes tombstones which are not useful for Thanos. NOTE: It should be used after writing any block by any Thanos component, otherwise we will miss crucial metadata.

func ReadMetaFile

func ReadMetaFile(dir string) (*Meta, error)

ReadMetaFile reads the given meta from <dir>/meta.json.

type ThanosMeta

type ThanosMeta struct {
	Labels     map[string]string `json:"labels"`
	Downsample struct {
		Resolution int64 `json:"resolution"`
	} `json:"downsample"`
}

ThanosMeta holds block meta information specific to Thanos.