indexheader

package
v0.34.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 19, 2024 License: Apache-2.0 Imports: 31 Imported by: 1

Documentation

Index

Constants

View Source 
const (
	// BinaryFormatV1 represents first version of index-header file.
	BinaryFormatV1 = 1

	// MagicIndex are 4 bytes at the head of an index-header file.
	MagicIndex = 0xBAAAD792
)

Variables

View Source 
var NotFoundRange = index.Range{Start: -1, End: -1}
View Source 
var NotFoundRangeErr = errors.New("range not found")

NotFoundRangeErr is an error returned by PostingsOffset when there is no posting for given name and value pairs.

Functions

func AlwaysEagerDownloadIndexHeader added in v0.34.0 

func AlwaysEagerDownloadIndexHeader(meta *metadata.Meta) bool

AlwaysEagerDownloadIndexHeader always eagerly download index header.

func AlwaysLazyDownloadIndexHeader added in v0.34.0 

func AlwaysLazyDownloadIndexHeader(meta *metadata.Meta) bool

AlwaysLazyDownloadIndexHeader always lazily download index header.

func WrapWithParallel added in v0.33.0 

func WrapWithParallel(b objstore.BucketReader, tmpDir string) objstore.BucketReader

func WriteBinary added in v0.11.0 

func WriteBinary(ctx context.Context, bkt objstore.BucketReader, id ulid.ULID, filename string) ([]byte, error)

WriteBinary build index header from the pieces of index in object storage, and cached in file if necessary.

Types

type BinaryReader added in v0.11.0 

type BinaryReader struct {
	// contains filtered or unexported fields
}

func NewBinaryReader added in v0.11.0 

func NewBinaryReader(ctx context.Context, logger log.Logger, bkt objstore.BucketReader, dir string, id ulid.ULID, postingOffsetsInMemSampling int, metrics *BinaryReaderMetrics) (*BinaryReader, error)

NewBinaryReader loads or builds new index-header if not present on disk.

func (*BinaryReader) Close added in v0.11.0 

func (r *BinaryReader) Close() error

func (*BinaryReader) IndexVersion added in v0.11.0 

func (r *BinaryReader) IndexVersion() (int, error)

func (*BinaryReader) LabelNames added in v0.11.0 

func (r *BinaryReader) LabelNames() ([]string, error)

func (*BinaryReader) LabelValues added in v0.11.0 

func (r *BinaryReader) LabelValues(name string) ([]string, error)

func (*BinaryReader) LookupSymbol added in v0.11.0 

func (r *BinaryReader) LookupSymbol(ctx context.Context, o uint32) (string, error)

func (*BinaryReader) PostingsOffset added in v0.11.0 

func (r *BinaryReader) PostingsOffset(name, value string) (index.Range, error)

TODO(bwplotka): Get advantage of multi value offset fetch.

func (*BinaryReader) PostingsOffsets added in v0.33.0 

func (r *BinaryReader) PostingsOffsets(name string, values ...string) ([]index.Range, error)

PostingsOffsets implements Reader.

type BinaryReaderMetrics added in v0.34.0 

type BinaryReaderMetrics struct {
	// contains filtered or unexported fields
}

LazyBinaryReaderMetrics holds metrics tracked by LazyBinaryReader.

func NewBinaryReaderMetrics added in v0.34.0 

func NewBinaryReaderMetrics(reg prometheus.Registerer) *BinaryReaderMetrics

NewBinaryReaderMetrics makes new BinaryReaderMetrics.

type BinaryTOC added in v0.11.0 

type BinaryTOC struct {
	// Symbols holds start to the same symbols section as index related to this index header.
	Symbols uint64
	// PostingsOffsetTable holds start to the same Postings Offset Table section as index related to this index header.
	PostingsOffsetTable uint64
}

BinaryTOC is a table of content for index-header file.

type FileWriter added in v0.11.0 

type FileWriter struct {
	// contains filtered or unexported fields
}

func NewFileWriter added in v0.11.0 

func NewFileWriter(name string, size int) (*FileWriter, error)

TODO(bwplotka): Added size to method, upstream this.

func (*FileWriter) Close added in v0.11.0 

func (fw *FileWriter) Close() error

func (*FileWriter) Flush added in v0.11.0 

func (fw *FileWriter) Flush() error

func (*FileWriter) Pos added in v0.11.0 

func (fw *FileWriter) Pos() uint64

func (*FileWriter) Remove added in v0.11.0 

func (fw *FileWriter) Remove() error

func (*FileWriter) Sync added in v0.31.0 

func (fw *FileWriter) Sync() error

func (*FileWriter) Write added in v0.11.0 

func (fw *FileWriter) Write(bufs ...[]byte) error

type IndexHeaderLazyDownloadStrategy added in v0.34.0 

type IndexHeaderLazyDownloadStrategy string

IndexHeaderLazyDownloadStrategy specifies how to download index headers lazily. Only used when lazy mmap is enabled. 

const (
	// EagerDownloadStrategy always disables lazy downloading index headers.
	EagerDownloadStrategy IndexHeaderLazyDownloadStrategy = "eager"
	// LazyDownloadStrategy always lazily download index headers.
	LazyDownloadStrategy IndexHeaderLazyDownloadStrategy = "lazy"
)

func (IndexHeaderLazyDownloadStrategy) StrategyToDownloadFunc added in v0.34.0 

func (s IndexHeaderLazyDownloadStrategy) StrategyToDownloadFunc() LazyDownloadIndexHeaderFunc

type LazyBinaryReader added in v0.17.0 

type LazyBinaryReader struct {
	// contains filtered or unexported fields
}

LazyBinaryReader wraps BinaryReader and loads (mmap) the index-header only upon the first Reader function is called.

func NewLazyBinaryReader added in v0.17.0 

func NewLazyBinaryReader(
	ctx context.Context,
	logger log.Logger,
	bkt objstore.BucketReader,
	dir string,
	id ulid.ULID,
	postingOffsetsInMemSampling int,
	metrics *LazyBinaryReaderMetrics,
	binaryReaderMetrics *BinaryReaderMetrics,
	onClosed func(*LazyBinaryReader),
	lazyDownload bool,
) (*LazyBinaryReader, error)

NewLazyBinaryReader makes a new LazyBinaryReader. If the index-header does not exist on the local disk at dir location, this function will build it downloading required sections from the full index stored in the bucket. However, this function doesn't load (mmap) the index-header; it will be loaded at first Reader function call.

func (*LazyBinaryReader) Close added in v0.17.0 

func (r *LazyBinaryReader) Close() error

Close implements Reader. It unloads the index-header from memory (releasing the mmap area), but a subsequent call to any other Reader function will automatically reload it.

func (*LazyBinaryReader) IndexVersion added in v0.17.0 

func (r *LazyBinaryReader) IndexVersion() (int, error)

IndexVersion implements Reader.

func (*LazyBinaryReader) LabelNames added in v0.17.0 

func (r *LazyBinaryReader) LabelNames() ([]string, error)

LabelNames implements Reader.

func (*LazyBinaryReader) LabelValues added in v0.17.0 

func (r *LazyBinaryReader) LabelValues(name string) ([]string, error)

LabelValues implements Reader.

func (*LazyBinaryReader) LookupSymbol added in v0.17.0 

func (r *LazyBinaryReader) LookupSymbol(ctx context.Context, o uint32) (string, error)

LookupSymbol implements Reader.

func (*LazyBinaryReader) PostingsOffset added in v0.17.0 

func (r *LazyBinaryReader) PostingsOffset(name, value string) (index.Range, error)

PostingsOffset implements Reader.

func (*LazyBinaryReader) PostingsOffsets added in v0.33.0 

func (r *LazyBinaryReader) PostingsOffsets(name string, values ...string) ([]index.Range, error)

PostingsOffsets implements Reader.

type LazyBinaryReaderMetrics added in v0.17.0 

type LazyBinaryReaderMetrics struct {
	// contains filtered or unexported fields
}

LazyBinaryReaderMetrics holds metrics tracked by LazyBinaryReader.

func NewLazyBinaryReaderMetrics added in v0.17.0 

func NewLazyBinaryReaderMetrics(reg prometheus.Registerer) *LazyBinaryReaderMetrics

NewLazyBinaryReaderMetrics makes new LazyBinaryReaderMetrics.

type LazyDownloadIndexHeaderFunc added in v0.34.0 

type LazyDownloadIndexHeaderFunc func(meta *metadata.Meta) bool

LazyDownloadIndexHeaderFunc is used to determinte whether to download the index header lazily or not by checking its block metadata. Usecase can be by time or by index file size.

type MemoryWriter added in v0.31.0 

type MemoryWriter struct {
	// contains filtered or unexported fields
}

func NewMemoryWriter added in v0.31.0 

func NewMemoryWriter(id ulid.ULID, size int) *MemoryWriter

func (*MemoryWriter) Buffer added in v0.31.0 

func (mw *MemoryWriter) Buffer() []byte

func (*MemoryWriter) Close added in v0.31.0 

func (mw *MemoryWriter) Close() error

func (*MemoryWriter) Flush added in v0.31.0 

func (mw *MemoryWriter) Flush() error

func (*MemoryWriter) Pos added in v0.31.0 

func (mw *MemoryWriter) Pos() uint64

func (*MemoryWriter) Sync added in v0.31.0 

func (mw *MemoryWriter) Sync() error

func (*MemoryWriter) Write added in v0.31.0 

func (mw *MemoryWriter) Write(bufs ...[]byte) error

type Part added in v0.33.0 

type Part interface {
	Read(buf []byte) (int, error)
	Write(buf []byte) (int, error)
	Flush() error
}

type PosWriter added in v0.31.0 

type PosWriter interface {
	Pos() uint64
	Write(bufs ...[]byte) error
	Flush() error
	Sync() error
	Close() error
}

type PosWriterWithBuffer added in v0.32.0 

type PosWriterWithBuffer interface {
	PosWriter
	Buffer() []byte
}

type Reader 

type Reader interface {
	io.Closer

	// IndexVersion returns version of index.
	IndexVersion() (int, error)

	// PostingsOffsets returns start and end offsets for postings for given name and values.
	// Input values need to be sorted.
	// If the requested label name doesn't exist, then no posting and error will be returned.
	// If the requested label name exists, but some values don't exist, the corresponding index range
	// will be set to -1 for both start and end.
	PostingsOffsets(name string, value ...string) ([]index.Range, error)

	// PostingsOffset returns start and end offsets of postings for given name and value.
	// The end offset might be bigger than the actual posting ending, but not larger than the whole index file.
	// NotFoundRangeErr is returned when no index can be found for given name and value.
	PostingsOffset(name string, value string) (index.Range, error)

	// LookupSymbol returns string based on given reference.
	// Error is return if the symbol can't be found.
	LookupSymbol(ctx context.Context, o uint32) (string, error)

	// LabelValues returns all label values for given label name or error.
	// If no values are found for label name, or label name does not exists,
	// then empty string is returned and no error.
	LabelValues(name string) ([]string, error)

	// LabelNames returns all label names in sorted order.
	LabelNames() ([]string, error)
}

Reader is an interface allowing to read essential, minimal number of index fields from the small portion of index file called header.

type ReaderPool added in v0.17.0 

type ReaderPool struct {
	// contains filtered or unexported fields
}

ReaderPool is used to istantiate new index-header readers and keep track of them. When the lazy reader is enabled, the pool keeps track of all instantiated readers and automatically close them once the idle timeout is reached. A closed lazy reader will be automatically re-opened upon next usage.

func NewReaderPool added in v0.17.0 

func NewReaderPool(logger log.Logger, lazyReaderEnabled bool, lazyReaderIdleTimeout time.Duration, metrics *ReaderPoolMetrics, lazyDownloadFunc LazyDownloadIndexHeaderFunc) *ReaderPool

NewReaderPool makes a new ReaderPool.

func (*ReaderPool) Close added in v0.17.0 

func (p *ReaderPool) Close()

Close the pool and stop checking for idle readers. No reader tracked by this pool will be closed. It's the caller responsibility to close readers.

func (*ReaderPool) NewBinaryReader added in v0.17.0 

func (p *ReaderPool) NewBinaryReader(ctx context.Context, logger log.Logger, bkt objstore.BucketReader, dir string, id ulid.ULID, postingOffsetsInMemSampling int, meta *metadata.Meta) (Reader, error)

NewBinaryReader creates and returns a new binary reader. If the pool has been configured with lazy reader enabled, this function will return a lazy reader. The returned lazy reader is tracked by the pool and automatically closed once the idle timeout expires.

type ReaderPoolMetrics added in v0.23.0 

type ReaderPoolMetrics struct {
	// contains filtered or unexported fields
}

ReaderPoolMetrics holds metrics tracked by ReaderPool.

func NewReaderPoolMetrics added in v0.23.0 

func NewReaderPoolMetrics(reg prometheus.Registerer) *ReaderPoolMetrics

NewReaderPoolMetrics makes new ReaderPoolMetrics.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.