Documentation ¶
Overview ¶
Package index provides access to files through multilevel indexes.
A multilevel index contains one or more levels where the lowest level contains file type index entries which directly reference file content and the above levels contain range type index entries which directly reference a range of index entries and indirectly reference ranges of file content. Multilevel indexes are created using writers which provide functionality for creating indexes for new files or creating indexes based on other indexes (rooted by file or range type indexes). Reading a multilevel index requires setting up a reader which provide various indexing strategies and filters.
Index ¶
- Constants
- Variables
- func Generate(s string) []string
- func Merge(ctx context.Context, storage *chunk.Storage, indexes []*Index, ...) error
- func Perm(a []rune, f func([]rune))
- func PointsTo(idx *Index) []chunk.ID
- func SizeBytes(idx *Index) int64
- type Cache
- type File
- func (*File) Descriptor() ([]byte, []int)deprecated
- func (x *File) GetDataRefs() []*chunk.DataRef
- func (x *File) GetDatum() string
- func (x *File) MarshalLogObject(enc zapcore.ObjectEncoder) error
- func (*File) ProtoMessage()
- func (x *File) ProtoReflect() protoreflect.Message
- func (x *File) Reset()
- func (x *File) String() string
- func (m *File) Validate() error
- func (m *File) ValidateAll() error
- type FileMultiError
- type FileValidationError
- type Index
- func (*Index) Descriptor() ([]byte, []int)deprecated
- func (x *Index) GetFile() *File
- func (x *Index) GetNumFiles() int64
- func (x *Index) GetPath() string
- func (x *Index) GetRange() *Range
- func (x *Index) GetSizeBytes() int64
- func (x *Index) MarshalLogObject(enc zapcore.ObjectEncoder) error
- func (*Index) ProtoMessage()
- func (x *Index) ProtoReflect() protoreflect.Message
- func (x *Index) Reset()
- func (x *Index) String() string
- func (m *Index) Validate() error
- func (m *Index) ValidateAll() error
- type IndexMultiError
- type IndexValidationError
- type Option
- type PathRange
- type Range
- func (*Range) Descriptor() ([]byte, []int)deprecated
- func (x *Range) GetChunkRef() *chunk.DataRef
- func (x *Range) GetLastPath() string
- func (x *Range) GetOffset() int64
- func (x *Range) MarshalLogObject(enc zapcore.ObjectEncoder) error
- func (*Range) ProtoMessage()
- func (x *Range) ProtoReflect() protoreflect.Message
- func (x *Range) Reset()
- func (x *Range) String() string
- func (m *Range) Validate() error
- func (m *Range) ValidateAll() error
- type RangeMultiError
- type RangeValidationError
- type Reader
- type ShardConfig
- type Writer
Constants ¶
const ( // DefaultShardNumThreshold is the default for the NumFiles threshold that must // be met before a shard is created. DefaultShardNumThreshold = 1000000 // DefaultShardSizeThreshold is the default for the SizeBytes threshold that must // be met before a shard is created. DefaultShardSizeThreshold = units.GB )
const (
DefaultBatchThreshold = units.MB
)
Variables ¶
var File_internal_storage_fileset_index_index_proto protoreflect.FileDescriptor
Functions ¶
func Generate ¶
Generate generates the permutations of the passed in string and returns them sorted.
Types ¶
type File ¶
type File struct { Datum string `protobuf:"bytes,1,opt,name=datum,proto3" json:"datum,omitempty"` DataRefs []*chunk.DataRef `protobuf:"bytes,2,rep,name=data_refs,json=dataRefs,proto3" json:"data_refs,omitempty"` // contains filtered or unexported fields }
func (*File) Descriptor
deprecated
func (*File) GetDataRefs ¶
func (*File) MarshalLogObject ¶
func (x *File) MarshalLogObject(enc zapcore.ObjectEncoder) error
func (*File) ProtoMessage ¶
func (*File) ProtoMessage()
func (*File) ProtoReflect ¶ added in v2.7.0
func (x *File) ProtoReflect() protoreflect.Message
func (*File) Validate ¶ added in v2.8.0
Validate checks the field values on File with the rules defined in the proto definition for this message. If any rules are violated, the first error encountered is returned, or nil if there are no violations.
func (*File) ValidateAll ¶ added in v2.8.0
ValidateAll checks the field values on File with the rules defined in the proto definition for this message. If any rules are violated, the result is a list of violation errors wrapped in FileMultiError, or nil if none found.
type FileMultiError ¶ added in v2.8.0
type FileMultiError []error
FileMultiError is an error wrapping multiple validation errors returned by File.ValidateAll() if the designated constraints aren't met.
func (FileMultiError) AllErrors ¶ added in v2.8.0
func (m FileMultiError) AllErrors() []error
AllErrors returns a list of validation violation errors.
func (FileMultiError) Error ¶ added in v2.8.0
func (m FileMultiError) Error() string
Error returns a concatenation of all the error messages it wraps.
type FileValidationError ¶ added in v2.8.0
type FileValidationError struct {
// contains filtered or unexported fields
}
FileValidationError is the validation error returned by File.Validate if the designated constraints aren't met.
func (FileValidationError) Cause ¶ added in v2.8.0
func (e FileValidationError) Cause() error
Cause function returns cause value.
func (FileValidationError) Error ¶ added in v2.8.0
func (e FileValidationError) Error() string
Error satisfies the builtin error interface
func (FileValidationError) ErrorName ¶ added in v2.8.0
func (e FileValidationError) ErrorName() string
ErrorName returns error name.
func (FileValidationError) Field ¶ added in v2.8.0
func (e FileValidationError) Field() string
Field function returns field value.
func (FileValidationError) Key ¶ added in v2.8.0
func (e FileValidationError) Key() bool
Key function returns key value.
func (FileValidationError) Reason ¶ added in v2.8.0
func (e FileValidationError) Reason() string
Reason function returns reason value.
type Index ¶
type Index struct { Path string `protobuf:"bytes,1,opt,name=path,proto3" json:"path,omitempty"` // NOTE: range and file are mutually exclusive. Range *Range `protobuf:"bytes,2,opt,name=range,proto3" json:"range,omitempty"` File *File `protobuf:"bytes,3,opt,name=file,proto3" json:"file,omitempty"` // NOTE: num_files and size_bytes did not exist in older versions of 2.x, so // they will not be set. NumFiles int64 `protobuf:"varint,4,opt,name=num_files,json=numFiles,proto3" json:"num_files,omitempty"` SizeBytes int64 `protobuf:"varint,5,opt,name=size_bytes,json=sizeBytes,proto3" json:"size_bytes,omitempty"` // contains filtered or unexported fields }
Index stores an index to and metadata about a range of files or a file.
func (*Index) Descriptor
deprecated
func (*Index) GetNumFiles ¶
func (*Index) GetSizeBytes ¶
func (*Index) MarshalLogObject ¶
func (x *Index) MarshalLogObject(enc zapcore.ObjectEncoder) error
func (*Index) ProtoMessage ¶
func (*Index) ProtoMessage()
func (*Index) ProtoReflect ¶ added in v2.7.0
func (x *Index) ProtoReflect() protoreflect.Message
func (*Index) Validate ¶ added in v2.8.0
Validate checks the field values on Index with the rules defined in the proto definition for this message. If any rules are violated, the first error encountered is returned, or nil if there are no violations.
func (*Index) ValidateAll ¶ added in v2.8.0
ValidateAll checks the field values on Index with the rules defined in the proto definition for this message. If any rules are violated, the result is a list of violation errors wrapped in IndexMultiError, or nil if none found.
type IndexMultiError ¶ added in v2.8.0
type IndexMultiError []error
IndexMultiError is an error wrapping multiple validation errors returned by Index.ValidateAll() if the designated constraints aren't met.
func (IndexMultiError) AllErrors ¶ added in v2.8.0
func (m IndexMultiError) AllErrors() []error
AllErrors returns a list of validation violation errors.
func (IndexMultiError) Error ¶ added in v2.8.0
func (m IndexMultiError) Error() string
Error returns a concatenation of all the error messages it wraps.
type IndexValidationError ¶ added in v2.8.0
type IndexValidationError struct {
// contains filtered or unexported fields
}
IndexValidationError is the validation error returned by Index.Validate if the designated constraints aren't met.
func (IndexValidationError) Cause ¶ added in v2.8.0
func (e IndexValidationError) Cause() error
Cause function returns cause value.
func (IndexValidationError) Error ¶ added in v2.8.0
func (e IndexValidationError) Error() string
Error satisfies the builtin error interface
func (IndexValidationError) ErrorName ¶ added in v2.8.0
func (e IndexValidationError) ErrorName() string
ErrorName returns error name.
func (IndexValidationError) Field ¶ added in v2.8.0
func (e IndexValidationError) Field() string
Field function returns field value.
func (IndexValidationError) Key ¶ added in v2.8.0
func (e IndexValidationError) Key() bool
Key function returns key value.
func (IndexValidationError) Reason ¶ added in v2.8.0
func (e IndexValidationError) Reason() string
Reason function returns reason value.
type Option ¶
type Option func(r *Reader)
Option configures an index reader.
func WithPrefix ¶
WithPrefix sets a prefix filter for the read.
func WithShardConfig ¶
func WithShardConfig(config *ShardConfig) Option
WithShardConfig sets the sharding configuration.
type PathRange ¶
type PathRange struct {
Lower, Upper string
}
PathRange is a range of paths. The range is inclusive, exclusive: [Lower, Upper).
type Range ¶
type Range struct { Offset int64 `protobuf:"varint,1,opt,name=offset,proto3" json:"offset,omitempty"` LastPath string `protobuf:"bytes,2,opt,name=last_path,json=lastPath,proto3" json:"last_path,omitempty"` ChunkRef *chunk.DataRef `protobuf:"bytes,3,opt,name=chunk_ref,json=chunkRef,proto3" json:"chunk_ref,omitempty"` // contains filtered or unexported fields }
func (*Range) Descriptor
deprecated
func (*Range) GetChunkRef ¶
func (*Range) GetLastPath ¶
func (*Range) MarshalLogObject ¶
func (x *Range) MarshalLogObject(enc zapcore.ObjectEncoder) error
func (*Range) ProtoMessage ¶
func (*Range) ProtoMessage()
func (*Range) ProtoReflect ¶ added in v2.7.0
func (x *Range) ProtoReflect() protoreflect.Message
func (*Range) Validate ¶ added in v2.8.0
Validate checks the field values on Range with the rules defined in the proto definition for this message. If any rules are violated, the first error encountered is returned, or nil if there are no violations.
func (*Range) ValidateAll ¶ added in v2.8.0
ValidateAll checks the field values on Range with the rules defined in the proto definition for this message. If any rules are violated, the result is a list of violation errors wrapped in RangeMultiError, or nil if none found.
type RangeMultiError ¶ added in v2.8.0
type RangeMultiError []error
RangeMultiError is an error wrapping multiple validation errors returned by Range.ValidateAll() if the designated constraints aren't met.
func (RangeMultiError) AllErrors ¶ added in v2.8.0
func (m RangeMultiError) AllErrors() []error
AllErrors returns a list of validation violation errors.
func (RangeMultiError) Error ¶ added in v2.8.0
func (m RangeMultiError) Error() string
Error returns a concatenation of all the error messages it wraps.
type RangeValidationError ¶ added in v2.8.0
type RangeValidationError struct {
// contains filtered or unexported fields
}
RangeValidationError is the validation error returned by Range.Validate if the designated constraints aren't met.
func (RangeValidationError) Cause ¶ added in v2.8.0
func (e RangeValidationError) Cause() error
Cause function returns cause value.
func (RangeValidationError) Error ¶ added in v2.8.0
func (e RangeValidationError) Error() string
Error satisfies the builtin error interface
func (RangeValidationError) ErrorName ¶ added in v2.8.0
func (e RangeValidationError) ErrorName() string
ErrorName returns error name.
func (RangeValidationError) Field ¶ added in v2.8.0
func (e RangeValidationError) Field() string
Field function returns field value.
func (RangeValidationError) Key ¶ added in v2.8.0
func (e RangeValidationError) Key() bool
Key function returns key value.
func (RangeValidationError) Reason ¶ added in v2.8.0
func (e RangeValidationError) Reason() string
Reason function returns reason value.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader is used for reading a multilevel index.
func (*Reader) Shards ¶
Shards creates shards for the index based on the sharding configuration provided to the reader. Sharding takes advantage of the NumFiles and SizeBytes index metadata to efficiently traverse the multilevel index. A subtree is traversed only when a split point exists within it, which we know based on the NumFiles and SizeBytes values at the root of each subtree.
type ShardConfig ¶
ShardConfig is a sharding configuration. NumFiles is the number of files to target for each shard. SizeBytes is the size, in bytes, to target for each shard.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer is used for creating a multilevel index into a serialized file set. Each index level is a stream of byte length encoded index entries that are stored in chunk storage. Both file and range type indexes can be written to a writer. New levels above the written indexes will be created when the serialized indexes reach the batching threshold.
func (*Writer) WriteIndex ¶
WriteIndex writes an index entry.