index

package
v0.11.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 11, 2022 License: Apache-2.0 Imports: 22 Imported by: 4

Documentation

Overview

Package index manages content indices.

Index

Constants

View Source 
const (
	// Version1 identifies version 1 of the index, without content-level compression.
	Version1 = 1
)
View Source 
const (
	// Version2 identifies version 2 of the index, supporting content-level compression.
	Version2 = 2
)

Variables

View Source 
var AllIDs = IDRange{"", maxIDCharacterPlus1}

AllIDs is an IDRange that contains all valid IDs. nolint:gochecknoglobals

View Source 
var AllNonPrefixedIDs = IDRange{"0", "g"}

AllNonPrefixedIDs is an IDRange that contains all valid IDs non-prefixed IDs ('0' .. 'f'). nolint:gochecknoglobals

View Source 
var AllPrefixedIDs = IDRange{"g", maxIDCharacterPlus1}

AllPrefixedIDs is an IDRange that contains all valid IDs prefixed IDs ('g' .. 'z'). nolint:gochecknoglobals

View Source 
var EmptyID = ID{} // nolint:gochecknoglobals

EmptyID represents empty content ID.

Functions

This section is empty.

Types

type Builder 

type Builder map[ID]Info

Builder prepares and writes content index.

func (Builder) Add 

func (b Builder) Add(i Info)

Add adds a new entry to the builder or conditionally replaces it if the timestamp is greater.

func (Builder) Build 

func (b Builder) Build(output io.Writer, version int) error

Build writes the pack index to the provided output.

func (Builder) BuildShards 

func (b Builder) BuildShards(indexVersion int, stable bool, shardSize int) ([]gather.Bytes, func(), error)

BuildShards builds the set of index shards ensuring no more than the provided number of contents are in each index. Returns shard bytes and function to clean up after the shards have been written.

func (Builder) BuildStable 

func (b Builder) BuildStable(output io.Writer, version int) error

BuildStable writes the pack index to the provided output.

func (Builder) Clone 

func (b Builder) Clone() Builder

Clone returns a deep Clone of the Builder.

type FormatV1 

type FormatV1 struct {
	Version    byte   // format version number must be 0x01
	KeySize    byte   // size of each key in bytes
	EntrySize  uint16 // size of each entry in bytes, big-endian
	EntryCount uint32 // number of sorted (key,value) entries that follow

	Entries []struct {
		Key   []byte // key bytes (KeySize)
		Entry indexEntryInfoV1
	}

	ExtraData []byte // extra data
}

FormatV1 describes a format of a single pack index. The actual structure is not used, it's purely for documentation purposes. The struct is byte-aligned.

type FormatV2 

type FormatV2 struct {
	Header struct {
		Version           byte   // format version number must be 0x02
		KeySize           byte   // size of each key in bytes
		EntrySize         uint16 // size of each entry in bytes, big-endian
		EntryCount        uint32 // number of sorted (key,value) entries that follow
		EntriesOffset     uint32 // offset where `Entries` begins
		FormatInfosOffset uint32 // offset where `Formats` begins
		NumFormatInfos    uint32
		PacksOffset       uint32 // offset where `Packs` begins
		NumPacks          uint32
		BaseTimestamp     uint32 // base timestamp in unix seconds
	}

	Entries []struct {
		Key   []byte // key bytes (KeySize)
		Entry indexV2EntryInfo
	}

	// each entry contains offset+length of the name of the pack blob, so that each entry can refer to the index
	// and it resolves to a name.
	Packs []struct {
		PackNameLength byte   // length of the filename
		PackNameOffset uint32 // offset to data (within extra data)
	}

	// each entry represents unique content format.
	Formats []indexV2FormatInfo

	ExtraData []byte // extra data
}

FormatV2 describes a format of a single pack index. The actual structure is not used, it's purely for documentation purposes. The struct is byte-aligned.

type ID 

type ID struct {
	// contains filtered or unexported fields
}

ID is an identifier of content in content-addressable storage.

func IDFromHash added in v0.11.0 

func IDFromHash(prefix IDPrefix, hash []byte) (ID, error)

IDFromHash constructs content ID from a prefix and a hash.

func ParseID added in v0.11.0 

func ParseID(s string) (ID, error)

ParseID parses and validates the content ID.

func (ID) AppendToLogBuffer added in v0.11.0 

func (i ID) AppendToLogBuffer(sb *logging.Buffer)

AppendToLogBuffer appends content ID to log buffer.

func (ID) HasPrefix 

func (i ID) HasPrefix() bool

HasPrefix determines if the given ID has a non-empty prefix.

func (ID) Hash added in v0.11.0 

func (i ID) Hash() []byte

Hash returns the hash part of content ID.

func (ID) MarshalJSON added in v0.11.0 

func (i ID) MarshalJSON() ([]byte, error)

MarshalJSON implements JSON serialization.

func (ID) Prefix 

func (i ID) Prefix() IDPrefix

Prefix returns a one-character prefix of a content ID or an empty string.

func (ID) String added in v0.11.0 

func (i ID) String() string

String returns a string representation of ID.

func (*ID) UnmarshalJSON added in v0.11.0 

func (i *ID) UnmarshalJSON(v []byte) error

UnmarshalJSON implements JSON deserialization.

type IDPrefix added in v0.11.0 

type IDPrefix string

IDPrefix represents a content ID prefix (empty string or single character between 'g' and 'z').

func (IDPrefix) ValidateSingle added in v0.11.0 

func (p IDPrefix) ValidateSingle() error

ValidateSingle returns an error if a given prefix is invalid.

type IDRange 

type IDRange struct {
	StartID IDPrefix // inclusive
	EndID   IDPrefix // exclusive
}

IDRange represents a range of IDs.

func PrefixRange 

func PrefixRange(prefix IDPrefix) IDRange

PrefixRange returns ID range that contains all IDs with a given prefix.

func (IDRange) Contains 

func (r IDRange) Contains(id ID) bool

Contains determines whether given ID is in the range.

type Index 

type Index interface {
	io.Closer
	ApproximateCount() int
	GetInfo(contentID ID) (Info, error)

	// invoked the provided callback for all entries such that entry.ID >= startID and entry.ID < endID
	Iterate(r IDRange, cb func(Info) error) error
}

Index is a read-only index of packed contents.

func Open 

func Open(data []byte, closer func() error, v1PerContentOverhead uint32) (Index, error)

Open reads an Index from a given reader. The caller must call Close() when the index is no longer used.

type Info 

type Info interface {
	GetContentID() ID
	GetPackBlobID() blob.ID
	GetTimestampSeconds() int64
	Timestamp() time.Time
	GetOriginalLength() uint32
	GetPackedLength() uint32
	GetPackOffset() uint32
	GetDeleted() bool
	GetFormatVersion() byte
	GetCompressionHeaderID() compression.HeaderID
	GetEncryptionKeyID() byte
}

Info is an information about a single piece of content managed by Manager.

type InfoStruct 

type InfoStruct struct {
	ContentID           ID                   `json:"contentID"`
	PackBlobID          blob.ID              `json:"packFile,omitempty"`
	TimestampSeconds    int64                `json:"time"`
	OriginalLength      uint32               `json:"originalLength"`
	PackedLength        uint32               `json:"length"`
	PackOffset          uint32               `json:"packOffset,omitempty"`
	Deleted             bool                 `json:"deleted"`
	FormatVersion       byte                 `json:"formatVersion"`
	CompressionHeaderID compression.HeaderID `json:"compression,omitempty"`
	EncryptionKeyID     byte                 `json:"encryptionKeyID,omitempty"`
}

InfoStruct is an implementation of Info based on a structure.

func ToInfoStruct 

func ToInfoStruct(i Info) *InfoStruct

ToInfoStruct converts the provided Info to *InfoStruct.

func (*InfoStruct) GetCompressionHeaderID 

func (i *InfoStruct) GetCompressionHeaderID() compression.HeaderID

GetCompressionHeaderID implements the Info interface.

func (*InfoStruct) GetContentID 

func (i *InfoStruct) GetContentID() ID

GetContentID implements the Info interface.

func (*InfoStruct) GetDeleted 

func (i *InfoStruct) GetDeleted() bool

GetDeleted implements the Info interface.

func (*InfoStruct) GetEncryptionKeyID 

func (i *InfoStruct) GetEncryptionKeyID() byte

GetEncryptionKeyID implements the Info interface.

func (*InfoStruct) GetFormatVersion 

func (i *InfoStruct) GetFormatVersion() byte

GetFormatVersion implements the Info interface.

func (*InfoStruct) GetOriginalLength 

func (i *InfoStruct) GetOriginalLength() uint32

GetOriginalLength implements the Info interface.

func (*InfoStruct) GetPackBlobID 

func (i *InfoStruct) GetPackBlobID() blob.ID

GetPackBlobID implements the Info interface.

func (*InfoStruct) GetPackOffset 

func (i *InfoStruct) GetPackOffset() uint32

GetPackOffset implements the Info interface.

func (*InfoStruct) GetPackedLength 

func (i *InfoStruct) GetPackedLength() uint32

GetPackedLength implements the Info interface.

func (*InfoStruct) GetTimestampSeconds 

func (i *InfoStruct) GetTimestampSeconds() int64

GetTimestampSeconds implements the Info interface.

func (*InfoStruct) Timestamp 

func (i *InfoStruct) Timestamp() time.Time

Timestamp implements the Info interface.

type Merged 

type Merged []Index

Merged is an implementation of Index that transparently merges returns from underlying Indexes.

func (Merged) ApproximateCount 

func (m Merged) ApproximateCount() int

ApproximateCount implements Index interface.

func (Merged) Close 

func (m Merged) Close() error

Close closes all underlying indexes.

func (Merged) GetInfo 

func (m Merged) GetInfo(id ID) (Info, error)

GetInfo returns information about a single content. If a content is not found, returns (nil,nil).

func (Merged) Iterate 

func (m Merged) Iterate(r IDRange, cb func(i Info) error) error

Iterate invokes the provided callback for all unique content IDs in the underlying sources until either all contents have been visited or until an error is returned by the callback.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.