Version: v0.8.4 Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Apr 26, 2021 License: Apache-2.0 Imports: 13 Imported by: 1



Package object implements repository support for content-addressable objects of arbitrary size.



This section is empty.


View Source
var ErrObjectNotFound = errors.New("object not found")

ErrObjectNotFound is returned when an object cannot be found.


func VerifyObject added in v0.8.0

func VerifyObject(ctx context.Context, cr contentReader, oid ID) ([]content.ID, error)

VerifyObject ensures that all objects backing ObjectID are present in the repository and returns the content IDs of which it is composed.


type Format

type Format struct {
	Splitter string `json:"splitter,omitempty"` // splitter used to break objects into pieces of content

Format describes the format of objects in a repository.

type HasObjectID

type HasObjectID interface {
	ObjectID() ID

HasObjectID exposes the identifier of an object.

type ID

type ID string

ID is an identifier of a repository object. Repository objects can be stored.

1. In a single content block, this is the most common case for small objects. 2. In a series of content blocks with an indirect block pointing at them (multiple indirections are allowed).

This is used for larger files. Object IDs using indirect blocks start with "I"

func Compressed added in v0.4.0

func Compressed(objectID ID) ID

Compressed returns object ID with 'Z' prefix indicating it's compressed.

func DirectObjectID

func DirectObjectID(contentID content.ID) ID

DirectObjectID returns direct object ID based on the provided block ID.

func IndirectObjectID

func IndirectObjectID(indexObjectID ID) ID

IndirectObjectID returns indirect object ID based on the underlying index object ID.

func ParseID

func ParseID(s string) (ID, error)

ParseID converts the specified string into object ID.

func (ID) ContentID

func (i ID) ContentID() (id content.ID, compressed, ok bool)

ContentID returns the ID of the underlying content.

func (ID) IndexObjectID

func (i ID) IndexObjectID() (ID, bool)

IndexObjectID returns the object ID of the underlying index object.

func (ID) String

func (i ID) String() string

String returns string representation of ObjectID that is suitable for displaying in the UI.

func (ID) Validate

func (i ID) Validate() error

Validate checks the ID format for validity and reports any errors.

type Manager

type Manager struct {
	Format Format
	// contains filtered or unexported fields

Manager implements a content-addressable storage on top of blob storage.

func NewObjectManager

func NewObjectManager(ctx context.Context, bm contentManager, f Format) (*Manager, error)

NewObjectManager creates an ObjectManager with the specified content manager and format.

func (*Manager) Close added in v0.6.0

func (om *Manager) Close() error

Close closes the object manager.

func (*Manager) Concatenate added in v0.7.0

func (om *Manager) Concatenate(ctx context.Context, objectIDs []ID) (ID, error)

Concatenate creates an object that's a result of concatenation of other objects. This is more efficient than reading and rewriting the objects because Concatenate can efficiently merge index entries without reading the underlying contents.

This function exists primarily to facilitate efficient parallel uploads of very large files (>1GB). Due to bottleneck of splitting which is inherently sequential, we can only one use CPU core for each Writer, which limits throughput.

For example when uploading a 100 GB file it is beneficial to independently upload sections of [0..25GB), [25..50GB), [50GB..75GB) and [75GB..100GB) and concatenate them together as this allows us to run four splitters in parallel utilizing more CPU cores. Because some split points now start at fixed bounaries and not content-specific, this causes some slight loss of deduplication at concatenation points (typically 1-2 contents, usually <10MB), so this method should only be used for very large files where this overhead is relatively small.

func (*Manager) NewWriter

func (om *Manager) NewWriter(ctx context.Context, opt WriterOptions) Writer

NewWriter creates an ObjectWriter for writing to the repository.

type Reader

type Reader interface {
	Length() int64

Reader allows reading, seeking, getting the length of and closing of a repository object.

func Open added in v0.8.0

func Open(ctx context.Context, r contentReader, objectID ID) (Reader, error)

Open creates new ObjectReader for reading given object from a repository.

type Writer

type Writer interface {

	// Checkpoint returns ID of an object consisting of all contents written to storage so far.
	// This may not include some data buffered in the writer.
	// In case nothing has been written yet, returns empty object ID.
	Checkpoint() (ID, error)

	// Result returns object ID representing all bytes written to the writer.
	Result() (ID, error)

Writer allows writing content to the storage and supports automatic deduplication and encryption of written data.

type WriterOptions

type WriterOptions struct {
	Description string
	Prefix      content.ID // empty string or a single-character ('g'..'z')
	Compressor  compression.Name
	AsyncWrites int // allow up to N content writes to be asynchronous

WriterOptions can be passed to Repository.NewWriter().

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
t or T : Toggle theme light dark auto
y or Y : Canonical URL