engine

package
v0.0.0-...-79ba347 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 1, 2023 License: GPL-3.0 Imports: 28 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrEndOfListing = shard.ErrEndOfListing

ErrEndOfListing is returned from an object listing with cursor when the storage can't return any more objects after the provided cursor. Use nil cursor object to start listing again.

Functions

func ContainerSize 

func ContainerSize(e *StorageEngine, id cid.ID) (uint64, error)

ContainerSize calls ContainerSize method on engine to calculate sum of estimation container sizes among all shards.

func Get 

func Get(storage *StorageEngine, addr oid.Address) (*objectSDK.Object, error)

Get reads object from local storage by provided address.

func GetRange 

func GetRange(storage *StorageEngine, addr oid.Address, rng *objectSDK.Range) ([]byte, error)

GetRange reads object payload range from local storage by provided address. 

func Head(storage *StorageEngine, addr oid.Address) (*objectSDK.Object, error)

Head reads object header from local storage by provided address.

func HeadRaw 

func HeadRaw(storage *StorageEngine, addr oid.Address, raw bool) (*objectSDK.Object, error)

HeadRaw reads object header from local storage by provided address and raw flag.

func List 

func List(storage *StorageEngine, limit uint64) ([]oid.Address, error)

List returns `limit` available physically storage object addresses in engine. If limit is zero, then returns all available object addresses.

func ListContainers 

func ListContainers(e *StorageEngine) ([]cid.ID, error)

ListContainers calls ListContainers method on engine to get a unique container IDs presented in the engine objects.

func Put 

func Put(storage *StorageEngine, obj *objectSDK.Object) error

Put writes provided object to local storage.

func Select 

func Select(storage *StorageEngine, cnr cid.ID, fs object.SearchFilters) ([]oid.Address, error)

Select selects objects from local storage using provided filters.

Types

type ContainerSizePrm 

type ContainerSizePrm struct {
	// contains filtered or unexported fields
}

ContainerSizePrm groups parameters of ContainerSize operation.

func (*ContainerSizePrm) SetContainerID 

func (p *ContainerSizePrm) SetContainerID(cnr cid.ID)

SetContainerID sets the identifier of the container to estimate the size.

type ContainerSizeRes 

type ContainerSizeRes struct {
	// contains filtered or unexported fields
}

ContainerSizeRes resulting values of ContainerSize operation.

func (ContainerSizeRes) Size 

func (r ContainerSizeRes) Size() uint64

Size returns calculated estimation of the container size.

type Cursor 

type Cursor struct {
	// contains filtered or unexported fields
}

Cursor is a type for continuous object listing.

type DeletePrm 

type DeletePrm struct {
	// contains filtered or unexported fields
}

DeletePrm groups the parameters of Delete operation.

func (*DeletePrm) WithAddress 

func (p *DeletePrm) WithAddress(addr oid.Address)

WithAddress is a Delete option to set the addresses of the objects to delete.

Option is required.

func (*DeletePrm) WithForceRemoval 

func (p *DeletePrm) WithForceRemoval()

WithForceRemoval is a Delete option to remove an object despite any restrictions imposed on deleting that object. Expected to be used only in control service.

type DeleteRes 

type DeleteRes struct{}

DeleteRes groups the resulting values of Delete operation.

type EvacuateShardPrm 

type EvacuateShardPrm struct {
	// contains filtered or unexported fields
}

EvacuateShardPrm represents parameters for the EvacuateShard operation.

func (*EvacuateShardPrm) WithFaultHandler 

func (p *EvacuateShardPrm) WithFaultHandler(f func(oid.Address, *objectSDK.Object) error)

WithFaultHandler sets handler to call for objects which cannot be saved on other shards.

func (*EvacuateShardPrm) WithIgnoreErrors 

func (p *EvacuateShardPrm) WithIgnoreErrors(ignore bool)

WithIgnoreErrors sets flag to ignore errors.

func (*EvacuateShardPrm) WithShardIDList 

func (p *EvacuateShardPrm) WithShardIDList(id []*shard.ID)

WithShardIDList sets shard ID.

type EvacuateShardRes 

type EvacuateShardRes struct {
	// contains filtered or unexported fields
}

EvacuateShardRes represents result of the EvacuateShard operation.

func (EvacuateShardRes) Count 

func (p EvacuateShardRes) Count() int

Count returns amount of evacuated objects. Objects for which handler returned no error are also assumed evacuated.

type FlushWriteCachePrm 

type FlushWriteCachePrm struct {
	// contains filtered or unexported fields
}

FlushWriteCachePrm groups the parameters of FlushWriteCache operation.

func (*FlushWriteCachePrm) SetIgnoreErrors 

func (p *FlushWriteCachePrm) SetIgnoreErrors(ignore bool)

SetIgnoreErrors sets errors ignore flag..

func (*FlushWriteCachePrm) SetShardID 

func (p *FlushWriteCachePrm) SetShardID(id *shard.ID)

SetShardID is an option to set shard ID.

Option is required.

type FlushWriteCacheRes 

type FlushWriteCacheRes struct{}

FlushWriteCacheRes groups the resulting values of FlushWriteCache operation.

type GetPrm 

type GetPrm struct {
	// contains filtered or unexported fields
}

GetPrm groups the parameters of Get operation.

func (*GetPrm) WithAddress 

func (p *GetPrm) WithAddress(addr oid.Address)

WithAddress is a Get option to set the address of the requested object.

Option is required.

type GetRes 

type GetRes struct {
	// contains filtered or unexported fields
}

GetRes groups the resulting values of Get operation.

func (GetRes) Object 

func (r GetRes) Object() *objectSDK.Object

Object returns the requested object.

type HeadPrm 

type HeadPrm struct {
	// contains filtered or unexported fields
}

HeadPrm groups the parameters of Head operation.

func (*HeadPrm) WithAddress 

func (p *HeadPrm) WithAddress(addr oid.Address)

WithAddress is a Head option to set the address of the requested object.

Option is required.

func (*HeadPrm) WithRaw 

func (p *HeadPrm) WithRaw(raw bool)

WithRaw is a Head option to set raw flag value. If flag is unset, then Head returns the header of the virtual object, otherwise it returns SplitInfo of the virtual object.

type HeadRes 

type HeadRes struct {
	// contains filtered or unexported fields
}

HeadRes groups the resulting values of Head operation.

func (HeadRes) Header 

func (r HeadRes) Header() *objectSDK.Object

Header returns the requested object header.

Instance has empty payload.

type Info 

type Info struct {
	Shards []shard.Info
}

Info groups the information about StorageEngine.

type InhumePrm 

type InhumePrm struct {
	// contains filtered or unexported fields
}

InhumePrm encapsulates parameters for inhume operation.

func (*InhumePrm) MarkAsGarbage 

func (p *InhumePrm) MarkAsGarbage(addrs ...oid.Address)

MarkAsGarbage marks an object to be physically removed from local storage.

Should not be called along with WithTarget.

func (*InhumePrm) WithForceRemoval 

func (p *InhumePrm) WithForceRemoval()

WithForceRemoval inhumes objects specified via MarkAsGarbage with GC mark without any object restrictions checks.

func (*InhumePrm) WithTarget 

func (p *InhumePrm) WithTarget(tombstone oid.Address, addrs ...oid.Address)

WithTarget sets a list of objects that should be inhumed and tombstone address as the reason for inhume operation.

tombstone should not be nil, addr should not be empty. Should not be called along with MarkAsGarbage.

type InhumeRes 

type InhumeRes struct{}

InhumeRes encapsulates results of inhume operation.

type ListContainersPrm 

type ListContainersPrm struct{}

ListContainersPrm groups parameters of ListContainers operation.

type ListContainersRes 

type ListContainersRes struct {
	// contains filtered or unexported fields
}

ListContainersRes groups the resulting values of ListContainers operation.

func (ListContainersRes) Containers 

func (r ListContainersRes) Containers() []cid.ID

Containers returns a list of identifiers of the containers in which local objects are stored.

type ListWithCursorPrm 

type ListWithCursorPrm struct {
	// contains filtered or unexported fields
}

ListWithCursorPrm contains parameters for ListWithCursor operation.

func (*ListWithCursorPrm) WithCount 

func (p *ListWithCursorPrm) WithCount(count uint32)

WithCount sets the maximum amount of addresses that ListWithCursor should return.

func (*ListWithCursorPrm) WithCursor 

func (p *ListWithCursorPrm) WithCursor(cursor *Cursor)

WithCursor sets a cursor for ListWithCursor operation. For initial request ignore this param or use nil value. For consecutive requests, use value from ListWithCursorRes.

type ListWithCursorRes 

type ListWithCursorRes struct {
	// contains filtered or unexported fields
}

ListWithCursorRes contains values returned from ListWithCursor operation.

func (ListWithCursorRes) AddressList 

func (l ListWithCursorRes) AddressList() []objectcore.AddressWithType

AddressList returns addresses selected by ListWithCursor operation.

func (ListWithCursorRes) Cursor 

func (l ListWithCursorRes) Cursor() *Cursor

Cursor returns cursor for consecutive listing requests.

type MetricRegister 

type MetricRegister interface {
	AddListContainersDuration(d time.Duration)
	AddEstimateContainerSizeDuration(d time.Duration)
	AddDeleteDuration(d time.Duration)
	AddExistsDuration(d time.Duration)
	AddGetDuration(d time.Duration)
	AddHeadDuration(d time.Duration)
	AddInhumeDuration(d time.Duration)
	AddPutDuration(d time.Duration)
	AddRangeDuration(d time.Duration)
	AddSearchDuration(d time.Duration)
	AddListObjectsDuration(d time.Duration)

	SetObjectCounter(shardID, objectType string, v uint64)
	AddToObjectCounter(shardID, objectType string, delta int)

	SetReadonly(shardID string, readonly bool)

	AddToContainerSize(cnrID string, size int64)
	AddToPayloadCounter(shardID string, size int64)
}

type Option 

type Option func(*cfg)

Option represents StorageEngine's constructor option.

func WithErrorThreshold 

func WithErrorThreshold(sz uint32) Option

WithErrorThreshold returns an option to specify size amount of errors after which shard is moved to read-only mode.

func WithLogger 

func WithLogger(l *logger.Logger) Option

WithLogger returns option to set StorageEngine's logger.

func WithMetrics 

func WithMetrics(v MetricRegister) Option

func WithShardPoolSize 

func WithShardPoolSize(sz uint32) Option

WithShardPoolSize returns option to specify size of worker pool for each shard.

type PutPrm 

type PutPrm struct {
	// contains filtered or unexported fields
}

PutPrm groups the parameters of Put operation.

func (*PutPrm) WithObject 

func (p *PutPrm) WithObject(obj *objectSDK.Object)

WithObject is a Put option to set object to save.

Option is required.

type PutRes 

type PutRes struct{}

PutRes groups the resulting values of Put operation.

type ReConfiguration 

type ReConfiguration struct {
	// contains filtered or unexported fields
}

func (*ReConfiguration) AddShard 

func (rCfg *ReConfiguration) AddShard(id string, opts []shard.Option)

AddShard adds a shard for the reconfiguration. Shard identifier is calculated from paths used in blobstor.

func (*ReConfiguration) SetErrorsThreshold 

func (rCfg *ReConfiguration) SetErrorsThreshold(errorsThreshold uint32)

SetErrorsThreshold sets a size amount of errors after which shard is moved to read-only mode.

func (*ReConfiguration) SetShardPoolSize 

func (rCfg *ReConfiguration) SetShardPoolSize(shardPoolSize uint32)

SetShardPoolSize sets a size of worker pool for each shard.

type RngPrm 

type RngPrm struct {
	// contains filtered or unexported fields
}

RngPrm groups the parameters of GetRange operation.

func (*RngPrm) WithAddress 

func (p *RngPrm) WithAddress(addr oid.Address)

WithAddress is a GetRng option to set the address of the requested object.

Option is required.

func (*RngPrm) WithPayloadRange 

func (p *RngPrm) WithPayloadRange(rng *objectSDK.Range)

WithPayloadRange is a GetRange option to set range of requested payload data.

Missing an option or calling with zero length is equivalent to getting the full payload range.

type RngRes 

type RngRes struct {
	// contains filtered or unexported fields
}

RngRes groups the resulting values of GetRange operation.

func (RngRes) Object 

func (r RngRes) Object() *objectSDK.Object

Object returns the requested object part.

Instance payload contains the requested range of the original object.

type SelectPrm 

type SelectPrm struct {
	// contains filtered or unexported fields
}

SelectPrm groups the parameters of Select operation.

func (*SelectPrm) WithContainerID 

func (p *SelectPrm) WithContainerID(cnr cid.ID)

WithContainerID is a Select option to set the container id to search in.

func (*SelectPrm) WithFilters 

func (p *SelectPrm) WithFilters(fs object.SearchFilters)

WithFilters is a Select option to set the object filters.

type SelectRes 

type SelectRes struct {
	// contains filtered or unexported fields
}

SelectRes groups the resulting values of Select operation.

func (SelectRes) AddressList 

func (r SelectRes) AddressList() []oid.Address

AddressList returns list of addresses of the selected objects.

type StorageEngine 

type StorageEngine struct {
	// contains filtered or unexported fields
}

StorageEngine represents FrostFS local storage engine.

func New 

func New(opts ...Option) *StorageEngine

New creates, initializes and returns new StorageEngine instance.

func (*StorageEngine) AddShard 

func (e *StorageEngine) AddShard(opts ...shard.Option) (*shard.ID, error)

AddShard adds a new shard to the storage engine.

Returns any error encountered that did not allow adding a shard. Otherwise returns the ID of the added shard.

func (*StorageEngine) BlockExecution 

func (e *StorageEngine) BlockExecution(err error) error

BlockExecution blocks the execution of any data-related operation. All blocked ops will return err. To resume the execution, use ResumeExecution method.

Сan be called regardless of the fact of the previous blocking. If execution wasn't blocked, releases all resources similar to Close. Can be called concurrently with Close and any data related method (waits for all executions to complete). Returns error if any Close has been called before.

Must not be called concurrently with either Open or Init.

Note: technically passing nil error will resume the execution, otherwise, it is recommended to call ResumeExecution for this.

func (*StorageEngine) Close 

func (e *StorageEngine) Close() error

Close releases all StorageEngine's components. Waits for all data-related operations to complete. After the call, all the next ones will fail.

The method MUST only be called when the application exits.

func (*StorageEngine) ContainerSize 

func (e *StorageEngine) ContainerSize(prm ContainerSizePrm) (res ContainerSizeRes, err error)

ContainerSize returns the sum of estimation container sizes among all shards.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) Delete 

func (e *StorageEngine) Delete(prm DeletePrm) (res DeleteRes, err error)

Delete marks the objects to be removed.

Returns an error if executions are blocked (see BlockExecution).

Returns apistatus.ObjectLocked if at least one object is locked. In this case no object from the list is marked to be deleted.

NOTE: Marks any object to be deleted (despite any prohibitions on operations with that object) if WithForceRemoval option has been provided.

func (*StorageEngine) DumpInfo 

func (e *StorageEngine) DumpInfo() (i Info)

DumpInfo returns information about the StorageEngine.

func (*StorageEngine) DumpShard 

func (e *StorageEngine) DumpShard(id *shard.ID, prm shard.DumpPrm) error

DumpShard dumps objects from the shard with provided identifier.

Returns an error if shard is not read-only.

func (*StorageEngine) Evacuate 

func (e *StorageEngine) Evacuate(prm EvacuateShardPrm) (EvacuateShardRes, error)

Evacuate moves data from one shard to the others. The shard being moved must be in read-only mode.

func (*StorageEngine) FlushWriteCache 

func (e *StorageEngine) FlushWriteCache(p FlushWriteCachePrm) (FlushWriteCacheRes, error)

FlushWriteCache flushes write-cache on a single shard.

func (*StorageEngine) Get 

func (e *StorageEngine) Get(prm GetPrm) (res GetRes, err error)

Get reads an object from local storage.

Returns any error encountered that did not allow to completely read the object part.

Returns an error of type apistatus.ObjectNotFound if the requested object is missing in local storage. Returns an error of type apistatus.ObjectAlreadyRemoved if the object has been marked as removed.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) GetRange 

func (e *StorageEngine) GetRange(prm RngPrm) (res RngRes, err error)

GetRange reads part of an object from local storage.

Returns any error encountered that did not allow to completely read the object part.

Returns an error of type apistatus.ObjectNotFound if the requested object is missing in local storage. Returns an error of type apistatus.ObjectAlreadyRemoved if the requested object is inhumed. Returns ErrRangeOutOfBounds if the requested object range is out of bounds.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) HandleNewEpoch 

func (e *StorageEngine) HandleNewEpoch(epoch uint64)

HandleNewEpoch notifies every shard about NewEpoch event.

func (*StorageEngine) Head 

func (e *StorageEngine) Head(prm HeadPrm) (res HeadRes, err error)

Head reads object header from local storage.

Returns any error encountered that did not allow to completely read the object header.

Returns an error of type apistatus.ObjectNotFound if the requested object is missing in local storage. Returns an error of type apistatus.ObjectAlreadyRemoved if the requested object was inhumed.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) Inhume 

func (e *StorageEngine) Inhume(prm InhumePrm) (res InhumeRes, err error)

Inhume calls metabase. Inhume method to mark an object as removed. It won't be removed physically from the shard until `Delete` operation.

Allows inhuming non-locked objects only. Returns apistatus.ObjectLocked if at least one object is locked.

NOTE: Marks any object as removed (despite any prohibitions on operations with that object) if WithForceRemoval option has been provided.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) Init 

func (e *StorageEngine) Init() error

Init initializes all StorageEngine's components.

func (*StorageEngine) List 

func (e *StorageEngine) List(limit uint64) (res SelectRes, err error)

List returns `limit` available physically storage object addresses in engine. If limit is zero, then returns all available object addresses.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) ListContainers 

func (e *StorageEngine) ListContainers(_ ListContainersPrm) (res ListContainersRes, err error)

ListContainers returns a unique container IDs presented in the engine objects.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) ListWithCursor 

func (e *StorageEngine) ListWithCursor(prm ListWithCursorPrm) (ListWithCursorRes, error)

ListWithCursor lists physical objects available in the engine starting from the cursor. It includes regular, tombstone and storage group objects. Does not include inhumed objects. Use cursor value from the response for consecutive requests.

Returns ErrEndOfListing if there are no more objects to return or count parameter set to zero.

func (*StorageEngine) Lock 

func (e *StorageEngine) Lock(idCnr cid.ID, locker oid.ID, locked []oid.ID) error

Lock marks objects as locked with another object. All objects from the specified container.

Allows locking regular objects only (otherwise returns apistatus.LockNonRegularObject).

Locked list should be unique. Panics if it is empty.

func (*StorageEngine) Open 

func (e *StorageEngine) Open() error

Open opens all StorageEngine's components.

func (*StorageEngine) Put 

func (e *StorageEngine) Put(prm PutPrm) (res PutRes, err error)

Put saves the object to local storage.

Returns any error encountered that did not allow to completely save the object.

Returns an error if executions are blocked (see BlockExecution).

Returns an error of type apistatus.ObjectAlreadyRemoved if the object has been marked as removed.

func (*StorageEngine) Reload 

func (e *StorageEngine) Reload(rcfg ReConfiguration) error

Reload reloads StorageEngine's configuration in runtime.

func (*StorageEngine) RestoreShard 

func (e *StorageEngine) RestoreShard(id *shard.ID, prm shard.RestorePrm) error

RestoreShard restores objects from dump to the shard with provided identifier.

Returns an error if shard is not read-only.

func (*StorageEngine) ResumeExecution 

func (e *StorageEngine) ResumeExecution() error

ResumeExecution resumes the execution of any data-related operation. To block the execution, use BlockExecution method.

Сan be called regardless of the fact of the previous blocking. If execution was blocked, prepares all resources similar to Open. Can be called concurrently with Close and any data related method (waits for all executions to complete). Returns error if any Close has been called before.

Must not be called concurrently with either Open or Init.

func (*StorageEngine) Select 

func (e *StorageEngine) Select(prm SelectPrm) (res SelectRes, err error)

Select selects the objects from local storage that match select parameters.

Returns any error encountered that did not allow to completely select the objects.

Returns an error if executions are blocked (see BlockExecution).

func (*StorageEngine) SetShardMode 

func (e *StorageEngine) SetShardMode(id *shard.ID, m mode.Mode, resetErrorCounter bool) error

SetShardMode sets mode of the shard with provided identifier.

Returns an error if shard mode was not set, or shard was not found in storage engine.

func (*StorageEngine) TreeAddByPath 

func (e *StorageEngine) TreeAddByPath(d pilorama.CIDDescriptor, treeID string, attr string, path []string, m []pilorama.KeyValue) ([]pilorama.Move, error)

TreeAddByPath implements the pilorama.Forest interface.

func (*StorageEngine) TreeApply 

func (e *StorageEngine) TreeApply(d pilorama.CIDDescriptor, treeID string, m *pilorama.Move, backgroundSync bool) error

TreeApply implements the pilorama.Forest interface.

func (*StorageEngine) TreeDrop 

func (e *StorageEngine) TreeDrop(cid cidSDK.ID, treeID string) error

TreeDrop implements the pilorama.Forest interface.

func (*StorageEngine) TreeExists 

func (e *StorageEngine) TreeExists(cid cidSDK.ID, treeID string) (bool, error)

TreeExists implements the pilorama.Forest interface.

func (*StorageEngine) TreeGetByPath 

func (e *StorageEngine) TreeGetByPath(cid cidSDK.ID, treeID string, attr string, path []string, latest bool) ([]pilorama.Node, error)

TreeGetByPath implements the pilorama.Forest interface.

func (*StorageEngine) TreeGetChildren 

func (e *StorageEngine) TreeGetChildren(cid cidSDK.ID, treeID string, nodeID pilorama.Node) ([]uint64, error)

TreeGetChildren implements the pilorama.Forest interface.

func (*StorageEngine) TreeGetMeta 

func (e *StorageEngine) TreeGetMeta(cid cidSDK.ID, treeID string, nodeID pilorama.Node) (pilorama.Meta, uint64, error)

TreeGetMeta implements the pilorama.Forest interface.

func (*StorageEngine) TreeGetOpLog 

func (e *StorageEngine) TreeGetOpLog(cid cidSDK.ID, treeID string, height uint64) (pilorama.Move, error)

TreeGetOpLog implements the pilorama.Forest interface.

func (*StorageEngine) TreeList 

func (e *StorageEngine) TreeList(cid cidSDK.ID) ([]string, error)

TreeList implements the pilorama.Forest interface.

func (*StorageEngine) TreeMove 

func (e *StorageEngine) TreeMove(d pilorama.CIDDescriptor, treeID string, m *pilorama.Move) (*pilorama.Move, error)

TreeMove implements the pilorama.Forest interface.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.