metabase

package
v1.102.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 16, 2024 License: AGPL-3.0 Imports: 33 Imported by: 7

Documentation

Overview

Package metabase implements storing objects and segements.

Index

Constants

View Source 
const (
	Delimiter        = '/'
	LastSegmentName  = "l"
	LastSegmentIndex = uint32(math.MaxUint32)
)

Common constants for segment keys.

View Source 
const (
	// Pending means that the object is being uploaded or that the client failed during upload.
	// The failed upload may be continued in the future.
	Pending = ObjectStatus(1)

	// CommittedUnversioned means that the object is finished and should be visible for general listing.
	CommittedUnversioned = ObjectStatus(3)
	// CommittedVersioned means that the object is finished and should be visible for general listing.
	CommittedVersioned = ObjectStatus(4)
	// DeleteMarkerVersioned is inserted when an object is deleted in a versioning enabled bucket.
	DeleteMarkerVersioned = ObjectStatus(5)
	// DeleteMarkerUnversioned is inserted when an unversioned object is deleted in a versioning suspended bucket.
	DeleteMarkerUnversioned = ObjectStatus(6)

	// Prefix is an ephemeral status used during non-recursive listing.
	Prefix = ObjectStatus(7)
)
View Source 
const CopySegmentLimit = int64(10000)

CopySegmentLimit is the maximum number of segments that can be copied.

View Source 
const DefaultVersion = Version(1)

DefaultVersion represents default version 1.

View Source 
const DelimiterNext = "0"

DelimiterNext is the string that comes immediately after Delimiter="/".

View Source 
const ListLimit = intLimitRange(1000)

ListLimit is the maximum number of items the client can request for listing.

View Source 
const ListVerifyLimit = intLimitRange(100000)

ListVerifyLimit is the maximum number of items the client can request for listing.

View Source 
const MaxVersion = Version(math.MaxInt64 - 64)

MaxVersion represents maximum version. Version in DB is represented as INT8.

It uses `MaxInt64 - 64` to avoid issues with `-MaxVersion`.

View Source 
const MoveSegmentLimit = int64(10000)

MoveSegmentLimit is the maximum number of segments that can be moved.

View Source 
const NextVersion = Version(0)

NextVersion means that the version should be chosen automatically.

View Source 
const PendingVersion = Version(0)

PendingVersion represents version that is used for pending objects (with UsePendingObjects).

Variables

View Source 
var (
	// ErrObjectNotFound is used to indicate that the object does not exist.
	ErrObjectNotFound = errs.Class("object not found")
	// ErrInvalidRequest is used to indicate invalid requests.
	ErrInvalidRequest = errs.Class("metabase: invalid request")
	// ErrFailedPrecondition is used to indicate that some conditions in the request has failed.
	ErrFailedPrecondition = errs.Class("metabase: failed precondition")
	// ErrConflict is used to indicate conflict with the request.
	ErrConflict = errs.Class("metabase: conflict")
)
View Source 
var (
	// Error is the default error for metabase.
	Error = errs.Class("metabase")
	// ErrObjectAlreadyExists is used to indicate that object already exists.
	ErrObjectAlreadyExists = errs.Class("object already exists")
	// ErrPendingObjectMissing is used to indicate a pending object is no longer accessible.
	ErrPendingObjectMissing = errs.Class("pending object missing")
	// ErrPermissionDenied general error for denying permission.
	ErrPermissionDenied = errs.Class("permission denied")
	// ErrMethodNotAllowed general error when operation is not allowed.
	ErrMethodNotAllowed = errs.Class("method not allowed")
)
View Source 
var ErrSegmentNotFound = errs.Class("segment not found")

ErrSegmentNotFound is an error class for non-existing segment.

View Source 
var ErrValueChanged = errs.Class("value changed")

ErrValueChanged is returned when the current value of the key does not match the oldValue in UpdateSegmentPieces.

Functions

func EqualAliasPieces 

func EqualAliasPieces(xs, ys AliasPieces) bool

EqualAliasPieces compares whether xs and ys are equal.

Types

type Adapter added in v1.102.2 

type Adapter interface {
	BeginObjectNextVersion(context.Context, BeginObjectNextVersion, *Object) error
	GetObjectLastCommitted(ctx context.Context, opts GetObjectLastCommitted, object *Object) error
	TestingBeginObjectExactVersion(ctx context.Context, opts BeginObjectExactVersion, object *Object) error
	TestingBatchInsertSegments(ctx context.Context, segments []RawSegment) (err error)
}

Adapter is a low level extension point to use datasource related queries. TODO: we may need separated adapter for segments/objects/etc.

type AliasPiece 

type AliasPiece struct {
	Number uint16
	Alias  NodeAlias
}

AliasPiece is a piece with alias node ID.

type AliasPieces 

type AliasPieces []AliasPiece

AliasPieces is a slice of AliasPiece.

func (AliasPieces) Bytes 

func (aliases AliasPieces) Bytes() ([]byte, error)

Bytes compresses alias pieces to a slice of bytes.

func (*AliasPieces) Scan 

func (aliases *AliasPieces) Scan(src interface{}) error

Scan implements the database/sql Scanner interface.

func (*AliasPieces) SetBytes 

func (aliases *AliasPieces) SetBytes(data []byte) error

SetBytes decompresses alias pieces from a slice of bytes.

func (AliasPieces) Value 

func (aliases AliasPieces) Value() (driver.Value, error)

Value implements the database/sql/driver Valuer interface.

type BeginCopyObject added in v1.50.1 

type BeginCopyObject struct {
	ObjectLocation
	Version Version

	// VerifyLimits holds a callback by which the caller can interrupt the copy
	// if it turns out the copy would exceed a limit.
	VerifyLimits func(encryptedObjectSize int64, nSegments int64) error
}

BeginCopyObject holds all data needed begin copy object method.

type BeginCopyObjectResult added in v1.50.1 

type BeginCopyObjectResult BeginMoveCopyResults

BeginCopyObjectResult holds data needed to begin copy object.

type BeginMoveCopyResults added in v1.65.1 

type BeginMoveCopyResults struct {
	StreamID                  uuid.UUID
	Version                   Version
	EncryptedMetadata         []byte
	EncryptedMetadataKeyNonce []byte
	EncryptedMetadataKey      []byte
	EncryptedKeysNonces       []EncryptedKeyAndNonce
	EncryptionParameters      storj.EncryptionParameters
}

BeginMoveCopyResults holds all data needed to begin move and copy object methods.

type BeginMoveObject added in v1.39.4 

type BeginMoveObject struct {
	ObjectLocation
}

BeginMoveObject holds all data needed begin move object method.

type BeginMoveObjectResult added in v1.39.4 

type BeginMoveObjectResult BeginMoveCopyResults

BeginMoveObjectResult holds data needed to begin move object.

type BeginObjectExactVersion 

type BeginObjectExactVersion struct {
	ObjectStream

	ExpiresAt              *time.Time
	ZombieDeletionDeadline *time.Time

	EncryptedMetadata             []byte // optional
	EncryptedMetadataNonce        []byte // optional
	EncryptedMetadataEncryptedKey []byte // optional

	Encryption storj.EncryptionParameters
}

BeginObjectExactVersion contains arguments necessary for starting an object upload.

func (*BeginObjectExactVersion) Verify added in v1.43.1 

func (opts *BeginObjectExactVersion) Verify() error

Verify verifies get object reqest fields.

type BeginObjectNextVersion 

type BeginObjectNextVersion struct {
	ObjectStream

	ExpiresAt              *time.Time
	ZombieDeletionDeadline *time.Time

	EncryptedMetadata             []byte // optional
	EncryptedMetadataNonce        []byte // optional
	EncryptedMetadataEncryptedKey []byte // optional

	Encryption storj.EncryptionParameters
}

BeginObjectNextVersion contains arguments necessary for starting an object upload.

func (*BeginObjectNextVersion) Verify added in v1.43.1 

func (opts *BeginObjectNextVersion) Verify() error

Verify verifies get object request fields.

type BeginSegment 

type BeginSegment struct {
	ObjectStream

	Position SegmentPosition

	// TODO: unused field, can remove
	RootPieceID storj.PieceID

	Pieces Pieces

	ObjectExistsChecked bool
}

BeginSegment contains options to verify, whether a new segment upload can be started.

type BucketEmpty 

type BucketEmpty struct {
	ProjectID  uuid.UUID
	BucketName string
}

BucketEmpty contains arguments necessary for checking if bucket is empty.

type BucketLocation 

type BucketLocation struct {
	ProjectID  uuid.UUID
	BucketName string
}

BucketLocation defines a bucket that belongs to a project.

func ParseBucketPrefix 

func ParseBucketPrefix(prefix BucketPrefix) (BucketLocation, error)

ParseBucketPrefix parses BucketPrefix.

func ParseCompactBucketPrefix 

func ParseCompactBucketPrefix(compactPrefix []byte) (BucketLocation, error)

ParseCompactBucketPrefix parses BucketPrefix.

func (BucketLocation) CompactPrefix 

func (loc BucketLocation) CompactPrefix() []byte

CompactPrefix converts bucket location into bucket prefix with compact project ID.

func (BucketLocation) Prefix 

func (loc BucketLocation) Prefix() BucketPrefix

Prefix converts bucket location into bucket prefix.

func (BucketLocation) Verify 

func (loc BucketLocation) Verify() error

Verify object location fields.

type BucketPrefix 

type BucketPrefix string

BucketPrefix consists of <project id>/<bucket name>.

type BucketTally added in v1.66.1 

type BucketTally struct {
	BucketLocation

	ObjectCount        int64
	PendingObjectCount int64

	TotalSegments int64
	TotalBytes    int64

	MetadataSize int64
}

BucketTally contains information about aggregate data stored in a bucket.

type CockroachAdapter added in v1.102.2 

type CockroachAdapter struct {
	PostgresAdapter
}

CockroachAdapter uses Cockroach related SQL queries.

type CollectBucketTallies added in v1.66.1 

type CollectBucketTallies struct {
	From               BucketLocation
	To                 BucketLocation
	AsOfSystemTime     time.Time
	AsOfSystemInterval time.Duration
	Now                time.Time
}

CollectBucketTallies contains arguments necessary for looping through objects in metabase.

func (*CollectBucketTallies) Verify added in v1.66.1 

func (opts *CollectBucketTallies) Verify() error

Verify verifies CollectBucketTallies request fields.

type CommitInlineSegment 

type CommitInlineSegment struct {
	ObjectStream

	Position SegmentPosition

	ExpiresAt *time.Time

	EncryptedKeyNonce []byte
	EncryptedKey      []byte

	PlainOffset   int64 // offset in the original data stream
	PlainSize     int32 // size before encryption
	EncryptedETag []byte

	InlineData []byte
}

CommitInlineSegment contains all necessary information about the segment.

type CommitObject 

type CommitObject struct {
	ObjectStream

	Encryption storj.EncryptionParameters

	// this flag controls if we want to set metadata fields with CommitObject
	// it's possible to set metadata with BeginObject request so we need to
	// be explicit if we would like to set it with CommitObject which will
	// override any existing metadata.
	OverrideEncryptedMetadata     bool
	EncryptedMetadata             []byte // optional
	EncryptedMetadataNonce        []byte // optional
	EncryptedMetadataEncryptedKey []byte // optional

	DisallowDelete bool

	// Versioned indicates whether an object is allowed to have multiple versions.
	Versioned bool
}

CommitObject contains arguments necessary for committing an object.

func (*CommitObject) Verify added in v1.43.1 

func (c *CommitObject) Verify() error

Verify verifies reqest fields.

type CommitObjectWithSegments 

type CommitObjectWithSegments struct {
	ObjectStream

	EncryptedMetadata             []byte
	EncryptedMetadataNonce        []byte
	EncryptedMetadataEncryptedKey []byte

	// TODO: this probably should use segment ranges rather than individual items
	Segments []SegmentPosition

	// DisallowDelete indicates whether the user is allowed to overwrite
	// the previous unversioned object.
	DisallowDelete bool

	// Versioned indicates whether an object is allowed to have multiple versions.
	Versioned bool
}

CommitObjectWithSegments contains arguments necessary for committing an object.

TODO: not ready for production.

type CommitSegment 

type CommitSegment struct {
	ObjectStream

	Position    SegmentPosition
	RootPieceID storj.PieceID

	ExpiresAt *time.Time

	EncryptedKeyNonce []byte
	EncryptedKey      []byte

	PlainOffset   int64 // offset in the original data stream
	PlainSize     int32 // size before encryption
	EncryptedSize int32 // segment size after encryption

	EncryptedETag []byte

	Redundancy storj.RedundancyScheme

	Pieces Pieces

	Placement storj.PlacementConstraint
}

CommitSegment contains all necessary information about the segment.

type Config added in v1.42.2 

type Config struct {
	ApplicationName  string
	MinPartSize      memory.Size
	MaxNumberOfParts int

	// TODO remove this flag when server-side copy implementation will be finished
	ServerSideCopy         bool
	ServerSideCopyDisabled bool
	UseListObjectsIterator bool

	TestingUniqueUnversioned bool
}

Config is a configuration struct for part validation.

type DB 

type DB struct {
	// contains filtered or unexported fields
}

DB implements a database for storing objects and segments.

func Open 

func Open(ctx context.Context, log *zap.Logger, connstr string, config Config) (*DB, error)

Open opens a connection to metabase.

func (*DB) BeginCopyObject added in v1.50.1 

func (db *DB) BeginCopyObject(ctx context.Context, opts BeginCopyObject) (_ BeginCopyObjectResult, err error)

BeginCopyObject collects all data needed to begin object copy procedure.

func (*DB) BeginMoveObject added in v1.39.4 

func (db *DB) BeginMoveObject(ctx context.Context, opts BeginMoveObject) (_ BeginMoveObjectResult, err error)

BeginMoveObject collects all data needed to begin object move procedure.

func (*DB) BeginObjectNextVersion 

func (db *DB) BeginObjectNextVersion(ctx context.Context, opts BeginObjectNextVersion) (object Object, err error)

BeginObjectNextVersion adds a pending object to the database, with automatically assigned version.

func (*DB) BeginSegment 

func (db *DB) BeginSegment(ctx context.Context, opts BeginSegment) (err error)

BeginSegment verifies, whether a new segment upload can be started.

func (*DB) BucketEmpty 

func (db *DB) BucketEmpty(ctx context.Context, opts BucketEmpty) (empty bool, err error)

BucketEmpty returns true if bucket does not contain objects (pending or committed). This method doesn't check bucket existence.

func (*DB) CheckVersion 

func (db *DB) CheckVersion(ctx context.Context) error

CheckVersion checks the database is the correct version.

func (*DB) ChooseAdapter added in v1.102.2 

func (db *DB) ChooseAdapter(projectID uuid.UUID) Adapter

ChooseAdapter selects the right adapter based on configuration.

func (*DB) Close 

func (db *DB) Close() error

Close closes the connection to database.

func (*DB) CollectBucketTallies added in v1.66.1 

func (db *DB) CollectBucketTallies(ctx context.Context, opts CollectBucketTallies) (result []BucketTally, err error)

CollectBucketTallies collect limited bucket tallies from given bucket locations.

func (*DB) CommitInlineSegment 

func (db *DB) CommitInlineSegment(ctx context.Context, opts CommitInlineSegment) (err error)

CommitInlineSegment commits inline segment to the database.

func (*DB) CommitObject 

func (db *DB) CommitObject(ctx context.Context, opts CommitObject) (object Object, err error)

CommitObject adds a pending object to the database. If another committed object is under target location it will be deleted.

func (*DB) CommitObjectWithSegments 

func (db *DB) CommitObjectWithSegments(ctx context.Context, opts CommitObjectWithSegments) (object Object, deletedSegments []DeletedSegmentInfo, err error)

CommitObjectWithSegments commits pending object to the database.

TODO: not ready for production.

func (*DB) CommitSegment 

func (db *DB) CommitSegment(ctx context.Context, opts CommitSegment) (err error)

CommitSegment commits segment to the database.

func (*DB) DeleteBucketObjects 

func (db *DB) DeleteBucketObjects(ctx context.Context, opts DeleteBucketObjects) (deletedObjectCount int64, err error)

DeleteBucketObjects deletes all objects in the specified bucket. Deletion performs in batches, so in case of error while processing, this method will return the number of objects deleted to the moment when an error occurs.

func (*DB) DeleteExpiredObjects 

func (db *DB) DeleteExpiredObjects(ctx context.Context, opts DeleteExpiredObjects) (err error)

DeleteExpiredObjects deletes all objects that expired before expiredBefore.

func (*DB) DeleteObjectExactVersion 

func (db *DB) DeleteObjectExactVersion(ctx context.Context, opts DeleteObjectExactVersion) (result DeleteObjectResult, err error)

DeleteObjectExactVersion deletes an exact object version.

func (*DB) DeleteObjectLastCommitted added in v1.63.1 

func (db *DB) DeleteObjectLastCommitted(
	ctx context.Context, opts DeleteObjectLastCommitted,
) (result DeleteObjectResult, err error)

DeleteObjectLastCommitted deletes an object last committed version.

func (*DB) DeleteObjectsAllVersions 

func (db *DB) DeleteObjectsAllVersions(ctx context.Context, opts DeleteObjectsAllVersions) (result DeleteObjectResult, err error)

DeleteObjectsAllVersions deletes all versions of multiple objects from the same bucket.

func (*DB) DeletePendingObject 

func (db *DB) DeletePendingObject(ctx context.Context, opts DeletePendingObject) (result DeleteObjectResult, err error)

DeletePendingObject deletes a pending object with specified version and streamID.

func (*DB) DeleteZombieObjects added in v1.30.1 

func (db *DB) DeleteZombieObjects(ctx context.Context, opts DeleteZombieObjects) (err error)

DeleteZombieObjects deletes all objects that zombie deletion deadline passed. TODO will be removed when objects table will be free from pending objects.

func (*DB) DestroyTables 

func (db *DB) DestroyTables(ctx context.Context) error

DestroyTables deletes all tables.

TODO: remove this, only for bootstrapping.

func (*DB) EnsureNodeAliases 

func (db *DB) EnsureNodeAliases(ctx context.Context, opts EnsureNodeAliases) (err error)

EnsureNodeAliases ensures that the supplied node ID-s have a alias. It's safe to concurrently try and create node ID-s for the same NodeID.

func (*DB) FinishCopyObject added in v1.50.1 

func (db *DB) FinishCopyObject(ctx context.Context, opts FinishCopyObject) (object Object, err error)

FinishCopyObject accepts new encryption keys for copied object and insert the corresponding new object ObjectKey and segments EncryptedKey. It returns the object at the destination location.

func (*DB) FinishMoveObject added in v1.40.3 

func (db *DB) FinishMoveObject(ctx context.Context, opts FinishMoveObject) (err error)

FinishMoveObject accepts new encryption keys for moved object and updates the corresponding object ObjectKey and segments EncryptedKey.

func (*DB) GetLatestObjectLastSegment 

func (db *DB) GetLatestObjectLastSegment(ctx context.Context, opts GetLatestObjectLastSegment) (segment Segment, err error)

GetLatestObjectLastSegment returns an object last segment information.

func (*DB) GetObjectExactVersion 

func (db *DB) GetObjectExactVersion(ctx context.Context, opts GetObjectExactVersion) (_ Object, err error)

GetObjectExactVersion returns object information for exact version.

func (*DB) GetObjectLastCommitted added in v1.62.1 

func (db *DB) GetObjectLastCommitted(ctx context.Context, opts GetObjectLastCommitted) (_ Object, err error)

GetObjectLastCommitted returns object information for last committed version.

func (*DB) GetSegmentByPosition 

func (db *DB) GetSegmentByPosition(ctx context.Context, opts GetSegmentByPosition) (segment Segment, err error)

GetSegmentByPosition returns information about segment on the specified position.

func (*DB) GetStreamPieceCountByNodeID 

func (db *DB) GetStreamPieceCountByNodeID(ctx context.Context, opts GetStreamPieceCountByNodeID) (result map[storj.NodeID]int64, err error)

GetStreamPieceCountByNodeID returns piece count by node id.

func (*DB) GetTableStats added in v1.32.2 

func (db *DB) GetTableStats(ctx context.Context, opts GetTableStats) (result TableStats, err error)

GetTableStats gathers information about the metabase tables, currently only "segments" table.

func (*DB) Implementation added in v1.32.2 

func (db *DB) Implementation() dbutil.Implementation

Implementation rturns the database implementation.

func (*DB) IterateLoopObjects 

func (db *DB) IterateLoopObjects(ctx context.Context, opts IterateLoopObjects, fn func(context.Context, LoopObjectsIterator) error) (err error)

IterateLoopObjects iterates through all objects in metabase.

func (*DB) IterateLoopSegments added in v1.31.1 

func (db *DB) IterateLoopSegments(ctx context.Context, opts IterateLoopSegments, fn func(context.Context, LoopSegmentsIterator) error) (err error)

IterateLoopSegments iterates through all segments in metabase.

func (*DB) IterateObjectsAllVersionsWithStatus 

func (db *DB) IterateObjectsAllVersionsWithStatus(ctx context.Context, opts IterateObjectsWithStatus, fn func(context.Context, ObjectsIterator) error) (err error)

IterateObjectsAllVersionsWithStatus iterates through all versions of all objects with specified status.

func (*DB) IterateObjectsAllVersionsWithStatusAscending added in v1.96.4 

func (db *DB) IterateObjectsAllVersionsWithStatusAscending(ctx context.Context, opts IterateObjectsWithStatus, fn func(context.Context, ObjectsIterator) error) (err error)

IterateObjectsAllVersionsWithStatusAscending iterates through all versions of all objects with specified status. Ordered from oldest to latest. TODO this method was copied (and renamed) from v1.95.1 as a workaround for issues with metabase.ListObject performance. It should be removed when problem with metabase.ListObject will be fixed.

func (*DB) IteratePendingObjectsByKey 

func (db *DB) IteratePendingObjectsByKey(ctx context.Context, opts IteratePendingObjectsByKey, fn func(context.Context, ObjectsIterator) error) (err error)

IteratePendingObjectsByKey iterates through all streams of pending objects with the same ObjectKey.

func (*DB) LatestNodesAliasMap added in v1.66.1 

func (db *DB) LatestNodesAliasMap(ctx context.Context) (_ *NodeAliasMap, err error)

LatestNodesAliasMap returns the latest mapping between storj.NodeID and NodeAlias.

func (*DB) ListBucketsStreamIDs added in v1.70.1 

func (db *DB) ListBucketsStreamIDs(ctx context.Context, opts ListBucketsStreamIDs) (ListBucketsStreamIDsResult, error)

ListBucketsStreamIDs lists the streamIDs of a list of buckets.

func (*DB) ListNodeAliases 

func (db *DB) ListNodeAliases(ctx context.Context) (_ []NodeAliasEntry, err error)

ListNodeAliases lists all node alias mappings.

func (*DB) ListObjects added in v1.65.1 

func (db *DB) ListObjects(ctx context.Context, opts ListObjects) (result ListObjectsResult, err error)

ListObjects lists objects.

func (*DB) ListObjectsWithIterator added in v1.102.2 

func (db *DB) ListObjectsWithIterator(ctx context.Context, opts ListObjects) (result ListObjectsResult, err error)

ListObjectsWithIterator lists objects.

func (*DB) ListSegments 

func (db *DB) ListSegments(ctx context.Context, opts ListSegments) (result ListSegmentsResult, err error)

ListSegments lists specified stream segments.

func (*DB) ListStreamPositions 

func (db *DB) ListStreamPositions(ctx context.Context, opts ListStreamPositions) (result ListStreamPositionsResult, err error)

ListStreamPositions lists specified stream segment positions.

func (*DB) ListVerifySegments added in v1.64.1 

func (db *DB) ListVerifySegments(ctx context.Context, opts ListVerifySegments) (result ListVerifySegmentsResult, err error)

ListVerifySegments lists specified stream segments.

func (*DB) MigrateToLatest 

func (db *DB) MigrateToLatest(ctx context.Context) error

MigrateToLatest migrates database to the latest version.

func (*DB) Now added in v1.30.1 

func (db *DB) Now(ctx context.Context) (time.Time, error)

Now returns time on the database.

func (*DB) Ping 

func (db *DB) Ping(ctx context.Context) error

Ping checks whether connection has been established.

func (*DB) PostgresMigration 

func (db *DB) PostgresMigration() *migrate.Migration

PostgresMigration returns steps needed for migrating postgres database.

func (*DB) TestMigrateToLatest added in v1.57.1 

func (db *DB) TestMigrateToLatest(ctx context.Context) error

TestMigrateToLatest replaces the migration steps with only one step to create metabase db.

func (*DB) TestingAllObjects 

func (db *DB) TestingAllObjects(ctx context.Context) (objects []Object, err error)

TestingAllObjects gets all objects. Use only for testing purposes.

func (*DB) TestingAllSegments 

func (db *DB) TestingAllSegments(ctx context.Context) (segments []Segment, err error)

TestingAllSegments gets all segments. Use only for testing purposes.

func (*DB) TestingBatchInsertObjects added in v1.98.1 

func (db *DB) TestingBatchInsertObjects(ctx context.Context, objects []RawObject) (err error)

TestingBatchInsertObjects batch inserts objects for testing. This implementation does no verification on the correctness of objects.

func (*DB) TestingBatchInsertSegments added in v1.102.2 

func (db *DB) TestingBatchInsertSegments(ctx context.Context, segments []RawSegment) (err error)

TestingBatchInsertSegments batch inserts segments for testing. This implementation does no verification on the correctness of segments.

func (*DB) TestingBeginObjectExactVersion added in v1.90.1 

func (db *DB) TestingBeginObjectExactVersion(ctx context.Context, opts BeginObjectExactVersion) (committed Object, err error)

TestingBeginObjectExactVersion adds a pending object to the database, with specific version.

func (*DB) TestingDeleteAll 

func (db *DB) TestingDeleteAll(ctx context.Context) (err error)

TestingDeleteAll deletes all objects and segments from the database.

func (*DB) TestingGetState 

func (db *DB) TestingGetState(ctx context.Context) (_ *RawState, err error)

TestingGetState returns the state of the database.

func (*DB) TestingSetCleanup added in v1.31.1 

func (db *DB) TestingSetCleanup(cleanup func() error)

TestingSetCleanup is used to set the callback for cleaning up test database.

func (*DB) UnderlyingTagSQL 

func (db *DB) UnderlyingTagSQL() tagsql.DB

UnderlyingTagSQL returns *tagsql.DB. TODO: remove.

func (*DB) UpdateObjectLastCommittedMetadata added in v1.91.2 

func (db *DB) UpdateObjectLastCommittedMetadata(ctx context.Context, opts UpdateObjectLastCommittedMetadata) (err error)

UpdateObjectLastCommittedMetadata updates an object metadata.

func (*DB) UpdateSegmentPieces 

func (db *DB) UpdateSegmentPieces(ctx context.Context, opts UpdateSegmentPieces) (err error)

UpdateSegmentPieces updates pieces for specified segment. If provided old pieces won't match current database state update will fail.

type DeleteBucketObjects 

type DeleteBucketObjects struct {
	Bucket    BucketLocation
	BatchSize int
}

DeleteBucketObjects contains arguments for deleting a whole bucket.

type DeleteExpiredObjects 

type DeleteExpiredObjects struct {
	ExpiredBefore      time.Time
	AsOfSystemInterval time.Duration
	BatchSize          int
}

DeleteExpiredObjects contains all the information necessary to delete expired objects and segments.

type DeleteObjectExactVersion 

type DeleteObjectExactVersion struct {
	Version Version
	ObjectLocation
}

DeleteObjectExactVersion contains arguments necessary for deleting an exact version of object.

func (*DeleteObjectExactVersion) Verify 

func (obj *DeleteObjectExactVersion) Verify() error

Verify delete object fields.

type DeleteObjectLastCommitted added in v1.63.1 

type DeleteObjectLastCommitted struct {
	ObjectLocation

	Versioned bool
	Suspended bool
}

DeleteObjectLastCommitted contains arguments necessary for deleting last committed version of object.

func (*DeleteObjectLastCommitted) Verify added in v1.63.1 

func (obj *DeleteObjectLastCommitted) Verify() error

Verify delete object last committed fields.

type DeleteObjectResult 

type DeleteObjectResult struct {
	// Removed contains the list of objects that were removed from the metabase.
	Removed []Object
	// Markers contains the delete markers that were added.
	Markers []Object
}

DeleteObjectResult result of deleting object.

type DeleteObjectsAllVersions 

type DeleteObjectsAllVersions struct {
	Locations []ObjectLocation
}

DeleteObjectsAllVersions contains arguments necessary for deleting all versions of multiple objects from the same bucket.

func (*DeleteObjectsAllVersions) Verify 

func (delete *DeleteObjectsAllVersions) Verify() error

Verify delete objects fields.

type DeletePendingObject 

type DeletePendingObject struct {
	ObjectStream
}

DeletePendingObject contains arguments necessary for deleting a pending object.

func (*DeletePendingObject) Verify 

func (opts *DeletePendingObject) Verify() error

Verify verifies delete pending object fields validity.

type DeleteZombieObjects added in v1.30.1 

type DeleteZombieObjects struct {
	DeadlineBefore     time.Time
	InactiveDeadline   time.Time
	AsOfSystemInterval time.Duration
	BatchSize          int
}

DeleteZombieObjects contains all the information necessary to delete zombie objects and segments.

type DeletedSegmentInfo 

type DeletedSegmentInfo struct {
	RootPieceID storj.PieceID
	Pieces      Pieces
}

DeletedSegmentInfo info about deleted segment.

type EncryptedKeyAndNonce added in v1.39.4 

type EncryptedKeyAndNonce struct {
	Position          SegmentPosition
	EncryptedKeyNonce []byte
	EncryptedKey      []byte
}

EncryptedKeyAndNonce holds single segment position, encrypted key and nonce.

type EnsureNodeAliases 

type EnsureNodeAliases struct {
	Nodes []storj.NodeID
}

EnsureNodeAliases contains arguments necessary for creating NodeAlias-es.

type FinishCopyObject added in v1.50.1 

type FinishCopyObject struct {
	ObjectStream
	NewBucket             string
	NewEncryptedObjectKey ObjectKey
	NewStreamID           uuid.UUID

	OverrideMetadata             bool
	NewEncryptedMetadata         []byte
	NewEncryptedMetadataKeyNonce storj.Nonce
	NewEncryptedMetadataKey      []byte

	NewSegmentKeys []EncryptedKeyAndNonce

	// NewDisallowDelete indicates whether the user is allowed to delete an existing unversioned object.
	NewDisallowDelete bool

	// NewVersioned indicates that the object allows multiple versions.
	NewVersioned bool

	// VerifyLimits holds a callback by which the caller can interrupt the copy
	// if it turns out completing the copy would exceed a limit.
	// It will be called only once.
	VerifyLimits func(encryptedObjectSize int64, nSegments int64) error
}

FinishCopyObject holds all data needed to finish object copy.

func (FinishCopyObject) NewLocation added in v1.92.1 

func (finishCopy FinishCopyObject) NewLocation() ObjectLocation

NewLocation returns the new object location.

func (FinishCopyObject) Verify added in v1.50.1 

func (finishCopy FinishCopyObject) Verify() error

Verify verifies metabase.FinishCopyObject data.

type FinishMoveObject added in v1.40.3 

type FinishMoveObject struct {
	ObjectStream

	NewBucket             string
	NewSegmentKeys        []EncryptedKeyAndNonce
	NewEncryptedObjectKey ObjectKey
	// Optional. Required if object has metadata.
	NewEncryptedMetadataKeyNonce storj.Nonce
	NewEncryptedMetadataKey      []byte

	// NewDisallowDelete indicates whether the user is allowed to delete an existing unversioned object.
	NewDisallowDelete bool

	// NewVersioned indicates that the object allows multiple versions.
	NewVersioned bool
}

FinishMoveObject holds all data needed to finish object move.

func (FinishMoveObject) NewLocation added in v1.92.1 

func (finishMove FinishMoveObject) NewLocation() ObjectLocation

NewLocation returns the new object location.

func (FinishMoveObject) Verify added in v1.40.3 

func (finishMove FinishMoveObject) Verify() error

Verify verifies metabase.FinishMoveObject data.

type GetLatestObjectLastSegment 

type GetLatestObjectLastSegment struct {
	ObjectLocation
}

GetLatestObjectLastSegment contains arguments necessary for fetching a last segment information.

type GetObjectExactVersion 

type GetObjectExactVersion struct {
	Version Version
	ObjectLocation
}

GetObjectExactVersion contains arguments necessary for fetching an information about exact object version.

func (*GetObjectExactVersion) Verify 

func (obj *GetObjectExactVersion) Verify() error

Verify verifies get object request fields.

type GetObjectLastCommitted added in v1.62.1 

type GetObjectLastCommitted struct {
	ObjectLocation
}

GetObjectLastCommitted contains arguments necessary for fetching an object information for last committed version.

type GetSegmentByPosition 

type GetSegmentByPosition struct {
	StreamID uuid.UUID
	Position SegmentPosition
}

GetSegmentByPosition contains arguments necessary for fetching a segment on specific position.

func (*GetSegmentByPosition) Verify 

func (seg *GetSegmentByPosition) Verify() error

Verify verifies get segment request fields.

type GetStreamPieceCountByNodeID 

type GetStreamPieceCountByNodeID struct {
	StreamID uuid.UUID
}

GetStreamPieceCountByNodeID contains arguments for GetStreamPieceCountByNodeID.

type GetTableStats added in v1.32.2 

type GetTableStats struct {
	AsOfSystemInterval time.Duration
}

GetTableStats contains arguments necessary for getting table statistics.

type IterateCursor 

type IterateCursor struct {
	Key     ObjectKey
	Version Version
}

IterateCursor is a cursor used during iteration through objects.

The cursor is exclusive.

type IterateLoopObjects 

type IterateLoopObjects struct {
	BatchSize int

	AsOfSystemTime     time.Time
	AsOfSystemInterval time.Duration
}

IterateLoopObjects contains arguments necessary for listing objects in metabase.

func (*IterateLoopObjects) Verify 

func (opts *IterateLoopObjects) Verify() error

Verify verifies get object request fields.

type IterateLoopSegments added in v1.31.1 

type IterateLoopSegments struct {
	BatchSize          int
	AsOfSystemTime     time.Time
	AsOfSystemInterval time.Duration
	StartStreamID      uuid.UUID
	EndStreamID        uuid.UUID
}

IterateLoopSegments contains arguments necessary for listing segments in metabase.

func (*IterateLoopSegments) Verify added in v1.31.1 

func (opts *IterateLoopSegments) Verify() error

Verify verifies segments request fields.

type IterateObjectsWithStatus 

type IterateObjectsWithStatus struct {
	ProjectID             uuid.UUID
	BucketName            string
	Recursive             bool
	BatchSize             int
	Prefix                ObjectKey
	Cursor                IterateCursor
	Pending               bool
	IncludeCustomMetadata bool
	IncludeSystemMetadata bool
}

IterateObjectsWithStatus contains arguments necessary for listing objects in a bucket.

func (*IterateObjectsWithStatus) Verify 

func (opts *IterateObjectsWithStatus) Verify() error

Verify verifies get object request fields.

type IteratePendingObjectsByKey 

type IteratePendingObjectsByKey struct {
	ObjectLocation
	BatchSize int
	Cursor    StreamIDCursor
}

IteratePendingObjectsByKey contains arguments necessary for listing pending objects by ObjectKey.

func (*IteratePendingObjectsByKey) Verify 

func (opts *IteratePendingObjectsByKey) Verify() error

Verify verifies get object request fields.

type ListBucketsStreamIDs added in v1.70.1 

type ListBucketsStreamIDs struct {
	BucketList     ListVerifyBucketList
	CursorBucket   BucketLocation
	CursorStreamID uuid.UUID
	Limit          int

	AsOfSystemTime     time.Time
	AsOfSystemInterval time.Duration
}

ListBucketsStreamIDs contains arguments necessary for listing stream segments from buckets.

type ListBucketsStreamIDsResult added in v1.70.1 

type ListBucketsStreamIDsResult struct {
	StreamIDs  []uuid.UUID
	Counts     []int
	LastBucket BucketLocation
}

ListBucketsStreamIDsResult is the result of listing segments of a list of buckets.

type ListObjects added in v1.65.1 

type ListObjects struct {
	ProjectID             uuid.UUID
	BucketName            string
	Recursive             bool
	Limit                 int
	Prefix                ObjectKey
	Cursor                ListObjectsCursor
	Pending               bool
	AllVersions           bool
	IncludeCustomMetadata bool
	IncludeSystemMetadata bool
}

ListObjects contains arguments necessary for listing objects.

For Pending = false, the versions are in descending order. For Pending = true, the versions are in ascending order.

func (*ListObjects) Verify added in v1.65.1 

func (opts *ListObjects) Verify() error

Verify verifies get object request fields.

func (*ListObjects) VersionAscending added in v1.101.1 

func (opts *ListObjects) VersionAscending() bool

VersionAscending returns whether the versions in the result are in ascending order.

type ListObjectsCursor added in v1.65.1 

type ListObjectsCursor IterateCursor

ListObjectsCursor is a cursor used during iteration through objects.

type ListObjectsResult added in v1.65.1 

type ListObjectsResult struct {
	Objects []ObjectEntry
	More    bool
}

ListObjectsResult result of listing objects.

type ListSegments 

type ListSegments struct {
	StreamID uuid.UUID
	Cursor   SegmentPosition
	Limit    int

	Range *StreamRange
}

ListSegments contains arguments necessary for listing stream segments.

type ListSegmentsResult 

type ListSegmentsResult struct {
	Segments []Segment
	More     bool
}

ListSegmentsResult result of listing segments.

type ListStreamPositions 

type ListStreamPositions struct {
	StreamID uuid.UUID
	Cursor   SegmentPosition
	Limit    int

	Range *StreamRange
}

ListStreamPositions contains arguments necessary for listing stream segments.

type ListStreamPositionsResult 

type ListStreamPositionsResult struct {
	Segments []SegmentPositionInfo
	More     bool
}

ListStreamPositionsResult result of listing segments.

type ListVerifyBucketList added in v1.70.1 

type ListVerifyBucketList struct {
	Buckets []BucketLocation
}

ListVerifyBucketList represents a list of buckets.

func (*ListVerifyBucketList) Add added in v1.70.1 

func (list *ListVerifyBucketList) Add(projectID uuid.UUID, bucketName string)

Add adds a (projectID, bucketName) to the list of buckets to be checked.

type ListVerifySegments added in v1.64.1 

type ListVerifySegments struct {
	CursorStreamID uuid.UUID
	CursorPosition SegmentPosition

	StreamIDs []uuid.UUID
	Limit     int

	CreatedAfter  *time.Time
	CreatedBefore *time.Time

	AsOfSystemTime     time.Time
	AsOfSystemInterval time.Duration
}

ListVerifySegments contains arguments necessary for listing stream segments.

type ListVerifySegmentsResult added in v1.64.1 

type ListVerifySegmentsResult struct {
	Segments []VerifySegment
}

ListVerifySegmentsResult is the result of ListVerifySegments.

type LoopObjectEntry 

type LoopObjectEntry struct {
	ObjectStream                       // metrics, repair, tally
	Status                ObjectStatus // verify
	CreatedAt             time.Time    // temp used by metabase-createdat-migration
	ExpiresAt             *time.Time   // tally
	SegmentCount          int32        // metrics
	TotalEncryptedSize    int64        // tally
	EncryptedMetadataSize int          // tally
}

LoopObjectEntry contains information about object needed by metainfo loop.

func (LoopObjectEntry) Expired added in v1.34.1 

func (o LoopObjectEntry) Expired(now time.Time) bool

Expired checks if object is expired relative to now.

type LoopObjectsIterator 

type LoopObjectsIterator interface {
	Next(ctx context.Context, item *LoopObjectEntry) bool
}

LoopObjectsIterator iterates over a sequence of LoopObjectEntry items.

type LoopSegmentEntry 

type LoopSegmentEntry struct {
	StreamID      uuid.UUID
	Position      SegmentPosition
	CreatedAt     time.Time // non-nillable
	ExpiresAt     *time.Time
	RepairedAt    *time.Time // repair
	RootPieceID   storj.PieceID
	EncryptedSize int32 // size of the whole segment (not a piece)
	PlainOffset   int64 // verify
	PlainSize     int32 // verify
	AliasPieces   AliasPieces
	Redundancy    storj.RedundancyScheme
	Pieces        Pieces
	Placement     storj.PlacementConstraint
}

LoopSegmentEntry contains information about segment metadata needed by metainfo loop.

func (LoopSegmentEntry) Inline 

func (s LoopSegmentEntry) Inline() bool

Inline returns true if segment is inline.

type LoopSegmentsIterator added in v1.31.1 

type LoopSegmentsIterator interface {
	Next(ctx context.Context, item *LoopSegmentEntry) bool
}

LoopSegmentsIterator iterates over a sequence of LoopSegmentEntry items.

type NodeAlias 

type NodeAlias int32

NodeAlias is a metabase local alias for NodeID-s to reduce segment table size.

type NodeAliasCache 

type NodeAliasCache struct {
	// contains filtered or unexported fields
}

NodeAliasCache is a write-through cache for looking up node ID and alias mapping.

func NewNodeAliasCache 

func NewNodeAliasCache(db NodeAliasDB) *NodeAliasCache

NewNodeAliasCache creates a new cache using the specified database.

func (*NodeAliasCache) Aliases 

func (cache *NodeAliasCache) Aliases(ctx context.Context, nodes []storj.NodeID) ([]NodeAlias, error)

Aliases returns node aliases corresponding to the node ID-s and returns an error when node is missing.

func (*NodeAliasCache) ConvertAliasesToPieces 

func (cache *NodeAliasCache) ConvertAliasesToPieces(ctx context.Context, aliasPieces AliasPieces) (_ Pieces, err error)

ConvertAliasesToPieces converts alias pieces to pieces.

func (*NodeAliasCache) EnsureAliases added in v1.65.1 

func (cache *NodeAliasCache) EnsureAliases(ctx context.Context, nodes []storj.NodeID) ([]NodeAlias, error)

EnsureAliases returns node aliases corresponding to the node ID-s, adding missing node ID-s to the database when needed.

func (*NodeAliasCache) EnsurePiecesToAliases added in v1.65.1 

func (cache *NodeAliasCache) EnsurePiecesToAliases(ctx context.Context, pieces Pieces) (_ AliasPieces, err error)

EnsurePiecesToAliases converts pieces to alias pieces and automatically adds storage node to alias table when necessary.

func (*NodeAliasCache) Latest added in v1.66.1 

func (cache *NodeAliasCache) Latest(ctx context.Context) (_ *NodeAliasMap, err error)

Latest returns the latest NodeAliasMap.

func (*NodeAliasCache) Nodes 

func (cache *NodeAliasCache) Nodes(ctx context.Context, aliases []NodeAlias) ([]storj.NodeID, error)

Nodes returns node ID-s corresponding to the aliases, refreshing the cache once when an alias is missing. This results in an error when the alias is not in the database.

type NodeAliasDB 

type NodeAliasDB interface {
	EnsureNodeAliases(ctx context.Context, opts EnsureNodeAliases) error
	ListNodeAliases(ctx context.Context) (_ []NodeAliasEntry, err error)
}

NodeAliasDB is an interface for looking up node alises.

type NodeAliasEntry 

type NodeAliasEntry struct {
	ID    storj.NodeID
	Alias NodeAlias
}

NodeAliasEntry is a mapping between NodeID and NodeAlias.

type NodeAliasMap 

type NodeAliasMap struct {
	// contains filtered or unexported fields
}

NodeAliasMap contains bidirectional mapping between node ID and a NodeAlias.

func NewNodeAliasMap 

func NewNodeAliasMap(entries []NodeAliasEntry) *NodeAliasMap

NewNodeAliasMap creates a new alias map from the given entries.

func (*NodeAliasMap) Alias added in v1.66.1 

func (m *NodeAliasMap) Alias(node storj.NodeID) (x NodeAlias, ok bool)

Alias returns alias for the given node ID.

func (*NodeAliasMap) Aliases 

func (m *NodeAliasMap) Aliases(nodes []storj.NodeID) (xs []NodeAlias, missing []storj.NodeID)

Aliases returns aliases-s for the given node ID-s and node ID-s that are not in this map.

func (*NodeAliasMap) ContainsAll 

func (m *NodeAliasMap) ContainsAll(nodeIDs []storj.NodeID, nodeAliases []NodeAlias) bool

ContainsAll returns true when the table contains all entries.

func (*NodeAliasMap) Max added in v1.70.1 

func (m *NodeAliasMap) Max() NodeAlias

Max returns the largest node alias in this map, -1 otherwise. Contrast with Size.

func (*NodeAliasMap) Merge 

func (m *NodeAliasMap) Merge(other *NodeAliasMap)

Merge merges the other map into m.

func (*NodeAliasMap) Node 

func (m *NodeAliasMap) Node(alias NodeAlias) (x storj.NodeID, ok bool)

Node returns NodeID for the given alias.

func (*NodeAliasMap) Nodes 

func (m *NodeAliasMap) Nodes(aliases []NodeAlias) (xs []storj.NodeID, missing []NodeAlias)

Nodes returns NodeID-s for the given aliases and aliases that are not in this map.

func (*NodeAliasMap) Size 

func (m *NodeAliasMap) Size() int

Size returns the number of entries in this map. Contrast with Max.

type Object 

type Object RawObject

Object object metadata. TODO define separated struct.

func (*Object) IsMigrated 

func (obj *Object) IsMigrated() bool

IsMigrated returns whether the object comes from PointerDB. Pointer objects are special that they are missing some information.

  • TotalPlainSize = 0 and FixedSegmentSize = 0.
  • Segment.PlainOffset = 0, Segment.PlainSize = 0

func (*Object) StreamVersionID added in v1.95.1 

func (obj *Object) StreamVersionID() StreamVersionID

StreamVersionID returns byte representation of object stream version id.

type ObjectEntry 

type ObjectEntry struct {
	IsPrefix bool

	ObjectKey ObjectKey
	Version   Version
	StreamID  uuid.UUID

	CreatedAt time.Time
	ExpiresAt *time.Time

	Status       ObjectStatus
	SegmentCount int32

	EncryptedMetadataNonce        []byte
	EncryptedMetadata             []byte
	EncryptedMetadataEncryptedKey []byte

	TotalPlainSize     int64
	TotalEncryptedSize int64
	FixedSegmentSize   int32

	Encryption storj.EncryptionParameters
}

ObjectEntry contains information about an item in a bucket.

func (ObjectEntry) Less added in v1.96.2 

func (entry ObjectEntry) Less(other ObjectEntry) bool

Less implements sorting on object entries.

func (ObjectEntry) LessVersionAsc added in v1.100.2 

func (entry ObjectEntry) LessVersionAsc(other ObjectEntry) bool

LessVersionAsc implements sorting on object entries.

func (ObjectEntry) StreamVersionID added in v1.95.1 

func (entry ObjectEntry) StreamVersionID() StreamVersionID

StreamVersionID returns byte representation of object stream version id.

type ObjectKey 

type ObjectKey string

ObjectKey is an encrypted object key encoded using Path Component Encoding. It is not ascii safe.

func (*ObjectKey) Scan added in v1.41.1 

func (o *ObjectKey) Scan(value interface{}) error

Scan extracts a ObjectKey from a database field.

func (ObjectKey) Value added in v1.41.1 

func (o ObjectKey) Value() (driver.Value, error)

Value converts a ObjectKey to a database field.

type ObjectLocation 

type ObjectLocation struct {
	ProjectID  uuid.UUID
	BucketName string
	ObjectKey  ObjectKey
}

ObjectLocation is decoded object key information.

func (ObjectLocation) Bucket 

func (obj ObjectLocation) Bucket() BucketLocation

Bucket returns bucket location this object belongs to.

func (ObjectLocation) Verify 

func (obj ObjectLocation) Verify() error

Verify object location fields.

type ObjectStatus 

type ObjectStatus byte

ObjectStatus defines the status that the object is in.

There are two types of objects:

  • Regular (i.e. Committed), which is used for storing data.
  • Delete Marker, which is used to show that an object has been deleted, while preserving older versions.

Each object can be in two states:

  • Pending, meaning that it's still being uploaded.
  • Committed, meaning it has finished uploading. Delete Markers are always considered committed, because they do not require committing.

There are two options for versioning:

  • Unversioned, there's only one allowed per project, bucket and encryption key.
  • Versioned, there can be any number of such objects for a given project, bucket and encryption key.

These lead to a few meaningful distinct statuses, listed below.

func (ObjectStatus) IsDeleteMarker added in v1.91.2 

func (status ObjectStatus) IsDeleteMarker() bool

IsDeleteMarker return whether the status is a delete marker.

func (ObjectStatus) String added in v1.96.2 

func (status ObjectStatus) String() string

String returns textual representation of status.

type ObjectStream 

type ObjectStream struct {
	ProjectID  uuid.UUID
	BucketName string
	ObjectKey  ObjectKey
	Version    Version
	StreamID   uuid.UUID
}

ObjectStream uniquely defines an object and stream.

func (ObjectStream) Less added in v1.91.2 

func (obj ObjectStream) Less(b ObjectStream) bool

Less implements sorting on object streams. Where ProjectID asc, BucketName asc, ObjectKey asc, Version desc.

func (ObjectStream) LessVersionAsc added in v1.100.2 

func (obj ObjectStream) LessVersionAsc(b ObjectStream) bool

LessVersionAsc implements sorting on object streams. Where ProjectID asc, BucketName asc, ObjectKey asc, Version asc.

func (*ObjectStream) Location 

func (obj *ObjectStream) Location() ObjectLocation

Location returns object location.

func (*ObjectStream) Verify 

func (obj *ObjectStream) Verify() error

Verify object stream fields.

type ObjectsIterator 

type ObjectsIterator interface {
	Next(ctx context.Context, item *ObjectEntry) bool
}

ObjectsIterator iterates over a sequence of ObjectEntry items.

type PendingObjectEntry added in v1.87.1 

type PendingObjectEntry struct {
	IsPrefix bool

	ObjectKey ObjectKey
	StreamID  uuid.UUID

	CreatedAt time.Time
	ExpiresAt *time.Time

	EncryptedMetadataNonce        []byte
	EncryptedMetadata             []byte
	EncryptedMetadataEncryptedKey []byte

	Encryption storj.EncryptionParameters
}

PendingObjectEntry contains information about an pending object item in a bucket.

type PendingObjectStream added in v1.85.1 

type PendingObjectStream struct {
	ProjectID  uuid.UUID
	BucketName string
	ObjectKey  ObjectKey
	StreamID   uuid.UUID
}

PendingObjectStream uniquely defines an pending object and stream.

func (*PendingObjectStream) Verify added in v1.85.1 

func (obj *PendingObjectStream) Verify() error

Verify object stream fields.

type PendingObjectsIterator added in v1.87.1 

type PendingObjectsIterator interface {
	Next(ctx context.Context, item *PendingObjectEntry) bool
}

PendingObjectsIterator iterates over a sequence of PendingObjectEntry items.

type Piece 

type Piece struct {
	Number      uint16
	StorageNode storj.NodeID
}

Piece defines information for a segment piece.

type Pieces 

type Pieces []Piece

Pieces defines information for pieces.

func (Pieces) Add added in v1.36.1 

func (p Pieces) Add(piecesToAdd Pieces) (Pieces, error)

Add adds the specified pieces and returns the updated Pieces.

func (Pieces) Equal 

func (p Pieces) Equal(pieces Pieces) bool

Equal checks if Pieces structures are equal.

func (Pieces) FindByNum added in v1.67.1 

func (p Pieces) FindByNum(pieceNum int) (_ Piece, found bool)

FindByNum finds a piece among the Pieces with the given piece number. If no such piece is found, `found` will be returned false.

func (Pieces) Len 

func (p Pieces) Len() int

Len is the number of pieces.

func (Pieces) Less 

func (p Pieces) Less(i, j int) bool

Less reports whether the piece with index i should sort before the piece with index j.

func (Pieces) Remove added in v1.36.1 

func (p Pieces) Remove(piecesToRemove Pieces) (Pieces, error)

Remove removes the specified pieces from the original pieces and returns the updated Pieces.

func (*Pieces) Scan 

func (pieces *Pieces) Scan(value interface{}) error

Scan implements sql.Scanner interface.

func (Pieces) Swap 

func (p Pieces) Swap(i, j int)

Swap swaps the pieces with indexes i and j.

func (Pieces) Update added in v1.36.1 

func (p Pieces) Update(piecesToAdd, piecesToRemove Pieces) (Pieces, error)

Update adds piecesToAdd pieces and removes piecesToRemove pieces from the original pieces struct and returns the updated Pieces.

It removes the piecesToRemove only if all piece number, node id match.

When adding a piece, it checks if the piece already exists using the piece Number If a piece already exists, it returns an empty pieces struct and an error.

func (Pieces) Value 

func (pieces Pieces) Value() (driver.Value, error)

Value implements sql/driver.Valuer interface.

func (Pieces) Verify 

func (p Pieces) Verify() error

Verify verifies pieces.

type PostgresAdapter added in v1.102.2 

type PostgresAdapter struct {
	// contains filtered or unexported fields
}

PostgresAdapter uses Cockroach related SQL queries.

func (*PostgresAdapter) BeginObjectNextVersion added in v1.102.2 

func (p *PostgresAdapter) BeginObjectNextVersion(ctx context.Context, opts BeginObjectNextVersion, object *Object) error

BeginObjectNextVersion implements Adapter.

func (*PostgresAdapter) GetObjectLastCommitted added in v1.102.2 

func (p *PostgresAdapter) GetObjectLastCommitted(ctx context.Context, opts GetObjectLastCommitted, object *Object) error

GetObjectLastCommitted implements Adapter.

func (*PostgresAdapter) TestingBatchInsertSegments added in v1.102.2 

func (p *PostgresAdapter) TestingBatchInsertSegments(ctx context.Context, segments []RawSegment) (err error)

TestingBatchInsertSegments implements postgres adapter.

func (*PostgresAdapter) TestingBeginObjectExactVersion added in v1.102.2 

func (p *PostgresAdapter) TestingBeginObjectExactVersion(ctx context.Context, opts BeginObjectExactVersion, object *Object) error

TestingBeginObjectExactVersion implements Adapter.

type RawCopy added in v1.50.1 

type RawCopy struct {
	StreamID         uuid.UUID
	AncestorStreamID uuid.UUID
}

RawCopy contains a copy that is stored in the database.

type RawObject 

type RawObject struct {
	ObjectStream

	CreatedAt time.Time
	ExpiresAt *time.Time

	Status       ObjectStatus
	SegmentCount int32

	EncryptedMetadataNonce        []byte
	EncryptedMetadata             []byte
	EncryptedMetadataEncryptedKey []byte

	// TotalPlainSize is 0 for a migrated object.
	TotalPlainSize     int64
	TotalEncryptedSize int64
	// FixedSegmentSize is 0 for a migrated object.
	FixedSegmentSize int32

	Encryption storj.EncryptionParameters

	// ZombieDeletionDeadline defines when the pending raw object should be deleted from the database.
	// This is as a safeguard against objects that failed to upload and the client has not indicated
	// whether they want to continue uploading or delete the already uploaded data.
	ZombieDeletionDeadline *time.Time
}

RawObject defines the full object that is stored in the database. It should be rarely used directly.

type RawSegment 

type RawSegment struct {
	StreamID uuid.UUID
	Position SegmentPosition

	CreatedAt  time.Time // non-nillable
	RepairedAt *time.Time
	ExpiresAt  *time.Time

	RootPieceID       storj.PieceID
	EncryptedKeyNonce []byte
	EncryptedKey      []byte

	EncryptedSize int32 // size of the whole segment (not a piece)
	// PlainSize is 0 for a migrated object.
	PlainSize int32
	// PlainOffset is 0 for a migrated object.
	PlainOffset   int64
	EncryptedETag []byte

	Redundancy storj.RedundancyScheme

	InlineData []byte
	Pieces     Pieces

	Placement storj.PlacementConstraint
}

RawSegment defines the full segment that is stored in the database. It should be rarely used directly.

type RawState 

type RawState struct {
	Objects  []RawObject
	Segments []RawSegment
}

RawState contains full state of a table.

type Segment 

type Segment RawSegment

Segment segment metadata. TODO define separated struct.

func (Segment) Expired added in v1.34.1 

func (s Segment) Expired(now time.Time) bool

Expired checks if segment is expired relative to now.

func (Segment) Inline 

func (s Segment) Inline() bool

Inline returns true if segment is inline.

func (Segment) PieceSize added in v1.74.1 

func (s Segment) PieceSize() int64

PieceSize returns calculated piece size for segment.

type SegmentIterator 

type SegmentIterator func(ctx context.Context, segment *LoopSegmentEntry) bool

SegmentIterator returns the next segment.

type SegmentKey 

type SegmentKey []byte

SegmentKey is an encoded metainfo key. This is used as the key in pointerdb key-value store.

type SegmentLocation 

type SegmentLocation struct {
	ProjectID  uuid.UUID
	BucketName string
	ObjectKey  ObjectKey
	Position   SegmentPosition
}

SegmentLocation is decoded segment key information.

func ParseSegmentKey 

func ParseSegmentKey(encoded SegmentKey) (SegmentLocation, error)

ParseSegmentKey parses an segment key into segment location.

func (SegmentLocation) Bucket 

func (seg SegmentLocation) Bucket() BucketLocation

Bucket returns bucket location this segment belongs to.

func (SegmentLocation) Encode 

func (seg SegmentLocation) Encode() SegmentKey

Encode converts segment location into a segment key.

func (SegmentLocation) Object 

func (seg SegmentLocation) Object() ObjectLocation

Object returns the object location associated with this segment location.

func (SegmentLocation) Verify 

func (seg SegmentLocation) Verify() error

Verify segment location fields.

type SegmentPosition 

type SegmentPosition struct {
	Part  uint32
	Index uint32
}

SegmentPosition is segment part and index combined.

func SegmentPositionFromEncoded 

func SegmentPositionFromEncoded(v uint64) SegmentPosition

SegmentPositionFromEncoded decodes an uint64 into a SegmentPosition.

func (SegmentPosition) Encode 

func (pos SegmentPosition) Encode() uint64

Encode encodes a segment position into an uint64, that can be stored in a database.

func (SegmentPosition) Less 

func (pos SegmentPosition) Less(b SegmentPosition) bool

Less returns whether pos should before b.

func (*SegmentPosition) Scan 

func (params *SegmentPosition) Scan(value interface{}) error

Scan implements sql.Scanner interface.

func (SegmentPosition) Value 

func (params SegmentPosition) Value() (driver.Value, error)

Value implements sql/driver.Valuer interface.

type SegmentPositionInfo 

type SegmentPositionInfo struct {
	Position SegmentPosition
	// PlainSize is 0 for a migrated object.
	PlainSize int32
	// PlainOffset is 0 for a migrated object.
	PlainOffset       int64
	CreatedAt         *time.Time // TODO: make it non-nilable after we migrate all existing segments to have creation time
	EncryptedETag     []byte
	EncryptedKeyNonce []byte
	EncryptedKey      []byte
}

SegmentPositionInfo contains information for segment position.

type StreamIDCursor 

type StreamIDCursor struct {
	StreamID uuid.UUID
}

StreamIDCursor is a cursor used during iteration through streamIDs of a pending object.

type StreamRange 

type StreamRange struct {
	PlainStart int64
	PlainLimit int64 // limit is exclusive
}

StreamRange allows to limit stream positions based on the plain offsets.

type StreamVersionID added in v1.95.1 

type StreamVersionID uuid.UUID

StreamVersionID represents combined Version and StreamID suffix for purposes of public API. First 8 bytes represents Version and rest are object StreamID suffix. TODO(ver): we may consider renaming this type to VersionID but to do that we would need to rename metabase.Version into metabase.SequenceNumber/metabase.Sequence to avoid confusion.

func StreamVersionIDFromBytes added in v1.95.1 

func StreamVersionIDFromBytes(bytes []byte) (_ StreamVersionID, err error)

StreamVersionIDFromBytes decodes stream version id from bytes.

func (StreamVersionID) Bytes added in v1.95.1 

func (s StreamVersionID) Bytes() []byte

Bytes returnes stream version id bytes.

func (StreamVersionID) StreamIDSuffix added in v1.95.1 

func (s StreamVersionID) StreamIDSuffix() []byte

StreamIDSuffix returns StreamID suffix encoded into stream version id.

func (StreamVersionID) Version added in v1.95.1 

func (s StreamVersionID) Version() Version

Version returns Version encoded into stream version id.

type TableStats added in v1.32.2 

type TableStats struct {
	SegmentCount int64
}

TableStats contains information about the metabase status.

type UpdateObjectLastCommittedMetadata added in v1.91.2 

type UpdateObjectLastCommittedMetadata struct {
	ObjectLocation
	StreamID uuid.UUID

	EncryptedMetadata             []byte
	EncryptedMetadataNonce        []byte
	EncryptedMetadataEncryptedKey []byte
}

UpdateObjectLastCommittedMetadata contains arguments necessary for replacing an object metadata.

func (*UpdateObjectLastCommittedMetadata) Verify added in v1.91.2 

func (obj *UpdateObjectLastCommittedMetadata) Verify() error

Verify object stream fields.

type UpdateSegmentPieces 

type UpdateSegmentPieces struct {
	StreamID uuid.UUID
	Position SegmentPosition

	OldPieces Pieces

	NewRedundancy storj.RedundancyScheme
	NewPieces     Pieces

	NewRepairedAt time.Time // sets new time of last segment repair (optional).
}

UpdateSegmentPieces contains arguments necessary for updating segment pieces.

type VerifySegment added in v1.64.1 

type VerifySegment struct {
	StreamID uuid.UUID
	Position SegmentPosition

	CreatedAt  time.Time
	RepairedAt *time.Time

	RootPieceID storj.PieceID
	Redundancy  storj.RedundancyScheme

	AliasPieces AliasPieces
}

VerifySegment result of listing segments for verifying remote segments.

type Version 

type Version int64

Version is used to uniquely identify objects with the same key.

Source Files

View all Source files

Directories

Path Synopsis
rangedlooptest
Package zombiedeletion contains the functions needed to run zombie objects deletion chore.
 Package zombiedeletion contains the functions needed to run zombie objects deletion chore.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.