objstore

package
v1.1.0-beta.0...-fba827a Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 28, 2026 License: Apache-2.0 Imports: 73 Imported by: 0

Documentation

Index

Constants

View Source 
const (

	// LocalURIPrefix represents the local storage prefix.
	LocalURIPrefix = "file://"
)
View Source 
const (
	// TombstoneSize is the tombstone size.
	TombstoneSize int64 = -1
)

Variables

This section is empty.

Functions

func CloneDefaultHTTPTransport 

func CloneDefaultHTTPTransport() (*http.Transport, bool)

CloneDefaultHTTPTransport close config.

func Create 

func Create(ctx context.Context, backend *backuppb.StorageBackend, sendCreds bool) (storeapi.Storage, error)

Create creates Storage.

Please consider using `New` in the future.

func DefineFlags 

func DefineFlags(flags *pflag.FlagSet)

DefineFlags adds flags to the flag set corresponding to all backend options.

func ExtractQueryParameters 

func ExtractQueryParameters(u *url.URL, options any)

ExtractQueryParameters moves the query parameters of the URL into the options using reflection.

The options must be a pointer to a struct which contains only string or bool fields (more types will be supported in the future), and tagged for JSON serialization.

All of the URL's query parameters will be removed after calling this method.

func FormatBackendURL 

func FormatBackendURL(backend *backuppb.StorageBackend) (u url.URL)

FormatBackendURL obtains the raw URL which can be used the reconstruct the backend. The returned URL does not contain options for further configurating the backend. This is to avoid exposing secret tokens.

func GetActiveUploadWorkerCount 

func GetActiveUploadWorkerCount() int64

GetActiveUploadWorkerCount returns the active upload worker count.

func GetDefaultHTTPClient 

func GetDefaultHTTPClient(concurrency uint) *http.Client

GetDefaultHTTPClient Different from `http.DefaultTransport`, set the `MaxIdleConns` and `MaxIdleConnsPerHost` to the actual request concurrency to reuse tcp connection as much as possible.

func HiddenFlagsForStream 

func HiddenFlagsForStream(flags *pflag.FlagSet)

HiddenFlagsForStream hidden flags for stream cmd.

func InterceptDecompressReader 

func InterceptDecompressReader(
	fileReader objectio.Reader,
	compressType compressedio.CompressType,
	cfg compressedio.DecompressConfig,
) (objectio.Reader, error)

InterceptDecompressReader intercepts the reader and wraps it with a decompress reader on the given Reader. Note that the returned Reader does not have the property that Seek(0, io.SeekCurrent) equals total bytes Read() if the decompress reader is used.

func IsLocal 

func IsLocal(u *url.URL) bool

IsLocal returns true if the URL is a local file path.

func IsLocalPath 

func IsLocalPath(p string) (bool, error)

IsLocalPath returns true if the path is a local file path.

func IsS3Like 

func IsS3Like(u *url.URL) bool

IsS3Like returns true if the URL is an S3 like store URL.

func JSONEffects 

func JSONEffects(es []Effect, output io.Writer) error

JSONEffects converts a slices of effects into json. The json will be a tagged union: `{"type": $go_type_name, "effect": $effect}`

func New 

func New(ctx context.Context, backend *backuppb.StorageBackend, opts *storeapi.Options) (storeapi.Storage, error)

New creates an Storage with options.

func NewFromURL 

func NewFromURL(ctx context.Context, uri string) (storeapi.Storage, error)

NewFromURL creates an Storage from URL.

func NewLimitedInterceptReader 

func NewLimitedInterceptReader(
	fileReader objectio.Reader,
	compressType compressedio.CompressType,
	cfg compressedio.DecompressConfig,
	n int64,
) (objectio.Reader, error)

NewLimitedInterceptReader creates a decompress reader with limit n.

func NewWithDefaultOpt 

func NewWithDefaultOpt(ctx context.Context, backend *backuppb.StorageBackend) (storeapi.Storage, error)

NewWithDefaultOpt creates Storage with default options.

func ParseBackend 

func ParseBackend(rawURL string, options *BackendOptions) (*backuppb.StorageBackend, error)

ParseBackend constructs a structured backend description from the storage URL.

func ParseBackendFromURL 

func ParseBackendFromURL(u *url.URL, options *BackendOptions) (*backuppb.StorageBackend, error)

ParseBackendFromURL constructs a structured backend description from the *url.URL.

func ParseRawURL 

func ParseRawURL(rawURL string) (*url.URL, error)

ParseRawURL parse raw url to url object.

func ReadDataInRange 

func ReadDataInRange(
	ctx context.Context,
	storage storeapi.Storage,
	name string,
	start int64,
	p []byte,
) (n int, err error)

ReadDataInRange reads data from storage in range [start, start+len(p)).

func SaveJSONEffectsToTmp 

func SaveJSONEffectsToTmp(es []Effect) (string, error)

SaveJSONEffectsToTmp save to tmp.

func UnmarshalDir 

func UnmarshalDir[T any](ctx context.Context, walkOpt *storeapi.WalkOption, s storeapi.Storage, unmarshal func(target *T, name string, content []byte) error) iter.TryNextor[*T]

UnmarshalDir iterates over a prefix, then "unmarshal" the content of each file it met with the unmarshal function. Returning an iterator that yields the unmarshaled content. The "unmarshal" function should put the result of unmarshalling to the `target` argument.

func ValidateCloudStorageURI 

func ValidateCloudStorageURI(ctx context.Context, uri string) error

ValidateCloudStorageURI makes validation for tidb_cloud_storage_uri.

func WithCompression 

func WithCompression(inner storeapi.Storage, compressionType compressedio.CompressType, cfg compressedio.DecompressConfig) storeapi.Storage

WithCompression returns an Storage with compress option

Types

type AzblobBackendOptions 

type AzblobBackendOptions struct {
	Endpoint        string `json:"endpoint" toml:"endpoint"`
	AccountName     string `json:"account-name" toml:"account-name"`
	AccountKey      string `json:"account-key" toml:"account-key"`
	AccessTier      string `json:"access-tier" toml:"access-tier"`
	SASToken        string `json:"sas-token" toml:"sas-token"`
	EncryptionScope string `json:"encryption-scope" toml:"encryption-scope"`
	EncryptionKey   string `json:"encryption-key" toml:"encryption-key"`
}

AzblobBackendOptions is the options for Azure Blob storage.

type AzureBlobStorage 

type AzureBlobStorage struct {
	// contains filtered or unexported fields
}

AzureBlobStorage is a storage engine that stores data in Azure Blob Storage.

func (*AzureBlobStorage) Close 

func (*AzureBlobStorage) Close()

Close implements the Storage interface.

func (*AzureBlobStorage) CopyFrom 

func (s *AzureBlobStorage) CopyFrom(ctx context.Context, e storeapi.Storage, spec storeapi.CopySpec) error

CopyFrom implements Copier.

func (*AzureBlobStorage) Create 

func (s *AzureBlobStorage) Create(_ context.Context, name string, _ *storeapi.WriterOption) (objectio.Writer, error)

Create implements the StorageWriter interface.

func (*AzureBlobStorage) DeleteFile 

func (s *AzureBlobStorage) DeleteFile(ctx context.Context, name string) error

DeleteFile deletes the file with the given name.

func (*AzureBlobStorage) DeleteFiles 

func (s *AzureBlobStorage) DeleteFiles(ctx context.Context, names []string) error

DeleteFiles deletes the files with the given names.

func (*AzureBlobStorage) FileExists 

func (s *AzureBlobStorage) FileExists(ctx context.Context, name string) (bool, error)

FileExists checks if a file exists in Azure Blob Storage.

func (*AzureBlobStorage) MarkStrongConsistency 

func (*AzureBlobStorage) MarkStrongConsistency()

MarkStrongConsistency implements Storage.

func (*AzureBlobStorage) Open 

func (s *AzureBlobStorage) Open(ctx context.Context, name string, o *storeapi.ReaderOption) (objectio.Reader, error)

Open implements the StorageReader interface.

func (*AzureBlobStorage) PresignFile 

func (*AzureBlobStorage) PresignFile(_ context.Context, _ string, _ time.Duration) (string, error)

PresignFile implements storeapi.Storage interface.

func (*AzureBlobStorage) ReadFile 

func (s *AzureBlobStorage) ReadFile(ctx context.Context, name string) ([]byte, error)

ReadFile reads a file from Azure Blob Storage.

func (*AzureBlobStorage) Rename 

func (s *AzureBlobStorage) Rename(ctx context.Context, oldFileName, newFileName string) error

Rename implements the StorageWriter interface.

func (*AzureBlobStorage) URI 

func (s *AzureBlobStorage) URI() string

URI implements the StorageReader interface.

func (*AzureBlobStorage) WalkDir 

func (s *AzureBlobStorage) WalkDir(ctx context.Context, opt *storeapi.WalkOption, fn func(path string, size int64) error) error

WalkDir implements the StorageReader interface.

func (*AzureBlobStorage) WriteFile 

func (s *AzureBlobStorage) WriteFile(ctx context.Context, name string, data []byte) error

WriteFile writes a file to Azure Blob Storage.

type BackendOptions 

type BackendOptions struct {
	S3     s3like.S3BackendOptions `json:"s3" toml:"s3"`
	GCS    GCSBackendOptions       `json:"gcs" toml:"gcs"`
	Azblob AzblobBackendOptions    `json:"azblob" toml:"azblob"`
}

BackendOptions further configures the storage backend not expressed by the storage URL.

func (*BackendOptions) ParseFromFlags 

func (options *BackendOptions) ParseFromFlags(flags *pflag.FlagSet) error

ParseFromFlags obtains the backend options from the flag set.

type Batched 

type Batched struct {
	storeapi.Storage
	// contains filtered or unexported fields
}

Batched is a wrapper of an external storage that suspends all write operations ("effects"). If `Close()` without calling `Commit()`, nothing will happen in the underlying external storage. In that case, we have done a "dry run".

You may use `ReadOnlyEffects()` to get the history of the effects. But don't modify the returned slice!

You may use `Commit()` to execute all suspended effects.

func Batch 

func Batch(s storeapi.Storage) *Batched

Batch wraps an external storage instance to a batched version.

func (*Batched) CleanEffects 

func (d *Batched) CleanEffects()

CleanEffects cleans all suspended effects.

func (*Batched) Commit 

func (d *Batched) Commit(ctx context.Context) error

Commit performs all effects recorded so long in the REAL external storage. This will cleanup all of the suspended effects.

func (*Batched) Create 

func (d *Batched) Create(ctx context.Context, path string, option *storeapi.WriterOption) (objectio.Writer, error)

Create implements the Storage interface.

func (*Batched) DeleteFile 

func (d *Batched) DeleteFile(ctx context.Context, name string) error

DeleteFile implements the Storage interface.

func (*Batched) DeleteFiles 

func (d *Batched) DeleteFiles(ctx context.Context, names []string) error

DeleteFiles implements the Storage interface.

func (*Batched) PresignFile 

func (d *Batched) PresignFile(ctx context.Context, fileName string, expire time.Duration) (string, error)

PresignFile implements the Storage interface by delegating to the underlying storage.

func (*Batched) ReadOnlyEffects 

func (d *Batched) ReadOnlyEffects() []Effect

ReadOnlyEffects Fetch all effects from the batched storage.

**The returned slice should not be modified.**

func (*Batched) Rename 

func (d *Batched) Rename(ctx context.Context, oldName, newName string) error

Rename implements the Storage interface.

func (*Batched) WriteFile 

func (d *Batched) WriteFile(ctx context.Context, name string, data []byte) error

WriteFile implements the Storage interface.

type ClientBuilder 

type ClientBuilder interface {
	// Example of serviceURL: https://<your_storage_account>.blob.core.windows.net
	GetServiceClient() (*azblob.Client, error)
	GetAccountName() string
	GetServiceURL() string
}

ClientBuilder provides common method to build a service client.

type CompleteMultipartUpload 

type CompleteMultipartUpload struct {
	XMLName xml.Name `xml:"CompleteMultipartUpload"`
	Text    string   `xml:",chardata"`
	Parts   []Part   `xml:"Part"`
}

CompleteMultipartUpload is the complete multipart upload structure.

type EffDeleteFile 

type EffDeleteFile string

EffDeleteFile is the side effect of a call to `DeleteFile`.

type EffDeleteFiles 

type EffDeleteFiles struct {
	Files []string `json:"files"`
}

EffDeleteFiles is the side effect of a call to `DeleteFiles`.

type EffPut 

type EffPut struct {
	File    string `json:"file"`
	Content []byte `json:"content"`
}

EffPut is the side effect of a call to `WriteFile`.

type EffRename 

type EffRename struct {
	From string `json:"from"`
	To   string `json:"to"`
}

EffRename is the side effect of a call to `Rename`.

type Effect 

type Effect any

Effect is an side effect that happens in the batch storage.

type ErrLocked 

type ErrLocked struct {
	Meta LockMeta
}

ErrLocked is the error returned when the lock is held by others.

func (ErrLocked) Error 

func (e ErrLocked) Error() string

Error return the error.

type GCSBackendOptions 

type GCSBackendOptions struct {
	Endpoint        string `json:"endpoint" toml:"endpoint"`
	StorageClass    string `json:"storage-class" toml:"storage-class"`
	PredefinedACL   string `json:"predefined-acl" toml:"predefined-acl"`
	CredentialsFile string `json:"credentials-file" toml:"credentials-file"`
}

GCSBackendOptions are options for configuration the GCS storage.

type GCSStorage 

type GCSStorage struct {
	// contains filtered or unexported fields
}

GCSStorage defines some standard operations for BR/Lightning on the GCS storage. It implements the `Storage` interface.

func NewGCSStorage 

func NewGCSStorage(ctx context.Context, gcs *backuppb.GCS, opts *storeapi.Options) (*GCSStorage, error)

NewGCSStorage creates a GCS external storage implementation.

func (*GCSStorage) Close 

func (s *GCSStorage) Close()

Close implements Storage interface.

func (*GCSStorage) CopyFrom 

func (s *GCSStorage) CopyFrom(ctx context.Context, e storeapi.Storage, spec storeapi.CopySpec) error

CopyFrom implements Copier.

func (*GCSStorage) Create 

func (s *GCSStorage) Create(ctx context.Context, name string, wo *storeapi.WriterOption) (objectio.Writer, error)

Create implements Storage interface.

func (*GCSStorage) DeleteFile 

func (s *GCSStorage) DeleteFile(ctx context.Context, name string) error

DeleteFile delete the file in storage

func (*GCSStorage) DeleteFiles 

func (s *GCSStorage) DeleteFiles(ctx context.Context, names []string) error

DeleteFiles delete the files in storage. If the file does not exist, we will ignore it.

func (*GCSStorage) FileExists 

func (s *GCSStorage) FileExists(ctx context.Context, name string) (bool, error)

FileExists return true if file exists.

func (*GCSStorage) GetBucketHandle 

func (s *GCSStorage) GetBucketHandle() *storage.BucketHandle

GetBucketHandle gets the handle to the GCS API on the bucket.

func (*GCSStorage) GetOptions 

func (s *GCSStorage) GetOptions() *backuppb.GCS

GetOptions gets the external storage operations for the GCS.

func (*GCSStorage) MarkStrongConsistency 

func (s *GCSStorage) MarkStrongConsistency()

MarkStrongConsistency implements Storage interface.

func (*GCSStorage) Open 

func (s *GCSStorage) Open(ctx context.Context, path string, o *storeapi.ReaderOption) (objectio.Reader, error)

Open a Reader by file path.

func (*GCSStorage) PresignFile 

func (s *GCSStorage) PresignFile(ctx context.Context, fileName string, expire time.Duration) (string, error)

PresignFile implements storeapi.Storage interface.

func (*GCSStorage) ReadFile 

func (s *GCSStorage) ReadFile(ctx context.Context, name string) ([]byte, error)

ReadFile reads the file from the storage and returns the contents.

func (*GCSStorage) Rename 

func (s *GCSStorage) Rename(ctx context.Context, oldFileName, newFileName string) error

Rename file name from oldFileName to newFileName.

func (*GCSStorage) Reset 

func (s *GCSStorage) Reset(ctx context.Context) error

Reset resets the GCS storage. Reset should not be used concurrently with Close.

func (*GCSStorage) URI 

func (s *GCSStorage) URI() string

URI implements Storage interface.

func (*GCSStorage) WalkDir 

func (s *GCSStorage) WalkDir(ctx context.Context, opt *storeapi.WalkOption, fn func(string, int64) error) error

WalkDir traverse all the files in a dir.

fn is the function called for each regular file visited by WalkDir. The first argument is the file path that can be used in `Open` function; the second argument is the size in byte of the file determined by path.

func (*GCSStorage) WriteFile 

func (s *GCSStorage) WriteFile(ctx context.Context, name string, data []byte) error

WriteFile writes data to a file to storage.

type GCSWriter 

type GCSWriter struct {
	// contains filtered or unexported fields
}

GCSWriter uses XML multipart upload API to upload a single file. https://cloud.google.com/storage/docs/multipart-uploads. GCSWriter will attempt to cancel uploads that fail due to an exception. If the upload fails in a way that precludes cancellation, such as a hardware failure, process termination, or power outage, then the incomplete upload may persist indefinitely. To mitigate this, set the `AbortIncompleteMultipartUpload` with a nonzero `Age` in bucket lifecycle rules, or refer to the XML API documentation linked above to learn more about how to list and delete individual downloads.

func NewGCSWriter 

func NewGCSWriter(
	ctx context.Context,
	cli *storage.Client,
	uri string,
	partSize int64,
	parallelCnt int,
	bucketName string,
) (*GCSWriter, error)

NewGCSWriter returns a GCSWriter which uses GCS multipart upload API behind the scene.

func (*GCSWriter) Close 

func (w *GCSWriter) Close() error

Close finishes the upload.

func (*GCSWriter) Write 

func (w *GCSWriter) Write(p []byte) (n int, err error)

Write uploads given bytes as a part to Google Cloud Storage. Write is not concurrent safe.

type HDFSStorage 

type HDFSStorage struct {
	// contains filtered or unexported fields
}

HDFSStorage represents HDFS storage.

func NewHDFSStorage 

func NewHDFSStorage(remote string) *HDFSStorage

NewHDFSStorage creates a new HDFS storage.

func (*HDFSStorage) Close 

func (*HDFSStorage) Close()

Close implements Storage interface.

func (*HDFSStorage) Create 

func (*HDFSStorage) Create(_ context.Context, _ string, _ *storeapi.WriterOption) (objectio.Writer, error)

Create opens a file writer by path. path is relative path to storage base path

func (*HDFSStorage) DeleteFile 

func (*HDFSStorage) DeleteFile(_ context.Context, _ string) error

DeleteFile delete the file in storage

func (*HDFSStorage) DeleteFiles 

func (*HDFSStorage) DeleteFiles(_ context.Context, _ []string) error

DeleteFiles deletes files in storage

func (*HDFSStorage) FileExists 

func (s *HDFSStorage) FileExists(_ context.Context, name string) (bool, error)

FileExists return true if file exists

func (*HDFSStorage) Open 

func (*HDFSStorage) Open(_ context.Context, _ string, _ *storeapi.ReaderOption) (objectio.Reader, error)

Open a Reader by file path. path is relative path to storage base path

func (*HDFSStorage) PresignFile 

func (*HDFSStorage) PresignFile(_ context.Context, _ string, _ time.Duration) (string, error)

PresignFile implements storeapi.Storage interface.

func (*HDFSStorage) ReadFile 

func (*HDFSStorage) ReadFile(_ context.Context, _ string) ([]byte, error)

ReadFile reads a complete file from storage, similar to os.ReadFile

func (*HDFSStorage) Rename 

func (*HDFSStorage) Rename(_ context.Context, _, _ string) error

Rename a file name from oldFileName to newFileName.

func (*HDFSStorage) URI 

func (s *HDFSStorage) URI() string

URI returns the base path as a URI

func (*HDFSStorage) WalkDir 

func (*HDFSStorage) WalkDir(_ context.Context, _ *storeapi.WalkOption, _ func(path string, size int64) error) error

WalkDir traverse all the files in a dir.

fn is the function called for each regular file visited by WalkDir. The argument `path` is the file path that can be used in `Open` function; the argument `size` is the size in byte of the file determined by path.

func (*HDFSStorage) WriteFile 

func (s *HDFSStorage) WriteFile(_ context.Context, name string, data []byte) error

WriteFile writes a complete file to storage, similar to os.WriteFile

type InitiateMultipartUploadResult 

type InitiateMultipartUploadResult struct {
	XMLName  xml.Name `xml:"InitiateMultipartUploadResult"`
	Text     string   `xml:",chardata"`
	Xmlns    string   `xml:"xmlns,attr"`
	Bucket   string   `xml:"Bucket"`
	Key      string   `xml:"Key"`
	UploadID string   `xml:"UploadId"`
}

InitiateMultipartUploadResult initiate multipart upload result structure.

type LocalStorage 

type LocalStorage struct {

	// Whether ignoring ENOINT while deleting.
	// Don't fail when deleting an unexist file is more like
	// a normal Storage implementation does.
	IgnoreEnoentForDelete bool
	// contains filtered or unexported fields
}

LocalStorage represents local file system storage.

export for using in tests.

func NewLocalStorage 

func NewLocalStorage(base string) (*LocalStorage, error)

NewLocalStorage return a LocalStorage at directory `base`.

export for test.

func (*LocalStorage) Base 

func (l *LocalStorage) Base() string

Base returns the base dir used by this local storage.

func (*LocalStorage) Close 

func (*LocalStorage) Close()

Close implements Storage interface.

func (*LocalStorage) CopyFrom 

func (l *LocalStorage) CopyFrom(ctx context.Context, e storeapi.Storage, spec storeapi.CopySpec) error

CopyFrom implements Storage interface.

func (*LocalStorage) Create 

func (l *LocalStorage) Create(_ context.Context, name string, _ *storeapi.WriterOption) (objectio.Writer, error)

Create implements Storage interface.

func (*LocalStorage) DeleteFile 

func (l *LocalStorage) DeleteFile(_ context.Context, name string) error

DeleteFile deletes the file.

func (*LocalStorage) DeleteFiles 

func (l *LocalStorage) DeleteFiles(ctx context.Context, names []string) error

DeleteFiles deletes the files.

func (*LocalStorage) FileExists 

func (l *LocalStorage) FileExists(_ context.Context, name string) (bool, error)

FileExists implement Storage.FileExists.

func (*LocalStorage) Open 

func (l *LocalStorage) Open(_ context.Context, path string, o *storeapi.ReaderOption) (objectio.Reader, error)

Open a Reader by file path, path is a relative path to base path.

func (*LocalStorage) PresignFile 

func (l *LocalStorage) PresignFile(_ context.Context, fileName string, _ time.Duration) (string, error)

PresignFile implements storeapi.Storage interface. For local storage, returns the file name only (basename) since presigned URL is not applicable.

func (*LocalStorage) ReadFile 

func (l *LocalStorage) ReadFile(_ context.Context, name string) ([]byte, error)

ReadFile reads the file from the storage and returns the contents.

func (*LocalStorage) Rename 

func (l *LocalStorage) Rename(_ context.Context, oldFileName, newFileName string) error

Rename implements Storage interface.

func (*LocalStorage) URI 

func (l *LocalStorage) URI() string

URI returns the base path as an URI with a file:/// prefix.

func (*LocalStorage) WalkDir 

func (l *LocalStorage) WalkDir(_ context.Context, opt *storeapi.WalkOption, fn func(string, int64) error) error

WalkDir traverse all the files in a dir.

fn is the function called for each regular file visited by WalkDir. The first argument is the file path that can be used in `Open` function; the second argument is the size in byte of the file determined by path.

func (*LocalStorage) WriteFile 

func (l *LocalStorage) WriteFile(_ context.Context, name string, data []byte) error

WriteFile writes data to a file to storage.

type LockMeta 

type LockMeta struct {
	LockedAt   time.Time `json:"locked_at"`
	LockerHost string    `json:"locker_host"`
	LockerPID  int       `json:"locker_pid"`
	TxnID      []byte    `json:"txn_id"`
	Hint       string    `json:"hint"`
}

LockMeta is the meta information of a lock.

func MakeLockMeta 

func MakeLockMeta(hint string) LockMeta

MakeLockMeta creates a LockMeta by the current node's metadata. Including current time and hostname, etc..

func (LockMeta) String 

func (l LockMeta) String() string

String implements fmt.Stringer interface.

type Locker 

type Locker = func(ctx context.Context, storage storeapi.Storage, path, hint string) (lock RemoteLock, err error)

Locker is a locker.

type MemStorage 

type MemStorage struct {
	// contains filtered or unexported fields
}

MemStorage represents a in-memory storage.

func NewMemStorage 

func NewMemStorage() *MemStorage

NewMemStorage creates a new in-memory storage.

func (*MemStorage) Close 

func (s *MemStorage) Close()

Close implements Storage interface.

func (*MemStorage) Create 

func (s *MemStorage) Create(ctx context.Context, name string, _ *storeapi.WriterOption) (objectio.Writer, error)

Create creates a file and returning a writer to write data into. When the writer is closed, the data is stored in the file. It implements the `Storage` interface

func (*MemStorage) DeleteFile 

func (s *MemStorage) DeleteFile(ctx context.Context, name string) error

DeleteFile delete the file in storage It implements the `Storage` interface

func (*MemStorage) DeleteFiles 

func (s *MemStorage) DeleteFiles(ctx context.Context, names []string) error

DeleteFiles delete the files in storage It implements the `Storage` interface

func (*MemStorage) FileExists 

func (s *MemStorage) FileExists(ctx context.Context, name string) (bool, error)

FileExists return true if file exists. It implements the `Storage` interface

func (*MemStorage) Open 

func (s *MemStorage) Open(ctx context.Context, filePath string, o *storeapi.ReaderOption) (objectio.Reader, error)

Open opens a Reader by file path. It implements the `Storage` interface

func (*MemStorage) PresignFile 

func (s *MemStorage) PresignFile(_ context.Context, fileName string, _ time.Duration) (string, error)

PresignFile implements storeapi.Storage interface. For in-memory storage, returns the file name only (basename).

func (*MemStorage) ReadFile 

func (s *MemStorage) ReadFile(ctx context.Context, name string) ([]byte, error)

ReadFile reads the storage file. It implements the `Storage` interface

func (*MemStorage) Rename 

func (s *MemStorage) Rename(ctx context.Context, oldFileName, newFileName string) error

Rename renames a file name to another file name. It implements the `Storage` interface

func (*MemStorage) URI 

func (*MemStorage) URI() string

URI returns the URI of the storage.

func (*MemStorage) WalkDir 

func (s *MemStorage) WalkDir(ctx context.Context, opt *storeapi.WalkOption, fn func(string, int64) error) error

WalkDir traverse all the files in a dir. It implements the `Storage` interface

func (*MemStorage) WriteFile 

func (s *MemStorage) WriteFile(ctx context.Context, name string, data []byte) error

WriteFile file to storage. It implements the `Storage` interface

type NoopWriter 

type NoopWriter struct{}

NoopWriter is a writer that does nothing.

func (NoopWriter) Close 

func (NoopWriter) Close(_ context.Context) error

Close implements objectio.Writer interface.

func (NoopWriter) Write 

func (NoopWriter) Write(_ context.Context, p []byte) (int, error)

Write implements objectio.Writer interface.

type Part 

type Part struct {
	Text       string `xml:",chardata"`
	PartNumber int    `xml:"PartNumber"`
	ETag       string `xml:"ETag"`
}

Part is a part.

type RemoteLock 

type RemoteLock struct {
	// contains filtered or unexported fields
}

RemoteLock is the remote lock.

func LockWithRetry 

func LockWithRetry(ctx context.Context, locker Locker, storage storeapi.Storage, path, hint string) (
	lock RemoteLock, err error)

LockWithRetry lock with retry.

func TryLockRemote 

func TryLockRemote(ctx context.Context, storage storeapi.Storage, path, hint string) (lock RemoteLock, err error)

TryLockRemote tries to create a "lock file" at the external storage. If success, we will create a file at the path provided. So others may not access the file then. Will return a `ErrLocked` if there is another process already creates the lock file. This isn't a strict lock like flock in linux: that means, the lock might be forced removed by manually deleting the "lock file" in external storage.

func TryLockRemoteRead 

func TryLockRemoteRead(ctx context.Context, storage storeapi.Storage, path, hint string) (lock RemoteLock, err error)

TryLockRemoteRead try lock.

func TryLockRemoteWrite 

func TryLockRemoteWrite(ctx context.Context, storage storeapi.Storage, path, hint string) (lock RemoteLock, err error)

TryLockRemoteWrite try lock.

func (*RemoteLock) String 

func (l *RemoteLock) String() string

String implements fmt.Stringer interface.

func (RemoteLock) Unlock 

func (l RemoteLock) Unlock(ctx context.Context) error

Unlock removes the lock file at the specified path. Removing that file will release the lock.

func (RemoteLock) UnlockOnCleanUp 

func (l RemoteLock) UnlockOnCleanUp(ctx context.Context)

UnlockOnCleanUp unlock the lock on clean up.

type VerifyWriteContext 

type VerifyWriteContext struct {
	context.Context
	Target  string
	Storage storeapi.Storage
	TxnID   uuid.UUID
}

VerifyWriteContext is the verify write context

func (*VerifyWriteContext) IntentFileName 

func (cx *VerifyWriteContext) IntentFileName() string

IntentFileName return the intent file name.

Source Files

View all Source files

Directories

Path Synopsis
Package mockobjstore is a generated GoMock package.
 Package mockobjstore is a generated GoMock package.
mock
Package mock is a generated GoMock package.
 Package mock is a generated GoMock package.
mock
Package mock is a generated GoMock package.
 Package mock is a generated GoMock package.
mock
Package mock is a generated GoMock package.
 Package mock is a generated GoMock package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.