storage

Documentation

Overview

Package storage provides an abstraction layer to object storage. It defines the abstract object storage interface and provides S3 and GCS implementations with support for conditional writes, storage classes, and encryption.

Index

• Variables
• func PutIfAbsent(ctx context.Context, store ObjectStorage, key string, data []byte) error
• func PutStreamIfAbsent(ctx context.Context, store ObjectStorage, key string, data io.ReadSeeker, ...) error
• type FailingStore
  • func NewFailingStore(inner ObjectStorage) *FailingStore
  • func (f *FailingStore) Delete(ctx context.Context, key string) error
  • func (f *FailingStore) Get(ctx context.Context, key string) ([]byte, string, error)
  • func (f *FailingStore) GetStream(ctx context.Context, key string) (io.ReadCloser, error)
  • func (f *FailingStore) List(ctx context.Context, prefix string) ([]ObjectInfo, error)
  • func (f *FailingStore) Put(ctx context.Context, key string, data []byte) error
  • func (f *FailingStore) PutIfMatch(ctx context.Context, key string, data []byte, etag string) error
  • func (f *FailingStore) PutStream(ctx context.Context, key string, r io.Reader, size int64) error
  • func (f *FailingStore) PutStreamIfMatch(ctx context.Context, key string, r io.Reader, size int64, etag string) error
  • func (f *FailingStore) SetFailGet(fail bool)
  • func (f *FailingStore) SetFailPut(fail bool)
• type MemoryStore
  • func NewMemoryStore() *MemoryStore
  • func (m *MemoryStore) Delete(_ context.Context, key string) error
  • func (m *MemoryStore) Get(_ context.Context, key string) ([]byte, string, error)
  • func (m *MemoryStore) GetStream(_ context.Context, key string) (io.ReadCloser, error)
  • func (m *MemoryStore) List(_ context.Context, prefix string) ([]ObjectInfo, error)
  • func (m *MemoryStore) Put(_ context.Context, key string, data []byte) error
  • func (m *MemoryStore) PutIfMatch(_ context.Context, key string, data []byte, etag string) error
  • func (m *MemoryStore) PutStream(_ context.Context, key string, r io.Reader, _ int64) error
  • func (m *MemoryStore) PutStreamIfMatch(_ context.Context, key string, r io.Reader, _ int64, etag string) error
• type ObjectInfo
• type ObjectStorage
  • func New(cfg *config.Config, logger *slog.Logger) (ObjectStorage, func(), error)

Variables

var (
	ErrNotFound     = s3lect.ErrStorageNotFound
	ErrPrecondition = s3lect.ErrStoragePrecondition
)

Common storage errors — use s3lect's errors for compatibility.

Functions

func PutIfAbsent

func PutIfAbsent(ctx context.Context, store ObjectStorage, key string, data []byte) error

PutIfAbsent creates an object only if it does not already exist. If the object already exists with identical contents, it returns nil. If it exists with different contents, it returns an error.

func PutStreamIfAbsent

func PutStreamIfAbsent(ctx context.Context, store ObjectStorage, key string, data io.ReadSeeker, size int64) error

PutStreamIfAbsent creates an object from a stream only if it does not already exist. If the object already exists with identical contents, it returns nil. If it exists with different contents, it returns an error.

Types

type FailingStore

type FailingStore struct {
	// contains filtered or unexported fields
}

FailingStore wraps an ObjectStorage implementation and can be toggled to fail specific operations on demand. It is intended for use in integration tests that exercise error handling and recovery paths.

func NewFailingStore

func NewFailingStore(inner ObjectStorage) *FailingStore

NewFailingStore returns a FailingStore that delegates to inner.

func (*FailingStore) Delete

func (f *FailingStore) Delete(ctx context.Context, key string) error

Delete delegates to the inner store.

func (*FailingStore) Get

func (f *FailingStore) Get(ctx context.Context, key string) ([]byte, string, error)

Get delegates to the inner store, or returns an error if get failures are enabled.

func (*FailingStore) GetStream

func (f *FailingStore) GetStream(ctx context.Context, key string) (io.ReadCloser, error)

GetStream delegates to the inner store, or returns an error if get failures are enabled.

func (*FailingStore) List

func (f *FailingStore) List(ctx context.Context, prefix string) ([]ObjectInfo, error)

List delegates to the inner store.

func (*FailingStore) Put

func (f *FailingStore) Put(ctx context.Context, key string, data []byte) error

Put delegates to the inner store, or returns an error if put failures are enabled.

func (*FailingStore) PutIfMatch

func (f *FailingStore) PutIfMatch(ctx context.Context, key string, data []byte, etag string) error

PutIfMatch delegates to the inner store, or returns an error if put failures are enabled.

func (*FailingStore) PutStream

func (f *FailingStore) PutStream(ctx context.Context, key string, r io.Reader, size int64) error

PutStream delegates to the inner store, or returns an error if put failures are enabled.

func (*FailingStore) PutStreamIfMatch

func (f *FailingStore) PutStreamIfMatch(ctx context.Context, key string, r io.Reader, size int64, etag string) error

PutStreamIfMatch delegates to the inner store, or returns an error if put failures are enabled.

func (*FailingStore) SetFailGet

func (f *FailingStore) SetFailGet(fail bool)

SetFailGet toggles whether Get and GetStream calls fail.

func (*FailingStore) SetFailPut

func (f *FailingStore) SetFailPut(fail bool)

SetFailPut toggles whether Put, PutIfMatch, PutStream, and PutStreamIfMatch calls fail.

type MemoryStore

type MemoryStore struct {
	// contains filtered or unexported fields
}

MemoryStore is an in-memory ObjectStorage for use in tests. It is safe for concurrent use. ETags are computed as MD5 hex digests of the stored content, matching S3 ETag semantics.

func NewMemoryStore

func NewMemoryStore() *MemoryStore

NewMemoryStore returns an empty MemoryStore.

func (*MemoryStore) Delete

func (m *MemoryStore) Delete(_ context.Context, key string) error

func (*MemoryStore) Get

func (m *MemoryStore) Get(_ context.Context, key string) ([]byte, string, error)

func (*MemoryStore) GetStream

func (m *MemoryStore) GetStream(_ context.Context, key string) (io.ReadCloser, error)

func (*MemoryStore) List

func (m *MemoryStore) List(_ context.Context, prefix string) ([]ObjectInfo, error)

func (*MemoryStore) Put

func (m *MemoryStore) Put(_ context.Context, key string, data []byte) error

func (*MemoryStore) PutIfMatch

func (m *MemoryStore) PutIfMatch(_ context.Context, key string, data []byte, etag string) error

func (*MemoryStore) PutStream

func (m *MemoryStore) PutStream(_ context.Context, key string, r io.Reader, _ int64) error

func (*MemoryStore) PutStreamIfMatch

func (m *MemoryStore) PutStreamIfMatch(_ context.Context, key string, r io.Reader, _ int64, etag string) error

type ObjectInfo

type ObjectInfo struct {
	Key  string
	Size int64
}

ObjectInfo represents metadata about an object in storage.

type ObjectStorage

type ObjectStorage interface {
	// Get retrieves an object and returns its contents and ETag.
	// Returns ErrNotFound when the key does not exist.
	Get(ctx context.Context, key string) ([]byte, string, error)

	// Put stores an object in storage.
	Put(ctx context.Context, key string, data []byte) error

	// PutIfMatch stores an object only if the ETag matches.
	// An empty etag means the object must not exist (create-only).
	// Returns ErrPrecondition when the precondition is not met.
	PutIfMatch(ctx context.Context, key string, data []byte, etag string) error

	// GetStream retrieves an object as a stream.
	// Returns ErrNotFound when the key does not exist.
	GetStream(ctx context.Context, key string) (io.ReadCloser, error)

	// PutStream stores an object from a stream.
	PutStream(ctx context.Context, key string, r io.Reader, size int64) error

	// PutStreamIfMatch stores an object from a stream only if the ETag matches.
	// An empty etag means the object must not exist (create-only).
	// Returns ErrPrecondition when the precondition is not met.
	PutStreamIfMatch(ctx context.Context, key string, r io.Reader, size int64, etag string) error

	// Delete removes the object at the given key.
	Delete(ctx context.Context, key string) error

	// List returns all object keys matching the given prefix.
	List(ctx context.Context, prefix string) ([]ObjectInfo, error)
}

ObjectStorage defines the interface for object storage operations. Buffered methods (Get, Put, PutIfMatch) operate on whole objects in memory and are intended for small blobs/files. Streaming methods (GetStream, PutStream, PutStreamIfMatch) are intended for large data files.

func New

func New(cfg *config.Config, logger *slog.Logger) (ObjectStorage, func(), error)

New creates an ObjectStorage provider based on config. Returns the provider, a cleanup function, and any error.