Documentation
¶
Overview ¶
Package s3 implements a backend.Backend over an S3-compatible object store. The store-specific calls are isolated behind the small ObjectStore interface: the AWS SDK adapter (NewAWS) implements it for real S3, and tests implement it with an in-memory fake, so all of the Backend's mapping logic — key prefixing, sorted listing, 404 → backend.ErrNotExist translation, conditional-put, and existence-checked delete — is exercised by the shared backend conformance suite without a live bucket.
The object store is the durable, stateless tier: nodes hold no authoritative state, so the read path is reconstructed entirely from objects (DESIGN.md §3, §11). Whole-object Get/Put is sufficient because a part maps to one key prefix with one object per column/marks/manifest.
Index ¶
- Variables
- type AWSAPI
- type Backend
- func (b *Backend) Delete(ctx context.Context, key string) error
- func (*Backend) IsEphemeral() bool
- func (b *Backend) List(ctx context.Context, prefix string) ([]string, error)
- func (b *Backend) PutIfAbsent(ctx context.Context, key string, data []byte) (bool, error)
- func (b *Backend) Read(ctx context.Context, key string) ([]byte, error)
- func (b *Backend) Write(ctx context.Context, key string, data []byte) error
- type ObjectStore
- type Option
Constants ¶
This section is empty.
Variables ¶
var ErrObjectNotFound = errors.New("s3: object not found")
ErrObjectNotFound is returned (wrapped) by ObjectStore.GetObject for an absent key. The Backend translates it to backend.ErrNotExist.
Functions ¶
This section is empty.
Types ¶
type AWSAPI ¶
type AWSAPI interface {
GetObject(ctx context.Context, in *awss3.GetObjectInput, optFns ...func(*awss3.Options)) (*awss3.GetObjectOutput, error)
PutObject(ctx context.Context, in *awss3.PutObjectInput, optFns ...func(*awss3.Options)) (*awss3.PutObjectOutput, error)
HeadObject(ctx context.Context, in *awss3.HeadObjectInput, optFns ...func(*awss3.Options)) (*awss3.HeadObjectOutput, error)
DeleteObject(ctx context.Context, in *awss3.DeleteObjectInput, optFns ...func(*awss3.Options)) (*awss3.DeleteObjectOutput, error)
ListObjectsV2(ctx context.Context, in *awss3.ListObjectsV2Input, optFns ...func(*awss3.Options)) (*awss3.ListObjectsV2Output, error)
}
AWSAPI is the subset of the aws-sdk-go-v2 *s3.Client the NewAWS adapter uses. The real client satisfies it; tests can fake it. It also satisfies s3.ListObjectsV2APIClient (used by the paginator).
type Backend ¶
type Backend struct {
// contains filtered or unexported fields
}
Backend is a backend.Backend over an ObjectStore. Keys are stored under an optional root prefix so several datasets can share one bucket.
func New ¶
func New(store ObjectStore, keyPrefix string, opts ...Option) *Backend
New returns a Backend over store, rooting all keys under keyPrefix (which may be empty). Pass WithRetry to make it resilient to a lossy/slow endpoint (per-attempt timeouts, bounded retries, hedged GETs).
func (*Backend) Delete ¶
Delete removes key, returning an backend.ErrNotExist-wrapping error if it is absent. Because S3 DeleteObject is idempotent, existence is checked first to honor the contract.
func (*Backend) IsEphemeral ¶
IsEphemeral reports false: objects persist in the store.
func (*Backend) PutIfAbsent ¶
PutIfAbsent stores data under key only if it does not already exist.
func (*Backend) Read ¶
Read returns the value stored under key, or an backend.ErrNotExist-wrapping error.
type ObjectStore ¶
type ObjectStore interface {
// GetObject returns the object bytes, or an error wrapping [ErrObjectNotFound] if absent.
GetObject(ctx context.Context, key string) ([]byte, error)
// PutObject stores data under key, overwriting any existing object.
PutObject(ctx context.Context, key string, data []byte) error
// PutObjectIfAbsent stores data under key only if it does not exist, returning whether it
// was created (the conditional write — S3 If-None-Match: *).
PutObjectIfAbsent(ctx context.Context, key string, data []byte) (bool, error)
// HeadObject reports whether key exists.
HeadObject(ctx context.Context, key string) (bool, error)
// DeleteObject removes key. It is idempotent (no error if the key is absent), mirroring
// S3 DeleteObject.
DeleteObject(ctx context.Context, key string) error
// ListObjects returns every key under prefix (the implementation paginates internally).
// Order is unspecified; the [Backend] sorts.
ListObjects(ctx context.Context, prefix string) ([]string, error)
}
ObjectStore is the minimal object-store API the Backend needs. It is intentionally thin and S3-shaped (delete is idempotent, list is by prefix); the Backend layers the backend.Backend contract on top. Implementations must be safe for concurrent use.
func NewAWS ¶
func NewAWS(api AWSAPI, bucket string) ObjectStore
NewAWS returns an ObjectStore backed by an aws-sdk-go-v2 S3 client over the given bucket. Compose it with New to get a backend.Backend:
store := s3.NewAWS(awss3.NewFromConfig(cfg), "my-bucket") b := s3.New(store, "oteldb/")
This adapter is verified by compilation and exercised against real/MinIO S3 in integration tests; the Backend's contract logic is covered by the conformance suite over the in-memory fake.
type Option ¶ added in v0.5.0
type Option func(*config)
Option configures a Backend at construction.
func WithRetry ¶ added in v0.5.0
func WithRetry(c reliability.RetryConfig) Option
WithRetry makes the backend survive an unreliable S3 endpoint: each call gets a per-attempt timeout (so a hung request is abandoned instead of stalling for the provider's full timeout) and bounded retries, and GETs are additionally *hedged* — a slow read is re-issued on a fresh connection and the first response wins. Use reliability.LossyEnvironment for noisy networks. The zero config leaves the bare store (the AWS SDK's own retryer still applies).
Source Files