objstore

package module
v0.8.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 1, 2026 License: MIT Imports: 12 Imported by: 0

README

objstore

go.dev reference CI

objstore is object storage with compare-and-swap, for Go.

S3 and Google Cloud Storage can write an object only if it doesn't exist yet, or only if it hasn't changed since you read it. That's compare-and-swap, and it's enough to build a write-ahead log, a leader lease, or a manifest that many processes update, with nothing but a bucket. objstore gives you a Store with those conditional writes, and builds the log, the lease and a read cache on top of it.

Objects are meant to be written once and never changed, except by deletion. The conditional writes cover the few that must change.

Install

go get github.com/axiomhq/objstore

Each provider is its own package, so a binary links only the SDKs it uses.

  • aws/s3 is AWS S3, and anything that speaks the S3 API: Cloudflare R2, MinIO, Ceph, Hetzner.
  • gcp/gcs is Google Cloud Storage.
  • fs is a local directory. Every object is fsynced before it's acknowledged. Unix only, and not Solaris or AIX.

Usage

Open a store
s, err := s3.Open(ctx, s3.Config{
	Endpoint: "https://s3.us-east-1.amazonaws.com",
	Bucket:   "my-bucket",
}, objstore.Config{})
if err != nil {
	return err
}

The other providers look the same.

s, err := gcs.Open(ctx, gcs.Config{Bucket: "my-bucket"}, objstore.Config{})

s := fs.Open("/var/lib/data", "my-bucket", objstore.Config{})

For R2, set Endpoint to https://<account-id>.r2.cloudflarestorage.com and AWS_REGION=auto. Anything else that implements objstore.Backend becomes a Store with objstore.Open.

objstore.Config paces requests (RequestsPerSecond) and bounds the writes in flight (MaxInflightWrites). Both exist because a burst past what a provider serves doesn't fail, it stalls, and takes your heartbeats down with it.

Call s.EnsureBucket(ctx) if the bucket might not exist yet. Then, before the first write, call s.CheckConditionalWrites(ctx). It spends three requests proving the store actually honours If-None-Match. A store that ignores the header would let two log writers both win the same sequence, and you'd rather find that out at startup.

Compare-and-swap
for {
	body, etag, err := s.GetWithETag(ctx, "manifest")
	if err != nil {
		return err
	}
	ok, err := s.PutIfMatch(ctx, "manifest", update(body), etag)
	switch {
	case errors.Is(err, objstore.ErrConflict):
		continue // the store never decided; retry
	case err != nil:
		return err
	case !ok:
		continue // someone else won; read again
	}
	return nil
}

PutIfMatch returns false when someone else got there first. ErrConflict is different: S3 answered 409 because another conditional write on the same key was in flight, and never decided yours. Retry both.

PutIfAbsent is the write-once version: it returns false if the key already existed. GetIfChanged polls a key against the ETag you last saw. On S3 and GCS that's one conditional request; the file store reads the whole file either way. A missing key returns an error wrapping objstore.ErrNotFound.

Heartbeats and log commits shouldn't wait behind bulk uploads. Mark their context with objstore.Urgent(ctx), and their calls skip pacing and the write bound. On the file store they're also written whole, instead of in chunks.

To see where a request spent its time, attach a Timings to its context.

var t objstore.Timings
ctx = objstore.WithTimings(ctx, &t)
// ... calls on s ...
logger.LogAttrs(ctx, slog.LevelInfo, "job", t.Attrs()...)

The file store fills in every call; S3 and GCS fill in only the time spent waiting on the write bound.

Ranged reads, paginated and delimited listing, and batch delete are on pkg.go.dev.

Write-ahead log

Package wal is a write-ahead log on a Store. Concurrent appends are batched into one entry per commit interval, one second by default, and each entry claims the next sequence number with a conditional PUT.

w := wal.NewWriter(s, "log/", 1, func(seq uint64, at time.Time, records []wal.Bytes) {
	// Runs for every durable entry, before its callers are acked.
})
defer w.Close()

if err := w.Append(ctx, []wal.Bytes{wal.Bytes("hello")}); err != nil {
	return err
}

A record is anything with Size and AppendTo; wal.Bytes is one that's already encoded. Read the log back with Walk.

err := wal.Walk(ctx, s, "log/", 0, 0, wal.Decode, func(e wal.Entry[[][]byte]) error {
	for _, page := range e.Pages {
		for _, record := range page {
			fmt.Printf("%d: %s\n", e.Seq, record)
		}
	}
	return nil
})

Nothing is listed. The sequence numbers are the catalog: a walk GETs after+1, after+2, and so on, and stops at the first missing key. That works because a writer advances only on a proven outcome, so a crashed writer leaves the log short, never holey.

Append returning nil means the records are durable and onCommit has run. Most errors mean they weren't written and won't be. Two don't: ErrUnresolved, and the context's own error when ctx ends first. Then the records may be durable, so recover by replaying the log from your checkpoint, not by appending them again. When the store can't keep up, Append fails fast with ErrOverloaded instead of queueing without bound.

A prefix has one live writer, and keeping it that way is your job (a lease, say). Conditional PUT detects a second writer, as ErrLostRace; it doesn't prevent one. The package docs cover every error, and when each one finishes the writer.

Leases

Package lease is a single-holder lease on one key, for leader election or a single writer.

l, err := lease.Acquire(ctx, s, "jobs/leader", lease.OwnerID(), lease.DefaultTTL)
if errors.Is(err, lease.ErrNotOwner) {
	return nil // someone else holds it; try again later
}
if err != nil {
	return err
}
defer l.Release(context.WithoutCancel(ctx))

ctx, cancel := context.WithCancel(ctx)
defer cancel()
l.Start(cancel) // runs once, if we lose the lease

for {
	if err := l.Valid(); err != nil {
		return err
	}
	if err := doGuardedWork(ctx); err != nil {
		return err
	}
}

The lease renews itself every TTL/4. Release hands it over at once, so the next process doesn't wait out the TTL.

Every write carries a fresh nonce. When a PUT lands but its answer is lost, the lease reads the object back, and its own nonce there means it still holds the lease. Nothing is guessed. A holder that's partitioned or crashed blocks takeover until its stored expiry plus TTL/2, and the holder's own deadline runs on its monotonic clock from before the PUT. So it stops acting no later than a taker may start, as long as the fleet shares one TTL and clocks disagree by less than TTL/2.

A lease check can't fence a write that's already in flight. Guard durable writes with their own compare-and-swap, on an ETag or a sequence number, not with a check of the lease.

A long-lived caller that retries should keep one handle, from lease.New, and call its Acquire method in a loop. lease.Shared shares one lease among holders in a process by reference count.

Cache and ranged reads

Package cache puts memory and an optional disk tier in front of a Store. Concurrent misses of one key share one GET.

disk, err := cache.NewDisk(dir, 10<<30) // 10 GiB on disk
if err != nil {
	return err
}
c := cache.New(s, 1<<30, disk, cache.Keys{}) // 1 GiB in memory

b, err := c.FetchWith(ctx, key, func(ctx context.Context) ([]byte, error) {
	return s.Get(ctx, key)
})

Objects are immutable, so the cache never revalidates a key. Cache.Put writes through both tiers, so a process reads back what it wrote from cache. Keys under ns/<name>/ belong to namespace <name>, and InvalidateNamespace drops one namespace from every tier without touching the others.

Package rangeread reads many byte ranges through the cache, coalescing nearby ones into one GET.

r, err := rangeread.New(s, c, rangeread.Config{})
if err != nil {
	return err
}
scope, err := r.FetchRanges(ctx, loads)
if err != nil {
	return err
}
b, ok := cache.Scoped(scope, loads[0].Key)
Encryption

Tell the store which KMS key each object gets.

s = s.WithKMSKeys(func(ctx context.Context, key string) (string, error) {
	tenant, _, _ := strings.Cut(key, "/")
	return keyFor(tenant), nil // "" means the bucket's own policy
})

Then write and read as usual. S3 stores each keyed object with SSE-KMS under that key, GCS with that kmsKeyName. Reads need no key. Rotating a key in KMS needs nothing from you; disabling or deleting one makes its objects fail with an error wrapping objstore.ErrAccessDenied.

The file store ignores keys and encrypts nothing; s.KMS() reports false there. s3.Config's SSE and KMSKeyID still set one policy for the whole bucket.

Testing your code

Package storetest injects faults, and storetest/bucket opens a fresh bucket per test, dropped at cleanup. That's a temp directory by default, or S3 when OBJSTORE_TEST_S3 is set.

func TestManifestSurvivesLostAnswer(t *testing.T) {
	s, f := bucket.NewFaulty(t)

	// The first PutIfMatch on the manifest lands, then reports an error.
	f.Set(storetest.Plan{Op: storetest.OpPutIfMatch, N: 1, Key: "manifest", Mode: storetest.Ambiguous})

	// ... run the code under test against s ...

	if f.Fired() != 1 {
		t.Fatal("fault never fired")
	}
}

Fail errors before the write, Ambiguous writes and then errors, Hang blocks until the call's context ends, and Pause blocks until f.Resume(). f.SetShape adds seeded latency, bandwidth limits and errors to every call, and f.WatchRewrites flags any key rewritten with different bytes. storetest.NewKMS records which key encrypted each object, and can revoke one.

Writing your own Backend? Wrap it with objstore.Open and run storetest.Conformance(t, s) on it. It's the suite every provider here passes. storetest imports no provider, so it links neither the AWS nor the GCS SDK.

Migrating from v0.4

v0.4 v0.5
S3 in package objstore package s3 at objstore/aws/s3; GCS (new) is objstore/gcp/gcs
objstore.New(ctx, endpoint, bucket) s3.Open(ctx, s3.Config{Endpoint: endpoint, Bucket: bucket}, objstore.Config{}); for file://root, fs.Open(root, bucket, objstore.Config{})
Config.Endpoint, Bucket, SSE, KMSKeyID, AllowedEndpoints, RequestTimeout s3.Config
Store.SSE(), objstore.ErrEndpointDenied (*s3.Backend).SSE() (from s3.New), s3.ErrEndpointDenied
s.ConfigureCMEK(p), InstallNamespaceKey, RotateNamespaceKey, CheckNamespaceKey, package kms gone: s = s.WithKMSKeys(fn), and the store encrypts (see Encryption)
Timings.LogAttrs() Timings.Attrs()
l.Release(), ref.Release() l.Release(ctx), ref.Release(ctx)
l.Log = logger l.SetLogger(logger)
l, err = l.Acquire(ctx) err = l.Acquire(ctx)
ByteCache.Generation.Load(), Resident.Generation.Load() GenerationOf(key) (per namespace)
r := rangeread.New(...); r.Cfg r, err := rangeread.New(...); r.Config()
storetest.New(t), storetest.NewFaulty(t) bucket.New(t), bucket.NewFaulty(t) (objstore/storetest/bucket)

Objects that v0.4 sealed client-side stay ciphertext: v0.5 doesn't decrypt them. See CHANGELOG.md for the rest.

Development

go test -race ./...

Without any configuration, every suite runs on a file bucket in a temp directory, and the S3 and GCS clients run against fake servers. Set these to run against real stores; each test is skipped when its variable is unset.

  • OBJSTORE_TEST_S3 is an S3 endpoint, e.g. MinIO at http://localhost:9000. AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_REGION default to MinIO's minioadmin, minioadmin and us-east-1 when unset.
  • OBJSTORE_TEST_R2_ENDPOINT is https://<account-id>.r2.cloudflarestorage.com, with an R2 API token in AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, and AWS_REGION=auto. It runs TestR2 in ./aws/s3.
  • OBJSTORE_TEST_GCS_PROJECT is a GCP project where Application Default Credentials can create buckets. It runs TestConformanceReal in ./gcp/gcs against a fresh objstore-test-<nanos> bucket, dropped afterwards.

MinIO creates the object on a PutIfMatch of a missing key, where AWS S3 and R2 answer 404, so skip that subtest there.

OBJSTORE_TEST_S3=http://localhost:9000 go test -race -skip '/ETagCASMissing' ./...

CI runs gofmt, go mod tidy -diff, vet, staticcheck and the race tests on Linux, the same tests against MinIO, and the tests on macOS. It also cross-vets for windows, darwin, illumos, solaris and aix/ppc64.

License

MIT, see LICENSE.

Documentation

Overview

Package objstore is one object-storage bucket with compare-and-swap: conditional PUT (PutIfAbsent, PutIfMatch) and GET (GetIfChanged), the ETag with the body, paginated listing and batch delete. Each provider is its own package, so a binary links only the SDKs it uses: package aws/s3 (AWS S3, and R2, MinIO, Ceph and Hetzner, which speak the S3 API), package gcp/gcs (Google Cloud Storage) and package fs, a durable local store with chunked write-back. Each builds a Backend and hands it to Open.

Objects are meant to be written once and never mutated except by deletion; the conditional writes cover the few that must change. ListPrefixes returns the child prefixes under a prefix (S3's delimiter), so discovering a thousand top-level names costs one request, not one per object.

Store.WithKMSKeys names a KMS key per object; S3 (SSE-KMS) and GCS (kmsKeyName) encrypt it under that key.

Package wal is a write-ahead log on a Store, package cache a memory and disk cache in front of one, package rangeread a coalescing ranged reader over the cache, package lease a single-holder lease on one key.

Index

Constants

View Source 
const MaxListPage = 1000

MaxListPage is the largest limit ListPage and ListPrefixesPage accept, S3's own page size.

Variables

View Source 
var ErrAccessDenied = errors.New("store: access denied")

ErrAccessDenied is wrapped by every error for a request the store refused (403), or for an object whose KMS key is disabled, deleted or out of reach.

View Source 
var ErrConflict = errors.New("store: conditional write conflict, retry")

ErrConflict is a conditional write whose outcome the backend never decided: another conditional write on the same key was in flight (S3's 409 ConditionalRequestConflict). The object may or may not have been written; the caller retries. It is never a lost race, which is (false, nil).

View Source 
var ErrInvalidKey = errors.New("store: invalid key")

ErrInvalidKey is wrapped by every error for a key a backend refuses to store (the file backend rejects empty elements, "." and "..", and its own reserved file names).

View Source 
var ErrNotFound = errors.New("store: object not found")

ErrNotFound is wrapped by every error for a missing object or bucket.

View Source 
var ErrRange = errors.New("store: invalid object range")

ErrRange is wrapped by every error for a range outside the object, or an offset or length that is not a valid range at all.

Functions

func IsUrgent added in v0.5.0 

func IsUrgent(ctx context.Context) bool

IsUrgent reports whether ctx was marked Urgent. Backends consult it for their own fast paths (the file store writes an urgent object whole).

func KMSKey added in v0.5.0 

func KMSKey(ctx context.Context) string

KMSKey is the KMS key id a write under ctx carries, "" for none. Backends read it; so do backend wrappers (storetest.KMS).

func OpErr added in v0.5.0 

func OpErr(op, key string, err error) error

OpErr attaches the operation and the key to every error leaving a backend. A bare SDK error names neither, so a failure surfaced three layers up — mid-compaction, mid-replay, inside a query's fan-out — says only that storage was unhappy about something. %w keeps ErrNotFound unwrappable by errors.Is. nil stays nil.

func Urgent 

func Urgent(ctx context.Context) context.Context

Urgent returns ctx marked as latency-critical (a lease heartbeat, a log commit, a manifest swap). Requests under it:

  • skip request pacing (Config.RequestsPerSecond);
  • skip the write bound (Config.MaxInflightWrites);
  • on the file backend, are written whole instead of written back in chunks, and so never wait for their own write-back.

func WithKMSKey added in v0.5.0 

func WithKMSKey(ctx context.Context, id string) context.Context

WithKMSKey returns ctx whose writes are encrypted server-side with KMS key id: S3 SSE-KMS with that key id, GCS kmsKeyName. On a Store with a KMSKeyFunc (WithKMSKeys) the function decides; it can read id with KMSKey.

func WithTimings 

func WithTimings(ctx context.Context, t *Timings) context.Context

WithTimings has the store add the calls made under ctx to t.

Types

type Backend 

type Backend interface {
	Put(ctx context.Context, key string, data []byte) error
	PutIfAbsent(ctx context.Context, key string, data []byte) (bool, error)
	Get(ctx context.Context, key string) ([]byte, error)
	GetRange(ctx context.Context, key string, offset, length int64) ([]byte, error)
	GetWithETag(ctx context.Context, key string) ([]byte, string, error)
	GetIfChanged(ctx context.Context, key, etag string) ([]byte, string, bool, error)
	PutIfMatch(ctx context.Context, key string, data []byte, etag string) (bool, error)
	ListPage(ctx context.Context, prefix, after string, limit int) ([]string, string, error)
	ListPrefixesPage(ctx context.Context, prefix, after string, limit int) ([]string, string, error)
	Delete(ctx context.Context, key string) error
	DeleteMany(ctx context.Context, keys ...string) error
	EnsureBucket(ctx context.Context) error
	DropBucket(ctx context.Context) error
}

Backend is the object-store contract every provider package satisfies (fs, s3, gcs); storetest.Conformance runs against each. It may grow.

Every error a backend returns is wrapped by OpErr. A failed condition is (false, nil), a missing object wraps ErrNotFound, a bad range wraps ErrRange. The ok of a conditional write is meaningful only when err == nil, except that a backend may report (true, err) when the object is visible but not proven durable (the file backend, when the directory sync after the publish fails); a retry of the same write settles it. ETags are opaque. Listing is lexical, after is exclusive, next is the last key returned when more remain and "" otherwise. The file backend reserves terminal .lock and .tmp-* files; dot-prefixed directory elements remain valid so documented namespace names work.

type Call 

type Call int

Call is a kind of work the store does for a caller: the file store's system calls, and the write gate every backend shares. 

const (
	CallGate      Call = iota // waiting for a write slot (Config.MaxInflightWrites)
	CallCreate                // creating an object's temp file (and its directory)
	CallWrite                 // write(2) of an object's temp file
	CallWriteBack             // sync_file_range of a chunk (writeBackChunk)
	CallFsync                 // fsync of an object's temp file
	CallLock                  // the key's stripe lock
	CallLink                  // rename(2) or link(2) into place
	CallDirSync               // fsync of a directory after a publish
	CallRead                  // an object read, whole or ranged
	CallList                  // a listing's directory walk
	CallDelete                // unlink(2)

)

func (Call) String added in v0.5.0 

func (c Call) String() string

String is the Call's short name, as used in Attrs ("gate", "fsync", ...).

type Config 

type Config struct {
	// RequestsPerSecond paces every request to the object store; 0 = no
	// pacing. A burst past what the provider serves does not fail, it
	// stalls: at ~900 requests/s against Hetzner Object Storage every
	// in-flight request hung for 13-15 s, the lease renewal among them, and
	// the namespace was fenced. Pacing below that keeps a prefetch burst
	// and the heartbeat both moving.
	RequestsPerSecond float64
	// MaxInflightWrites bounds the object writes and deletes in flight at
	// once, except those marked Urgent (lease heartbeats, log commits,
	// manifest heads); 0 = 16, negative = unbounded. A background job
	// publishes thousands of objects in a burst; without a bound they queue
	// ahead of the write path's own small requests on the store.
	MaxInflightWrites int
}

Config is what Open adds around any backend. Provider-specific settings (endpoint, credentials, bucket-wide server-side encryption) live in each provider package's own Config.

type KMSKeyFunc added in v0.5.0 

type KMSKeyFunc func(ctx context.Context, key string) (string, error)

KMSKeyFunc names the KMS key an object is written with; "" writes it under the bucket's own encryption policy. It runs on every Put, PutIfAbsent and PutIfMatch, before the write takes a slot.

type Store 

type Store struct {
	// contains filtered or unexported fields
}

Store is one bucket of write-once objects with the OCC primitives a log and a manifest build on (PutIfAbsent to append, PutIfMatch to swap).

func Open added in v0.5.0 

func Open(b Backend, cfg Config) *Store

Open returns a Store over b with cfg's pacing, write bound and KMS keys. The provider packages (fs, s3, gcs) each have an Open that builds the backend and calls this.

func (*Store) CheckConditionalWrites added in v0.5.0 

func (s *Store) CheckConditionalWrites(ctx context.Context) error

CheckConditionalWrites proves the backend honours If-None-Match: a first PutIfAbsent of a fresh _probe/ifnonematch/ key must create it and a second must not. A store that ignores the header would let two log writers both win one sequence. Three requests; call it once before writing.

func (*Store) Close added in v0.8.0 

func (s *Store) Close() error

Close releases the provider's client, if its backend has one to release (gcs.Backend does); otherwise it does nothing. Stores from WithBackend share the backend: close one of them, once, when all are done.

func (*Store) Delete 

func (s *Store) Delete(ctx context.Context, key string) error

Delete removes key. A missing key is not an error.

func (*Store) DeleteMany 

func (s *Store) DeleteMany(ctx context.Context, keys ...string) error

DeleteMany removes keys. Missing keys are not an error. Empty input is a no-op.

func (*Store) DropBucket 

func (s *Store) DropBucket(ctx context.Context) error

DropBucket deletes every object and then the bucket itself (test teardown; the harness's fresh bucket per test must not outlive the test). A bucket that is already gone is not an error.

func (*Store) EnsureBucket 

func (s *Store) EnsureBucket(ctx context.Context) error

EnsureBucket creates the bucket if missing (dev/test convenience).

func (*Store) Get 

func (s *Store) Get(ctx context.Context, key string) ([]byte, error)

Get returns the whole object at key; a missing object wraps ErrNotFound.

func (*Store) GetIfChanged 

func (s *Store) GetIfChanged(ctx context.Context, key, etag string) ([]byte, string, bool, error)

GetIfChanged conditionally reads one revision. Empty etag is unconditional. An unchanged object returns nil bytes, its ETag, true, nil. A changed object returns its complete bytes and ETag with false. Absence is ErrNotFound; failures never report unchanged. ETags are opaque backend values.

func (*Store) GetRange 

func (s *Store) GetRange(ctx context.Context, key string, offset, length int64) ([]byte, error)

GetRange reads exactly length bytes starting at offset. Short/out-of-bounds ranges fail instead of returning a plausible partial index block.

func (*Store) GetWithETag 

func (s *Store) GetWithETag(ctx context.Context, key string) ([]byte, string, error)

GetWithETag returns the object and its ETag for conditional replacement.

func (*Store) ID added in v0.6.0 

func (s *Store) ID() string

ID names the bucket: Stores over the same bucket, opened apart or seen through WithBackend, share it. A backend without an ID method gets a random one per Open.

func (*Store) KMS added in v0.5.0 

func (s *Store) KMS() bool

KMS reports whether the backend applies per-object KMS keys (WithKMSKeys). The file backend does not: it ignores them and encrypts nothing.

func (*Store) List 

func (s *Store) List(ctx context.Context, prefix string) ([]string, error)

List returns all keys under prefix, lexically sorted.

func (*Store) ListPage 

func (s *Store) ListPage(ctx context.Context, prefix, after string, limit int) ([]string, string, error)

ListPage returns at most limit (1 to MaxListPage) keys under prefix, lexically after after. next is the last returned key when another page remains, otherwise empty.

func (*Store) ListPrefixes 

func (s *Store) ListPrefixes(ctx context.Context, prefix string) ([]string, error)

ListPrefixes returns the immediate child prefixes under prefix — S3's CommonPrefixes with Delimiter "/", each ending in "/", lexically sorted. Discovering top-level names costs one request per page of names instead of one key per object in the bucket. A prefix appears only if at least one object lives under it, on every backend.

func (*Store) ListPrefixesPage 

func (s *Store) ListPrefixesPage(ctx context.Context, prefix, after string, limit int) ([]string, string, error)

ListPrefixesPage is one page of ListPrefixes: at most limit (1 to MaxListPage) prefixes lexically after after, and the cursor as in ListPage.

func (*Store) Put 

func (s *Store) Put(ctx context.Context, key string, data []byte) error

Put writes key unconditionally, replacing any object there.

func (*Store) PutIfAbsent 

func (s *Store) PutIfAbsent(ctx context.Context, key string, data []byte) (bool, error)

PutIfAbsent writes key only if it does not already exist. Returns false when the key existed. This is the OCC primitive a write-ahead log builds on. See Backend for the one case where ok is true alongside an error.

func (*Store) PutIfMatch 

func (s *Store) PutIfMatch(ctx context.Context, key string, data []byte, etag string) (bool, error)

PutIfMatch replaces key only if its current ETag equals etag — compare-and- swap, the manifest-swap primitive. (false, nil) = precondition failed: the object changed under us, or no longer exists. See Backend for the one case where ok is true alongside an error.

func (*Store) WithBackend 

func (s *Store) WithBackend(wrap func(Backend) Backend) *Store

WithBackend returns a Store over wrap(s's backend) with s's KMS keys and write bound: the same bucket seen through a wrapper.

func (*Store) WithKMSKeys added in v0.5.0 

func (s *Store) WithKMSKeys(fn KMSKeyFunc) *Store

WithKMSKeys returns s with every write asking fn for its object's KMS key. fn decides: its answer replaces any WithKMSKey on the write's context, which fn can read with KMSKey. The store encrypts and decrypts: reads need no key, and an object whose key is disabled or revoked fails with ErrAccessDenied. The file backend ignores keys (KMS reports false).

type Timings 

type Timings struct {
	// contains filtered or unexported fields
}

Timings sums the wall time the store spent in each Call for the calls made under one context (WithTimings), and how many there were: a background job can log its own. Concurrent calls add up, so the sum can pass the span's wall time. The file store fills every Call; S3 only the gate.

func TimingsOf added in v0.5.0 

func TimingsOf(ctx context.Context) *Timings

TimingsOf returns the Timings ctx carries, or nil. For backends.

func (*Timings) Attrs added in v0.5.0 

func (t *Timings) Attrs() []slog.Attr

Attrs is t as slog attributes: sys_<call>_ms and sys_<call>_n per Call. A nil t has none.

func (*Timings) Since added in v0.5.0 

func (t *Timings) Since(op Call, start time.Time)

Since adds the time from start to op; a nil t records nothing. For backends.

Source Files

View all Source files

Directories

Path Synopsis
aws
s3
Package s3 is the S3 backend for objstore.
 Package s3 is the S3 backend for objstore.
Package cache is a process's read path to immutable objects in a Store: a striped in-memory byte cache (ByteCache) with decoded values riding on their bytes, an optional disposable disk tier with 4 KiB block checksums (Disk), one gate bounding concurrent store GETs, and one singleflight across concurrent misses of a key (Cache).
 Package cache is a process's read path to immutable objects in a Store: a striped in-memory byte cache (ByteCache) with decoded values riding on their bytes, an optional disposable disk tier with 4 KiB block checksums (Disk), one gate bounding concurrent store GETs, and one singleflight across concurrent misses of a key (Cache).
Package fs is the file backend for objstore: one bucket as a directory of write-once files.
 Package fs is the file backend for objstore: one bucket as a directory of write-once files.
gcp
gcs
Package gcs is the Google Cloud Storage backend for objstore.
 Package gcs is the Google Cloud Storage backend for objstore.
Package lease is a single-holder lease on one object in a Store: leader election, or a single writer, on S3 or any store with conditional PUT.
 Package lease is a single-holder lease on one object in a Store: leader election, or a single writer, on S3 or any store with conditional PUT.
Package rangeread reads immutable objects by byte range through a cache.Cache: Plan unions and coalesces the extents of one read stage under gap, extra-byte and range-size limits, and Reader.FetchRanges runs the plan on bounded workers, publishing each requested child in the returned context (cache.Scoped) and caching it by its logical key.
 Package rangeread reads immutable objects by byte range through a cache.Cache: Plan unions and coalesces the extents of one read stage under gap, extra-byte and range-size limits, and Reader.FetchRanges runs the plan on bounded workers, publishing each requested child in the returned context (cache.Scoped) and caching it by its logical key.
Package storetest provides provider-free test helpers: a fault-injecting, metering wrapper around a Store, and Conformance, the suite every backend passes.
 Package storetest provides provider-free test helpers: a fault-injecting, metering wrapper around a Store, and Conformance, the suite every backend passes.
bucket
Package bucket opens a fresh test bucket on the file backend or, with OBJSTORE_TEST_S3, on S3.
 Package bucket opens a fresh test bucket on the file backend or, with OBJSTORE_TEST_S3, on S3.
Package wal is a write-ahead log in object storage: ordered, immutable pages sequenced by conditional PUT.
 Package wal is a write-ahead log in object storage: ordered, immutable pages sequenced by conditional PUT.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.