objectstore

package module
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 4, 2026 License: MIT Imports: 8 Imported by: 0

README

go-objectstore 🪣

🧭 S3-compatible object storage for Go: a small Store interface with streaming put and get, pagination, presigned URLs and copy, an S3 backend tested on MinIO, and an in-memory fake.

Editor images, attachments and rendered PDFs all end up in object storage: AWS S3 in one place, MinIO or SeaweedFS in another. go-objectstore puts them behind one eight-method interface, so application code never touches an SDK, its unit tests run on an in-memory fake, and the fake is held to the same contract as the real backend.

✨ Highlights

  • 🌊 Streaming both ways: a Put buffers one part at a time whatever the object size, and a Get can read a byte range.
  • 🧩 No SDK in the root: objectstore depends only on go-apperr; aws-sdk-go-v2 comes in only when you import s3store.
  • 🧪 One contract, two backends: objectstoretest.Run is the suite. s3store passes it on real MinIO (path-style, virtual-host and Content-MD5 runs) and memstore passes it in memory.
  • 🔗 Presigned URLs that work: PresignGet and PresignPut return the method, URL, required headers and expiry, ready for a browser or a plain http.Client, the fake included.
  • 🔐 Integrity and encryption: a SHA-256 (or Content-MD5) checksum on every upload request, and SSE-S3, SSE-KMS or SSE-C on every object.
  • 🚦 Errors you can branch on: ErrNotFound, ErrAlreadyExists, ErrPermissionDenied, ErrUnavailable, ErrTooLarge and ErrInvalid from every backend, with WithCodes for your go-apperr codes.

📦 Install

go get github.com/Bugs5382/go-objectstore

🚀 Usage

Build an S3 store once per bucket and pass it around as an objectstore.Store:

store, err := s3store.New(s3store.Config{
    Endpoint:        "http://seaweedfs-s3.example.org:8333", // empty for AWS S3
    Region:          "us-east-1",
    Bucket:          "documents",
    AccessKeyID:     os.Getenv("S3_ACCESS_KEY"),
    SecretAccessKey: os.Getenv("S3_SECRET_KEY"),
    PathStyle:       true, // MinIO and SeaweedFS; AWS wants virtual-host style
    Encryption:      s3store.Encryption{Mode: s3store.SSES3},
    MaxObjectSize:   100 << 20,
})

info, err := store.Put(ctx, "pdf/2026/report.pdf", file, objectstore.PutOptions{
    ContentType: "application/pdf",
    Metadata:    map[string]string{"document": "42"},
    IfNotExists: true,
})

obj, err := store.Get(ctx, "pdf/2026/report.pdf", objectstore.GetOptions{Offset: 0, Length: 1024})
defer obj.Body.Close()

link, err := store.PresignGet(ctx, "pdf/2026/report.pdf", 15*time.Minute)
// hand link.URL to the browser

On AWS with IAM roles or IRSA, load the SDK config yourself and pass Credentials: awsCfg.Credentials instead of static keys.

Method What it does
Put(ctx, key, body, PutOptions) Streams a body. Content type, metadata, Size, MaxSize, IfNotExists, cache and disposition headers.
Get(ctx, key, GetOptions) Opens the object or a range; Object carries the full size, the offset and the body length.
Stat(ctx, key) Size, ETag, last modified, content type and metadata.
Delete(ctx, key) Removes the object; a missing key is not an error.
List(ctx, ListOptions) One page: prefix, delimiter, page token, limit (up to 1000).
Copy(ctx, src, dst, CopyOptions) Server-side copy, keeping or replacing the metadata.
PresignGet(ctx, key, expiry) A signed download, 1 second to 7 days.
PresignPut(ctx, key, expiry, PresignPutOptions) A signed upload, optionally pinned to one content type.

objectstore.All(ctx, store, opts) walks every page as an iterator.

🧱 Design

  • 📐 Small interface, rich options: eight methods, each with an options struct, so new knobs do not break implementations.
  • 📏 Limits before bytes: a declared Size over the limit fails before the body is read; an undeclared one fails as soon as it passes the limit. A body shorter or longer than its declared Size fails with ErrInvalid. Nothing is stored in either case.
  • 🧹 No orphaned parts: a failed or cancelled multipart upload is aborted, even after the caller's context is gone.
  • 🛑 Cancellation everywhere: every call checks its context, an upload stops mid-stream, and a Get body returns context.Canceled once its context is done.
  • 🧾 Checksums by the store: s3store computes the checksum of each request itself and turns off the SDK's default CRC32 trailers, which some S3-compatible services reject. Full-object reads turn on the SDK's response checksum validation.
  • 📋 Copy pins the source: the copy sends the source's ETag, so a source replaced mid-copy fails the copy rather than mixing versions. Objects over 5 GiB are copied part by part.
  • 🔏 Presigning and SSE-C don't mix: an SSE-C request must carry the customer key, so presigning fails with ErrInvalid on an SSE-C store. SSE-S3 and SSE-KMS presigned uploads return the encryption headers the upload must send.
  • 🏷️ Codes are the caller's: the library never picks go-apperr codes. Wrap a store with objectstore.WithCodes(store, objectstore.Codes{NotFound: 1404, ...}) to attach yours.
  • 🔭 No logging inside: objectstore.Observe(store, fn) reports every call (operation, key, duration, bytes, error) for your logger, metrics or tracer.
Keys and metadata

Keys are 1 to 1024 bytes of valid UTF-8. Keep path segments under 255 bytes and don't store an object and a prefix with the same name (a and a/b): MinIO's file-system layout refuses both, though AWS allows them. Metadata keys use letters, digits and -, come back in lower case, and values are printable ASCII, 2 KiB in total. Listed objects carry key, size, ETag and last modified only; call Stat for the rest.

🧪 Testing your code

Use memstore in unit tests. Serve its handler when the code under test follows presigned URLs:

var store *memstore.Store
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    store.Handler().ServeHTTP(w, r)
}))
defer srv.Close()
store = memstore.New(memstore.WithBaseURL(srv.URL+"/bucket"), memstore.WithMaxObjectSize(10<<20))

Writing another backend? Run the contract against it:

func TestContract(t *testing.T) {
    objectstoretest.Run(t, func(t *testing.T) objectstore.Store { return newStore(t) })
}

🛠 Develop

task build             # go build ./...
task test              # go test ./... (no Docker needed)
task test:integration  # also runs s3store against MinIO in Docker (testcontainers)
task lint              # gofmt check + golangci-lint + yamllint
task license           # check MIT headers (golic)

The integration tests use a community MinIO build (pgsty/minio), since upstream MinIO images are no longer published; set OBJECTSTORE_MINIO_IMAGE to use another. Each run makes its own throwaway credentials, and the SSE-C tests run MinIO over TLS with a certificate made for the run.

⚖️ License

MIT (c) 2026 Shane

Documentation

Overview

Package objectstore is S3-compatible object storage behind one small interface, for files such as editor images, attachments and rendered PDFs.

The Store interface

Store has eight methods: Put and Get stream (a Put never buffers more than one part, a Get can read a byte range), Stat reads an object's info, Delete is idempotent, List returns one page at a time with an opaque page token and optional delimiter grouping, Copy runs on the server, and PresignGet and PresignPut return Presigned requests that a browser or a plain http.Client can make until they expire.

Backends

This package imports no SDK: code that takes a Store, and its tests on memstore, never pull in aws-sdk-go-v2.

Errors

Every backend reports failures with the same sentinels, wrapped with the operation, the key and the backend's cause: ErrNotFound, ErrAlreadyExists, ErrPermissionDenied, ErrUnavailable, ErrTooLarge and ErrInvalid. Branch with errors.Is. A cancelled or expired context is returned as the context's own error. WithCodes attaches the application's go-apperr codes to the sentinels, since a library should not pick codes for its callers.

Limits and cancellation

PutOptions.MaxSize caps one upload and each backend has a store-wide cap; an upload that passes either fails with ErrTooLarge and leaves nothing behind, and a declared PutOptions.Size over the cap fails before any byte is read. Every call honours its context, including mid-upload and while the caller reads a Get body.

Observability

The package does not log. Observe wraps any Store and reports each call (operation, key, duration, bytes, error) to a function, so the application logs, counts or traces store traffic with its own tools.

Example
package main

import (
	"context"
	"errors"
	"fmt"
	"io"
	"strings"

	"github.com/Bugs5382/go-objectstore"
	"github.com/Bugs5382/go-objectstore/memstore"
)

func main() {
	ctx := context.Background()
	var store objectstore.Store = memstore.New()

	_, err := store.Put(ctx, "attachments/42/notes.txt", strings.NewReader("hello"), objectstore.PutOptions{
		ContentType: "text/plain",
		Metadata:    map[string]string{"document": "42"},
		MaxSize:     10 << 20,
	})
	if err != nil {
		fmt.Println(err)
		return
	}

	obj, err := store.Get(ctx, "attachments/42/notes.txt", objectstore.GetOptions{})
	if err != nil {
		fmt.Println(err)
		return
	}
	defer func() { _ = obj.Body.Close() }()
	body, _ := io.ReadAll(obj.Body)
	fmt.Println(string(body), obj.ContentType, obj.Metadata["document"])

	_, err = store.Stat(ctx, "attachments/42/missing.txt")
	fmt.Println(errors.Is(err, objectstore.ErrNotFound))
}
Output:
hello text/plain 42
true

Index

Examples

Constants

View Source
const (
	MinPresignExpiry = time.Second
	MaxPresignExpiry = 7 * 24 * time.Hour
)

Presigned URL lifetimes are bounded the way S3 bounds them.

View Source
const MaxListLimit = 1000

MaxListLimit is the largest page a List returns.

Variables

View Source
var (
	// ErrNotFound means the object (or the bucket) does not exist.
	ErrNotFound = errors.New("objectstore: not found")
	// ErrAlreadyExists means a Put with IfNotExists found an object there.
	ErrAlreadyExists = errors.New("objectstore: already exists")
	// ErrPermissionDenied means the backend refused the credentials or the
	// request, or a presigned URL was expired or altered.
	ErrPermissionDenied = errors.New("objectstore: permission denied")
	// ErrUnavailable means the backend could not be reached or failed; a
	// retry later may succeed.
	ErrUnavailable = errors.New("objectstore: unavailable")
	// ErrTooLarge means an upload exceeded the per-call or the store limit.
	ErrTooLarge = errors.New("objectstore: too large")
	// ErrInvalid means the key, the options or the body length were wrong.
	ErrInvalid = errors.New("objectstore: invalid request")
)

Every Store reports failures with these sentinels, wrapped with the operation, the key and the backend's cause. A cancelled or expired context is reported as the context's own error instead.

Functions

func All

func All(ctx context.Context, s Store, opts ListOptions) iter.Seq2[Info, error]

All walks every page of a listing and yields each object, then stops at the first error. Prefixes are not yielded; use List with a Delimiter for those.

Example
package main

import (
	"context"
	"fmt"
	"strings"

	"github.com/Bugs5382/go-objectstore"
	"github.com/Bugs5382/go-objectstore/memstore"
)

func main() {
	ctx := context.Background()
	store := memstore.New()
	for _, k := range []string{"pdf/1", "pdf/2", "pdf/3"} {
		if _, err := store.Put(ctx, k, strings.NewReader("x"), objectstore.PutOptions{}); err != nil {
			fmt.Println(err)
			return
		}
	}
	for info, err := range objectstore.All(ctx, store, objectstore.ListOptions{Prefix: "pdf/", Limit: 2}) {
		if err != nil {
			fmt.Println(err)
			return
		}
		fmt.Println(info.Key, info.Size)
	}
}
Output:
pdf/1 1
pdf/2 1
pdf/3 1

Types

type Codes

type Codes struct {
	NotFound         int
	AlreadyExists    int
	PermissionDenied int
	Unavailable      int
	TooLarge         int
	Invalid          int
}

Codes are the consumer's go-apperr codes, one per sentinel. A zero field leaves that error uncoded.

type CopyOptions

type CopyOptions struct {
	// ReplaceMetadata gives dst the ContentType and Metadata below instead of
	// the source's. An empty ContentType keeps the source's.
	ReplaceMetadata bool
	ContentType     string
	Metadata        map[string]string
}

CopyOptions tune a Copy.

type Event

type Event struct {
	Op Op
	// Key is the object key; for a List it is the prefix, for a Copy the
	// destination.
	Key      string
	Duration time.Duration
	// Bytes is the object size for a Put, Stat or Copy, the length of Body
	// for a Get, and the number of entries for a List.
	Bytes int64
	Err   error
}

Event reports one finished Store call.

type GetOptions

type GetOptions struct {
	// Offset is the first byte to read. An offset at or past the end of a
	// non-empty object fails with ErrInvalid.
	Offset int64
	// Length is the number of bytes to read; zero reads to the end. A range
	// that runs past the end is cut short at the end.
	Length int64
}

GetOptions select a byte range. The zero value reads the whole object.

type Info

type Info struct {
	Key  string
	Size int64
	// ETag is the backend's entity tag without quotes. It changes whenever the
	// content changes; it is an MD5 of the content only for single-part
	// uploads, so treat it as opaque.
	ETag         string
	LastModified time.Time
	ContentType  string
	// Metadata holds the user metadata, keys in lower case.
	Metadata map[string]string
}

Info describes a stored object.

type ListOptions

type ListOptions struct {
	// Prefix limits the listing to keys that start with it.
	Prefix string
	// Delimiter groups keys that share a prefix up to the next delimiter into
	// Page.Prefixes, the way folders work. Usually "/" or empty.
	Delimiter string
	// PageToken is the NextPageToken of the previous page; empty starts at
	// the beginning.
	PageToken string
	// Limit is the page size, objects and prefixes together, from 1 to 1000.
	// Zero means 1000.
	Limit int
}

ListOptions select one page of a listing.

type Object

type Object struct {
	// Info describes the whole object: Size is the full size, not the
	// length of Body.
	Info
	// Offset is the position of the first byte of Body within the object.
	Offset int64
	// Length is the number of bytes Body yields.
	Length int64
	Body   io.ReadCloser
}

Object is an open object, or the part of it that GetOptions selected.

type Op

type Op string

Op names a Store operation in an Event.

const (
	OpPut        Op = "put"
	OpGet        Op = "get"
	OpStat       Op = "stat"
	OpDelete     Op = "delete"
	OpList       Op = "list"
	OpCopy       Op = "copy"
	OpPresignGet Op = "presign_get"
	OpPresignPut Op = "presign_put"
)

The operations an Event reports.

type Page

type Page struct {
	Objects  []Info
	Prefixes []string
	// NextPageToken is empty on the last page.
	NextPageToken string
}

Page is one page of a listing. Listed objects carry Key, Size, ETag and LastModified only; call Stat for the content type and metadata.

type PresignPutOptions

type PresignPutOptions struct {
	// ContentType is signed into the URL, so the upload must send exactly
	// this Content-Type. Empty leaves it unsigned.
	ContentType string
}

PresignPutOptions tune a presigned upload.

type Presigned

type Presigned struct {
	Method string
	URL    *url.URL
	// Header lists the headers the request must send, exactly as given.
	Header  http.Header
	Expires time.Time
}

Presigned is a signed request a plain HTTP client can make.

type PutOptions

type PutOptions struct {
	// ContentType defaults to application/octet-stream.
	ContentType string
	// Metadata is stored with the object. Keys are case-insensitive and are
	// returned in lower case; keys use letters, digits and '-', and values are
	// printable ASCII, 2 KiB in total.
	Metadata map[string]string
	// Size is the exact length of body when it is known, which lets the
	// backend reject an oversized upload before reading it. Zero means
	// unknown. A body that ends early or runs long fails with ErrInvalid.
	Size int64
	// MaxSize caps this upload. Zero leaves only the store's own limit.
	MaxSize int64
	// IfNotExists makes the Put fail with ErrAlreadyExists rather than
	// replace an existing object.
	IfNotExists bool
	// CacheControl and ContentDisposition are stored as the object's
	// Cache-Control and Content-Disposition headers.
	CacheControl       string
	ContentDisposition string
}

PutOptions tune a Put.

type Store

type Store interface {
	// Put streams body to key. It reads body until io.EOF and never buffers
	// more than the backend's part size. An existing object is replaced unless
	// PutOptions.IfNotExists is set.
	Put(ctx context.Context, key string, body io.Reader, opts PutOptions) (Info, error)
	// Get opens the object at key, or the byte range opts selects. The caller
	// closes Object.Body.
	Get(ctx context.Context, key string, opts GetOptions) (*Object, error)
	// Stat returns the object's info without its content.
	Stat(ctx context.Context, key string) (Info, error)
	// Delete removes the object at key. Deleting a missing key succeeds.
	Delete(ctx context.Context, key string) error
	// List returns one page of the objects under opts.Prefix, in key order.
	List(ctx context.Context, opts ListOptions) (Page, error)
	// Copy copies src to dst inside the store, without moving the content
	// through the caller.
	Copy(ctx context.Context, src, dst string, opts CopyOptions) (Info, error)
	// PresignGet returns a URL that downloads key with a plain HTTP GET until
	// it expires. The object does not have to exist yet.
	PresignGet(ctx context.Context, key string, expiry time.Duration) (Presigned, error)
	// PresignPut returns a URL that uploads key with a plain HTTP PUT until it
	// expires. The request must carry the returned headers.
	PresignPut(ctx context.Context, key string, expiry time.Duration, opts PresignPutOptions) (Presigned, error)
}

Store is a bucket of objects addressed by key. Every implementation is safe for concurrent use, honours context cancellation, and reports failures with the sentinel errors in this package, so callers branch with errors.Is and never on a backend's own error types.

func Observe

func Observe(s Store, fn func(Event)) Store

Observe wraps s and calls fn after every call with what happened, so the consumer logs, counts or traces store traffic with its own tools. fn runs on the calling goroutine and must not block.

func WithCodes

func WithCodes(s Store, c Codes) Store

WithCodes wraps s so every error it returns carries the matching code from c through apperr.Coded. errors.Is still finds the sentinel and the cause.

Directories

Path Synopsis
internal
check
Package check holds the request validation and error shaping that every backend in this module shares, so they accept and refuse the same inputs.
Package check holds the request validation and error shaping that every backend in this module shares, so they accept and refuse the same inputs.
Package memstore is an in-memory objectstore.Store for tests.
Package memstore is an in-memory objectstore.Store for tests.
Package objectstoretest is the contract every objectstore.Store passes.
Package objectstoretest is the contract every objectstore.Store passes.
Package s3store is the objectstore.Store for AWS S3 and S3-compatible services such as MinIO and SeaweedFS, built on aws-sdk-go-v2.
Package s3store is the objectstore.Store for AWS S3 and S3-compatible services such as MinIO and SeaweedFS, built on aws-sdk-go-v2.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL