Documentation
¶
Overview ¶
Package objectstore is S3-compatible object storage behind one small interface, for files such as editor images, attachments and rendered PDFs.
The Store interface ¶
Store has eight methods: Put and Get stream (a Put never buffers more than one part, a Get can read a byte range), Stat reads an object's info, Delete is idempotent, List returns one page at a time with an opaque page token and optional delimiter grouping, Copy runs on the server, and PresignGet and PresignPut return Presigned requests that a browser or a plain http.Client can make until they expire.
Backends ¶
- github.com/Bugs5382/go-objectstore/s3store: AWS S3, MinIO, SeaweedFS and other S3-compatible services, on aws-sdk-go-v2. Custom endpoints, path-style or virtual-host addressing, SSE-S3, SSE-KMS and SSE-C, and a SHA-256 or Content-MD5 checksum on every upload request.
- github.com/Bugs5382/go-objectstore/memstore: an in-memory fake for unit tests, with an http.Handler that serves its presigned URLs.
- github.com/Bugs5382/go-objectstore/objectstoretest: the contract suite. Both backends pass it, and a new backend proves itself with it.
This package imports no SDK: code that takes a Store, and its tests on memstore, never pull in aws-sdk-go-v2.
Errors ¶
Every backend reports failures with the same sentinels, wrapped with the operation, the key and the backend's cause: ErrNotFound, ErrAlreadyExists, ErrPermissionDenied, ErrUnavailable, ErrTooLarge and ErrInvalid. Branch with errors.Is. A cancelled or expired context is returned as the context's own error. WithCodes attaches the application's go-apperr codes to the sentinels, since a library should not pick codes for its callers.
Limits and cancellation ¶
PutOptions.MaxSize caps one upload and each backend has a store-wide cap; an upload that passes either fails with ErrTooLarge and leaves nothing behind, and a declared PutOptions.Size over the cap fails before any byte is read. Every call honours its context, including mid-upload and while the caller reads a Get body.
Observability ¶
The package does not log. Observe wraps any Store and reports each call (operation, key, duration, bytes, error) to a function, so the application logs, counts or traces store traffic with its own tools.
Example ¶
package main
import (
"context"
"errors"
"fmt"
"io"
"strings"
"github.com/Bugs5382/go-objectstore"
"github.com/Bugs5382/go-objectstore/memstore"
)
func main() {
ctx := context.Background()
var store objectstore.Store = memstore.New()
_, err := store.Put(ctx, "attachments/42/notes.txt", strings.NewReader("hello"), objectstore.PutOptions{
ContentType: "text/plain",
Metadata: map[string]string{"document": "42"},
MaxSize: 10 << 20,
})
if err != nil {
fmt.Println(err)
return
}
obj, err := store.Get(ctx, "attachments/42/notes.txt", objectstore.GetOptions{})
if err != nil {
fmt.Println(err)
return
}
defer func() { _ = obj.Body.Close() }()
body, _ := io.ReadAll(obj.Body)
fmt.Println(string(body), obj.ContentType, obj.Metadata["document"])
_, err = store.Stat(ctx, "attachments/42/missing.txt")
fmt.Println(errors.Is(err, objectstore.ErrNotFound))
}
Output: hello text/plain 42 true
Index ¶
Examples ¶
Constants ¶
const ( MinPresignExpiry = time.Second MaxPresignExpiry = 7 * 24 * time.Hour )
Presigned URL lifetimes are bounded the way S3 bounds them.
const MaxListLimit = 1000
MaxListLimit is the largest page a List returns.
Variables ¶
var ( // ErrNotFound means the object (or the bucket) does not exist. ErrNotFound = errors.New("objectstore: not found") // ErrAlreadyExists means a Put with IfNotExists found an object there. ErrAlreadyExists = errors.New("objectstore: already exists") // ErrPermissionDenied means the backend refused the credentials or the // request, or a presigned URL was expired or altered. ErrPermissionDenied = errors.New("objectstore: permission denied") // retry later may succeed. ErrUnavailable = errors.New("objectstore: unavailable") // ErrTooLarge means an upload exceeded the per-call or the store limit. ErrTooLarge = errors.New("objectstore: too large") // ErrInvalid means the key, the options or the body length were wrong. ErrInvalid = errors.New("objectstore: invalid request") )
Every Store reports failures with these sentinels, wrapped with the operation, the key and the backend's cause. A cancelled or expired context is reported as the context's own error instead.
Functions ¶
func All ¶
All walks every page of a listing and yields each object, then stops at the first error. Prefixes are not yielded; use List with a Delimiter for those.
Example ¶
package main
import (
"context"
"fmt"
"strings"
"github.com/Bugs5382/go-objectstore"
"github.com/Bugs5382/go-objectstore/memstore"
)
func main() {
ctx := context.Background()
store := memstore.New()
for _, k := range []string{"pdf/1", "pdf/2", "pdf/3"} {
if _, err := store.Put(ctx, k, strings.NewReader("x"), objectstore.PutOptions{}); err != nil {
fmt.Println(err)
return
}
}
for info, err := range objectstore.All(ctx, store, objectstore.ListOptions{Prefix: "pdf/", Limit: 2}) {
if err != nil {
fmt.Println(err)
return
}
fmt.Println(info.Key, info.Size)
}
}
Output: pdf/1 1 pdf/2 1 pdf/3 1
Types ¶
type Codes ¶
Codes are the consumer's go-apperr codes, one per sentinel. A zero field leaves that error uncoded.
type CopyOptions ¶
type CopyOptions struct {
// ReplaceMetadata gives dst the ContentType and Metadata below instead of
// the source's. An empty ContentType keeps the source's.
ReplaceMetadata bool
ContentType string
Metadata map[string]string
}
CopyOptions tune a Copy.
type Event ¶
type Event struct {
Op Op
// Key is the object key; for a List it is the prefix, for a Copy the
// destination.
Key string
Duration time.Duration
// Bytes is the object size for a Put, Stat or Copy, the length of Body
// for a Get, and the number of entries for a List.
Bytes int64
Err error
}
Event reports one finished Store call.
type GetOptions ¶
type GetOptions struct {
// Offset is the first byte to read. An offset at or past the end of a
// non-empty object fails with ErrInvalid.
Offset int64
// Length is the number of bytes to read; zero reads to the end. A range
// that runs past the end is cut short at the end.
Length int64
}
GetOptions select a byte range. The zero value reads the whole object.
type Info ¶
type Info struct {
Key string
Size int64
// ETag is the backend's entity tag without quotes. It changes whenever the
// content changes; it is an MD5 of the content only for single-part
// uploads, so treat it as opaque.
ETag string
LastModified time.Time
ContentType string
// Metadata holds the user metadata, keys in lower case.
Metadata map[string]string
}
Info describes a stored object.
type ListOptions ¶
type ListOptions struct {
// Prefix limits the listing to keys that start with it.
Prefix string
// Delimiter groups keys that share a prefix up to the next delimiter into
// Page.Prefixes, the way folders work. Usually "/" or empty.
Delimiter string
// PageToken is the NextPageToken of the previous page; empty starts at
// the beginning.
PageToken string
// Limit is the page size, objects and prefixes together, from 1 to 1000.
// Zero means 1000.
Limit int
}
ListOptions select one page of a listing.
type Object ¶
type Object struct {
// Info describes the whole object: Size is the full size, not the
// length of Body.
Info
// Offset is the position of the first byte of Body within the object.
Offset int64
// Length is the number of bytes Body yields.
Length int64
Body io.ReadCloser
}
Object is an open object, or the part of it that GetOptions selected.
type Page ¶
type Page struct {
Objects []Info
Prefixes []string
// NextPageToken is empty on the last page.
NextPageToken string
}
Page is one page of a listing. Listed objects carry Key, Size, ETag and LastModified only; call Stat for the content type and metadata.
type PresignPutOptions ¶
type PresignPutOptions struct {
// ContentType is signed into the URL, so the upload must send exactly
// this Content-Type. Empty leaves it unsigned.
ContentType string
}
PresignPutOptions tune a presigned upload.
type Presigned ¶
type Presigned struct {
Method string
URL *url.URL
// Header lists the headers the request must send, exactly as given.
Header http.Header
Expires time.Time
}
Presigned is a signed request a plain HTTP client can make.
type PutOptions ¶
type PutOptions struct {
// ContentType defaults to application/octet-stream.
ContentType string
// Metadata is stored with the object. Keys are case-insensitive and are
// returned in lower case; keys use letters, digits and '-', and values are
// printable ASCII, 2 KiB in total.
Metadata map[string]string
// Size is the exact length of body when it is known, which lets the
// backend reject an oversized upload before reading it. Zero means
// unknown. A body that ends early or runs long fails with ErrInvalid.
Size int64
// MaxSize caps this upload. Zero leaves only the store's own limit.
MaxSize int64
// IfNotExists makes the Put fail with ErrAlreadyExists rather than
// replace an existing object.
IfNotExists bool
// CacheControl and ContentDisposition are stored as the object's
// Cache-Control and Content-Disposition headers.
CacheControl string
ContentDisposition string
}
PutOptions tune a Put.
type Store ¶
type Store interface {
// Put streams body to key. It reads body until io.EOF and never buffers
// more than the backend's part size. An existing object is replaced unless
// PutOptions.IfNotExists is set.
Put(ctx context.Context, key string, body io.Reader, opts PutOptions) (Info, error)
// Get opens the object at key, or the byte range opts selects. The caller
// closes Object.Body.
Get(ctx context.Context, key string, opts GetOptions) (*Object, error)
// Stat returns the object's info without its content.
Stat(ctx context.Context, key string) (Info, error)
// Delete removes the object at key. Deleting a missing key succeeds.
Delete(ctx context.Context, key string) error
// List returns one page of the objects under opts.Prefix, in key order.
List(ctx context.Context, opts ListOptions) (Page, error)
// Copy copies src to dst inside the store, without moving the content
// through the caller.
Copy(ctx context.Context, src, dst string, opts CopyOptions) (Info, error)
// PresignGet returns a URL that downloads key with a plain HTTP GET until
// it expires. The object does not have to exist yet.
PresignGet(ctx context.Context, key string, expiry time.Duration) (Presigned, error)
// PresignPut returns a URL that uploads key with a plain HTTP PUT until it
// expires. The request must carry the returned headers.
PresignPut(ctx context.Context, key string, expiry time.Duration, opts PresignPutOptions) (Presigned, error)
}
Store is a bucket of objects addressed by key. Every implementation is safe for concurrent use, honours context cancellation, and reports failures with the sentinel errors in this package, so callers branch with errors.Is and never on a backend's own error types.
Directories
¶
| Path | Synopsis |
|---|---|
|
internal
|
|
|
check
Package check holds the request validation and error shaping that every backend in this module shares, so they accept and refuse the same inputs.
|
Package check holds the request validation and error shaping that every backend in this module shares, so they accept and refuse the same inputs. |
|
Package memstore is an in-memory objectstore.Store for tests.
|
Package memstore is an in-memory objectstore.Store for tests. |
|
Package objectstoretest is the contract every objectstore.Store passes.
|
Package objectstoretest is the contract every objectstore.Store passes. |
|
Package s3store is the objectstore.Store for AWS S3 and S3-compatible services such as MinIO and SeaweedFS, built on aws-sdk-go-v2.
|
Package s3store is the objectstore.Store for AWS S3 and S3-compatible services such as MinIO and SeaweedFS, built on aws-sdk-go-v2. |