objfs

package module
v0.0.0-...-5d2ee38 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 14, 2026 License: MIT Imports: 13 Imported by: 0

README

objfs

A lightweight object-storage abstraction for Go that is also an io/fs.FS.

Inspired by Thanos' objstore, objfs exposes one small Bucket interface that every backend implements. Because Bucket embeds io/fs.FS, a bucket drops straight into the standard library — fs.WalkDir, fs.ReadFile, http.FileServerFS, template.ParseFS — while still offering context-aware upload, delete, listing, and presigned URLs.

type Bucket interface {
    fs.FS // Open(name) — stdlib compatibility (uses context.Background)

    Get(ctx, name) (io.ReadCloser, error)
    GetRange(ctx, name, off, length) (io.ReadCloser, error)
    Upload(ctx, name, r, ...UploadOption) error
    Delete(ctx, name) error
    Stat(ctx, name) (Attributes, error)
    Exists(ctx, name) (bool, error)
    List(ctx, prefix, func(Attributes) error) error
    io.Closer
}

Presigned URLs are an optional capability — backends that support them implement Presigner; detect it with a type assertion or the PresignedGet / PresignedPut helpers.

io/fs extension interfaces

Every backend goes beyond the bare fs.FS:

  • fs.ReadFileFS — bucket.ReadFile(name) reads an object in one shot (a single GET, skipping the Stat that Open performs); fs.ReadFile routes through it automatically.
  • fs.SubFS — objfs.Sub(bucket, "tenants/acme") returns a full prefix-scoped Bucket (Upload/Delete/List/presign all scoped, List names stripped), richer than the Open-only view fs.Sub would give.
  • fs.ReadDirFS — bucket.ReadDir(dir) lists the immediate children of a directory, synthesising subdirectories from the / delimiter. Cloud backends use native delimited listing (S3 CommonPrefixes, GCS Delimiter, Azure hierarchical listing) so fs.WalkDir descends a tree level-by-level instead of scanning every key. objfs.ReadDir(ctx, b, dir) provides the same over any Bucket.
  • io.ReaderAt + io.Seeker — the file returned by Open supports random access, backed by range reads. This makes a remote object usable directly with archive/zip, range-serving, and other random-access readers without buffering it whole. ReadAt is safe for concurrent use.
// Read a zip stored in any bucket without downloading it whole:
f, _ := bucket.Open("archive.zip")
at, _ := bucket.Stat(ctx, "archive.zip")
zr, _ := zip.NewReader(f.(io.ReaderAt), at.Size)

Lightweight by design

The core module (github.com/armadakv/objfs) depends only on the standard library and ships the local-filesystem backend. Each cloud backend is a separate module under its own directory, so its heavy SDK is only pulled into your build when you import it:

Module Import Backend Presign
github.com/armadakv/objfs core + NewLocal local disk —
github.com/armadakv/objfs/s3 objfs/s3 Amazon S3 / S3-compatible ✅
github.com/armadakv/objfs/gcs objfs/gcs Google Cloud Storage ✅
github.com/armadakv/objfs/azblob objfs/azblob Azure Blob Storage ✅ 
go get github.com/armadakv/objfs          # core, stdlib only
go get github.com/armadakv/objfs/s3       # adds the AWS SDK

Usage

ctx := context.Background()

bucket, err := objfs.NewLocal("/var/data") // or s3.Open(...), gcs.Open(...), azblob.OpenWithSharedKey(...)
if err != nil {
    return err
}
defer bucket.Close()

// Upload.
if err := bucket.Upload(ctx, "reports/q3.pdf", r, objfs.WithContentType("application/pdf")); err != nil {
    return err
}

// Read back through the standard library — Bucket is an io/fs.FS.
data, err := fs.ReadFile(bucket, "reports/q3.pdf")
if err != nil {
    return err
}
fmt.Printf("read %d bytes\n", len(data))

// Stream a byte range ([off, off+length)).
rc, err := bucket.GetRange(ctx, "reports/q3.pdf", 0, 1024)
if err != nil {
    return err
}
defer rc.Close()

// List by prefix.
if err := bucket.List(ctx, "reports/", func(a objfs.Attributes) error {
    fmt.Println(a.Name, a.Size)
    return nil // return objfs.SkipAll to stop early
}); err != nil {
    return err
}

// Presigned URL (cloud backends only).
url, err := objfs.PresignedGet(ctx, bucket, "reports/q3.pdf", 15*time.Minute)
if errors.Is(err, objfs.ErrUnsupported) {
    // local backend and any backend without presigning support
} else if err != nil {
    return err
} else {
    fmt.Println(url)
}
Backend constructors
// S3 (and S3-compatible: MinIO, Cloudflare R2, ...)
import objfss3 "github.com/armadakv/objfs/s3"
s3Bucket, _ := objfss3.Open(ctx, "my-bucket")              // default AWS config chain
s3Bucket = objfss3.New(existingS3Client, "my-bucket")      // bring your own *s3.Client

// Google Cloud Storage
import objfsgcs "github.com/armadakv/objfs/gcs"
gcsBucket, _ := objfsgcs.Open(ctx, "my-bucket")            // Application Default Credentials

// Azure Blob Storage (shared key enables SAS presigning)
import objfsaz "github.com/armadakv/objfs/azblob"
azBucket, _ := objfsaz.OpenWithSharedKey(account, key, "my-container")
azBucket = objfsaz.New(existingAzureClient, "my-container") // BYO *azblob.Client

Semantics

  • Names are slash-separated keys satisfying fs.ValidPath (no leading slash, no ./..). This keeps every backend interchangeable with the io/fs helpers.
  • Not-found errors wrap fs.ErrNotExist, so errors.Is(err, fs.ErrNotExist) works uniformly across backends.
  • Delete of a missing object is a no-op (not an error), matching cloud object-store semantics.
  • The local backend confines all access to its root via os.Root — traversal via .., absolute paths, or symlinks is rejected.

Development

Each module is independent. The cloud modules use a replace directive pointing at ../ so they build against your local core checkout:

go test ./...            # core + local backend
(cd s3 && go build ./...)
(cd gcs && go build ./...)
(cd azblob && go build ./...)

The local backend is verified against the standard library's own testing/fstest.TestFS conformance checker.

Testing

A single conformance suite (objfstest.RunBucket) exercises every backend identically — round-trip, ranges, ReadFile, List, ReadDir, Sub, io.ReaderAt (via a real archive/zip read), not-found semantics, delete, and (where supported) a presigned GET fetched over HTTP.

  • Unit tests (no Docker) run the suite against the local backend: 
    go test ./...
  • Integration tests run the same suite against real service APIs in containers — MinIO (S3), fake-gcs-server (GCS), and Azurite (Azure Blob) — via testcontainers-go. They require a running Docker daemon and are gated behind the integration build tag: 
    (cd s3 && go test -tags=integration ./...)
(cd gcs && go test -tags=integration ./...)
(cd azblob && go test -tags=integration ./...)
    testcontainers is a test-only dependency of each provider module, so it is never pulled into your application build.

Documentation

Overview

Package objfs is a lightweight object-storage abstraction that is also an io/fs.FS.

Inspired by Thanos' objstore, objfs exposes a small Bucket interface that every backend implements. Because Bucket embeds io/fs.FS, any backend can be handed to the standard library — io/fs.WalkDir, io/fs.ReadFile, net/http.FileServerFS, templates, and so on — while still offering context-aware upload, delete, listing and (optionally) presigned URLs.

The core module depends only on the standard library and ships the local filesystem backend (NewLocal). Cloud backends live in opt-in submodules so their SDK dependencies are only pulled in when imported: 

github.com/armadakv/objfs/s3      // Amazon S3 (and S3-compatible)
github.com/armadakv/objfs/gcs     // Google Cloud Storage
github.com/armadakv/objfs/azblob  // Azure Blob Storage
Example

Example shows the core workflow: upload, read back through the io/fs.FS surface, and check for the optional presign capability — all backend-agnostic. 

package main

import (
	"context"
	"fmt"
	"io/fs"
	"strings"

	"github.com/armadakv/objfs"
)

func main() {
	ctx := context.Background()

	// Any backend works here; the local one needs only the standard library.
	var bucket objfs.Bucket
	bucket, _ = objfs.NewLocal("/tmp/objfs-example")
	defer bucket.Close()

	_ = bucket.Upload(ctx, "greetings/hello.txt", strings.NewReader("hi"))

	// Read it back via the standard library — bucket is an io/fs.FS.
	data, _ := fs.ReadFile(bucket, "greetings/hello.txt")
	fmt.Printf("contents: %s\n", data)

	// Presigning is an optional capability. Local does not support it.
	if _, err := objfs.PresignedGet(ctx, bucket, "greetings/hello.txt", 0); err != nil {
		fmt.Println("presign:", err)
	}

}

Output:
contents: hi
presign: objfs: presign GET: objfs: unsupported operation

Index

Examples

Constants

This section is empty.

Variables

View Source 
var ErrNotExist = fs.ErrNotExist

ErrNotExist is returned when an object does not exist. It aliases io/fs.ErrNotExist so that errors.Is(err, fs.ErrNotExist) holds for every backend, keeping objfs interchangeable with the standard library.

View Source 
var ErrUnsupported = errors.New("objfs: unsupported operation")

ErrUnsupported is returned by operations a backend does not implement, most commonly Presigner.PresignedURL on backends without signed-URL support.

View Source 
var SkipAll = fs.SkipAll

SkipAll is returned by a Bucket.List callback to stop iteration early without reporting an error. It mirrors io/fs.SkipAll.

Functions

func NewFile 

func NewFile(r io.ReadCloser, attrs Attributes) fs.File

NewFile returns an io/fs.File that reads from r and reports info from attrs. The returned file takes ownership of r and closes it on Close.

func NewRandomAccessFile 

func NewRandomAccessFile(b Bucket, attrs Attributes) fs.File

NewRandomAccessFile returns an io/fs.File for the object described by attrs whose bytes are fetched on demand from b via Bucket.GetRange. The returned file also satisfies io.ReaderAt and io.Seeker.

attrs.Name and attrs.Size must be populated; Size bounds Seek and ReadAt.

func PresignedGet 

func PresignedGet(ctx context.Context, b Bucket, name string, expiry time.Duration) (string, error)

PresignedGet is a convenience wrapper that returns a download URL for name, or an error wrapping ErrUnsupported if b is not a Presigner.

func PresignedPut 

func PresignedPut(ctx context.Context, b Bucket, name string, expiry time.Duration) (string, error)

PresignedPut is a convenience wrapper that returns an upload URL for name, or an error wrapping ErrUnsupported if b is not a Presigner.

func ReadDir 

func ReadDir(ctx context.Context, b Bucket, name string) ([]fs.DirEntry, error)

ReadDir lists the immediate children of the directory name within b, synthesising subdirectories from the "/" delimiter. It is the backend- agnostic implementation of io/fs.ReadDirFS built on Bucket.List; cloud backends override it with native delimiter listing for efficiency, but it is exported so it works against any Bucket.

Object stores have no empty directories, so a name with no children yields an empty slice rather than an error.

func ReadFile 

func ReadFile(ctx context.Context, b Bucket, name string) ([]byte, error)

ReadFile reads the named object from b in full. Backends expose it as their io/fs.ReadFileFS implementation; it issues a single Get (no preliminary Stat), which is cheaper than the Open+ReadAll fallback on stores where Open stats first.

func SortDirEntries 

func SortDirEntries(entries []fs.DirEntry)

SortDirEntries sorts entries by name, as io/fs.ReadDir requires. Backends call it before returning from ReadDir.

Types

type Attributes 

type Attributes struct {
	// Name is the object's slash-separated key.
	Name string
	// Size is the object size in bytes.
	Size int64
	// LastModified is the last-modified time, if known.
	LastModified time.Time
	// ContentType is the stored MIME type, if any.
	ContentType string
	// ETag is the backend's entity tag, if any.
	ETag string
	// Metadata holds backend user metadata, if any.
	Metadata map[string]string
}

Attributes describes a stored object.

type Bucket 

type Bucket interface {
	// FS provides Open(name) and, where the backend supports it, the richer
	// fs.ReadDirFS / fs.StatFS / fs.SubFS behaviours. Open uses
	// context.Background internally.
	fs.FS

	// Get returns a reader for the whole object. The caller must Close it.
	// It returns an error wrapping ErrNotExist if name does not exist.
	Get(ctx context.Context, name string) (io.ReadCloser, error)

	// GetRange returns a reader for the half-open byte range [off, off+length).
	// A negative length reads to the end of the object.
	GetRange(ctx context.Context, name string, off, length int64) (io.ReadCloser, error)

	// Upload stores the contents of r under name, creating intermediate
	// "directories" as needed. Existing objects are overwritten.
	Upload(ctx context.Context, name string, r io.Reader, opts ...UploadOption) error

	// Delete removes name. Deleting a missing object is not an error.
	Delete(ctx context.Context, name string) error

	// Stat returns metadata about name without reading its body.
	Stat(ctx context.Context, name string) (Attributes, error)

	// Exists reports whether name exists.
	Exists(ctx context.Context, name string) (bool, error)

	// List calls fn once per object whose name begins with prefix. Iteration
	// stops, and the error is returned, if fn returns a non-nil error;
	// returning [SkipAll] stops iteration without error.
	List(ctx context.Context, prefix string, fn func(Attributes) error) error

	// Close releases resources held by the backend (connection pools, clients).
	io.Closer
}

Bucket is a read-write object store that doubles as an io/fs.FS.

Object names are always slash-separated paths satisfying io/fs.ValidPath (no leading slash, no "." or ".." elements, no empty segments). The reusable io/fs helpers — ReadFile, WalkDir, Glob, Sub — therefore work against any Bucket via the embedded FS.

The context-free io/fs.FS Open exists for stdlib compatibility; prefer the context-aware methods (Get, Upload, ...) in application code.

func Sub 

func Sub(b Bucket, dir string) (Bucket, error)

Sub returns a Bucket rooted at dir within b: every name passed to the returned bucket is prefixed with dir before reaching b, and names reported by List have the prefix stripped. dir must satisfy io/fs.ValidPath.

The result is a full Bucket — Upload, Delete and (when b supports it) presigning all operate within the sub-tree — which is richer than the Open-only view that io/fs.Sub would produce. Nested Subs are flattened.

type FileInfo 

type FileInfo struct {
	// contains filtered or unexported fields
}

FileInfo adapts Attributes to io/fs.FileInfo.

func NewDirInfo 

func NewDirInfo(name string) FileInfo

NewDirInfo returns an io/fs.FileInfo describing a synthetic directory at name. Object stores have no real directories; backends synthesise them from key prefixes so that io/fs.WalkDir works.

func NewFileInfo 

func NewFileInfo(a Attributes) FileInfo

NewFileInfo returns an io/fs.FileInfo describing a regular object.

func (FileInfo) Attributes 

func (fi FileInfo) Attributes() Attributes

Attributes returns the object metadata backing this FileInfo.

func (FileInfo) IsDir 

func (fi FileInfo) IsDir() bool

func (FileInfo) ModTime 

func (fi FileInfo) ModTime() time.Time

func (FileInfo) Mode 

func (fi FileInfo) Mode() fs.FileMode

func (FileInfo) Name 

func (fi FileInfo) Name() string

func (FileInfo) Size 

func (fi FileInfo) Size() int64

func (FileInfo) Sys 

func (fi FileInfo) Sys() any

Sys returns the underlying Attributes.

type Local 

type Local struct {
	// contains filtered or unexported fields
}

Local is a Bucket backed by a directory on the local filesystem. It is part of the core module and depends only on the standard library, which makes it the natural choice for tests, local development and single-node deployments.

All access is confined to the root directory via os.Root: object names that would escape the root (via "..", absolute paths or symlinks) are rejected. Local does not implement Presigner; presign helpers return ErrUnsupported.

func NewLocal 

func NewLocal(dir string) (*Local, error)

NewLocal opens (creating it if necessary) the directory dir and returns a Bucket rooted there.

func (*Local) Close 

func (l *Local) Close() error

Close releases the underlying os.Root handle.

func (*Local) Delete 

func (l *Local) Delete(ctx context.Context, name string) error

Delete removes name. Removing a missing object is not an error.

func (*Local) Exists 

func (l *Local) Exists(ctx context.Context, name string) (bool, error)

Exists reports whether name exists.

func (*Local) Get 

func (l *Local) Get(ctx context.Context, name string) (io.ReadCloser, error)

Get returns a reader over the whole object.

func (*Local) GetRange 

func (l *Local) GetRange(ctx context.Context, name string, off, length int64) (io.ReadCloser, error)

GetRange returns a reader over [off, off+length); a negative length reads to the end of the object.

func (*Local) List 

func (l *Local) List(ctx context.Context, prefix string, fn func(Attributes) error) error

List walks the directory tree and reports every regular file whose name begins with prefix.

func (*Local) Open 

func (l *Local) Open(name string) (fs.File, error)

Open implements io/fs.FS. The returned file is an *os.File, so directories support ReadDir and regular files support Seek.

func (*Local) ReadDir 

func (l *Local) ReadDir(name string) ([]fs.DirEntry, error)

ReadDir implements io/fs.ReadDirFS, returning the immediate children of the directory name (real filesystem directories, so empty dirs are honoured).

func (*Local) ReadFile 

func (l *Local) ReadFile(name string) ([]byte, error)

ReadFile implements io/fs.ReadFileFS.

func (*Local) Stat 

func (l *Local) Stat(ctx context.Context, name string) (Attributes, error)

Stat returns metadata about name.

func (*Local) Sub 

func (l *Local) Sub(dir string) (fs.FS, error)

Sub implements io/fs.SubFS, returning a prefix-scoped Bucket.

func (*Local) Upload 

func (l *Local) Upload(ctx context.Context, name string, r io.Reader, _ ...UploadOption) error

Upload writes r to name, creating parent directories as needed.

type Operation 

type Operation int

Operation identifies the HTTP method a presigned URL authorises. 

const (
	// OpGet authorises downloading (HTTP GET) an object.
	OpGet Operation = iota
	// OpPut authorises uploading (HTTP PUT) an object.
	OpPut
)

func (Operation) String 

func (o Operation) String() string

type Presigner 

type Presigner interface {
	// PresignedURL returns a URL authorising op on name, valid for expiry.
	PresignedURL(ctx context.Context, name string, op Operation, expiry time.Duration) (string, error)
}

Presigner is an optional capability for backends that can mint time-limited URLs granting direct access to an object without further authentication.

Detect it with a type assertion, or use the package helpers PresignedGet and PresignedPut: 

if p, ok := bucket.(objfs.Presigner); ok {
	url, err := p.PresignedURL(ctx, "report.pdf", objfs.OpGet, time.Hour)
}

type UploadOption 

type UploadOption func(*UploadOptions)

An UploadOption customises UploadOptions.

func WithCacheControl 

func WithCacheControl(cc string) UploadOption

WithCacheControl sets the Cache-Control header.

func WithContentType 

func WithContentType(ct string) UploadOption

WithContentType sets the object's MIME type.

func WithMetadata 

func WithMetadata(m map[string]string) UploadOption

WithMetadata sets backend user metadata. Repeated calls merge.

type UploadOptions 

type UploadOptions struct {
	// ContentType sets the object's MIME type. When empty, backends may sniff
	// it from the name or content.
	ContentType string
	// CacheControl sets the Cache-Control header on backends that support it.
	CacheControl string
	// Metadata sets backend user metadata.
	Metadata map[string]string
}

UploadOptions configures a single Bucket.Upload call.

func ApplyUploadOptions 

func ApplyUploadOptions(opts []UploadOption) UploadOptions

ApplyUploadOptions builds an UploadOptions from opts. Backends call this at the top of Upload.

Source Files

View all Source files

Directories

Path Synopsis
Package objfstest provides a reusable conformance suite for objfs.Bucket implementations.
 Package objfstest provides a reusable conformance suite for objfs.Bucket implementations.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.