fs

README

fs

Simple S3-compatible storage server for development and testing.

Features

S3-Compatible Storage Server

A lightweight S3-compatible storage server for development and testing.

Quick Start:

# Install
go install github.com/go-faster/fs/cmd/fs@latest

# Start the server
fs s3

# Or with custom configuration
fs s3 --addr :9000 --root /data/s3

Features:

• Bucket operations (create, delete, list)
• Object operations (put, get, delete, list)
• File system-based storage
• Compatible with AWS CLI, MinIO client, and other S3 clients
• Health check endpoint

Example Usage:

# Using AWS CLI
export AWS_ENDPOINT_URL=http://localhost:8080
aws s3 mb s3://mybucket --endpoint-url $AWS_ENDPOINT_URL
aws s3 cp file.txt s3://mybucket/ --endpoint-url $AWS_ENDPOINT_URL

# Using cURL
curl -X PUT http://localhost:8080/mybucket
curl -X PUT -d "Hello!" http://localhost:8080/mybucket/hello.txt
curl http://localhost:8080/mybucket/hello.txt

See S3_README.md for detailed documentation.

Installation

go install github.com/go-faster/fs/cmd/fs@latest

Or build from source:

git clone https://github.com/go-faster/fs
cd fs
go build -o bin/fs ./cmd/fs

Usage

Quick Start

# Start S3 server with defaults
fs s3

# Show help
fs s3 --help

Configuration

The server supports both YAML configuration files and command-line flags:

# Using YAML configuration
fs s3 --config config.yaml

# Using command-line flags
fs s3 --addr :9000 --root /var/lib/s3data

# Mix both (flags override config file)
fs s3 --config config.yaml --addr :9000

# Generate example configuration
fs s3 --generate-config > my-config.yaml

See CONFIGURATION.md for detailed configuration documentation.

Example Configuration

server:
  addr: ":8080"
  read_timeout: 30s
  write_timeout: 30s
  idle_timeout: 120s
  health_path: "/health"

storage:
  root: ".s3data"
  type: "filesystem"

observability:
  service_name: "go-faster/fs"
  enable_request_logging: true
  enable_metrics: true
  enable_tracing: true

Use as a library

The S3 server is embeddable. Install the module and pick a storage backend — storagefs for the filesystem, storagemem for in-memory, or your own implementation of the fs.Storage interface:

go get github.com/go-faster/fs

The library core pulls in no observability stack — wrap the handler yourself (e.g. with otelhttp) via server.NewHandler or Config.WrapHandler.

Mount the handler into your own server

Use server.NewHandler when you already run an http.Server or mux and just want to expose the S3 API (optionally under a path prefix):

package main

import (
	"net/http"

	"github.com/go-faster/fs/server"
	"github.com/go-faster/fs/storagefs"
)

func main() {
	store, err := storagefs.New("/data")
	if err != nil {
		panic(err)
	}

	mux := http.NewServeMux()
	mux.Handle("/s3/", http.StripPrefix("/s3", server.NewHandler(store)))

	http.ListenAndServe(":8080", mux)
}

Run the turnkey server

Use server.New for a managed server with a health endpoint, request timeouts, optional bucket pre-creation and graceful shutdown driven by a context:

package main

import (
	"context"
	"os/signal"
	"syscall"

	"github.com/go-faster/fs/server"
	"github.com/go-faster/fs/storagemem"
)

func main() {
	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
	defer stop()

	srv, err := server.New(server.Config{
		Storage: storagemem.New(),
		Addr:    ":9000",
		Buckets: []string{"uploads"}, // pre-created if absent
	})
	if err != nil {
		panic(err)
	}

	// Serves until ctx is canceled, then drains in-flight requests.
	if err := srv.ListenAndServe(ctx); err != nil {
		panic(err)
	}
}

server.Config

Field	Default	Description
Storage	— (required)	Backend serving S3 operations (fs.Storage).
Addr	:8080	TCP address to listen on.
ReadTimeout / WriteTimeout / IdleTimeout	30s / 30s / 120s	Underlying http.Server timeouts.
HealthPath	/health	Plaintext health endpoint; "-" disables it.
Buckets	—	Buckets created (idempotently) before serving.
WrapHandler	—	Wrap the handler with middleware/observability (e.g. otelhttp.NewHandler).

See the server package reference for the full API and runnable examples.

Development

# Run tests
go test ./...

# Build
go build ./cmd/fs

# Run with coverage
./go.coverage.sh

License

Apache 2.0

Documentation

Overview

Package fs is a S3-compatible storage server implementation.

Index

• Variables
• type Bucket
• type CompleteMultipartUploadRequest
• type CompleteMultipartUploadResponse
• type CompletedPart
• type GetObjectResponse
• type MultipartUpload
• type Object
• type Part
• type PutObjectRequest
• type Service
• type Storage
• type UploadPartRequest

Variables

var (
	ErrBucketNotFound       = errors.New("bucket not found")
	ErrObjectNotFound       = errors.New("object not found")
	ErrUploadNotFound       = errors.New("upload not found")
	ErrInvalidBucketName    = errors.New("invalid bucket name")
	ErrUnsupportedOperation = errors.New("unsupported operation")
	ErrPreconditionFailed   = errors.New("precondition failed")
)

Types

type Bucket

type Bucket struct {
	Name         string
	CreationDate time.Time
}

Bucket represents an S3 bucket.

type CompleteMultipartUploadRequest

type CompleteMultipartUploadRequest struct {
	Bucket   string
	Key      string
	UploadID string
	Parts    []CompletedPart
}

CompleteMultipartUploadRequest represents a request to complete multipart upload.

type CompleteMultipartUploadResponse

type CompleteMultipartUploadResponse struct {
	Location string
	Bucket   string
	Key      string
	ETag     string
}

CompleteMultipartUploadResponse represents the response for completing multipart upload.

type CompletedPart

type CompletedPart struct {
	PartNumber int
	ETag       string
}

CompletedPart represents a completed part for completing multipart upload.

type GetObjectResponse

type GetObjectResponse struct {
	Reader       io.ReadCloser
	Size         int64
	LastModified time.Time
	ETag         string
	ContentType  string
}

GetObjectResponse represents the response for GetObject operation.

type MultipartUpload

type MultipartUpload struct {
	UploadID  string
	Bucket    string
	Key       string
	Initiated time.Time
}

MultipartUpload represents an in-progress multipart upload.

type Object

type Object struct {
	Key          string
	Size         int64
	LastModified time.Time
	ETag         string
}

Object represents an S3 object.

type Part

type Part struct {
	PartNumber int
	ETag       string
	Size       int64
}

Part represents a part of a multipart upload.

type PutObjectRequest

type PutObjectRequest struct {
	Reader io.Reader
	Bucket string
	Key    string
	Size   int64
}

type Service

type Service interface {
	ListBuckets(ctx context.Context) ([]Bucket, error)
	CreateBucket(ctx context.Context, bucket string) error
	DeleteBucket(ctx context.Context, bucket string) error
	BucketExists(ctx context.Context, bucket string) (bool, error)
	ListObjects(ctx context.Context, bucket, prefix string) ([]Object, error)
	PutObject(ctx context.Context, req *PutObjectRequest) error
	GetObject(ctx context.Context, bucket, key string) (*GetObjectResponse, error)
	DeleteObject(ctx context.Context, bucket, key string) error

	CreateMultipartUpload(ctx context.Context, bucket, key string) (*MultipartUpload, error)
	UploadPart(ctx context.Context, req *UploadPartRequest) (*Part, error)
	CompleteMultipartUpload(ctx context.Context, req *CompleteMultipartUploadRequest) (*CompleteMultipartUploadResponse, error)
	AbortMultipartUpload(ctx context.Context, bucket, key, uploadID string) error
}

Service defines the interface for S3-compatible storage operations.

type Storage

type Storage interface {
	ListBuckets(ctx context.Context) ([]Bucket, error)
	CreateBucket(ctx context.Context, bucket string) error
	DeleteBucket(ctx context.Context, bucket string) error
	BucketExists(ctx context.Context, bucket string) (bool, error)
	ListObjects(ctx context.Context, bucket, prefix string) ([]Object, error)
	PutObject(ctx context.Context, req *PutObjectRequest) error
	GetObject(ctx context.Context, bucket, key string) (*GetObjectResponse, error)
	DeleteObject(ctx context.Context, bucket, key string) error

	CreateMultipartUpload(ctx context.Context, bucket, key string) (*MultipartUpload, error)
	UploadPart(ctx context.Context, req *UploadPartRequest) (*Part, error)
	CompleteMultipartUpload(ctx context.Context, req *CompleteMultipartUploadRequest) (*CompleteMultipartUploadResponse, error)
	AbortMultipartUpload(ctx context.Context, bucket, key, uploadID string) error
}

Storage defines the interface for S3-compatible storage operations.

type UploadPartRequest

type UploadPartRequest struct {
	Bucket     string
	Key        string
	UploadID   string
	PartNumber int
	Reader     io.Reader
	Size       int64
}

UploadPartRequest represents a request to upload a part.