fs

README

fs

Simple S3-compatible storage server for development and testing.

Features

S3-Compatible Storage Server

A lightweight S3-compatible storage server for development and testing.

Quick Start:

# Install
go install github.com/go-faster/fs/cmd/fs@latest

# Start the server
fs s3

# Or with custom configuration
fs s3 --addr :9000 --root /data/s3

Features:

• Bucket operations (create, delete, list)
• Object operations (put, get, delete, list)
• File system-based storage
• Compatible with AWS CLI, MinIO client, and other S3 clients
• Health check endpoint

Example Usage:

# Using AWS CLI
export AWS_ENDPOINT_URL=http://localhost:8080
aws s3 mb s3://mybucket --endpoint-url $AWS_ENDPOINT_URL
aws s3 cp file.txt s3://mybucket/ --endpoint-url $AWS_ENDPOINT_URL

# Using cURL
curl -X PUT http://localhost:8080/mybucket
curl -X PUT -d "Hello!" http://localhost:8080/mybucket/hello.txt
curl http://localhost:8080/mybucket/hello.txt

See S3_README.md for detailed documentation.

Installation

go install github.com/go-faster/fs/cmd/fs@latest

Or build from source:

git clone https://github.com/go-faster/fs
cd fs
go build -o bin/fs ./cmd/fs

Usage

Quick Start

# Start S3 server with defaults
fs s3

# Show help
fs s3 --help

Configuration

The server supports both YAML configuration files and command-line flags:

# Using YAML configuration
fs s3 --config config.yaml

# Using command-line flags
fs s3 --addr :9000 --root /var/lib/s3data

# Mix both (flags override config file)
fs s3 --config config.yaml --addr :9000

# Generate example configuration
fs s3 --generate-config > my-config.yaml

See CONFIGURATION.md for detailed configuration documentation.

Example Configuration

server:
  addr: ":8080"
  read_timeout: 30s
  write_timeout: 30s
  idle_timeout: 120s
  health_path: "/health"

storage:
  root: ".s3data"
  type: "filesystem"

observability:
  service_name: "go-faster/fs"
  enable_request_logging: true
  enable_metrics: true
  enable_tracing: true

Use as a library

The S3 server is embeddable. Pick a storage backend (storagefs for filesystem, storagemem for in-memory, or your own implementation of fs.Storage) and either mount the bare http.Handler or run the turnkey server.Server. The library core pulls in no observability stack — wrap the handler yourself (e.g. with otelhttp) via Config.WrapHandler.

import (
	"net/http"

	"github.com/go-faster/fs/server"
	"github.com/go-faster/fs/storagefs"
)

func main() {
	store, err := storagefs.New("/data")
	if err != nil {
		panic(err)
	}

	// Low-level: mount into your own mux.
	h := server.NewHandler(store)
	http.ListenAndServe(":8080", h)

	// High-level: turnkey server with health, timeouts and graceful shutdown.
	srv, err := server.New(server.Config{Storage: store, Addr: ":9000"})
	if err != nil {
		panic(err)
	}
	srv.ListenAndServe(ctx) // serves until ctx is canceled, then drains
}

See the server package reference for the full API and runnable examples.

Development

# Run tests
go test ./...

# Build
go build ./cmd/fs

# Run with coverage
./go.coverage.sh

License

Apache 2.0

Documentation

Overview

Package fs is a S3-compatible storage server implementation.

Index

• Variables
• type Bucket
• type CompleteMultipartUploadRequest
• type CompleteMultipartUploadResponse
• type CompletedPart
• type GetObjectResponse
• type MultipartUpload
• type Object
• type Part
• type PutObjectRequest
• type Service
• type Storage
• type UploadPartRequest

Variables

var (
	ErrBucketNotFound       = errors.New("bucket not found")
	ErrObjectNotFound       = errors.New("object not found")
	ErrUploadNotFound       = errors.New("upload not found")
	ErrInvalidBucketName    = errors.New("invalid bucket name")
	ErrUnsupportedOperation = errors.New("unsupported operation")
)

Types

type Bucket

type Bucket struct {
	Name         string
	CreationDate time.Time
}

Bucket represents an S3 bucket.

type CompleteMultipartUploadRequest

type CompleteMultipartUploadRequest struct {
	Bucket   string
	Key      string
	UploadID string
	Parts    []CompletedPart
}

CompleteMultipartUploadRequest represents a request to complete multipart upload.

type CompleteMultipartUploadResponse

type CompleteMultipartUploadResponse struct {
	Location string
	Bucket   string
	Key      string
	ETag     string
}

CompleteMultipartUploadResponse represents the response for completing multipart upload.

type CompletedPart

type CompletedPart struct {
	PartNumber int
	ETag       string
}

CompletedPart represents a completed part for completing multipart upload.

type GetObjectResponse

type GetObjectResponse struct {
	Reader       io.ReadCloser
	Size         int64
	LastModified time.Time
	ETag         string
	ContentType  string
}

GetObjectResponse represents the response for GetObject operation.

type MultipartUpload

type MultipartUpload struct {
	UploadID  string
	Bucket    string
	Key       string
	Initiated time.Time
}

MultipartUpload represents an in-progress multipart upload.

type Object

type Object struct {
	Key          string
	Size         int64
	LastModified time.Time
	ETag         string
}

Object represents an S3 object.

type Part

type Part struct {
	PartNumber int
	ETag       string
	Size       int64
}

Part represents a part of a multipart upload.

type PutObjectRequest

type PutObjectRequest struct {
	Reader io.Reader
	Bucket string
	Key    string
	Size   int64
}

type Service

type Service interface {
	ListBuckets(ctx context.Context) ([]Bucket, error)
	CreateBucket(ctx context.Context, bucket string) error
	DeleteBucket(ctx context.Context, bucket string) error
	BucketExists(ctx context.Context, bucket string) (bool, error)
	ListObjects(ctx context.Context, bucket, prefix string) ([]Object, error)
	PutObject(ctx context.Context, req *PutObjectRequest) error
	GetObject(ctx context.Context, bucket, key string) (*GetObjectResponse, error)
	DeleteObject(ctx context.Context, bucket, key string) error

	CreateMultipartUpload(ctx context.Context, bucket, key string) (*MultipartUpload, error)
	UploadPart(ctx context.Context, req *UploadPartRequest) (*Part, error)
	CompleteMultipartUpload(ctx context.Context, req *CompleteMultipartUploadRequest) (*CompleteMultipartUploadResponse, error)
	AbortMultipartUpload(ctx context.Context, bucket, key, uploadID string) error
}

Service defines the interface for S3-compatible storage operations.

type Storage

type Storage interface {
	ListBuckets(ctx context.Context) ([]Bucket, error)
	CreateBucket(ctx context.Context, bucket string) error
	DeleteBucket(ctx context.Context, bucket string) error
	BucketExists(ctx context.Context, bucket string) (bool, error)
	ListObjects(ctx context.Context, bucket, prefix string) ([]Object, error)
	PutObject(ctx context.Context, req *PutObjectRequest) error
	GetObject(ctx context.Context, bucket, key string) (*GetObjectResponse, error)
	DeleteObject(ctx context.Context, bucket, key string) error

	CreateMultipartUpload(ctx context.Context, bucket, key string) (*MultipartUpload, error)
	UploadPart(ctx context.Context, req *UploadPartRequest) (*Part, error)
	CompleteMultipartUpload(ctx context.Context, req *CompleteMultipartUploadRequest) (*CompleteMultipartUploadResponse, error)
	AbortMultipartUpload(ctx context.Context, bucket, key, uploadID string) error
}

Storage defines the interface for S3-compatible storage operations.

type UploadPartRequest

type UploadPartRequest struct {
	Bucket     string
	Key        string
	UploadID   string
	PartNumber int
	Reader     io.Reader
	Size       int64
}

UploadPartRequest represents a request to upload a part.