stowry

package module
v1.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 17, 2026 License: MIT Imports: 18 Imported by: 0

README

Stowry

CI Go Report Card

A lightweight, self-hosted object storage server with AWS Signature V4 authentication.

Use cases: Local development, self-hosting, static site hosting, SPA deployment, simple file storage.

Features

  • AWS Sig V4 authentication - Uses AWS Signature V4 presigned URLs (not S3-compatible API)
  • Three server modes - Object storage API, static file server, or SPA host
  • Minimal dependencies - Single binary, SQLite or PostgreSQL for metadata
  • Soft deletion - Files are recoverable until cleanup runs
  • Atomic writes - No partial or corrupted files
  • Pluggable storage - Filesystem now, S3/GCS ready interface

Quick Start

# Using Docker
docker run -p 5708:5708 -v ./data:/data ghcr.io/sagarc03/stowry:latest

# Using binary
./stowry serve

Server starts at http://localhost:5708

Client SDKs

Generate presigned URLs to interact with Stowry:

Language Package Install
Go stowry-go go get github.com/sagarc03/stowry-go
Python stowrypy pip install stowrypy
JavaScript stowryjs npm install stowryjs

AWS SDKs (boto3, aws-sdk-go-v2, @aws-sdk/client-s3) also work for generating presigned URLs.

See examples/ for usage.

Installation

Docker
docker pull ghcr.io/sagarc03/stowry:latest

# With persistent storage
docker run -d \
  --name stowry \
  -p 5708:5708 \
  -v ./data:/data \
  -v ./stowry.db:/stowry.db \
  ghcr.io/sagarc03/stowry:latest
Binary

Download from Releases:

# Linux
curl -LO https://github.com/sagarc03/stowry/releases/latest/download/stowry_linux_amd64.tar.gz
tar xzf stowry_linux_amd64.tar.gz
./stowry serve

# macOS
curl -LO https://github.com/sagarc03/stowry/releases/latest/download/stowry_darwin_arm64.tar.gz
tar xzf stowry_darwin_arm64.tar.gz
./stowry serve
From Source
go install github.com/sagarc03/stowry/cmd/stowry@latest

CLI Commands

# Start the server
stowry serve [--port 5708] [--mode store|static|spa]

# Initialize metadata from existing files
stowry init [--storage ./data]

# Clean up soft-deleted files
stowry cleanup [--limit 100]
Global Flags
Flag Env Var Default Description
--config - config.yaml Config file path
--db-type STOWRY_DATABASE_TYPE sqlite Database type
--db-dsn STOWRY_DATABASE_DSN stowry.db Database connection
--storage STOWRY_STORAGE_PATH ./data Storage directory

Configuration

Create config.yaml:

server:
  port: 5708
  mode: store  # store | static | spa

database:
  type: sqlite      # sqlite | postgres
  dsn: stowry.db    # file path or connection string
  table: stowry_metadata

storage:
  path: ./data

# Optional: Authentication
auth:
  region: us-east-1
  service: s3
  keys:
    - access_key: YOUR_ACCESS_KEY
      secret_key: YOUR_SECRET_KEY

# Optional: Public access
access:
  public_read: false
  public_write: false

log:
  level: info  # debug | info | warn | error

Environment variables use STOWRY_ prefix: STOWRY_SERVER_PORT=8080

API

Upload
curl -X PUT http://localhost:5708/path/to/file.txt \
  -H "Content-Type: text/plain" \
  -d "Hello, World!"
Download
curl http://localhost:5708/path/to/file.txt
Delete
curl -X DELETE http://localhost:5708/path/to/file.txt
List Objects
curl "http://localhost:5708/?prefix=path/&limit=100"

Response:

{
  "items": [
    {
      "path": "path/to/file.txt",
      "content_type": "text/plain",
      "etag": "abc123...",
      "file_size_bytes": 13,
      "created_at": "2024-01-15T10:00:00Z",
      "updated_at": "2024-01-15T10:00:00Z"
    }
  ],
  "next_cursor": "..."
}
Authentication

When access.public_read or access.public_write is false, requests require AWS Signature V4 presigned URL parameters.

Generating Keys

Access keys and secret keys are arbitrary strings. Generate them with:

# Access key (20 chars)
openssl rand -hex 10 | tr '[:lower:]' '[:upper:]'

# Secret key (40 chars)
openssl rand -hex 20

Or use any password generator.

Presigned URL Format
?X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Credential=ACCESS_KEY/20240115/us-east-1/s3/aws4_request
&X-Amz-Date=20240115T100000Z
&X-Amz-Expires=3600
&X-Amz-SignedHeaders=host
&X-Amz-Signature=...

You can use S3 SDKs to generate presigned URL signatures, but note that Stowry's API is not S3-compatible.

Server Modes

Store (default)

Object storage API. Returns 404 for missing paths.

Static

Static file server. Serves index.html for directory paths:

  • /docs → /docs/index.html
SPA

Single Page Application mode. Returns /index.html for all 404s, enabling client-side routing.

Kubernetes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: stowry
spec:
  template:
    spec:
      securityContext:
        runAsUser: 65532
        runAsGroup: 65532
        fsGroup: 65532
      containers:
        - name: stowry
          image: ghcr.io/sagarc03/stowry:latest
          ports:
            - containerPort: 5708
          volumeMounts:
            - name: data
              mountPath: /data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: stowry-data

Development

# Run tests
make test

# Run linter
make lint

# Build binary
make build

# Run all checks
make check

Contributing

Contributions are welcome! Please follow these steps:

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Make your changes
  4. Run tests and linter (make check)
  5. Commit your changes (git commit -m 'Add amazing feature')
  6. Push to the branch (git push origin feature/amazing-feature)
  7. Open a Pull Request
Guidelines
  • Follow existing code style
  • Add tests for new features
  • Update documentation as needed
  • Keep commits focused and atomic

Changelog

See CHANGELOG.md for release history.

License

MIT

Documentation

Overview

Package stowry provides a lightweight object storage library with pluggable metadata backends and AWS Signature V4 authentication.

Stowry implements core object storage operations (create, get, delete, list) with support for soft deletion, atomic writes, and ETag-based integrity checks.

Key Components

  • StowryService: Main service combining metadata repository and file storage
  • MetaDataRepo: Interface for metadata persistence (PostgreSQL, SQLite)
  • FileStorage: Interface for file operations (filesystem, extensible to S3/GCS)
  • SignatureVerifier: AWS Signature V4 presigned URL verification

Server Modes

The library supports three server modes for different use cases:

  • ModeStore: Object storage API returning exact paths or 404
  • ModeStatic: Static file server with index.html fallback for directories
  • ModeSPA: Single Page Application mode returning /index.html for 404s

Example Usage 

service, err := stowry.NewStowryService(repo, storage, stowry.ModeStore)
if err != nil {
    log.Fatal(err)
}

// Create an object
metadata, err := service.Create(ctx, "path/to/file.txt", contentType, reader)

// Get an object
obj, err := service.Get(ctx, "path/to/file.txt")

See the http package for REST API implementation and the postgres/sqlite packages for metadata backend implementations.

Index

Constants

View Source 
const (
	SignatureAlgorithm = "AWS4-HMAC-SHA256"
	MaxExpiresSeconds  = 604800 // 7 days
	DateTimeFormat     = "20060102T150405Z"
	DateFormat         = "20060102"
)

Variables

View Source 
var (
	// ErrNotFound is returned when a resource is not found
	ErrNotFound = errors.New("not found")
	// ErrInternal is returned when an internal error occurs
	ErrInternal = errors.New("internal error")
	// ErrInvalidInput is returned when input validation fails
	ErrInvalidInput = errors.New("invalid input")
	// ErrUnauthorized is returned when authentication fails
	ErrUnauthorized = errors.New("unauthorized")
)

Functions

func EncodeCursor added in v1.0.0 

func EncodeCursor(createdAt time.Time, path string) string

EncodeCursor encodes cursor data to a base64 string for pagination.

func EscapeLikePattern added in v1.0.0 

func EscapeLikePattern(pattern string) string

EscapeLikePattern escapes special LIKE characters (%, _, \) to prevent SQL injection.

func IsValidPath 

func IsValidPath(p string) bool

IsValidPath validates that a path string meets the requirements for a storage path. It checks that the path:

  • is not empty or just "/"
  • starts with "/" (absolute path)
  • does not end with "/"
  • does not contain ".." (path traversal)
  • does not contain "//" (empty segments)
  • does not contain invalid characters: \ ? # ~
  • is valid UTF-8
  • does not contain "." segments (/., /./, or ending with /.)
  • does not contain null bytes, control characters (< 0x20), DEL (0x7f), or whitespace

Returns true if the path is valid, false otherwise.

func IsValidTableName 

func IsValidTableName(name string) bool

IsValidTableName checks if a table name is valid (lowercase, alphanumeric with underscores, max 63 chars).

Types

type CreateObject 

type CreateObject struct {
	Path        string
	ContentType string
}

type Cursor added in v1.0.0 

type Cursor struct {
	CreatedAt time.Time
	Path      string
}

Cursor represents pagination cursor data for list operations.

func DecodeCursor added in v1.0.0 

func DecodeCursor(cursor string) (Cursor, error)

DecodeCursor decodes a pagination cursor string back to cursor data.

type FileStorage 

type FileStorage interface {
	// Get retrieves a file from storage for reading.
	//
	// Parameters:
	//   - ctx: Context for cancellation and timeout
	//   - path: The object path to retrieve
	//
	// Returns:
	//   - io.ReadSeekCloser: Reader for file content with seek capability
	//   - error: ErrNotFound if file doesn't exist, or other storage errors
	//
	// The caller is responsible for closing the returned ReadSeekCloser.
	// Implementations should return a ReadSeekCloser to support range reads
	// and efficient streaming.
	Get(ctx context.Context, path string) (io.ReadSeekCloser, error)

	// Write stores content to a file at the specified path.
	// If a file already exists at the path, it should be overwritten.
	//
	// Parameters:
	//   - ctx: Context for cancellation and timeout
	//   - path: The destination path for the file
	//   - content: io.Reader providing the data to write
	//
	// Returns:
	//   - SaveResult: Contains bytes written and computed ETag/hash
	//   - error: Any storage or I/O error
	//
	// Implementations should:
	//   - Write atomically when possible (e.g., write to temp file then rename)
	//   - Compute an ETag or hash during write for integrity verification
	//   - Return accurate byte count of data written
	//   - Handle context cancellation gracefully and clean up partial writes
	//   - Create parent directories if they don't exist
	Write(ctx context.Context, path string, content io.Reader) (SaveResult, error)

	// Delete removes a file from storage.
	//
	// Parameters:
	//   - ctx: Context for cancellation and timeout
	//   - path: The object path to delete
	//
	// Returns:
	//   - error: ErrNotFound if file doesn't exist, or other storage errors
	//
	// Note: This only deletes the physical file, not its metadata.
	// Callers are responsible for coordinating file and metadata deletion.
	Delete(ctx context.Context, path string) error

	// List returns all objects currently in storage with their metadata.
	//
	// Parameters:
	//   - ctx: Context for cancellation and timeout
	//
	// Returns:
	//   - []ObjectEntry: Slice of all objects with path, size, ETag, and content type
	//   - error: Any storage or I/O error
	//
	// This method is typically used for:
	//   - Synchronizing metadata with physical storage (see StowryService.Populate)
	//   - Recovery operations after metadata loss
	//   - Storage health checks and auditing
	//
	// Implementations should:
	//   - Walk the entire storage tree recursively
	//   - Detect content type from file extensions or content inspection
	//   - Compute ETag/hash for each file
	//   - Return an empty slice (not nil) when storage is empty
	//
	// Warning: This can be expensive for large storage volumes. Use with caution
	// in production and consider implementing pagination for very large datasets.
	List(ctx context.Context) ([]ObjectEntry, error)
}

FileStorage defines the interface for physical file storage operations. Implementations can use local filesystem, S3, GCS, or any other storage backend.

All methods accept a context for cancellation and timeout control. Implementations should respect context cancellation during long-running operations like large file uploads or downloads.

type ListQuery 

type ListQuery struct {
	PathPrefix string
	Limit      int
	Cursor     string
}

type ListResult 

type ListResult struct {
	Items      []MetaData
	NextCursor string
}

type MetaData 

type MetaData struct {
	ID            uuid.UUID
	Path          string
	ContentType   string
	Etag          string
	FileSizeBytes int64
	CreatedAt     time.Time
	UpdatedAt     time.Time
}

type MetaDataRepo 

type MetaDataRepo interface {
	// Get retrieves metadata for a specific object by its path.
	//
	// Parameters:
	//   - ctx: Context for cancellation and timeout
	//   - path: The object path to look up
	//
	// Returns:
	//   - MetaData: The metadata entry if found
	//   - error: ErrNotFound if path doesn't exist, or other database errors
	Get(ctx context.Context, path string) (MetaData, error)

	// Upsert creates or updates metadata for an object.
	// If an entry with the same path exists, it updates the existing entry.
	// If no entry exists, it creates a new one.
	//
	// Parameters:
	//   - ctx: Context for cancellation and timeout
	//   - entry: ObjectEntry containing path, size, ETag, and content type
	//
	// Returns:
	//   - MetaData: The created or updated metadata entry with ID and timestamps
	//   - bool: true if a new entry was created, false if existing entry was updated
	//   - error: Any database or validation error
	Upsert(ctx context.Context, entry ObjectEntry) (MetaData, bool, error)

	// Delete removes metadata for a specific object by its path.
	//
	// Parameters:
	//   - ctx: Context for cancellation and timeout
	//   - path: The object path to delete
	//
	// Returns:
	//   - error: ErrNotFound if path doesn't exist, or other database errors
	Delete(ctx context.Context, path string) error

	// List retrieves a paginated list of metadata entries matching the query criteria.
	//
	// Parameters:
	//   - ctx: Context for cancellation and timeout
	//   - q: ListQuery with optional path prefix filter, limit, and cursor for pagination
	//
	// Returns:
	//   - ListResult: Contains matching metadata items and cursor for next page
	//   - error: Any database error
	List(ctx context.Context, q ListQuery) (ListResult, error)

	// ListPendingCleanup retrieves a paginated list of soft-deleted metadata entries
	// that have not yet been cleaned up (deleted_at IS NOT NULL AND cleaned_up_at IS NULL).
	//
	// Parameters:
	//   - ctx: Context for cancellation and timeout
	//   - q: ListQuery with optional path prefix filter, limit, and cursor for pagination
	//
	// Returns:
	//   - ListResult: Contains matching metadata items and cursor for next page
	//   - error: Any database error
	ListPendingCleanup(ctx context.Context, q ListQuery) (ListResult, error)

	// MarkCleanedUp marks a soft-deleted metadata entry as cleaned up by setting cleaned_up_at.
	// This should be called after the physical file has been deleted.
	//
	// Parameters:
	//   - ctx: Context for cancellation and timeout
	//   - id: The UUID of the metadata entry to mark as cleaned up
	//
	// Returns:
	//   - error: ErrNotFound if entry doesn't exist or isn't pending cleanup, or other database errors
	MarkCleanedUp(ctx context.Context, id uuid.UUID) error
}

MetaDataRepo defines the interface for managing object metadata persistence. Implementations must handle concurrent access safely and ensure data consistency.

All methods accept a context for cancellation and timeout control. Implementations should respect context cancellation and return appropriate errors.

type ObjectEntry 

type ObjectEntry struct {
	Path        string
	Size        int64
	ETag        string
	ContentType string
}

type SaveResult 

type SaveResult struct {
	BytesWritten int64
	Etag         string
}

type ServerMode 

type ServerMode string
const (
	ModeStore  ServerMode = "store"
	ModeStatic ServerMode = "static"
	ModeSPA    ServerMode = "spa"
)

func ParseServerMode 

func ParseServerMode(s string) (ServerMode, error)

func (ServerMode) IsValid 

func (m ServerMode) IsValid() bool

type SignatureVerifier 

type SignatureVerifier struct {
	Region          string
	Service         string
	AccessKeyLookup func(accessKey string) (secretKey string, found bool)
}

SignatureVerifier verifies AWS Signature V4 presigned URLs.

func NewSignatureVerifier 

func NewSignatureVerifier(region, service string, lookup func(string) (string, bool)) *SignatureVerifier

NewSignatureVerifier creates a new signature verifier.

Parameters:

  • region: AWS region (e.g., "us-east-1")
  • service: AWS service name (e.g., "s3")
  • lookup: Function to retrieve secret key by access key. Returns (secretKey, true) if found, ("", false) if not.

func (*SignatureVerifier) Verify 

func (v *SignatureVerifier) Verify(method, path string, query url.Values, headers http.Header) error

Verify verifies an AWS Signature V4 presigned URL.

This function implements AWS Signature Version 4 verification for presigned URLs, compatible with S3's authentication scheme. It validates all required query parameters, checks signature expiration, and verifies the HMAC-SHA256 signature.

Required query parameters:

  • X-Amz-Algorithm: Must be "AWS4-HMAC-SHA256"
  • X-Amz-Credential: Format "access_key/date/region/service/aws4_request"
  • X-Amz-Date: ISO8601 timestamp (YYYYMMDDTHHMMSSZ)
  • X-Amz-Expires: Validity duration in seconds (1-604800)
  • X-Amz-SignedHeaders: Semicolon-separated list of signed headers
  • X-Amz-Signature: Hex-encoded HMAC-SHA256 signature

The function performs the following validations:

  1. Presence of all required parameters
  2. Correct algorithm (AWS4-HMAC-SHA256)
  3. Valid timestamp format
  4. Expiration within allowed range (1 second to 7 days)
  5. Request not expired (current time before timestamp + expires)
  6. Credential format and component matching (date, region, service)
  7. Access key exists (via lookup function)
  8. Signature matches calculated signature

Parameters:

  • method: HTTP method (GET, PUT, DELETE, etc.)
  • path: Request path
  • query: URL query parameters including signature parameters
  • headers: HTTP headers from the request (used for signed header verification)

Returns an error if verification fails, nil if signature is valid.

Example: 

verifier := stowry.NewSignatureVerifier("us-east-1", "s3", lookupFunc)
err := verifier.Verify("GET", "/file.txt", r.URL.Query(), r.Header)
if err != nil {
    // Invalid signature
}

type StowryService 

type StowryService struct {
	// contains filtered or unexported fields
}

func NewStowryService 

func NewStowryService(repo MetaDataRepo, storage FileStorage, mode ServerMode) (*StowryService, error)

func (*StowryService) Create 

func (s *StowryService) Create(ctx context.Context, obj CreateObject, content io.Reader) (MetaData, error)

Create stores a new object in storage and creates its metadata entry. It performs comprehensive validation, writes the content to storage, and creates a corresponding metadata entry. If the metadata creation fails, the stored file is automatically cleaned up to prevent orphaned data.

The method performs the following steps:

  1. Validates context is not cancelled
  2. Validates input parameters (path, content type)
  3. Validates path using IsValidPath (prevents path traversal attacks)
  4. Writes content to storage and computes ETag
  5. Creates metadata entry
  6. On metadata failure, automatically deletes the stored file

Parameters:

  • ctx: Context for cancellation and timeout. If cancelled during storage write, the operation may still complete. Cleanup uses a separate background context.
  • obj: CreateObject containing path and content type
  • content: io.Reader providing the object data to store

Returns:

  • MetaData: The created metadata entry with ID, timestamps, and computed ETag
  • error: Any error encountered, including validation, storage, or metadata errors

Error types returned:

  • ErrInvalidInput: Empty path or content type
  • ErrInvalidInput: Path fails validation (contains .., //, invalid chars, etc.)
  • context.Canceled or context.DeadlineExceeded: Context was cancelled
  • Wrapped storage errors: Issues writing to storage
  • Wrapped metadata errors: Issues creating metadata entry

Concurrency safety: Safe for concurrent calls with different paths. Data consistency: If metadata creation fails, the stored file is automatically deleted using a background context with 30-second timeout to ensure cleanup completes even if the original context is cancelled.

func (*StowryService) Delete 

func (s *StowryService) Delete(ctx context.Context, path string) error

func (*StowryService) Get 

func (s *StowryService) Get(ctx context.Context, path string) (MetaData, io.ReadSeekCloser, error)

func (*StowryService) List 

func (s *StowryService) List(ctx context.Context, q ListQuery) (ListResult, error)

func (*StowryService) Populate 

func (s *StowryService) Populate(ctx context.Context) error

Populate synchronizes metadata from physical storage files. It lists all files in storage and creates or updates their corresponding metadata entries.

This method is typically used during initialization or recovery to ensure the metadata repository is in sync with actual files in storage. It processes all files sequentially and stops at the first error encountered.

Returns an error if:

  • Storage listing fails
  • Any metadata upsert operation fails
  • Context is cancelled during processing

Note: This operation is not atomic. If it fails partway through, some files may have been processed while others remain unprocessed.

func (*StowryService) Tombstone 

func (s *StowryService) Tombstone(ctx context.Context, q ListQuery) (int, error)

Tombstone permanently removes all soft-deleted files from storage and marks them as cleaned up. It processes all pending cleanup items by paginating through until none remain.

The method performs the following for each soft-deleted file:

  1. Deletes the physical file from storage
  2. Marks the metadata entry as cleaned up (sets cleaned_up_at)

If a file has already been deleted from storage (ErrNotFound), the method continues and marks it as cleaned up anyway - this handles the case where a previous cleanup attempt deleted the file but failed to mark the metadata.

Parameters:

  • ctx: Context for cancellation and timeout
  • q: ListQuery with optional path prefix filter and limit (cursor is managed internally)

Returns:

  • int: Total number of items cleaned up
  • error: Any error encountered during cleanup

type Tables 

type Tables struct {
	MetaData string
}

Tables holds configurable table names for metadata storage. This allows multi-tenant deployments to use different table names.

func (Tables) Validate 

func (t Tables) Validate() error

Validate checks that all required table names are set and valid.

Source Files

View all Source files

Directories

Path Synopsis
cmd
stowry command
Package filesystem provides a file system storage backend for stowry.
 Package filesystem provides a file system storage backend for stowry.
Package http provides HTTP server functionality for Stowry object storage.
 Package http provides HTTP server functionality for Stowry object storage.
Package postgres implements the repo interface for all the services
 Package postgres implements the repo interface for all the services
Package sqlite implements the repo interface using SQLite
 Package sqlite implements the repo interface using SQLite

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.