objectstorage

package
v0.4.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 21, 2025 License: MIT Imports: 12 Imported by: 0

Documentation

Overview

Package objectstorage provides a unified interface for object storage across multiple providers including local filesystem, MinIO, and Volcengine TOS.

Supported Providers

  • local: Local filesystem storage
  • minio: MinIO and S3-compatible storage
  • volcengine: Volcengine TOS (Toutiao Object Storage)

Basic Usage

Create a storage client: 

import objectstorage "github.com/poly-workshop/go-webmods/objectstorage"

storage, err := objectstorage.NewObjectStorage(objectstorage.Config{
    ProviderType: objectstorage.ProviderLocal,
    ProviderConfig: objectstorage.ProviderConfig{
        BasePath: "/data/uploads",
    },
})
if err != nil {
    panic(err)
}

Local Filesystem Provider

Store files on the local filesystem: 

storage, err := objectstorage.NewObjectStorage(objectstorage.Config{
    ProviderType: objectstorage.ProviderLocal,
    ProviderConfig: objectstorage.ProviderConfig{
        BasePath: "/var/data",  // Base directory for all files
    },
})

MinIO / S3-Compatible Provider

Connect to MinIO or any S3-compatible service: 

storage, err := objectstorage.NewObjectStorage(objectstorage.Config{
    ProviderType: objectstorage.ProviderMinio,
    ProviderConfig: objectstorage.ProviderConfig{
        Endpoint:  "minio.example.com:9000",
        Region:    "us-east-1",
        AccessKey: "minioadmin",
        SecretKey: "minioadmin",
        Bucket:    "mybucket",
        BasePath:  "uploads/",  // Optional prefix for all keys
    },
})

Volcengine TOS Provider

Connect to Volcengine TOS: 

storage, err := objectstorage.NewObjectStorage(objectstorage.Config{
    ProviderType: objectstorage.ProviderVolcengine,
    ProviderConfig: objectstorage.ProviderConfig{
        Endpoint:    "tos-cn-beijing.volces.com",
        Region:      "cn-beijing",
        AccessKey:   "your-access-key",
        SecretKey:   "your-secret-key",
        Bucket:      "mybucket",
        UseInternal: false,  // Use internal endpoint for in-region access
    },
})

Common Operations

Save a file: 

file, _ := os.Open("photo.jpg")
defer file.Close()
size, err := storage.Save("photos/photo.jpg", file)

List files in a directory: 

objects, err := storage.List("photos/")
for _, obj := range objects {
    info, _ := obj.Stat()
    fmt.Printf("%s - %d bytes\n", info.Name(), info.Size())
}

Open and read a file: 

obj, err := storage.Open("photos/photo.jpg")
if err != nil {
    panic(err)
}
defer obj.Close()

data, err := io.ReadAll(obj)
// Or use obj.Read(), obj.Seek() for streaming

Get file metadata: 

info, err := storage.Stat("photos/photo.jpg")
fmt.Printf("Name: %s, Size: %d, Modified: %s\n",
    info.Name(), info.Size(), info.ModTime())

Delete a file: 

err := storage.Delete("photos/photo.jpg")

Interface Design

The ObjectStorage interface provides a consistent API across all providers: 

type ObjectStorage interface {
    Save(path string, r io.Reader) (int64, error)
    List(path string) ([]Object, error)
    Open(path string) (Object, error)
    Stat(path string) (os.FileInfo, error)
    Delete(path string) error
}

Objects returned by Open() and List() implement io.ReadSeekCloser: 

type Object interface {
    io.ReadSeekCloser
    Stat() (os.FileInfo, error)
}

This allows seamless use with Go's standard I/O functions.

Configuration with Viper

Example configuration file (configs/default.yaml): 

objectstorage:
  provider: local
  base_path: /var/data/uploads

For production (configs/production.yaml): 

objectstorage:
  provider: minio
  endpoint: minio.prod.example.com:9000
  region: us-east-1
  access_key: ${MINIO_ACCESS_KEY}
  secret_key: ${MINIO_SECRET_KEY}
  bucket: prod-uploads

Loading configuration: 

import (
    "github.com/poly-workshop/go-webmods/app"
    objectstorage "github.com/poly-workshop/go-webmods/objectstorage"
)

app.Init(".")
cfg := app.Config()

storage, err := objectstorage.NewObjectStorage(objectstorage.Config{
    ProviderType: objectstorage.ProviderType(cfg.GetString("objectstorage.provider")),
    ProviderConfig: objectstorage.ProviderConfig{
        Endpoint:  cfg.GetString("objectstorage.endpoint"),
        Region:    cfg.GetString("objectstorage.region"),
        AccessKey: cfg.GetString("objectstorage.access_key"),
        SecretKey: cfg.GetString("objectstorage.secret_key"),
        Bucket:    cfg.GetString("objectstorage.bucket"),
        BasePath:  cfg.GetString("objectstorage.base_path"),
    },
})

Best Practices

  • Use local provider for development and testing
  • Use MinIO or cloud providers for production
  • Always close Object instances after use (defer obj.Close())
  • Use BasePath to organize files and support multi-tenancy
  • Validate file sizes before calling Save() to prevent abuse
  • Use Stat() to check file existence before Open()
  • Handle errors appropriately (file not found, permission denied, etc.)
  • For large files, use streaming (Open/Read) instead of loading into memory
  • Consider using signed URLs for direct client uploads (provider-specific)

Error Handling

Common errors:

  • File not found: Open() and Stat() return errors
  • Permission denied: All operations may return permission errors
  • Invalid configuration: NewObjectStorage() returns errors
  • Network errors: MinIO and TOS operations may fail on network issues

Always check returned errors: 

obj, err := storage.Open("file.txt")
if err != nil {
    if errors.Is(err, os.ErrNotExist) {
        // Handle file not found
    }
    return err
}
defer obj.Close()

Provider-Specific Notes

Local Provider:

  • Automatically creates directories as needed
  • BasePath is the root directory
  • Paths are relative to BasePath

MinIO Provider:

  • Supports any S3-compatible service
  • BasePath is used as a key prefix
  • Requires bucket to exist before use

Volcengine TOS Provider:

  • UseInternal=true uses internal endpoint (for in-region VMs)
  • Supports Volcengine-specific features
  • Region must match bucket region
Example

Example demonstrates basic object storage usage with the local provider. 

package main

import (
	"bytes"
	"fmt"
	"io"

	objectstorage "github.com/poly-workshop/go-webmods/objectstorage"
)

func main() {
	storage, err := objectstorage.NewObjectStorage(objectstorage.Config{
		ProviderType: objectstorage.ProviderLocal,
		ProviderConfig: objectstorage.ProviderConfig{
			BasePath: "/tmp/storage",
		},
	})
	if err != nil {
		panic(err)
	}

	// Save a file
	content := bytes.NewReader([]byte("Hello, World!"))
	size, err := storage.Save("hello.txt", content)
	if err != nil {
		panic(err)
	}

	fmt.Printf("Saved %d bytes\n", size)

	// Open and read the file
	obj, err := storage.Open("hello.txt")
	if err != nil {
		panic(err)
	}
	defer func() {
		_ = obj.Close()
	}()

	data, err := io.ReadAll(obj)
	if err != nil {
		panic(err)
	}

	fmt.Printf("Content: %s\n", string(data))
}

Output:

Saved 13 bytes
Content: Hello, World!
Example (Delete)

Example_delete demonstrates deleting a file from storage. 

package main

import (
	"bytes"
	"fmt"

	objectstorage "github.com/poly-workshop/go-webmods/objectstorage"
)

func main() {
	storage, err := objectstorage.NewObjectStorage(objectstorage.Config{
		ProviderType: objectstorage.ProviderLocal,
		ProviderConfig: objectstorage.ProviderConfig{
			BasePath: "/tmp/storage",
		},
	})
	if err != nil {
		panic(err)
	}

	// Save a file
	_, _ = storage.Save("temp.txt", bytes.NewReader([]byte("temporary")))

	// Delete the file
	err = storage.Delete("temp.txt")
	if err != nil {
		panic(err)
	}

	fmt.Println("File deleted")
}

Output:

File deleted
Example (List)

Example_list demonstrates listing objects in a directory. 

package main

import (
	"bytes"
	"fmt"

	objectstorage "github.com/poly-workshop/go-webmods/objectstorage"
)

func main() {
	storage, err := objectstorage.NewObjectStorage(objectstorage.Config{
		ProviderType: objectstorage.ProviderLocal,
		ProviderConfig: objectstorage.ProviderConfig{
			BasePath: "/tmp/storage",
		},
	})
	if err != nil {
		panic(err)
	}

	// Save some files
	_, _ = storage.Save("photos/photo1.jpg", bytes.NewReader([]byte("photo1")))
	_, _ = storage.Save("photos/photo2.jpg", bytes.NewReader([]byte("photo2")))

	// List files in directory
	objects, err := storage.List("photos/")
	if err != nil {
		panic(err)
	}

	fmt.Printf("Found %d files\n", len(objects))
	for _, obj := range objects {
		info, _ := obj.Stat()
		fmt.Printf("- %s (%d bytes)\n", info.Name(), info.Size())
	}
}
Example (Minio)

Example_minio demonstrates using MinIO or S3-compatible storage. 

package main

import (
	"fmt"

	objectstorage "github.com/poly-workshop/go-webmods/objectstorage"
)

func main() {
	storage, err := objectstorage.NewObjectStorage(objectstorage.Config{
		ProviderType: objectstorage.ProviderMinio,
		ProviderConfig: objectstorage.ProviderConfig{
			Endpoint:  "minio.example.com:9000",
			Region:    "us-east-1",
			AccessKey: "minioadmin",
			SecretKey: "minioadmin",
			Bucket:    "mybucket",
			BasePath:  "uploads/",
		},
	})
	if err != nil {
		panic(err)
	}

	_ = storage
	fmt.Println("MinIO storage configured")
}

Output:

MinIO storage configured
Example (Stat)

Example_stat demonstrates getting file metadata without downloading. 

package main

import (
	"bytes"
	"fmt"

	objectstorage "github.com/poly-workshop/go-webmods/objectstorage"
)

func main() {
	storage, err := objectstorage.NewObjectStorage(objectstorage.Config{
		ProviderType: objectstorage.ProviderLocal,
		ProviderConfig: objectstorage.ProviderConfig{
			BasePath: "/tmp/storage",
		},
	})
	if err != nil {
		panic(err)
	}

	// Save a file
	_, _ = storage.Save("document.pdf", bytes.NewReader([]byte("PDF content")))

	// Get file info
	info, err := storage.Stat("document.pdf")
	if err != nil {
		panic(err)
	}

	fmt.Printf("Name: %s\n", info.Name())
	fmt.Printf("Size: %d bytes\n", info.Size())
	fmt.Printf("Modified: %s\n", info.ModTime().Format("2006-01-02"))
}

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Config 

type Config struct {
	ProviderType
	ProviderConfig
}

type FileInfo 

type FileInfo struct {
	// contains filtered or unexported fields
}

func (FileInfo) IsDir 

func (i FileInfo) IsDir() bool

func (FileInfo) ModTime 

func (i FileInfo) ModTime() time.Time

func (FileInfo) Mode 

func (i FileInfo) Mode() os.FileMode

func (FileInfo) Name 

func (i FileInfo) Name() string

func (FileInfo) Size 

func (i FileInfo) Size() int64

func (FileInfo) Sys 

func (i FileInfo) Sys() any

type LocalObject 

type LocalObject struct {
	// contains filtered or unexported fields
}

LocalObject implements Object interface for local files

func (*LocalObject) Close 

func (o *LocalObject) Close() error

Close implements io.Closer

func (*LocalObject) Read 

func (o *LocalObject) Read(p []byte) (n int, err error)

Read implements io.Reader

func (*LocalObject) Seek 

func (o *LocalObject) Seek(offset int64, whence int) (int64, error)

Seek implements io.Seeker

func (*LocalObject) Stat 

func (o *LocalObject) Stat() (os.FileInfo, error)

Stat returns file information

type LocalObjectStorage 

type LocalObjectStorage struct {
	// contains filtered or unexported fields
}

LocalObjectStorage implements ObjectStorage interface for local file system

func NewLocalObjectStorage 

func NewLocalObjectStorage(config ProviderConfig) (*LocalObjectStorage, error)

NewLocalObjectStorage creates a new local object storage instance

func (*LocalObjectStorage) Delete 

func (s *LocalObjectStorage) Delete(path string) error

Delete deletes a file

func (*LocalObjectStorage) List 

func (s *LocalObjectStorage) List(path string) ([]Object, error)

List lists objects in the given path

func (*LocalObjectStorage) Open 

func (s *LocalObjectStorage) Open(path string) (Object, error)

Open opens a file for reading

func (*LocalObjectStorage) Save 

func (s *LocalObjectStorage) Save(path string, r io.Reader) (int64, error)

Save saves a file to the local storage

func (*LocalObjectStorage) Stat 

func (s *LocalObjectStorage) Stat(path string) (os.FileInfo, error)

Stat returns file information

type MinioObject 

type MinioObject struct {
	// contains filtered or unexported fields
}

func (*MinioObject) Close 

func (o *MinioObject) Close() error

func (*MinioObject) Read 

func (o *MinioObject) Read(p []byte) (n int, err error)

func (*MinioObject) Seek 

func (o *MinioObject) Seek(offset int64, whence int) (int64, error)

func (*MinioObject) Stat 

func (o *MinioObject) Stat() (os.FileInfo, error)

type MinioObjectStorage 

type MinioObjectStorage struct {
	// contains filtered or unexported fields
}

func NewMinioObjectStorage 

func NewMinioObjectStorage(config ProviderConfig) (*MinioObjectStorage, error)

func (*MinioObjectStorage) Delete 

func (s *MinioObjectStorage) Delete(path string) error

func (*MinioObjectStorage) List 

func (s *MinioObjectStorage) List(path string) ([]Object, error)

func (*MinioObjectStorage) Open 

func (s *MinioObjectStorage) Open(path string) (Object, error)

func (*MinioObjectStorage) Save 

func (s *MinioObjectStorage) Save(path string, r io.Reader) (int64, error)

func (*MinioObjectStorage) Stat 

func (s *MinioObjectStorage) Stat(path string) (os.FileInfo, error)

type Object 

type Object interface {
	io.ReadSeekCloser
	Stat() (os.FileInfo, error)
}

type ObjectStorage 

type ObjectStorage interface {
	Save(path string, r io.Reader) (int64, error)
	List(path string) ([]Object, error)
	Open(path string) (Object, error)
	Stat(path string) (os.FileInfo, error)
	Delete(path string) error
}

func NewObjectStorage 

func NewObjectStorage(cfg Config) (ObjectStorage, error)

type ProviderConfig 

type ProviderConfig struct {
	Endpoint    string
	Region      string
	AccessKey   string
	SecretKey   string
	Bucket      string
	BasePath    string
	UseInternal bool
}

type ProviderType 

type ProviderType string
const (
	ProviderLocal      ProviderType = "local"
	ProviderVolcengine ProviderType = "volcengine"
	ProviderMinio      ProviderType = "minio"
)

type TOSObject 

type TOSObject struct {
	// contains filtered or unexported fields
}

func (*TOSObject) Close 

func (o *TOSObject) Close() error

func (*TOSObject) Read 

func (o *TOSObject) Read(p []byte) (n int, err error)

func (*TOSObject) Seek 

func (o *TOSObject) Seek(offset int64, whence int) (int64, error)

func (*TOSObject) Stat 

func (o *TOSObject) Stat() (os.FileInfo, error)

type TOSObjectStorage 

type TOSObjectStorage struct {
	// contains filtered or unexported fields
}

func NewTOSObjectStorage 

func NewTOSObjectStorage(config ProviderConfig) (*TOSObjectStorage, error)

func (*TOSObjectStorage) Delete 

func (s *TOSObjectStorage) Delete(path string) error

func (*TOSObjectStorage) List 

func (s *TOSObjectStorage) List(path string) ([]Object, error)

func (*TOSObjectStorage) Open 

func (s *TOSObjectStorage) Open(path string) (Object, error)

func (*TOSObjectStorage) Save 

func (s *TOSObjectStorage) Save(path string, r io.Reader) (int64, error)

func (*TOSObjectStorage) Stat 

func (s *TOSObjectStorage) Stat(path string) (os.FileInfo, error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.