upload

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 7, 2026 License: MIT Imports: 26 Imported by: 0

README

go-upload

Object storage library for Go: upload, download, list, and multipart transfers across major cloud providers.

Install

go get github.com/andywang2011/go-upload@v0.1.0

Quick start

cfg := upload.ConfigDefault()
cfg.Provider.Name = upload.ProviderAliyun
cfg.Aliyun = upload.AliyunConfig{
	Endpoint:        "oss-cn-hangzhou.aliyuncs.com",
	AccessKeyID:     "...",
	AccessKeySecret: "...",
	Bucket:          "my-bucket",
}
store, err := upload.NewStorage(ctx, cfg)
if err != nil {
	// handle
}
defer store.Close()

Examples

cd go-upload/examples/aliyun && go run .
cd go-upload/examples/cos && go run .

Subpackages for browser STS upload: coscred (Tencent COS), osscred (Aliyun OSS).

License

MIT — see LICENSE.

Documentation

Overview

AWS S3 driver (provider: aws). Uses aws-sdk-go-v2/service/s3 under the hood.

AWS client construction (provider: aws).

Microsoft Azure Blob driver (provider: azure).

Tencent COS / WeChat COS driver (providers: tencent, wechat).

Package upload provides a thin, testable object storage client for major cloud providers.

Design constraints (for use in application frameworks):

  • No viper, no global singletons, no log.Fatal inside the library.
  • Storage interface covers upload, download, list, delete, and multipart transfers.
  • One Storage per process; reuse it across goroutines.
  • Call Close on shutdown.

Supported providers: Aliyun OSS, Tencent COS, WeChat (COS config), AWS S3, Azure Blob.

Aliyun OSS driver (provider: aliyun).

Index

Examples

Constants

View Source
const (
	ProviderAliyun  = "aliyun"
	ProviderTencent = "tencent"
	ProviderWechat  = "wechat"
	ProviderAWS     = "aws"
	ProviderAzure   = "azure"
)

Variables

View Source
var (
	ErrInvalidConfig  = errors.New("upload: invalid config")
	ErrNotInitialized = errors.New("upload: not initialized")
	ErrNotFound       = errors.New("upload: object not found")
	ErrUnsupported    = errors.New("upload: unsupported operation")
	ErrObjectTooLarge = errors.New("upload: object too large for simple upload")
	// ErrInvalidRange is returned for negative offsets, non-positive lengths, or offset+length overflow.
	ErrInvalidRange = errors.New("upload: invalid range")
)

Functions

func PublicURLFromHost

func PublicURLFromHost(host, key string) string

PublicURLFromHost joins CDN host and object key.

func SetMemObject

func SetMemObject(s Storage, key string, data []byte) error

SetMemObject seeds mem backend when buildBackend is overridden in tests.

Types

type AWSConfig

type AWSConfig struct {
	// Region is the AWS region (e.g. us-east-1).
	Region string
	// AccessKeyID is the AWS access key ID.
	AccessKeyID string
	// SecretAccessKey is the AWS secret access key.
	SecretAccessKey string
	// SessionToken is the optional STS session token.
	SessionToken string
	// Bucket is the S3 bucket name.
	Bucket string
	// Endpoint is an optional custom S3-compatible endpoint.
	Endpoint string
	// CDNHost is the optional CDN base URL for public object URLs.
	CDNHost string
}

AWSConfig is Amazon S3 settings.

type AliyunConfig

type AliyunConfig struct {
	// Endpoint is the OSS endpoint URL.
	Endpoint string
	// AccessKeyID is the Aliyun access key ID.
	AccessKeyID string
	// AccessKeySecret is the Aliyun access key secret.
	AccessKeySecret string
	// Bucket is the target OSS bucket name.
	Bucket string
	// CDNHost is the optional CDN base URL for public object URLs.
	CDNHost string
}

AliyunConfig is Aliyun OSS settings.

type AzureConfig

type AzureConfig struct {
	// AccountName is the storage account name.
	AccountName string
	// AccountKey is the storage account access key.
	AccountKey string
	// Container is the blob container name.
	Container string
	// Endpoint is an optional custom blob endpoint.
	Endpoint string
	// CDNHost is the optional CDN base URL for public object URLs.
	CDNHost string
}

AzureConfig is Azure Blob settings.

type COSConfig

type COSConfig struct {
	// Region is the COS region (e.g. ap-shanghai).
	Region string
	// SecretID is the Tencent Cloud secret ID.
	SecretID string
	// SecretKey is the Tencent Cloud secret key.
	SecretKey string
	// Bucket is the bucket name without app ID suffix.
	Bucket string
	// AppID is the Tencent Cloud app ID appended to the bucket.
	AppID string
	// CDNHost is the optional CDN base URL for public object URLs.
	CDNHost string
	// BaseURL is an optional custom API base URL.
	BaseURL string
}

COSConfig is Tencent/WeChat COS settings.

type Config

type Config struct {
	// Provider selects the active storage backend.
	Provider ProviderSettings
	// Aliyun holds Aliyun OSS credentials when Provider is aliyun.
	Aliyun AliyunConfig
	// Tencent holds Tencent COS credentials when Provider is tencent.
	Tencent COSConfig
	// Wechat holds WeChat COS credentials when Provider is wechat.
	Wechat COSConfig
	// AWS holds Amazon S3 credentials when Provider is aws.
	AWS AWSConfig
	// Azure holds Azure Blob credentials when Provider is azure.
	Azure AzureConfig
	// Multipart tunes chunked upload/download behavior.
	Multipart MultipartConfig
	// Retry controls startup connectivity retry.
	Retry RetryConfig
}

Config holds provider credentials and tuning.

func ConfigDefault

func ConfigDefault() Config

ConfigDefault returns tuning defaults (provider block must still be set).

func (*Config) Validate

func (c *Config) Validate() error

Validate applies defaults and checks provider-specific required fields.

type GetOption

type GetOption func(*getOptions)

GetOption configures downloads.

func GetWithMultipart

func GetWithMultipart() GetOption

GetWithMultipart forces ranged/multipart download.

type ListOption

type ListOption func(*listOptions)

ListOption configures list operations.

func WithContinuationToken

func WithContinuationToken(token string) ListOption

WithContinuationToken continues a previous list page.

func WithDelimiter

func WithDelimiter(d string) ListOption

WithDelimiter sets folder delimiter (e.g. "/").

func WithMaxKeys

func WithMaxKeys(n int) ListOption

WithMaxKeys limits keys per page.

type ListPage

type ListPage struct {
	NextContinuationToken string
	IsTruncated           bool
}

ListPage carries pagination for List.

type MultipartConfig

type MultipartConfig struct {
	// PartSize is the upload part size in bytes.
	PartSize int64
	// Concurrency is the number of parallel part uploads.
	Concurrency int
	// Threshold is the file size above which multipart upload is used.
	Threshold int64
	// DownloadPartSize is the chunk size for ranged downloads.
	DownloadPartSize int64
}

MultipartConfig controls chunked transfers.

type ObjectInfo

type ObjectInfo struct {
	Key          string
	Size         int64
	LastModified int64 // unix seconds when unknown
	ETag         string
}

ObjectInfo describes one listed object.

type ProviderSettings

type ProviderSettings struct {
	// Name is one of ProviderAliyun, ProviderTencent, ProviderWechat, ProviderAWS, ProviderAzure.
	Name string
}

ProviderSettings selects active backend.

type PutOption

type PutOption func(*putOptions)

PutOption configures uploads.

func WithContentLength

func WithContentLength(n int64) PutOption

WithContentLength sets known object size (enables streaming upload optimizations).

func WithContentType

func WithContentType(ct string) PutOption

WithContentType sets Content-Type.

func WithMultipart

func WithMultipart() PutOption

WithMultipart forces multipart upload regardless of threshold.

type RetryConfig

type RetryConfig struct {
	// MaxAttempts is the total number of connection attempts.
	MaxAttempts int
	// InitialDelay is the base delay before the first retry.
	InitialDelay time.Duration
	// MaxDelay caps exponential backoff delay.
	MaxDelay time.Duration
	// Multiplier scales delay after each failed attempt.
	Multiplier float64
}

RetryConfig controls startup connectivity retry.

func DefaultRetryConfig

func DefaultRetryConfig() RetryConfig

DefaultRetryConfig returns startup retry defaults.

type Storage

type Storage interface {
	Upload(ctx context.Context, key string, r io.Reader, opts ...PutOption) error
	Download(ctx context.Context, key string, w io.Writer, opts ...GetOption) error
	UploadLarge(ctx context.Context, key string, r io.Reader, size int64, opts ...PutOption) error
	DownloadLarge(ctx context.Context, key string, w io.Writer, size int64, opts ...GetOption) error
	List(ctx context.Context, prefix string, opts ...ListOption) ([]ObjectInfo, *ListPage, error)
	Delete(ctx context.Context, key string) error
	PublicURL(key string) string
	Close() error
}

Storage is the primary object storage abstraction.

func NewStorage

func NewStorage(ctx context.Context, cfg Config) (Storage, error)

NewStorage creates a storage client with optional startup ping retry. ctx bounds the startup ping and its retries.

Example
package main

import (
	"context"
	"github.com/andywang2011/go-upload"
)

func main() {
	cfg := upload.ConfigDefault()
	cfg.Provider.Name = upload.ProviderAliyun
	cfg.Aliyun = upload.AliyunConfig{
		Endpoint:        "oss-cn-hangzhou.aliyuncs.com",
		AccessKeyID:     "...",
		AccessKeySecret: "...",
		Bucket:          "my-bucket",
	}
	store, err := upload.NewStorage(context.Background(), cfg)
	if err != nil {
		return
	}
	defer store.Close()
}

type Uploader

type Uploader interface {
	Upload(ctx context.Context, path string, reader io.Reader) error
	CdnUrl(path string) string
	BaseURL() string
}

Uploader is the legacy upload-only interface used by demo-api.

func AsUploader

func AsUploader(s Storage) Uploader

AsUploader returns a Uploader view of Storage.

Directories

Path Synopsis
Package coscred issues temporary Tencent Cloud COS credentials for client-side uploads.
Package coscred issues temporary Tencent Cloud COS credentials for client-side uploads.
Package osscred issues Aliyun STS temporary credentials and OSS post policies for client-side uploads.
Package osscred issues Aliyun STS temporary credentials and OSS post policies for client-side uploads.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL