object_storage

package module
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 26, 2025 License: MIT Imports: 20 Imported by: 0

README

Generic Object Storage

A unified Go library for performing CRUD operations on cloud object storage services. Supports Google Cloud Storage (GCS) and Amazon S3 with a consistent interface.

Features

  • Unified Interface: Single IStorageBackend interface works with both GCS and S3
  • Full CRUD Operations: Get, Put, Delete, Copy, and List objects
  • Context Support: All operations accept context for cancellation and timeouts
  • Structured Errors: Consistent error handling with detailed error codes
  • Prefix Support: Organize objects with path prefixes
  • Testable: Interfaces designed for easy mocking in tests

Installation

go get github.com/piyushkumar96/generic-object-storage

Quick Start

Google Cloud Storage
package main

import (
    "context"
    "log"

    storage "github.com/piyushkumar96/generic-object-storage"
)

func main() {
    ctx := context.Background()

    // Create GCS backend
    // Uses Application Default Credentials (set GOOGLE_APPLICATION_CREDENTIALS)
    backend, err := storage.NewGoogleCSBackend(ctx, "my-bucket", "optional/prefix")
    if err != nil {
        log.Fatal(err)
    }

    // Upload an object
    if err := backend.PutObject(ctx, "path/to/file.txt", []byte("Hello, World!")); err != nil {
        log.Fatal(err)
    }

    // Get an object
    obj, err := backend.GetObject(ctx, "path/to/file.txt")
    if err != nil {
        log.Fatal(err)
    }
    log.Printf("Content: %s", string(obj.Content))

    // List objects
    objects, err := backend.GetObjects(ctx, "path/")
    if err != nil {
        log.Fatal(err)
    }
    for _, o := range objects {
        log.Printf("Found: %s", o.Path)
    }

    // Copy an object
    if err := backend.CopyObject(ctx, "path/to/file.txt", "path/to/copy.txt"); err != nil {
        log.Fatal(err)
    }

    // Delete an object
    if err := backend.DeleteObject(ctx, "path/to/file.txt"); err != nil {
        log.Fatal(err)
    }
}
Amazon S3
package main

import (
    "context"
    "log"

    storage "github.com/piyushkumar96/generic-object-storage"
    "github.com/aws/aws-sdk-go/aws/credentials"
)

func main() {
    ctx := context.Background()

    // Option 1: Use default credentials (env vars, ~/.aws/credentials, IAM role)
    backend, err := storage.NewS3Backend("my-bucket", "optional/prefix", "us-east-1", false)
    if err != nil {
        log.Fatal(err)
    }

    // Option 2: Use explicit credentials
    creds := credentials.NewStaticCredentials("ACCESS_KEY", "SECRET_KEY", "")
    backend, err = storage.NewS3BackendWithCredentials("my-bucket", "prefix", "us-east-1", false, creds)
    if err != nil {
        log.Fatal(err)
    }

    // Option 3: Use custom endpoint (MinIO, LocalStack, etc.)
    creds := credentials.NewStaticCredentials("minioadmin", "minioadmin", "")
    backend, err = storage.NewS3BackendWithEndpoint(
        "my-bucket",
        "prefix",
        "us-east-1",
        "http://localhost:9000",
        true, // disableSSL
        creds,
    )
    if err != nil {
        log.Fatal(err)
    }

    // All operations are identical to GCS
    if err := backend.PutObject(ctx, "test.txt", []byte("Hello from S3!")); err != nil {
        log.Fatal(err)
    }
}

API Reference

Interface
type IStorageBackend interface {
    GetObject(ctx context.Context, path string) (Object, *ae.AppError)
    GetObjects(ctx context.Context, prefix string) ([]Object, *ae.AppError)
    PutObject(ctx context.Context, path string, content []byte) *ae.AppError
    DeleteObject(ctx context.Context, path string) *ae.AppError
    CopyObject(ctx context.Context, srcPath, dstPath string) *ae.AppError
}
Object Structure
type Object struct {
    Meta         Metadata
    Path         string
    Content      []byte
    LastModified time.Time
}

type Metadata struct {
    Name    string
    Version string
}
Constructor Functions
Google Cloud Storage
// NewGoogleCSBackend creates a GCS backend using Application Default Credentials
func NewGoogleCSBackend(ctx context.Context, bucket string, prefix string) (*GoogleCSBackend, *ae.AppError)
Amazon S3
// NewS3Backend creates an S3 backend using default credential chain
func NewS3Backend(bucket string, prefix string, region string, disableSSL bool) (*S3Backend, *ae.AppError)

// NewS3BackendWithCredentials creates an S3 backend with explicit credentials
func NewS3BackendWithCredentials(bucket string, prefix string, region string, disableSSL bool, creds *credentials.Credentials) (*S3Backend, *ae.AppError)

// NewS3BackendWithEndpoint creates an S3 backend with custom endpoint (for S3-compatible services)
func NewS3BackendWithEndpoint(bucket string, prefix string, region string, endpoint string, disableSSL bool, creds *credentials.Credentials) (*S3Backend, *ae.AppError)

Error Handling

The library uses structured errors with error codes for easy identification:

GCS Error Codes
Code Description
ERR_OS_GCS_1000 Failed to initialize GCS client
ERR_OS_GCS_1001 Error getting objects from GCS
ERR_OS_GCS_1002 Error getting single object from GCS
ERR_OS_GCS_1003 Error putting object to GCS
ERR_OS_GCS_1004 Error deleting object from GCS
ERR_OS_GCS_1005 Error copying object in GCS
S3 Error Codes
Code Description
ERR_OS_S3_2000 Failed to initialize S3 client
ERR_OS_S3_2001 Error getting objects from S3
ERR_OS_S3_2002 Error getting single object from S3
ERR_OS_S3_2003 Error putting object to S3
ERR_OS_S3_2004 Error deleting object from S3
ERR_OS_S3_2005 Error copying object in S3

Authentication

Google Cloud Storage

GCS uses Application Default Credentials (ADC):

  1. Service Account Key (recommended for production):

    export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"

  2. User Credentials (for development):

    gcloud auth application-default login

  3. Workload Identity (for GKE)

Amazon S3

S3 supports multiple authentication methods via the AWS SDK credential chain:

  1. Environment Variables:

    export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export AWS_REGION="us-east-1"

  2. Shared Credentials File (~/.aws/credentials)

  3. IAM Role (for EC2, ECS, Lambda)

  4. Explicit Credentials (using NewS3BackendWithCredentials)

Testing with Mocks

The library includes mock implementations for testing:

import (
    "testing"
    "context"

    storage "github.com/piyushkumar96/generic-object-storage"
    "github.com/piyushkumar96/generic-object-storage/mocks"
)

func TestMyFunction(t *testing.T) {
    mockBackend := mocks.NewMockIStorageBackend(t)
    
    // Set up expectations
    mockBackend.On("GetObject", mock.Anything, "test.txt").Return(
        storage.Object{Content: []byte("test data")},
        nil,
    )
    
    // Use mockBackend in your tests
    obj, err := mockBackend.GetObject(context.Background(), "test.txt")
    // ... assertions
}

Examples

See the examples directory for complete working examples:

# Run GCS example
STORAGE_TYPE=gcs GCS_BUCKET=my-bucket go run examples/example.go

# Run S3 example  
STORAGE_TYPE=s3 S3_BUCKET=my-bucket AWS_REGION=us-east-1 go run examples/example.go

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	GoogleCSBackendClient = ae.GetCustomErr("ERR_OS_GCS_1000",
		"failed to initialise the gcs client", false)
	GCSGetObjects = ae.GetCustomErr("ERR_OS_GCS_1001",
		"error while getting objects from gcs bucket", false)
	GCSGetObject = ae.GetCustomErr("ERR_OS_GCS_1002",
		"error while getting object from gcs bucket", false)
	GCSPutObject = ae.GetCustomErr("ERR_OS_GCS_1003",
		"error while putting object to gcs bucket", false)
	GCSDeleteObject = ae.GetCustomErr("ERR_OS_GCS_1004",
		"error while deleting object from gcs bucket", false)
	GCSCopyObject = ae.GetCustomErr("ERR_OS_GCS_1005",
		"error while copying object in gcs bucket", false)
)

GCS (Google Cloud Storage) error definitions

View Source 
var (
	S3BackendClient = ae.GetCustomErr("ERR_OS_S3_2000",
		"failed to initialise the s3 client", false)
	S3GetObjects = ae.GetCustomErr("ERR_OS_S3_2001",
		"error while getting objects from s3 bucket", false)
	S3GetObject = ae.GetCustomErr("ERR_OS_S3_2002",
		"error while getting object from s3 bucket", false)
	S3PutObject = ae.GetCustomErr("ERR_OS_S3_2003",
		"error while putting object to s3 bucket", false)
	S3DeleteObject = ae.GetCustomErr("ERR_OS_S3_2004",
		"error while deleting object from s3 bucket", false)
	S3CopyObject = ae.GetCustomErr("ERR_OS_S3_2005",
		"error while copying object in s3 bucket", false)
)

S3 (Amazon S3) error definitions

Functions

This section is empty.

Types

type GoogleCSBackend 

type GoogleCSBackend struct {
	Prefix string
	Client IGCSClient
}

GoogleCSBackend is a storage backend for Google Cloud Storage

func NewGoogleCSBackend 

func NewGoogleCSBackend(ctx context.Context, bucket string, prefix string) (*GoogleCSBackend, *ae.AppError)

NewGoogleCSBackend creates a new instance of GoogleCSBackend

func (GoogleCSBackend) CopyObject 

func (b GoogleCSBackend) CopyObject(ctx context.Context, srcPath, dstPath string) *ae.AppError

CopyObject copy an object from Google Cloud Storage bucket one path to another

func (GoogleCSBackend) DeleteObject 

func (b GoogleCSBackend) DeleteObject(ctx context.Context, path string) *ae.AppError

DeleteObject removes an object from Google Cloud Storage bucket, at prefix

func (GoogleCSBackend) GetObject 

func (b GoogleCSBackend) GetObject(ctx context.Context, path string) (Object, *ae.AppError)

GetObject retrieves an object from Google Cloud Storage bucket, at prefix

func (GoogleCSBackend) GetObjects 

func (b GoogleCSBackend) GetObjects(ctx context.Context, prefix string) ([]Object, *ae.AppError)

GetObjects lists all objects in Google Cloud Storage bucket, at prefix

func (GoogleCSBackend) PutObject 

func (b GoogleCSBackend) PutObject(ctx context.Context, path string, content []byte) *ae.AppError

PutObject uploads an object to Google Cloud Storage bucket, at prefix

type IGCSClient 

type IGCSClient interface {
	Objects(ctx context.Context, q *storage.Query) *storage.ObjectIterator
	Object(name string) *storage.ObjectHandle
}

IGCSClient this interface is added to make Client ins GCS BucketHandle mock compatible for tests

type IS3Client 

type IS3Client interface {
	ListObjectsWithContext(ctx aws.Context, input *s3.ListObjectsInput, opts ...request.Option) (*s3.ListObjectsOutput, error)
	GetObjectWithContext(ctx aws.Context, input *s3.GetObjectInput, opts ...request.Option) (*s3.GetObjectOutput, error)
	DeleteObjectWithContext(ctx aws.Context, input *s3.DeleteObjectInput, opts ...request.Option) (*s3.DeleteObjectOutput, error)
	CopyObjectWithContext(ctx aws.Context, input *s3.CopyObjectInput, opts ...request.Option) (*s3.CopyObjectOutput, error)
}

IS3Client interface for S3 client operations - allows mocking in tests

type IS3Uploader 

type IS3Uploader interface {
	UploadWithContext(ctx aws.Context, input *s3manager.UploadInput, opts ...func(*s3manager.Uploader)) (*s3manager.UploadOutput, error)
}

IS3Uploader interface for S3 upload operations - allows mocking in tests

type IStorageBackend 

type IStorageBackend interface {
	// GetObject retrieves a single object from the storage bucket
	GetObject(ctx context.Context, path string) (Object, *ae.AppError)
	// GetObjects lists all objects at the given prefix
	GetObjects(ctx context.Context, prefix string) ([]Object, *ae.AppError)
	// PutObject uploads an object to the storage bucket
	PutObject(ctx context.Context, path string, content []byte) *ae.AppError
	// DeleteObject removes an object from the storage bucket
	DeleteObject(ctx context.Context, path string) *ae.AppError
	// CopyObject copies an object from source path to destination path
	CopyObject(ctx context.Context, srcPath, dstPath string) *ae.AppError
}

IStorageBackend defines the interface for storage backend implementations Both S3Backend and GoogleCSBackend implement this interface

type Metadata 

type Metadata struct {
	Name    string
	Version string
}

Metadata contains additional information about the object

type Object 

type Object struct {
	Meta         Metadata
	Path         string
	Content      []byte
	LastModified time.Time
}

Object represents a storage object with its metadata and content

type S3Backend 

type S3Backend struct {
	Bucket     string
	Client     IS3Client
	Downloader *s3manager.Downloader
	Prefix     string
	Uploader   IS3Uploader
}

S3Backend implements IStorageBackend for Amazon S3

func NewS3Backend 

func NewS3Backend(bucket string, prefix string, region string, disableSSL bool) (*S3Backend, *ae.AppError)

NewS3Backend creates a new instance of S3Backend using default credentials

func NewS3BackendWithCredentials 

func NewS3BackendWithCredentials(bucket string, prefix string, region string, disableSSL bool, creds *credentials.Credentials) (*S3Backend, *ae.AppError)

NewS3BackendWithCredentials creates a new instance of S3Backend with explicit credentials

func NewS3BackendWithEndpoint 

func NewS3BackendWithEndpoint(bucket string, prefix string, region string, endpoint string, disableSSL bool, creds *credentials.Credentials) (*S3Backend, *ae.AppError)

NewS3BackendWithEndpoint creates a new instance of S3Backend with custom endpoint (for S3-compatible services like MinIO)

func (*S3Backend) CopyObject 

func (b *S3Backend) CopyObject(ctx context.Context, srcPath, dstPath string) *ae.AppError

CopyObject copies an object within Amazon S3 bucket

func (*S3Backend) DeleteObject 

func (b *S3Backend) DeleteObject(ctx context.Context, path string) *ae.AppError

DeleteObject removes an object from Amazon S3 bucket

func (*S3Backend) GetObject 

func (b *S3Backend) GetObject(ctx context.Context, path string) (Object, *ae.AppError)

GetObject retrieves an object from Amazon S3 bucket

func (*S3Backend) GetObjects 

func (b *S3Backend) GetObjects(ctx context.Context, prefix string) ([]Object, *ae.AppError)

GetObjects lists all objects in Amazon S3 bucket at the given prefix

func (*S3Backend) PutObject 

func (b *S3Backend) PutObject(ctx context.Context, path string, content []byte) *ae.AppError

PutObject uploads an object to Amazon S3 bucket

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.