storage

package module

v0.7.0 Latest Latest Go to latest Published: Jun 2, 2026 License: MIT Imports: 21 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/tigrisdata/storage-go

Links

Open Source Insights

README ¶

Tigris Storage SDK for Go

Welcome to the Tigris Storage SDK for Go! This package contains high-level wrappers and helpers to help you take advantage of all of Tigris' features.

Overview

Tigris is a cloud storage service that provides a simple, scalable, and secure object storage solution. It is based on the S3 API, but has additional features that need these helpers.

This SDK provides the main storage package containing the Tigris client with S3-compatible methods plus Tigris-specific features like bucket forking, snapshots, and object renaming.

Installation

go get github.com/tigrisdata/storage-go

Quick Start

package main

import (
    "context"
    "fmt"
    "log"

    storage "github.com/tigrisdata/storage-go"
)

func main() {
    ctx := context.Background()

    // Create a new Tigris client
    client, err := storage.New(ctx)
    if err != nil {
        log.Fatal(err)
    }

    // Use the client like you would use AWS S3 client
    // plus Tigris-specific features (see below)
    fmt.Println("Connected to Tigris!")
}

Client Configuration

The New() function creates a new S3 client optimized for interactions with Tigris. It accepts functional options to configure the client:

client, err := storage.New(ctx,
    storage.WithFlyEndpoint(),           // Use fly.io optimized endpoint
    storage.WithGlobalEndpoint(),        // Use globally available endpoint (default)
    storage.WithRegion("auto"),          // Specify a region
    storage.WithAccessKeypair(key, secret), // Set access credentials
)

Configuration Options

Option	Description
`WithFlyEndpoint()`	Connect to Tigris' fly.io optimized endpoint. If you are deployed to fly.io, this zero-rates your traffic to Tigris.
`WithGlobalEndpoint()`	Connect to Tigris' globally available endpoint (`https://t3.storage.dev`). This is the default.
`WithRegion(region)`	Statically specify a region for interacting with Tigris. You will almost certainly never need this.
`WithAccessKeypair(accessKeyID, secretAccessKey)`	Specify a custom access key and secret access key. Useful when loading credentials from non-standard locations.

Bucket Features

Snapshots and Forks

Tigris supports bucket snapshots and forking, allowing you to create point-in-time copies of buckets and branch from them.

Create a Snapshot Enabled Bucket

output, err := client.CreateSnapshotEnabledBucket(ctx, &s3.CreateBucketInput{
    Bucket: aws.String("my-bucket"),
})

Create a Snapshot

output, err := client.CreateBucketSnapshot(ctx, "Initial backup", &s3.CreateBucketInput{
    Bucket: aws.String("my-bucket"),
})
// output.SnapshotVersion holds the version returned by Tigris.

Fork a Bucket

// Creates a new bucket "my-bucket-fork" as a fork of "my-bucket"
output, err := client.CreateBucketFork(ctx, "my-bucket", "my-bucket-fork")

List Snapshots

snapshots, err := client.ListBucketSnapshots(ctx, "my-bucket")

Get Fork/Snapshot Metadata

info, err := client.HeadBucketForkOrSnapshot(ctx, &s3.HeadBucketInput{
    Bucket: aws.String("my-bucket"),
})
// info.SnapshotsEnabled     - true if snapshots are enabled
// info.SourceBucket         - The bucket this was forked from
// info.SourceBucketSnapshot - The snapshot this was forked from
// info.IsForkParent         - true if there are forks of this bucket

Object Features

Rename Objects

Tigris supports in-place object renaming without copying data:

_, err := client.RenameObject(ctx, &s3.CopyObjectInput{
    Bucket:     aws.String("my-bucket"),
    CopySource: aws.String("my-bucket/old-name.txt"),
    Key:        aws.String("new-name.txt"),
})

Documentation

For more information on Tigris features, see:

License

See LICENSE for details.

Documentation ¶

Overview ¶

Package storage contains a Tigris client and helpers for interacting with Tigris.

Tigris is a cloud storage service that provides a simple, scalable, and secure object storage solution. It is based on the S3 API, but has additional features that need these helpers.

Index ¶

Constants
type BundleObjectsInput
type BundleObjectsOutput
type Client
- func New(ctx context.Context, options ...Option) (*Client, error)
type CreateBucketSnapshotOutput
type HeadBucketForkOrSnapshotOutput
type Option
type Options

Constants ¶

View Source

const (
	// BundleFormatTar is the tar archive format for bundle responses.
	BundleFormatTar = "tar"

	// BundleCompressionNone disables compression (default).
	BundleCompressionNone = "none"
	// BundleCompressionGzip enables gzip compression.
	BundleCompressionGzip = "gzip"
	// BundleCompressionZstd enables zstd compression.
	BundleCompressionZstd = "zstd"

	// BundleOnErrorSkip silently omits missing objects from the archive (default).
	BundleOnErrorSkip = "skip"
	// BundleOnErrorFail returns an error if any object is missing.
	BundleOnErrorFail = "fail"
)

View Source

const (
	GlobalEndpoint = "https://t3.storage.dev"
	FlyEndpoint    = "https://fly.storage.tigris.dev"
)

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type BundleObjectsInput ¶ added in v0.6.0

type BundleObjectsInput struct {
	// Bucket is the name of the bucket containing the objects. Required.
	Bucket string

	// Keys is the list of object keys to include in the bundle. Required.
	// Maximum 5,000 keys per request.
	Keys []string

	// Compression sets the compression algorithm for the response.
	// Valid values: "none" (default), "gzip", "zstd".
	Compression string

	// OnError controls behavior when objects are missing.
	// "skip" (default): omit missing objects and append __bundle_errors.json to the tar.
	// "fail": return an error before streaming if any object is missing.
	OnError string
}

BundleObjectsInput is the input for a BundleObjects request.

type BundleObjectsOutput ¶ added in v0.6.0

type BundleObjectsOutput struct {
	// Body is the streaming tar archive. Must be closed by the caller.
	Body io.ReadCloser

	// ContentType is the response Content-Type (e.g. "application/x-tar", "application/gzip").
	ContentType string

	// StatusCode is the HTTP status code of the response.
	StatusCode int
}

BundleObjectsOutput is the response from a BundleObjects request.

The Body contains a streaming tar archive. Callers are responsible for closing Body. Use archive/tar to iterate entries:

tr := tar.NewReader(output.Body)
for {
    hdr, err := tr.Next()
    if err == io.EOF { break }
    // process hdr.Name, tr
}

If compression was requested, wrap Body with the appropriate decompressor first:

gz, _ := gzip.NewReader(output.Body)
tr := tar.NewReader(gz)

type Client ¶

type Client struct {
	*s3.Client
}

Client is a wrapper around the AWS SDK S3 Client with additional methods for integration with Tigris.

func New ¶

func New(ctx context.Context, options ...Option) (*Client, error)

New returns a new S3 client optimized for interactions with Tigris.

Example ¶

package main

import (
	"context"
	"log"

	storage "github.com/tigrisdata/storage-go"

	_ "github.com/joho/godotenv/autoload"
)

func main() {
	ctx := context.Background()

	// Create a new Tigris client with default options
	client, err := storage.New(ctx)
	if err != nil {
		log.Fatal(err)
	}
	_ = client

	// Create a client with custom options
	client, err = storage.New(ctx,
		storage.WithFlyEndpoint(),    // Use fly.io optimized endpoint
		storage.WithGlobalEndpoint(), // Use globally available endpoint (default)
		storage.WithRegion("auto"),   // Specify a region
		// storage.WithAccessKeypair(key, secret), // Set access credentials
	)
	if err != nil {
		log.Fatal(err)
	}
	_ = client
}

Output:

func (*Client) BundleObjects ¶ added in v0.6.0

func (c *Client) BundleObjects(ctx context.Context, in *BundleObjectsInput) (*BundleObjectsOutput, error)

BundleObjects fetches multiple objects from a bucket as a streaming tar archive in a single HTTP request.

This is a Tigris extension to the S3 API, designed for ML training workloads that need to fetch thousands of objects per batch without per-object HTTP overhead.

The caller is responsible for closing the returned Body.

Example ¶

package main

import (
	"archive/tar"
	"context"
	"io"
	"log"

	storage "github.com/tigrisdata/storage-go"

	_ "github.com/joho/godotenv/autoload"
)

func main() {
	ctx := context.Background()

	client, err := storage.New(ctx)
	if err != nil {
		log.Fatal(err) // handle the error here
	}

	// Fetch multiple objects as a streaming tar archive in a single request.
	output, err := client.BundleObjects(ctx, &storage.BundleObjectsInput{
		Bucket: "my-dataset-bucket",
		Keys: []string{
			"train/img_001.jpg",
			"train/img_002.jpg",
			"train/img_003.jpg",
		},
	})
	if err != nil {
		log.Fatal(err) // handle the error here
	}
	defer output.Body.Close()

	// Iterate tar entries as they arrive.
	tr := tar.NewReader(output.Body)
	for {
		hdr, err := tr.Next()
		if err == io.EOF {
			break
		}
		if err != nil {
			log.Fatal(err) // handle the error here
		}

		// Read the object content.
		data, err := io.ReadAll(tr)
		if err != nil {
			log.Fatal(err) // handle the error here
		}

		_ = hdr.Name // object key, e.g. "train/img_001.jpg"
		_ = data     // object content
	}
}

Output:

func (*Client) CreateBucketFork ¶

func (c *Client) CreateBucketFork(ctx context.Context, source, target string, opts ...func(*s3.Options)) (*s3.CreateBucketOutput, error)

CreateBucketFork creates a fork of the source bucket named target.

If you want to specify an exact snapshot version to fork from, use tigrisheaders.WithSnapshotVersion.

Example ¶

package main

import (
	"context"
	"log"

	storage "github.com/tigrisdata/storage-go"

	_ "github.com/joho/godotenv/autoload"
)

func main() {
	ctx := context.Background()

	client, err := storage.New(ctx)
	if err != nil {
		log.Fatal(err)
	}

	// Creates a new bucket "my-bucket-fork" as a fork of "my-bucket"
	output, err := client.CreateBucketFork(ctx, "my-bucket", "my-bucket-fork")
	if err != nil {
		log.Fatal(err)
	}
	_ = output
}

Output:

func (*Client) CreateBucketSnapshot ¶

func (c *Client) CreateBucketSnapshot(ctx context.Context, description string, in *s3.CreateBucketInput, opts ...func(*s3.Options)) (*CreateBucketSnapshotOutput, error)

CreateBucketSnapshot creates a snapshot with the given description for a bucket. The returned output carries the snapshot version in SnapshotVersion.

Example ¶

package main

import (
	"context"
	"log"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/service/s3"

	storage "github.com/tigrisdata/storage-go"

	_ "github.com/joho/godotenv/autoload"
)

func main() {
	ctx := context.Background()

	client, err := storage.New(ctx)
	if err != nil {
		log.Fatal(err)
	}

	// Create a snapshot with a description
	output, err := client.CreateBucketSnapshot(ctx, "Initial backup", &s3.CreateBucketInput{
		Bucket: aws.String("my-bucket"),
	})
	if err != nil {
		log.Fatal(err)
	}
	_ = output
}

Output:

func (*Client) CreateSnapshotEnabledBucket ¶

func (c *Client) CreateSnapshotEnabledBucket(ctx context.Context, in *s3.CreateBucketInput, opts ...func(*s3.Options)) (*s3.CreateBucketOutput, error)

CreateSnapshotEnabledBucket creates a new bucket with the ability to take snapshots and fork the contents of it.

Example ¶

package main

import (
	"context"
	"log"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/service/s3"

	storage "github.com/tigrisdata/storage-go"

	_ "github.com/joho/godotenv/autoload"
)

func main() {
	ctx := context.Background()

	client, err := storage.New(ctx)
	if err != nil {
		log.Fatal(err)
	}

	// Create a bucket with snapshot support enabled
	output, err := client.CreateSnapshotEnabledBucket(ctx, &s3.CreateBucketInput{
		Bucket: aws.String("my-bucket"),
	})
	if err != nil {
		log.Fatal(err)
	}
	_ = output
}

Output:

func (*Client) HeadBucketForkOrSnapshot ¶

func (c *Client) HeadBucketForkOrSnapshot(ctx context.Context, in *s3.HeadBucketInput, opts ...func(*s3.Options)) (*HeadBucketForkOrSnapshotOutput, error)

HeadBucketForkOrSnapshot fetches the fork/snapshot metadata for a bucket.

For more information, see the Tigris documentation1.

Example ¶

package main

import (
	"context"
	"log"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/service/s3"

	storage "github.com/tigrisdata/storage-go"

	_ "github.com/joho/godotenv/autoload"
)

func main() {
	ctx := context.Background()

	client, err := storage.New(ctx)
	if err != nil {
		log.Fatal(err)
	}

	// Get fork/snapshot metadata for a bucket
	info, err := client.HeadBucketForkOrSnapshot(ctx, &s3.HeadBucketInput{
		Bucket: aws.String("my-bucket"),
	})
	if err != nil {
		log.Fatal(err)
	}

	_ = info.SnapshotsEnabled     // true if snapshots are enabled
	_ = info.SourceBucket         // The bucket this was forked from
	_ = info.SourceBucketSnapshot // The snapshot this was forked from
	_ = info.IsForkParent         // true if there are forks of this bucket
}

Output:

func (*Client) ListBucketSnapshots ¶

func (c *Client) ListBucketSnapshots(ctx context.Context, bucketName string, opts ...func(*s3.Options)) (*s3.ListBucketsOutput, error)

ListBucketSnapshots lists the snapshots for a bucket.

For more information, see the Tigris documentation1.

Example ¶

package main

import (
	"context"
	"log"

	storage "github.com/tigrisdata/storage-go"

	_ "github.com/joho/godotenv/autoload"
)

func main() {
	ctx := context.Background()

	client, err := storage.New(ctx)
	if err != nil {
		log.Fatal(err)
	}

	// List all snapshots for a bucket
	snapshots, err := client.ListBucketSnapshots(ctx, "my-bucket")
	if err != nil {
		log.Fatal(err)
	}
	_ = snapshots
}

Output:

func (*Client) RenameObject ¶

func (c *Client) RenameObject(ctx context.Context, in *s3.CopyObjectInput, opts ...func(*s3.Options)) (*s3.CopyObjectOutput, error)

RenameObject performs an in-place rename of objects instead of copying the data.

For more information, see the Tigris documentation1.

Example ¶

package main

import (
	"context"
	"log"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/service/s3"

	storage "github.com/tigrisdata/storage-go"

	_ "github.com/joho/godotenv/autoload"
)

func main() {
	ctx := context.Background()

	client, err := storage.New(ctx)
	if err != nil {
		log.Fatal(err)
	}

	// Rename an object in-place without copying data
	_, err = client.RenameObject(ctx, &s3.CopyObjectInput{
		Bucket:     aws.String("my-bucket"),
		CopySource: aws.String("my-bucket/old-name.txt"),
		Key:        aws.String("new-name.txt"),
	})
	if err != nil {
		log.Fatal(err)
	}
}

Output:

func (*Client) S3 ¶ added in v0.7.0

func (c *Client) S3() *s3.Client

S3 returns the underlying S3 client.

type CreateBucketSnapshotOutput ¶ added in v0.7.0

type CreateBucketSnapshotOutput struct {
	*s3.CreateBucketOutput

	// SnapshotVersion is the version identifier of the snapshot just created.
	// Empty if the server did not return a version header.
	SnapshotVersion string
}

CreateBucketSnapshotOutput is the response from CreateBucketSnapshot. It wraps the underlying s3.CreateBucketOutput and surfaces the snapshot version that Tigris returns in the X-Tigris-Snapshot-Version response header.

type HeadBucketForkOrSnapshotOutput ¶

type HeadBucketForkOrSnapshotOutput struct {
	SnapshotsEnabled     bool   // true if snapshots are enabled, otherwise false.
	SourceBucket         string // The name of the bucket this bucket was forked from.
	SourceBucketSnapshot string // The snapshot this bucket was forked from.
	IsForkParent         bool   // true if there are forks of this bucket, otherwise false.
}

HeadBucketForkOrSnapshotOutput records the fork/snapshot metadata for a bucket.

type Option ¶

type Option func(o *Options)

Option is a functional option for configuring the Tigris client.

func WithAccessKeypair ¶

func WithAccessKeypair(accessKeyID, secretAccessKey string) Option

WithAccessKeypair lets you specify a custom access key and secret access key for interfacing with Tigris.

This is useful when you need to load environment variables from somewhere other than the default AWS configuration path.

func WithEndpoint ¶

func WithEndpoint(endpoint string) Option

WithEndpoint sets a custom endpoint for connecting to Tigris.

This allows you to connect to a custom Tigris endpoint instead of the default global endpoint. Use this for:

Using a custom proxy or gateway
Testing against local development endpoints

For most use cases, consider using WithGlobalEndpoint or WithFlyEndpoint instead.

func WithFlyEndpoint ¶

func WithFlyEndpoint() Option

WithFlyEndpoint lets you connect to Tigris' fly.io optimized endpoint.

If you are deployed to fly.io, this zero-rates your traffic to Tigris.

If you are not deployed to fly.io, please use WithGlobalEndpoint instead.

func WithGlobalEndpoint ¶

func WithGlobalEndpoint() Option

WithGlobalEndpoint lets you connect to Tigris' globally available endpoint.

If you are deployed to fly.io, please use WithFlyEndpoint instead.

func WithPathStyle ¶

func WithPathStyle(enabled bool) Option

WithPathStyle configures whether to use path-style addressing for S3 requests.

By default, Tigris uses virtual-hosted-style addressing (e.g., https://bucket.t3.storage.dev). Path-style addressing (e.g., https://t3.storage.dev/bucket) may be needed for:

Compatibility with older S3 clients that don't support virtual-hosted-style
Working through certain proxies or load balancers that don't support virtual-hosted-style
Local development environments with custom DNS setups

Enable this only if you encounter issues with the default virtual-hosted-style addressing.

func WithRegion ¶

func WithRegion(region string) Option

WithRegion lets you statically specify a region for interacting with Tigris.

You will almost certainly never need this. This is here for development usecases where the default region is not "auto".

type Options ¶

type Options struct {
	BaseEndpoint string
	Region       string
	UsePathStyle bool

	AccessKeyID     string
	SecretAccessKey string
}

Options is the set of options that can be configured for the Tigris client.

Source Files ¶

View all Source files

Directories ¶

Path	Synopsis
simplestorage
tigrisheaders Package tigrisheaders contains Tigris-specific header helpers for the AWS S3 SDK and helpers for interacting with Tigris.	Package tigrisheaders contains Tigris-specific header helpers for the AWS S3 SDK and helpers for interacting with Tigris.

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL