storage

README

Tigris Storage SDK for Go

Welcome to the Tigris Storage SDK for Go! This package contains high-level wrappers and helpers to help you take advantage of all of Tigris' features.

Overview

Tigris is a cloud storage service that provides a simple, scalable, and secure object storage solution. It is based on the S3 API, but has additional features that need these helpers.

This SDK provides the main storage package containing the Tigris client with S3-compatible methods plus Tigris-specific features like bucket forking, snapshots, and object renaming.

Installation

go get github.com/tigrisdata/storage-go

Quick Start

package main

import (
    "context"
    "fmt"
    "log"

    storage "github.com/tigrisdata/storage-go"
)

func main() {
    ctx := context.Background()

    // Create a new Tigris client
    client, err := storage.New(ctx)
    if err != nil {
        log.Fatal(err)
    }

    // Use the client like you would use AWS S3 client
    // plus Tigris-specific features (see below)
    fmt.Println("Connected to Tigris!")
}

Client Configuration

The New() function creates a new S3 client optimized for interactions with Tigris. It accepts functional options to configure the client:

client, err := storage.New(ctx,
    storage.WithFlyEndpoint(),           // Use fly.io optimized endpoint
    storage.WithGlobalEndpoint(),        // Use globally available endpoint (default)
    storage.WithRegion("auto"),          // Specify a region
    storage.WithAccessKeypair(key, secret), // Set access credentials
)

Configuration Options

Option	Description
WithFlyEndpoint()	Connect to Tigris' fly.io optimized endpoint. If you are deployed to fly.io, this zero-rates your traffic to Tigris.
WithGlobalEndpoint()	Connect to Tigris' globally available endpoint (https://t3.storage.dev). This is the default.
WithRegion(region)	Statically specify a region for interacting with Tigris. You will almost certainly never need this.
WithAccessKeypair(accessKeyID, secretAccessKey)	Specify a custom access key and secret access key. Useful when loading credentials from non-standard locations.

Bucket Features

Snapshots and Forks

Tigris supports bucket snapshots and forking, allowing you to create point-in-time copies of buckets and branch from them.

Create a Snapshot Enabled Bucket

output, err := client.CreateSnapshotEnabledBucket(ctx, &s3.CreateBucketInput{
    Bucket: aws.String("my-bucket"),
})

Create a Snapshot

output, err := client.CreateBucketSnapshot(ctx, "Initial backup", &s3.CreateBucketInput{
    Bucket: aws.String("my-bucket"),
})

Fork a Bucket

// Creates a new bucket "my-bucket-fork" as a fork of "my-bucket"
output, err := client.CreateBucketFork(ctx, "my-bucket", "my-bucket-fork")

List Snapshots

snapshots, err := client.ListBucketSnapshots(ctx, "my-bucket")

Get Fork/Snapshot Metadata

info, err := client.HeadBucketForkOrSnapshot(ctx, &s3.HeadBucketInput{
    Bucket: aws.String("my-bucket"),
})
// info.SnapshotsEnabled     - true if snapshots are enabled
// info.SourceBucket         - The bucket this was forked from
// info.SourceBucketSnapshot - The snapshot this was forked from
// info.IsForkParent         - true if there are forks of this bucket

Object Features

Rename Objects

Tigris supports in-place object renaming without copying data:

_, err := client.RenameObject(ctx, &s3.CopyObjectInput{
    Bucket:     aws.String("my-bucket"),
    CopySource: aws.String("my-bucket/old-name.txt"),
    Key:        aws.String("new-name.txt"),
})

Documentation

For more information on Tigris features, see:

• Snapshots and Forks
• Object Rename
• Query Metadata
• Conditional Operations
• Regions

License

See LICENSE for details.

Documentation

Overview

Package storage contains a Tigris client and helpers for interacting with Tigris.

Tigris is a cloud storage service that provides a simple, scalable, and secure object storage solution. It is based on the S3 API, but has additional features that need these helpers.

Index

• Constants
• type BundleObjectsInput
• type BundleObjectsOutput
• type Client
  • func New(ctx context.Context, options ...Option) (*Client, error)
  • func (c *Client) BundleObjects(ctx context.Context, in *BundleObjectsInput) (*BundleObjectsOutput, error)
  • func (c *Client) CreateBucketFork(ctx context.Context, source, target string, opts ...func(*s3.Options)) (*s3.CreateBucketOutput, error)
  • func (c *Client) CreateBucketSnapshot(ctx context.Context, description string, in *s3.CreateBucketInput, ...) (*s3.CreateBucketOutput, error)
  • func (c *Client) CreateSnapshotEnabledBucket(ctx context.Context, in *s3.CreateBucketInput, opts ...func(*s3.Options)) (*s3.CreateBucketOutput, error)
  • func (c *Client) HeadBucketForkOrSnapshot(ctx context.Context, in *s3.HeadBucketInput, opts ...func(*s3.Options)) (*HeadBucketForkOrSnapshotOutput, error)
  • func (c *Client) ListBucketSnapshots(ctx context.Context, bucketName string, opts ...func(*s3.Options)) (*s3.ListBucketsOutput, error)
  • func (c *Client) RenameObject(ctx context.Context, in *s3.CopyObjectInput, opts ...func(*s3.Options)) (*s3.CopyObjectOutput, error)
• type HeadBucketForkOrSnapshotOutput
• type Option
  • func WithAccessKeypair(accessKeyID, secretAccessKey string) Option
  • func WithEndpoint(endpoint string) Option
  • func WithFlyEndpoint() Option
  • func WithGlobalEndpoint() Option
  • func WithPathStyle(enabled bool) Option
  • func WithRegion(region string) Option
• type Options

Examples

• Client.BundleObjects
• Client.CreateBucketFork
• Client.CreateBucketSnapshot
• Client.CreateSnapshotEnabledBucket
• Client.HeadBucketForkOrSnapshot
• Client.ListBucketSnapshots
• Client.RenameObject
• New

Constants

const (
	// BundleFormatTar is the tar archive format for bundle responses.
	BundleFormatTar = "tar"

	// BundleCompressionNone disables compression (default).
	BundleCompressionNone = "none"
	// BundleCompressionGzip enables gzip compression.
	BundleCompressionGzip = "gzip"
	// BundleCompressionZstd enables zstd compression.
	BundleCompressionZstd = "zstd"

	// BundleOnErrorSkip silently omits missing objects from the archive (default).
	BundleOnErrorSkip = "skip"
	// BundleOnErrorFail returns an error if any object is missing.
	BundleOnErrorFail = "fail"
)

const (
	GlobalEndpoint = "https://t3.storage.dev"
	FlyEndpoint    = "https://fly.storage.tigris.dev"
)

Types

type BundleObjectsInput

type BundleObjectsInput struct {
	// Bucket is the name of the bucket containing the objects. Required.
	Bucket string

	// Keys is the list of object keys to include in the bundle. Required.
	// Maximum 5,000 keys per request.
	Keys []string

	// Compression sets the compression algorithm for the response.
	// Valid values: "none" (default), "gzip", "zstd".
	Compression string

	// OnError controls behavior when objects are missing.
	// "skip" (default): omit missing objects and append __bundle_errors.json to the tar.
	// "fail": return an error before streaming if any object is missing.
	OnError string
}

BundleObjectsInput is the input for a BundleObjects request.

type BundleObjectsOutput

type BundleObjectsOutput struct {
	// Body is the streaming tar archive. Must be closed by the caller.
	Body io.ReadCloser

	// ContentType is the response Content-Type (e.g. "application/x-tar", "application/gzip").
	ContentType string

	// StatusCode is the HTTP status code of the response.
	StatusCode int
}

BundleObjectsOutput is the response from a BundleObjects request.

The Body contains a streaming tar archive. Callers are responsible for closing Body. Use archive/tar to iterate entries:

tr := tar.NewReader(output.Body)
for {
    hdr, err := tr.Next()
    if err == io.EOF { break }
    // process hdr.Name, tr
}

If compression was requested, wrap Body with the appropriate decompressor first:

gz, _ := gzip.NewReader(output.Body)
tr := tar.NewReader(gz)

type Client

type Client struct {
	*s3.Client
}

Client is a wrapper around the AWS SDK S3 Client with additional methods for integration with Tigris.

func New

func New(ctx context.Context, options ...Option) (*Client, error)

New returns a new S3 client optimized for interactions with Tigris.

Example

package main

import (
	"context"
	"log"

	storage "github.com/tigrisdata/storage-go"

	_ "github.com/joho/godotenv/autoload"
)

func main() {
	ctx := context.Background()

	// Create a new Tigris client with default options
	client, err := storage.New(ctx)
	if err != nil {
		log.Fatal(err)
	}
	_ = client

	// Create a client with custom options
	client, err = storage.New(ctx,
		storage.WithFlyEndpoint(),    // Use fly.io optimized endpoint
		storage.WithGlobalEndpoint(), // Use globally available endpoint (default)
		storage.WithRegion("auto"),   // Specify a region
		// storage.WithAccessKeypair(key, secret), // Set access credentials
	)
	if err != nil {
		log.Fatal(err)
	}
	_ = client
}

func (*Client) BundleObjects

func (c *Client) BundleObjects(ctx context.Context, in *BundleObjectsInput) (*BundleObjectsOutput, error)

BundleObjects fetches multiple objects from a bucket as a streaming tar archive in a single HTTP request.

This is a Tigris extension to the S3 API, designed for ML training workloads that need to fetch thousands of objects per batch without per-object HTTP overhead.

The caller is responsible for closing the returned Body.

Example

package main

import (
	"archive/tar"
	"context"
	"io"
	"log"

	storage "github.com/tigrisdata/storage-go"

	_ "github.com/joho/godotenv/autoload"
)

func main() {
	ctx := context.Background()

	client, err := storage.New(ctx)
	if err != nil {
		log.Fatal(err) // handle the error here
	}

	// Fetch multiple objects as a streaming tar archive in a single request.
	output, err := client.BundleObjects(ctx, &storage.BundleObjectsInput{
		Bucket: "my-dataset-bucket",
		Keys: []string{
			"train/img_001.jpg",
			"train/img_002.jpg",
			"train/img_003.jpg",
		},
	})
	if err != nil {
		log.Fatal(err) // handle the error here
	}
	defer output.Body.Close()

	// Iterate tar entries as they arrive.
	tr := tar.NewReader(output.Body)
	for {
		hdr, err := tr.Next()
		if err == io.EOF {
			break
		}
		if err != nil {
			log.Fatal(err) // handle the error here
		}

		// Read the object content.
		data, err := io.ReadAll(tr)
		if err != nil {
			log.Fatal(err) // handle the error here
		}

		_ = hdr.Name // object key, e.g. "train/img_001.jpg"
		_ = data     // object content
	}
}

func (*Client) CreateBucketFork

func (c *Client) CreateBucketFork(ctx context.Context, source, target string, opts ...func(*s3.Options)) (*s3.CreateBucketOutput, error)

CreateBucketFork creates a fork of the source bucket named target.

If you want to specify an exact snapshot version to fork from, use tigrisheaders.WithSnapshotVersion.

Example

package main

import (
	"context"
	"log"

	storage "github.com/tigrisdata/storage-go"

	_ "github.com/joho/godotenv/autoload"
)

func main() {
	ctx := context.Background()

	client, err := storage.New(ctx)
	if err != nil {
		log.Fatal(err)
	}

	// Creates a new bucket "my-bucket-fork" as a fork of "my-bucket"
	output, err := client.CreateBucketFork(ctx, "my-bucket", "my-bucket-fork")
	if err != nil {
		log.Fatal(err)
	}
	_ = output
}

func (*Client) CreateBucketSnapshot

func (c *Client) CreateBucketSnapshot(ctx context.Context, description string, in *s3.CreateBucketInput, opts ...func(*s3.Options)) (*s3.CreateBucketOutput, error)

CreateBucketSnapshot creates a snapshot with the given description for a bucket.

Example

package main

import (
	"context"
	"log"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/service/s3"

	storage "github.com/tigrisdata/storage-go"

	_ "github.com/joho/godotenv/autoload"
)

func main() {
	ctx := context.Background()

	client, err := storage.New(ctx)
	if err != nil {
		log.Fatal(err)
	}

	// Create a snapshot with a description
	output, err := client.CreateBucketSnapshot(ctx, "Initial backup", &s3.CreateBucketInput{
		Bucket: aws.String("my-bucket"),
	})
	if err != nil {
		log.Fatal(err)
	}
	_ = output
}

func (*Client) CreateSnapshotEnabledBucket

func (c *Client) CreateSnapshotEnabledBucket(ctx context.Context, in *s3.CreateBucketInput, opts ...func(*s3.Options)) (*s3.CreateBucketOutput, error)

CreateSnapshotEnabledBucket creates a new bucket with the ability to take snapshots and fork the contents of it.

Example

package main

import (
	"context"
	"log"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/service/s3"

	storage "github.com/tigrisdata/storage-go"

	_ "github.com/joho/godotenv/autoload"
)

func main() {
	ctx := context.Background()

	client, err := storage.New(ctx)
	if err != nil {
		log.Fatal(err)
	}

	// Create a bucket with snapshot support enabled
	output, err := client.CreateSnapshotEnabledBucket(ctx, &s3.CreateBucketInput{
		Bucket: aws.String("my-bucket"),
	})
	if err != nil {
		log.Fatal(err)
	}
	_ = output
}

func (*Client) HeadBucketForkOrSnapshot

func (c *Client) HeadBucketForkOrSnapshot(ctx context.Context, in *s3.HeadBucketInput, opts ...func(*s3.Options)) (*HeadBucketForkOrSnapshotOutput, error)

HeadBucketForkOrSnapshot fetches the fork/snapshot metadata for a bucket.

For more information, see the Tigris documentation1.

Example

package main

import (
	"context"
	"log"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/service/s3"

	storage "github.com/tigrisdata/storage-go"

	_ "github.com/joho/godotenv/autoload"
)

func main() {
	ctx := context.Background()

	client, err := storage.New(ctx)
	if err != nil {
		log.Fatal(err)
	}

	// Get fork/snapshot metadata for a bucket
	info, err := client.HeadBucketForkOrSnapshot(ctx, &s3.HeadBucketInput{
		Bucket: aws.String("my-bucket"),
	})
	if err != nil {
		log.Fatal(err)
	}

	_ = info.SnapshotsEnabled     // true if snapshots are enabled
	_ = info.SourceBucket         // The bucket this was forked from
	_ = info.SourceBucketSnapshot // The snapshot this was forked from
	_ = info.IsForkParent         // true if there are forks of this bucket
}

func (*Client) ListBucketSnapshots

func (c *Client) ListBucketSnapshots(ctx context.Context, bucketName string, opts ...func(*s3.Options)) (*s3.ListBucketsOutput, error)

ListBucketSnapshots lists the snapshots for a bucket.

For more information, see the Tigris documentation1.

Example

package main

import (
	"context"
	"log"

	storage "github.com/tigrisdata/storage-go"

	_ "github.com/joho/godotenv/autoload"
)

func main() {
	ctx := context.Background()

	client, err := storage.New(ctx)
	if err != nil {
		log.Fatal(err)
	}

	// List all snapshots for a bucket
	snapshots, err := client.ListBucketSnapshots(ctx, "my-bucket")
	if err != nil {
		log.Fatal(err)
	}
	_ = snapshots
}

func (*Client) RenameObject

func (c *Client) RenameObject(ctx context.Context, in *s3.CopyObjectInput, opts ...func(*s3.Options)) (*s3.CopyObjectOutput, error)

RenameObject performs an in-place rename of objects instead of copying the data.

For more information, see the Tigris documentation1.

Example

package main

import (
	"context"
	"log"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/service/s3"

	storage "github.com/tigrisdata/storage-go"

	_ "github.com/joho/godotenv/autoload"
)

func main() {
	ctx := context.Background()

	client, err := storage.New(ctx)
	if err != nil {
		log.Fatal(err)
	}

	// Rename an object in-place without copying data
	_, err = client.RenameObject(ctx, &s3.CopyObjectInput{
		Bucket:     aws.String("my-bucket"),
		CopySource: aws.String("my-bucket/old-name.txt"),
		Key:        aws.String("new-name.txt"),
	})
	if err != nil {
		log.Fatal(err)
	}
}

type HeadBucketForkOrSnapshotOutput

type HeadBucketForkOrSnapshotOutput struct {
	SnapshotsEnabled     bool   // true if snapshots are enabled, otherwise false.
	SourceBucket         string // The name of the bucket this bucket was forked from.
	SourceBucketSnapshot string // The snapshot this bucket was forked from.
	IsForkParent         bool   // true if there are forks of this bucket, otherwise false.
}

HeadBucketForkOrSnapshotOutput records the fork/snapshot metadata for a bucket.

type Option

type Option func(o *Options)

Option is a functional option for configuring the Tigris client.

func WithAccessKeypair

func WithAccessKeypair(accessKeyID, secretAccessKey string) Option

WithAccessKeypair lets you specify a custom access key and secret access key for interfacing with Tigris.

This is useful when you need to load environment variables from somewhere other than the default AWS configuration path.

func WithEndpoint

func WithEndpoint(endpoint string) Option

WithEndpoint sets a custom endpoint for connecting to Tigris.

This allows you to connect to a custom Tigris endpoint instead of the default global endpoint. Use this for:

• Using a custom proxy or gateway
• Testing against local development endpoints

For most use cases, consider using WithGlobalEndpoint or WithFlyEndpoint instead.

func WithFlyEndpoint

func WithFlyEndpoint() Option

WithFlyEndpoint lets you connect to Tigris' fly.io optimized endpoint.

If you are deployed to fly.io, this zero-rates your traffic to Tigris.

If you are not deployed to fly.io, please use WithGlobalEndpoint instead.

func WithGlobalEndpoint

func WithGlobalEndpoint() Option

WithGlobalEndpoint lets you connect to Tigris' globally available endpoint.

If you are deployed to fly.io, please use WithFlyEndpoint instead.

func WithPathStyle

func WithPathStyle(enabled bool) Option

WithPathStyle configures whether to use path-style addressing for S3 requests.

By default, Tigris uses virtual-hosted-style addressing (e.g., https://bucket.t3.storage.dev). Path-style addressing (e.g., https://t3.storage.dev/bucket) may be needed for:

• Compatibility with older S3 clients that don't support virtual-hosted-style
• Working through certain proxies or load balancers that don't support virtual-hosted-style
• Local development environments with custom DNS setups

Enable this only if you encounter issues with the default virtual-hosted-style addressing.

func WithRegion

func WithRegion(region string) Option

WithRegion lets you statically specify a region for interacting with Tigris.

You will almost certainly never need this. This is here for development usecases where the default region is not "auto".

type Options

type Options struct {
	BaseEndpoint string
	Region       string
	UsePathStyle bool

	AccessKeyID     string
	SecretAccessKey string
}

Options is the set of options that can be configured for the Tigris client.