Help us shape the future of Go! Take the 2026 Go Developer Survey

s3pe

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 24, 2026 License: MIT Imports: 22 Imported by: 0

README ¶

s3pe

CI crates.io PyPI CRAN Go Reference DOI

Read and write data on any S3-compatible object store (AWS S3, Garage, MinIO, Ceph, …) with the same nine verbs in R, Python, Rust and Go: the same defaults, the same signing, the same retries and the same integrity checks. The endpoint and the credentials come from the environment; the format comes from the file extension; every transfer prints one line.

Website: https://strategicprojects.github.io/s3pe/ · Português

Edition Install Docs
R install.packages("s3pe") · pak::pak("StrategicProjects/s3pe/r") pkgdown
Python pip install s3pe ("s3pe[pandas]" for tables) site · API
Rust cargo add s3pe · cargo install s3pe (the s3pe command) site · docs.rs
Go go get github.com/StrategicProjects/s3pe site · pkg.go.dev

The nine verbs

library(s3pe)                                           # R
use_endpoint("https://s3.example.org"); use_bucket("my-bucket")
meta  <- read_s3("data/manifest.json")                  # list, from JSON
sales <- read_s3("series/sales.csv.gz")                 # tibble
write_s3(sales, "reports/sales.parquet")
list_s3("data/")

import s3pe                                             # Python
meta  = s3pe.read_s3("s3://my-bucket/data/manifest.json")          # dict
sales = s3pe.read_s3("series/sales.csv.gz", bucket="my-bucket")    # DataFrame
s3pe.write_s3(sales, "s3://my-bucket/reports/sales.parquet")
s3pe.list_s3("s3://my-bucket/data/")

let meta = s3pe::read_s3("s3://my-bucket/data/manifest.json")?;    // Rust: Vec<u8>
s3pe::upload_s3("sales.parquet", "s3://my-bucket/reports/sales.parquet", None)?;
for o in s3pe::list_s3("s3://my-bucket/data/")? { println!("{} {}", o.size, o.key); }

meta, err := s3pe.ReadS3(ctx, "s3://my-bucket/data/manifest.json")  // Go: []byte
_, err = s3pe.UploadS3(ctx, "sales.parquet", "s3://my-bucket/reports/sales.parquet", "")
objects, err := s3pe.ListS3(ctx, "s3://my-bucket/data/")

s3pe ls s3://my-bucket/data/                                         # command line
s3pe get s3://my-bucket/data/regions.geojson
s3pe put sales.parquet s3://my-bucket/reports/
Verb Does
read_s3 / write_s3 move an object in memory: decoded by extension in R and Python, bytes in Rust and Go
download_s3 / upload_s3 move a file, streaming, written as .s3pe-part and renamed when complete
list_s3 every object under a prefix, all pages
exists_s3 / head_s3 whether an object exists; its size, type, date and ETag
delete_s3 delete (idempotent)
buckets_s3 the buckets the credentials can see
copy_s3 server-side copy
presign_s3 a URL that works without credentials for a while (GET or PUT)
sync_up_s3 / sync_down_s3 mirror a directory and a prefix by ETag, a manifest last

write_s3/upload_s3 take metadata and headers (so SSE is one header away where the store supports it); delete_s3 takes many keys at once; list_s3(delimiter = "/") lists one level. Files above 64 MiB go up as a multipart upload, parts in parallel, aborted cleanly on failure. No ACLs and no versioning, by design (DESIGN.md D19). Every edition also has a Client (R: s3_config()) for explicit settings.

Architecture

Configuration

Argument, then environment, then default:

Setting Environment Default
endpoint S3PE_ENDPOINT, then AWS_ENDPOINT_URL required (https://host[:port])
region AWS_REGION, then AWS_DEFAULT_REGION us-east-1
credentials AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN required
bucket S3PE_BUCKET none: write s3://bucket/key
quiet S3PE_QUIET=1 one log line per transfer

In R the usual place is ~/.Renviron (or use_endpoint() / use_bucket() for the session); in Python and the CLI any .env loader or the shell. Never put the key in a script or a repository.

What happens in every call

Dependable by default

  • Signing: AWS Signature V4, header style, over the real SHA-256 of the payload. No aws-chunked (which some stores reject), no unsigned payloads. The four signers reproduce the AWS documentation vector.
  • Retries: connection failures and HTTP 429/500/502/503/504, 3 tries, 1 s then 2 s. Never on other 4xx.
  • Typed errors: not found, access denied, other HTTP (with the S3 code), transport, configuration, integrity. exists_s3/head_s3 turn 404 into FALSE/None/nil.
  • Integrity: ETag vs MD5 after uploads; Content-Length and MD5 after downloads.
  • Tests: each edition runs against a local fake S3 that verifies every signature, and against a real store when S3PE_ENDPOINT and S3PE_TEST_BUCKET are set.

Design decisions are recorded in DESIGN.md; changes in CHANGELOG.md.

Repository layout

r/        R package (CRAN)                  rust/    Rust crate and CLI (crates.io)
python/   Python package, PyO3 on the crate  *.go     Go module (this directory)
site/     landing pages (GitHub Pages)       DESIGN.md decisions shared by the four

Authors

André Leite, Marcos Wasiliew, Hugo Vasconcelos, Carlos Amorim, Diogo Bezerra and Júlia Nascimento Barreto. MIT license. Cite with CITATION.cff.

Made in Pernambuco 🌵

s3pe is developed in Recife, Brazil.

Documentation ¶

Overview ¶

Package s3pe is a small, dependable client for S3-compatible object stores (AWS S3, Garage, MinIO, Ceph, ...). Standard library only, no cgo.

Nine verbs, one client, nothing else to learn: 

ctx := context.Background()
manifest, err := s3pe.ReadS3(ctx, "s3://my-bucket/data/manifest.json")
_, err = s3pe.WriteS3(ctx, "s3://my-bucket/reports/figure1.png", png, "")
objects, err := s3pe.ListS3(ctx, "s3://my-bucket/data/")
_, err = s3pe.UploadS3(ctx, "sales.parquet", "s3://my-bucket/etl/", "")
_, err = s3pe.DownloadS3(ctx, "s3://my-bucket/etl/sales.parquet", "sales.parquet")
ok, err := s3pe.ExistsS3(ctx, "s3://my-bucket/etl/manifest.json")
meta, err := s3pe.HeadS3(ctx, "s3://my-bucket/etl/manifest.json")
err = s3pe.DeleteS3(ctx, "s3://my-bucket/etl/old.parquet")
buckets, err := s3pe.BucketsS3(ctx)

And, since 0.3.0, a few more: CopyS3 (server-side copy), DeleteManyS3 (1000 keys per request), ListDirS3 (one level: objects and "folders"), PresignS3 (a URL that works without credentials for a while), SyncUpS3 and SyncDownS3 (a directory and a prefix kept equal by ETag, a manifest last). Client.WriteWith and Client.UploadWith take PutOptions: user metadata (x-amz-meta-*) and any other header, so Cache-Control, storage classes and server-side encryption headers reach stores that support them. Bodies above 64 MiB go up as a multipart upload, 16 MiB parts, four at a time, aborted on failure, with the final ETag checked (configurable in Config).

The package-level verbs share a client built from the environment. An explicit Client takes the same settings in a Config; a default bucket makes keys short: 

s3, err := s3pe.New(s3pe.Config{Endpoint: "https://s3.example.org", Bucket: "my-bucket"})
data, err := s3.Read(ctx, "data/manifest.json")

Configuration ¶

Anything not set in Config comes from the environment, then from the defaults: 

endpoint     S3PE_ENDPOINT, AWS_ENDPOINT_URL      required
region       AWS_REGION, AWS_DEFAULT_REGION       us-east-1
credentials  AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN   required
bucket       S3PE_BUCKET                          none: use s3://bucket/key
multipart    S3PE_MULTIPART_THRESHOLD, S3PE_PART_SIZE (bytes), S3PE_PARALLEL   64 MiB, 16 MiB, 4
quiet        S3PE_QUIET                           off

URLs are always path-style (https://host/bucket/key); a key may be an s3://bucket/key URL, which names the bucket for that call.

What "dependable" means here ¶

Requests are signed with AWS Signature V4 over the real SHA-256 of the payload (no aws-chunked, which some stores, Garage among them, reject); every x-amz-* header is part of the signature. Connection failures and HTTP 429/500/502/503/504 are retried twice, with a 1 s then 2 s pause; other errors are returned at once, typed: errors.Is(err, ErrNotFound), ErrAccessDenied, ErrConfig, ErrIntegrity, ErrTransport, ErrHTTP, with the details in an *Error. After an upload the server's ETag is compared with the local MD5 (part by part, and as a whole, for multipart uploads); after a download the byte count is compared with Content-Length and, when the ETag is an MD5, with the MD5 of what arrived. Downloads go to a .s3pe-part file renamed when complete. One line per transfer goes to the logger (standard error by default): 

↑ s3://my-bucket/etl/sales.parquet (12.3 MB, 0.8 s)

Out of scope, on purpose: ACLs (manage access with keys and policies) and object versioning.

This is the Go edition of s3pe; the same package exists as a Rust crate with a command line (cargo install s3pe), for Python (pip install s3pe) and for R (install.packages("s3pe")). All four follow one specification and reproduce the same signatures.

Example ¶

Endpoint and credentials come from the environment; keys are s3:// URLs. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/StrategicProjects/s3pe"
)

func main() {
	ctx := context.Background()
	manifest, err := s3pe.ReadS3(ctx, "s3://my-bucket/data/manifest.json")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(len(manifest))

	if _, err := s3pe.WriteS3(ctx, "s3://my-bucket/reports/table1.csv", []byte("a,b\n1,2\n"), ""); err != nil {
		log.Fatal(err)
	}
	objects, err := s3pe.ListS3(ctx, "s3://my-bucket/data/")
	if err != nil {
		log.Fatal(err)
	}
	for _, o := range objects {
		fmt.Println(o.Size, o.Key)
	}
}

Index ¶

Examples ¶

Constants ¶

View Source 
const (
	// DefaultRegion is the signing region used when none is given.
	DefaultRegion = "us-east-1"
	// DefaultTimeout is the time one call may take before it is abandoned.
	DefaultTimeout = 300 * time.Second
	// DefaultMaxTries is the number of attempts per call: the first and two retries.
	DefaultMaxTries = 3
	// DefaultMultipartThreshold: bodies larger than this go up as a multipart upload (D15).
	DefaultMultipartThreshold int64 = 64 * 1024 * 1024
	// DefaultPartSize is the size of each part of a multipart upload (D15).
	DefaultPartSize int64 = 16 * 1024 * 1024
	// DefaultParallel is the number of parts sent at the same time (D15).
	DefaultParallel = 4
	// MinPartSize is the smallest part S3 accepts, except for the last one.
	MinPartSize int64 = 5 * 1024 * 1024
	// MaxPresignSecs is the longest validity of a pre-signed URL (seven days).
	MaxPresignSecs int64 = 7 * 24 * 3600
)

Variables ¶

View Source 
var (
	// ErrConfig: missing endpoint, credentials or bucket, bad endpoint.
	ErrConfig = errors.New("configuration")
	// ErrTransport: the server could not be reached, or the connection broke (retried first).
	ErrTransport = errors.New("transport")
	// ErrNotFound: the object or bucket does not exist (HTTP 404).
	ErrNotFound = errors.New("not found")
	// ErrAccessDenied: the credentials do not allow this (HTTP 403).
	ErrAccessDenied = errors.New("access denied")
	// ErrHTTP: any other HTTP error; the *Error carries status, code and message.
	ErrHTTP = errors.New("http")
	// ErrIntegrity: the bytes that arrived are not the bytes that were sent.
	ErrIntegrity = errors.New("integrity")
)

Sentinel errors, to be tested with errors.Is.

Functions ¶

func ContentTypeFor ¶ 

func ContentTypeFor(key string) string

ContentTypeFor is the content type a key's extension suggests (application/octet-stream when unknown). Every edition of s3pe uses the same table.

func DeleteManyS3 ¶ added in v0.3.0 

func DeleteManyS3(ctx context.Context, keys []string) error

DeleteManyS3 deletes several objects in batches, with a client built from the environment.

func DeleteS3 ¶ 

func DeleteS3(ctx context.Context, key string) error

DeleteS3 deletes an object, with a client built from the environment.

func ExistsS3 ¶ 

func ExistsS3(ctx context.Context, key string) (bool, error)

ExistsS3 reports whether an object exists, with a client built from the environment.

func ExpectedETag ¶ added in v0.3.0 

func ExpectedETag(path string, threshold, partSize int64) (string, error)

ExpectedETag is the ETag the store would give a local file uploaded by this client: its MD5, or the multipart ETag (D15) when the file is larger than threshold, with parts of partSize.

func FormatSize ¶ 

func FormatSize(n int64) string

FormatSize renders a byte count with SI units and one decimal: "12.3 MB", "1.2 kB", "512 B".

func MultipartETag ¶ added in v0.3.0 

func MultipartETag(partMD5sHex []string) string

MultipartETag is the ETag S3 gives a multipart upload: md5(concat(part MD5 bytes))-N, from the hex MD5s of the parts.

func PresignS3 ¶ added in v0.3.0 

func PresignS3(key, method string, expires time.Duration) (string, error)

PresignS3 returns a pre-signed URL, with a client built from the environment.

func ReadS3 ¶ 

func ReadS3(ctx context.Context, key string) ([]byte, error)

ReadS3 reads an object into memory, with a client built from the environment.

Types ¶

type Bucket ¶ 

type Bucket struct {
	Name    string
	Created time.Time
}

Bucket is a bucket as listed by the store.

func BucketsS3 ¶ 

func BucketsS3(ctx context.Context) ([]Bucket, error)

BucketsS3 lists the buckets the credentials can see, with a client built from the environment.

type Client ¶ 

type Client struct {
	// contains filtered or unexported fields
}

Client is a configured connection to one S3 endpoint. Every method takes a key that is either relative to the client's bucket or a full s3://bucket/key URL. A Client is safe for concurrent use.

func FromEnv ¶ 

func FromEnv() (*Client, error)

FromEnv builds a client configured entirely from the environment.

func New ¶ 

func New(cfg Config) (*Client, error)

New builds a client from cfg, the environment and the defaults. It fails without an endpoint, without credentials or with an endpoint that is not scheme://host[:port].

Example ¶

An explicit client, with a default bucket, keeps the keys short. 

package main

import (
	"context"
	"log"

	"github.com/StrategicProjects/s3pe"
)

func main() {
	s3, err := s3pe.New(s3pe.Config{
		Endpoint:  "https://s3.example.org",
		AccessKey: "AKIA…",
		SecretKey: "…",
		Bucket:    "my-bucket",
	})
	if err != nil {
		log.Fatal(err)
	}
	ctx := context.Background()
	if _, err := s3.Upload(ctx, "sales.parquet", "etl/sales.parquet", ""); err != nil {
		log.Fatal(err)
	}
	if _, err := s3.Download(ctx, "etl/sales.parquet", "copy.parquet"); err != nil {
		log.Fatal(err)
	}
}

func (*Client) Bucket ¶ 

func (c *Client) Bucket() string

Bucket is the default bucket, empty if none.

func (*Client) Buckets ¶ 

func (c *Client) Buckets(ctx context.Context) ([]Bucket, error)

Buckets lists the buckets the credentials can see.

func (*Client) Copy ¶ added in v0.3.0 

func (c *Client) Copy(ctx context.Context, from, to string) (Transfer, error)

Copy copies an object inside the store (CopyObject, D17). Both keys may be s3:// URLs; the copy keeps the source's metadata.

Example ¶

A server-side copy keeps the source's metadata. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/StrategicProjects/s3pe"
)

func main() {
	s3, err := s3pe.FromEnv()
	if err != nil {
		log.Fatal(err)
	}
	t, err := s3.Copy(context.Background(), "s3://my-bucket/etl/sales.parquet", "s3://my-bucket/archive/sales-2026.parquet")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(t.URL())
}

func (*Client) Delete ¶ 

func (c *Client) Delete(ctx context.Context, key string) error

Delete deletes an object. Deleting what does not exist is not an error.

func (*Client) DeleteMany ¶ added in v0.3.0 

func (c *Client) DeleteMany(ctx context.Context, keys []string) error

DeleteMany deletes several objects with DeleteObjects, 1000 per request (D17). Keys may name different buckets. Any object the store could not delete is reported in one error of kind ErrHTTP with code DeleteErrors.

func (*Client) Download ¶ 

func (c *Client) Download(ctx context.Context, key, path string) (Transfer, error)

Download streams an object to a file. The bytes go to path + ".s3pe-part", renamed when complete, so a failure leaves nothing half-written at path. Missing parent directories are created.

func (*Client) Endpoint ¶ 

func (c *Client) Endpoint() string

Endpoint is the endpoint, as scheme://host[:port].

func (*Client) Exists ¶ 

func (c *Client) Exists(ctx context.Context, key string) (bool, error)

Exists reports whether an object exists.

func (*Client) Head ¶ 

func (c *Client) Head(ctx context.Context, key string) (*Object, error)

Head returns the metadata of an object, or nil, nil when it does not exist.

func (*Client) List ¶ 

func (c *Client) List(ctx context.Context, prefix string) ([]Object, error)

List lists every object under a prefix (all pages). prefix may be an s3://bucket/prefix URL, a bare prefix, or "" for the whole bucket.

func (*Client) ListDir ¶ added in v0.3.0 

func (c *Client) ListDir(ctx context.Context, prefix string) (Listing, error)

ListDir lists one level: the objects directly under the prefix and the common prefixes ("folders") below it, with the delimiter "/" (D17).

func (*Client) ListWith ¶ added in v0.3.0 

func (c *Client) ListWith(ctx context.Context, prefix, delimiter string) (Listing, error)

ListWith lists with an optional delimiter, all pages.

func (*Client) Presign ¶ added in v0.3.0 

func (c *Client) Presign(key, method string, expires time.Duration) (string, error)

Presign returns a URL that reads (GET) or writes (PUT) the object without credentials for expires, at most seven days (D16).

Example ¶

A pre-signed URL lets someone without credentials read the object for a while. 

package main

import (
	"fmt"
	"log"
	"time"

	"github.com/StrategicProjects/s3pe"
)

func main() {
	s3, err := s3pe.FromEnv()
	if err != nil {
		log.Fatal(err)
	}
	u, err := s3.Presign("s3://my-bucket/reports/report.pdf", "GET", time.Hour)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(u)
}

func (*Client) Read ¶ 

func (c *Client) Read(ctx context.Context, key string) ([]byte, error)

Read reads an object into memory.

Example ¶

A missing object is a typed error. 

package main

import (
	"context"
	"errors"
	"fmt"
	"log"

	"github.com/StrategicProjects/s3pe"
)

func main() {
	s3, err := s3pe.FromEnv()
	if err != nil {
		log.Fatal(err)
	}
	_, err = s3.Read(context.Background(), "s3://my-bucket/does-not-exist.json")
	fmt.Println(errors.Is(err, s3pe.ErrNotFound))
}

func (*Client) Resolve ¶ 

func (c *Client) Resolve(key string) (bucket, objectKey string, err error)

Resolve splits s3://bucket/key or applies the default bucket to a bare key.

func (*Client) SyncDown ¶ added in v0.3.0 

func (c *Client) SyncDown(ctx context.Context, prefix, dir string, opts SyncOptions) (SyncReport, error)

SyncDown downloads every object under prefix whose local copy is missing or differs; with Delete, removes local files that are not remote (D18).

func (*Client) SyncUp ¶ added in v0.3.0 

func (c *Client) SyncUp(ctx context.Context, dir, prefix string, opts SyncOptions) (SyncReport, error)

SyncUp uploads every file under dir whose remote ETag differs from the local one or which is missing remotely; see SyncOptions (D18).

Example ¶

SyncUp uploads what changed, the manifest last. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/StrategicProjects/s3pe"
)

func main() {
	s3, err := s3pe.FromEnv()
	if err != nil {
		log.Fatal(err)
	}
	r, err := s3.SyncUp(context.Background(), "./out", "s3://my-bucket/reports/", s3pe.SyncOptions{Last: "manifest.json"})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(len(r.Transferred), "uploaded,", len(r.Skipped), "unchanged")
}

func (*Client) Upload ¶ 

func (c *Client) Upload(ctx context.Context, path, key, contentType string) (Transfer, error)

Upload streams a file to an object. A key that is empty or ends in "/" gets the file's base name appended. An empty contentType is guessed from the key. Files above the multipart threshold go up in parts.

func (*Client) UploadWith ¶ added in v0.3.0 

func (c *Client) UploadWith(ctx context.Context, path, key string, opts PutOptions) (Transfer, error)

UploadWith streams a file to an object with metadata and headers (D17).

func (*Client) Write ¶ 

func (c *Client) Write(ctx context.Context, key string, data []byte, contentType string) (Transfer, error)

Write writes bytes to an object. An empty contentType is guessed from the key's extension. Bodies above the multipart threshold go up in parts.

func (*Client) WriteWith ¶ added in v0.3.0 

func (c *Client) WriteWith(ctx context.Context, key string, data []byte, opts PutOptions) (Transfer, error)

WriteWith writes bytes to an object with metadata and headers (D17).

Example ¶

Metadata and extra headers travel with the object; large bodies go up in parts. 

package main

import (
	"context"
	"log"

	"github.com/StrategicProjects/s3pe"
)

func main() {
	s3, err := s3pe.FromEnv()
	if err != nil {
		log.Fatal(err)
	}
	_, err = s3.WriteWith(context.Background(), "s3://my-bucket/reports/report.pdf", []byte("%PDF"), s3pe.PutOptions{
		Metadata: map[string]string{"author": "ana"},
		Headers:  map[string]string{"Cache-Control": "max-age=3600"},
	})
	if err != nil {
		log.Fatal(err)
	}
}

type Config ¶ 

type Config struct {
	// Endpoint is scheme://host[:port], no path. Required.
	Endpoint string
	// Region is the signing region (default us-east-1; on AWS, the bucket's region).
	Region string
	// AccessKey and SecretKey are required, here or in the environment.
	AccessKey, SecretKey string
	// SessionToken is for temporary credentials.
	SessionToken string
	// Bucket is used when a key is not an s3:// URL.
	Bucket string
	// Timeout limits one call (default 300 s).
	Timeout time.Duration
	// MaxTries is the number of attempts per call (default 3).
	MaxTries int
	// MultipartThreshold: bodies above it go up in parts (default 64 MiB).
	MultipartThreshold int64
	// PartSize of a multipart upload, at least 5 MiB (default 16 MiB).
	PartSize int64
	// Parallel is the number of parts sent at the same time (default 4,
	// or S3PE_PARALLEL).
	Parallel int
	// HTTPClient replaces the default one (which follows no redirects).
	HTTPClient *http.Client
	// Logger receives one line per transfer (default: standard error).
	Logger *log.Logger
	// Quiet silences the transfer lines; warnings about retries still go out.
	Quiet bool
}

Config configures a Client. Zero fields are taken from the environment (S3PE_ENDPOINT or AWS_ENDPOINT_URL, AWS_REGION or AWS_DEFAULT_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, S3PE_BUCKET, S3PE_MULTIPART_THRESHOLD, S3PE_PART_SIZE, S3PE_PARALLEL, S3PE_QUIET) and then from the defaults.

type Error ¶ 

type Error struct {
	// Status is the HTTP status, 0 when no response arrived.
	Status int
	// Code is the S3 error code (NoSuchKey, AccessDenied, …), empty if none.
	Code string
	// Message is the server's or the client's explanation.
	Message string
	// Bucket and Key name what was asked for (empty for bucket-level calls).
	Bucket, Key string
	// contains filtered or unexported fields
}

Error is what every failed call returns. errors.Is(err, ErrNotFound) and the other sentinels tell the kind; errors.As gives the details.

func (*Error) Error ¶ 

func (e *Error) Error() string

func (*Error) Is ¶ 

func (e *Error) Is(target error) bool

Is reports whether the error is of the sentinel kind target.

func (*Error) Unwrap ¶ 

func (e *Error) Unwrap() error

Unwrap returns the underlying error (network, file system, context), if any.

type Listing ¶ added in v0.3.0 

type Listing struct {
	// Objects directly under the prefix.
	Objects []Object
	// Prefixes are the common prefixes, each ending with the delimiter.
	Prefixes []string
}

Listing is one level of a bucket: the objects directly under a prefix and the common prefixes ("folders") below it (D17).

func ListDirS3 ¶ added in v0.3.0 

func ListDirS3(ctx context.Context, prefix string) (Listing, error)

ListDirS3 lists one level of a prefix, with a client built from the environment.

type Object ¶ 

type Object struct {
	// Key inside the bucket.
	Key string
	// Size in bytes.
	Size int64
	// LastModified is zero when the server's date could not be parsed.
	LastModified time.Time
	// ETag without the surrounding quotes.
	ETag string
	// ContentType is only known after Head.
	ContentType string
	// Metadata is the user metadata (x-amz-meta-*, names without the
	// prefix, lower case), only known after Head.
	Metadata map[string]string
}

Object is one object as listed or described by the store.

func HeadS3 ¶ 

func HeadS3(ctx context.Context, key string) (*Object, error)

HeadS3 returns an object's metadata (nil if absent), with a client built from the environment.

func ListS3 ¶ 

func ListS3(ctx context.Context, prefix string) ([]Object, error)

ListS3 lists the objects under a prefix, with a client built from the environment.

type PutOptions ¶ added in v0.3.0 

type PutOptions struct {
	// ContentType; guessed from the key when empty.
	ContentType string
	// Metadata is user metadata, sent as x-amz-meta-<name>.
	Metadata map[string]string
	// Headers are any other headers (Cache-Control, Content-Disposition,
	// x-amz-storage-class, x-amz-server-side-encryption, …). x-amz-* headers
	// are signed, the rest are not.
	Headers map[string]string
}

PutOptions are the options of WriteWith and UploadWith (D17).

type SyncOptions ¶ added in v0.3.0 

type SyncOptions struct {
	// Delete removes what exists on the destination but not on the source.
	Delete bool
	// Last is the relative path of the file to transfer last (a manifest).
	Last string
}

SyncOptions are the options of SyncUp and SyncDown (D18).

type SyncReport ¶ added in v0.3.0 

type SyncReport struct {
	// Transferred are the relative paths uploaded or downloaded.
	Transferred []string
	// Skipped are the relative paths already identical on both sides.
	Skipped []string
	// Deleted are the relative paths removed from the destination.
	Deleted []string
	// Elapsed is the wall time.
	Elapsed time.Duration
}

SyncReport is what a sync did.

func SyncDownS3 ¶ added in v0.3.0 

func SyncDownS3(ctx context.Context, prefix, dir string, opts SyncOptions) (SyncReport, error)

SyncDownS3 downloads a prefix to a directory, with a client built from the environment.

func SyncUpS3 ¶ added in v0.3.0 

func SyncUpS3(ctx context.Context, dir, prefix string, opts SyncOptions) (SyncReport, error)

SyncUpS3 uploads a directory to a prefix, with a client built from the environment.

type Transfer ¶ 

type Transfer struct {
	Bucket, Key string
	// Size is the number of bytes moved.
	Size int64
	// ETag reported by the server, without quotes.
	ETag string
	// Elapsed is the wall time of the whole call, retries included.
	Elapsed time.Duration
}

Transfer is what a completed upload, download or copy reports.

func CopyS3 ¶ added in v0.3.0 

func CopyS3(ctx context.Context, from, to string) (Transfer, error)

CopyS3 copies an object inside the store, with a client built from the environment.

func DownloadS3 ¶ 

func DownloadS3(ctx context.Context, key, path string) (Transfer, error)

DownloadS3 downloads an object to a file, with a client built from the environment.

func UploadS3 ¶ 

func UploadS3(ctx context.Context, path, key, contentType string) (Transfer, error)

UploadS3 uploads a file, with a client built from the environment.

func WriteS3 ¶ 

func WriteS3(ctx context.Context, key string, data []byte, contentType string) (Transfer, error)

WriteS3 writes bytes to an object, with a client built from the environment.

func (Transfer) URL ¶ 

func (t Transfer) URL() string

URL is s3://bucket/key.

Source Files ¶

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.