Help us shape the future of Go! Take the 2026 Go Developer Survey

s3pe

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 23, 2026 License: MIT Imports: 20 Imported by: 0

README

s3pe

CI crates.io PyPI CRAN Go Reference

Read and write data on s3.pe.gov.br, the S3 object store of the State of Pernambuco, Brazil (a Garage server), or on any S3-style store. The same nine verbs in R, Python, Rust and Go, with the same defaults, the same signing, the same retries and the same integrity checks. Credentials come from two environment variables; the format comes from the file extension; every transfer prints one line.

Website: https://strategicprojects.github.io/s3pe/ · Português

Edition Install Docs
R install.packages("s3pe") · pak::pak("StrategicProjects/s3pe/r") pkgdown
Python pip install s3pe ("s3pe[pandas]" for tables) API
Rust cargo add s3pe · cargo install s3pe (the s3pe command) docs.rs
Go go get github.com/StrategicProjects/s3pe pkg.go.dev

The nine verbs

library(s3pe)                                         # R
meta <- read_s3("s3://iaempe/dados/metadata.json")    # list, from JSON
icms <- read_s3("series/estado/icms.csv.gz", bucket = "horizonte-fiscal")  # tibble
write_s3(icms, "s3://iaempe/nota/icms.parquet")
list_s3("s3://iaempe/dados/")

import s3pe                                           # Python
meta = s3pe.read_s3("s3://iaempe/dados/metadata.json")          # dict
icms = s3pe.read_s3("series/estado/icms.csv.gz", bucket="horizonte-fiscal")  # DataFrame
s3pe.write_s3(icms, "s3://iaempe/nota/icms.parquet")
s3pe.list_s3("s3://iaempe/dados/")

let meta = s3pe::read_s3("s3://iaempe/dados/metadata.json")?;    // Rust: Vec<u8>
s3pe::upload_s3("icms.parquet", "s3://iaempe/nota/icms.parquet", None)?;
for o in s3pe::list_s3("s3://iaempe/dados/")? { println!("{} {}", o.size, o.key); }

meta, err := s3pe.ReadS3(ctx, "s3://iaempe/dados/metadata.json") // Go: []byte
_, err = s3pe.UploadS3(ctx, "icms.parquet", "s3://iaempe/nota/icms.parquet", "")
objects, err := s3pe.ListS3(ctx, "s3://iaempe/dados/")

s3pe ls s3://iaempe/dados/                                       # command line
s3pe get s3://iaempe/dados/pe_municipios.geojson
s3pe put painel.parquet s3://iaempe/etl/
Verb Does
read_s3 / write_s3 move an object in memory: decoded by extension in R and Python, bytes in Rust and Go
download_s3 / upload_s3 move a file, streaming, written as .s3pe-part and renamed when complete
list_s3 every object under a prefix, all pages
exists_s3 / head_s3 whether an object exists; its size, type, date and ETag
delete_s3 delete (idempotent)
buckets_s3 the buckets the credentials can see

Every edition also has a Client (R: s3_config()) for explicit settings.

Configuration

Argument, then environment, then default:

Setting Environment Default
endpoint S3PE_ENDPOINT, then AWS_ENDPOINT_URL https://s3.pe.gov.br
region AWS_REGION, then AWS_DEFAULT_REGION us-east-1 (the Garage's s3_region)
credentials AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN required
bucket S3PE_BUCKET none: write s3://bucket/key
quiet S3PE_QUIET=1 one log line per transfer

In R the usual place is ~/.Renviron; in Python and the CLI any .env loader or the shell. Never put the key in a script or a repository. Inside the condepe server use S3PE_ENDPOINT=http://127.0.0.1:9003.

Dependable by default

  • Signing: AWS Signature V4, header style, over the real SHA-256 of the payload. No aws-chunked (Garage rejects it), no unsigned payloads. The four signers reproduce the AWS documentation vector.
  • Retries: connection failures and HTTP 429/500/502/503/504, 3 tries, 1 s then 2 s. Never on other 4xx.
  • Typed errors: not found, access denied, other HTTP (with the S3 code), transport, configuration, integrity. exists_s3/head_s3 turn 404 into FALSE/None/nil.
  • Integrity: ETag vs MD5 after uploads; Content-Length and MD5 after downloads.
  • Tests: each edition runs against a local fake S3 that verifies every signature, and against the real server when S3PE_TEST_BUCKET is set.

Design decisions are recorded in DESIGN.md; changes in CHANGELOG.md.

Repository layout

r/        R package (CRAN)                  rust/    Rust crate and CLI (crates.io)
python/   Python package, PyO3 on the crate  *.go     Go module (this directory)
site/     landing page (GitHub Pages)        DESIGN.md decisions shared by the four

Authors

André Leite, Marcos Wasiliew, Hugo Vasconcelos, Carlos Amorim, Diogo Bezerra and Júlia Nascimento Barreto. MIT license. Cite with CITATION.cff.

Documentation

Overview

Package s3pe is a small, dependable S3 client made for s3.pe.gov.br, the Garage object store of the State of Pernambuco, Brazil, and usable with any S3-style store. Standard library only, no cgo.

Nine verbs, one client, nothing else to learn: 

ctx := context.Background()
manifest, err := s3pe.ReadS3(ctx, "s3://iaempe/dados/metadata.json")
_, err = s3pe.WriteS3(ctx, "s3://iaempe/nota/figura1.png", png, "")
objects, err := s3pe.ListS3(ctx, "s3://iaempe/dados/")
_, err = s3pe.UploadS3(ctx, "painel.parquet", "s3://iaempe/etl/", "")
_, err = s3pe.DownloadS3(ctx, "s3://iaempe/etl/painel.parquet", "painel.parquet")
ok, err := s3pe.ExistsS3(ctx, "s3://iaempe/etl/manifest.json")
meta, err := s3pe.HeadS3(ctx, "s3://iaempe/etl/manifest.json")
err = s3pe.DeleteS3(ctx, "s3://iaempe/etl/old.parquet")
buckets, err := s3pe.BucketsS3(ctx)

The package-level verbs share a client built from the environment. An explicit Client takes the same settings in a Config; a default bucket makes keys short: 

s3, err := s3pe.New(s3pe.Config{Bucket: "iaempe"})
data, err := s3.Read(ctx, "dados/metadata.json")

Configuration

Anything not set in Config comes from the environment, then from the defaults: 

endpoint     S3PE_ENDPOINT, AWS_ENDPOINT_URL      https://s3.pe.gov.br
region       AWS_REGION, AWS_DEFAULT_REGION       us-east-1
credentials  AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN   required
bucket       S3PE_BUCKET                          none: use s3://bucket/key
quiet        S3PE_QUIET                           off

URLs are always path-style (https://host/bucket/key); a key may be an s3://bucket/key URL, which names the bucket for that call.

What "dependable" means here

Requests are signed with AWS Signature V4 over the real SHA-256 of the payload (no aws-chunked, which Garage rejects). Connection failures and HTTP 429/500/502/503/504 are retried twice, with a 1 s then 2 s pause; other errors are returned at once, typed: errors.Is(err, ErrNotFound), ErrAccessDenied, ErrConfig, ErrIntegrity, ErrTransport, ErrHTTP, with the details in an *Error. After an upload the server's ETag is compared with the local MD5; after a download the byte count is compared with Content-Length and, when the ETag is an MD5, with the MD5 of what arrived. Downloads go to a .s3pe-part file renamed when complete. One line per transfer goes to the logger (standard error by default): 

↑ s3://iaempe/etl/painel.parquet (12.3 MB, 0.8 s)

This is the Go edition of s3pe; the same package exists as a Rust crate with a command line (cargo install s3pe), for Python (pip install s3pe) and for R (install.packages("s3pe")). All four follow one specification and reproduce the same signatures.

Example

Credentials and endpoint come from the environment; keys are s3:// URLs. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/StrategicProjects/s3pe"
)

func main() {
	ctx := context.Background()
	manifest, err := s3pe.ReadS3(ctx, "s3://iaempe/dados/metadata.json")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(len(manifest))

	if _, err := s3pe.WriteS3(ctx, "s3://iaempe/nota/figura1.csv", []byte("a,b\n1,2\n"), ""); err != nil {
		log.Fatal(err)
	}
	objects, err := s3pe.ListS3(ctx, "s3://iaempe/dados/")
	if err != nil {
		log.Fatal(err)
	}
	for _, o := range objects {
		fmt.Println(o.Size, o.Key)
	}
}

Index

Examples

Constants

View Source 
const (
	// DefaultEndpoint is used when none is given: Pernambuco's Garage server.
	DefaultEndpoint = "https://s3.pe.gov.br"
	// DefaultRegion is the signing region of that server.
	DefaultRegion = "us-east-1"
	// DefaultTimeout is the time one call may take before it is abandoned.
	DefaultTimeout = 300 * time.Second
	// DefaultMaxTries is the number of attempts per call: the first and two retries.
	DefaultMaxTries = 3
)

Variables

View Source 
var (
	// ErrConfig: missing credentials, missing bucket, bad endpoint.
	ErrConfig = errors.New("configuration")
	// ErrTransport: the server could not be reached, or the connection broke (retried first).
	ErrTransport = errors.New("transport")
	// ErrNotFound: the object or bucket does not exist (HTTP 404).
	ErrNotFound = errors.New("not found")
	// ErrAccessDenied: the credentials do not allow this (HTTP 403).
	ErrAccessDenied = errors.New("access denied")
	// ErrHTTP: any other HTTP error; the *Error carries status, code and message.
	ErrHTTP = errors.New("http")
	// ErrIntegrity: the bytes that arrived are not the bytes that were sent.
	ErrIntegrity = errors.New("integrity")
)

Sentinel errors, to be tested with errors.Is.

Functions

func ContentTypeFor 

func ContentTypeFor(key string) string

ContentTypeFor is the content type a key's extension suggests (application/octet-stream when unknown). Every edition of s3pe uses the same table.

func DeleteS3 

func DeleteS3(ctx context.Context, key string) error

DeleteS3 deletes an object, with a client built from the environment.

func ExistsS3 

func ExistsS3(ctx context.Context, key string) (bool, error)

ExistsS3 reports whether an object exists, with a client built from the environment.

func FormatSize 

func FormatSize(n int64) string

FormatSize renders a byte count with SI units and one decimal: "12.3 MB", "1.2 kB", "512 B".

func ReadS3 

func ReadS3(ctx context.Context, key string) ([]byte, error)

ReadS3 reads an object into memory, with a client built from the environment.

Types

type Bucket 

type Bucket struct {
	Name    string
	Created time.Time
}

Bucket is a bucket as listed by the store.

func BucketsS3 

func BucketsS3(ctx context.Context) ([]Bucket, error)

BucketsS3 lists the buckets the credentials can see, with a client built from the environment.

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client is a configured connection to one S3 endpoint. Every method takes a key that is either relative to the client's bucket or a full s3://bucket/key URL. A Client is safe for concurrent use.

func FromEnv 

func FromEnv() (*Client, error)

FromEnv builds a client configured entirely from the environment.

func New 

func New(cfg Config) (*Client, error)

New builds a client from cfg, the environment and the defaults. It fails without credentials or with an endpoint that is not scheme://host[:port].

Example

An explicit client, with a default bucket, keeps the keys short. 

package main

import (
	"context"
	"log"

	"github.com/StrategicProjects/s3pe"
)

func main() {
	s3, err := s3pe.New(s3pe.Config{
		Endpoint:  "https://s3.pe.gov.br",
		AccessKey: "GK…",
		SecretKey: "…",
		Bucket:    "iaempe",
	})
	if err != nil {
		log.Fatal(err)
	}
	ctx := context.Background()
	if _, err := s3.Upload(ctx, "painel.parquet", "etl/painel.parquet", ""); err != nil {
		log.Fatal(err)
	}
	if _, err := s3.Download(ctx, "etl/painel.parquet", "copy.parquet"); err != nil {
		log.Fatal(err)
	}
}

func (*Client) Bucket 

func (c *Client) Bucket() string

Bucket is the default bucket, empty if none.

func (*Client) Buckets 

func (c *Client) Buckets(ctx context.Context) ([]Bucket, error)

Buckets lists the buckets the credentials can see.

func (*Client) Delete 

func (c *Client) Delete(ctx context.Context, key string) error

Delete deletes an object. Deleting what does not exist is not an error.

func (*Client) Download 

func (c *Client) Download(ctx context.Context, key, path string) (Transfer, error)

Download streams an object to a file. The bytes go to path + ".s3pe-part", renamed when complete, so a failure leaves nothing half-written at path. Missing parent directories are created.

func (*Client) Endpoint 

func (c *Client) Endpoint() string

Endpoint is the endpoint, as scheme://host[:port].

func (*Client) Exists 

func (c *Client) Exists(ctx context.Context, key string) (bool, error)

Exists reports whether an object exists.

func (*Client) Head 

func (c *Client) Head(ctx context.Context, key string) (*Object, error)

Head returns the metadata of an object, or nil, nil when it does not exist.

func (*Client) List 

func (c *Client) List(ctx context.Context, prefix string) ([]Object, error)

List lists every object under a prefix (all pages). prefix may be an s3://bucket/prefix URL, a bare prefix, or "" for the whole bucket.

func (*Client) Read 

func (c *Client) Read(ctx context.Context, key string) ([]byte, error)

Read reads an object into memory.

Example

A missing object is a typed error. 

package main

import (
	"context"
	"errors"
	"fmt"
	"log"

	"github.com/StrategicProjects/s3pe"
)

func main() {
	s3, err := s3pe.FromEnv()
	if err != nil {
		log.Fatal(err)
	}
	_, err = s3.Read(context.Background(), "s3://iaempe/does-not-exist.json")
	fmt.Println(errors.Is(err, s3pe.ErrNotFound))
}

func (*Client) Resolve 

func (c *Client) Resolve(key string) (bucket, objectKey string, err error)

Resolve splits s3://bucket/key or applies the default bucket to a bare key.

func (*Client) Upload 

func (c *Client) Upload(ctx context.Context, path, key, contentType string) (Transfer, error)

Upload streams a file to an object (two passes over the file: one to hash, one to send). A key that is empty or ends in "/" gets the file's base name appended. An empty contentType is guessed from the key.

func (*Client) Write 

func (c *Client) Write(ctx context.Context, key string, data []byte, contentType string) (Transfer, error)

Write writes bytes to an object. An empty contentType is guessed from the key's extension. The server's ETag is checked against the local MD5.

type Config 

type Config struct {
	// Endpoint is scheme://host[:port], no path.
	Endpoint string
	// Region is the signing region (us-east-1 for the Garage of Pernambuco).
	Region string
	// AccessKey and SecretKey are required, here or in the environment.
	AccessKey, SecretKey string
	// SessionToken is for temporary credentials.
	SessionToken string
	// Bucket is used when a key is not an s3:// URL.
	Bucket string
	// Timeout limits one call (default 300 s).
	Timeout time.Duration
	// MaxTries is the number of attempts per call (default 3).
	MaxTries int
	// HTTPClient replaces the default one (which follows no redirects).
	HTTPClient *http.Client
	// Logger receives one line per transfer (default: standard error).
	Logger *log.Logger
	// Quiet silences the transfer lines; warnings about retries still go out.
	Quiet bool
}

Config configures a Client. Zero fields are taken from the environment (S3PE_ENDPOINT or AWS_ENDPOINT_URL, AWS_REGION or AWS_DEFAULT_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, S3PE_BUCKET, S3PE_QUIET) and then from the defaults.

type Error 

type Error struct {
	// Status is the HTTP status, 0 when no response arrived.
	Status int
	// Code is the S3 error code (NoSuchKey, AccessDenied, …), empty if none.
	Code string
	// Message is the server's or the client's explanation.
	Message string
	// Bucket and Key name what was asked for (empty for bucket-level calls).
	Bucket, Key string
	// contains filtered or unexported fields
}

Error is what every failed call returns. errors.Is(err, ErrNotFound) and the other sentinels tell the kind; errors.As gives the details.

func (*Error) Error 

func (e *Error) Error() string

func (*Error) Is 

func (e *Error) Is(target error) bool

Is reports whether the error is of the sentinel kind target.

func (*Error) Unwrap 

func (e *Error) Unwrap() error

Unwrap returns the underlying error (network, file system, context), if any.

type Object 

type Object struct {
	// Key inside the bucket.
	Key string
	// Size in bytes.
	Size int64
	// LastModified is zero when the server's date could not be parsed.
	LastModified time.Time
	// ETag without the surrounding quotes.
	ETag string
	// ContentType is only known after Head.
	ContentType string
}

Object is one object as listed or described by the store.

func HeadS3 

func HeadS3(ctx context.Context, key string) (*Object, error)

HeadS3 returns an object's metadata (nil if absent), with a client built from the environment.

func ListS3 

func ListS3(ctx context.Context, prefix string) ([]Object, error)

ListS3 lists the objects under a prefix, with a client built from the environment.

type Transfer 

type Transfer struct {
	Bucket, Key string
	// Size is the number of bytes moved.
	Size int64
	// ETag reported by the server, without quotes.
	ETag string
	// Elapsed is the wall time of the whole call, retries included.
	Elapsed time.Duration
}

Transfer is what a completed upload or download reports.

func DownloadS3 

func DownloadS3(ctx context.Context, key, path string) (Transfer, error)

DownloadS3 downloads an object to a file, with a client built from the environment.

func UploadS3 

func UploadS3(ctx context.Context, path, key, contentType string) (Transfer, error)

UploadS3 uploads a file, with a client built from the environment.

func WriteS3 

func WriteS3(ctx context.Context, key string, data []byte, contentType string) (Transfer, error)

WriteS3 writes bytes to an object, with a client built from the environment.

func (Transfer) URL 

func (t Transfer) URL() string

URL is s3://bucket/key.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.