s3

README

s3

A pure-Go (CGO_ENABLED=0), from-scratch S3-backed block store that satisfies the go-volumes/pool Backing interface. It maps a flat byte address space onto fixed-size chunk objects in any S3-compatible bucket, with an in-memory write-back cache, so a copy-on-write volume pool — or any block consumer — can run on object storage with no local file.

No external dependencies: the Go standard library only. No aws-sdk-go, no minio-go. AWS Signature Version 4 is implemented from scratch on crypto/hmac + crypto/sha256.

Packages

Package	Purpose
sigv4	AWS Signature Version 4 request signer (canonical request → string-to-sign → kDate→kRegion→kService→kSigning chain → Authorization). Validated byte-for-byte against the official AWS Signature Version 4 Test Suite.
client	Minimal S3 client over net/http: Get / GetRange / Put / PutWithOptions / Delete / Head / ObjectLockConfiguration / List (ListObjectsV2 with continuation tokens, 8 MiB per page, a token that does not advance is an error), path-style + virtual-host addressing, XML error parsing, 404 → ErrNotFound, injectable HTTP doer, context cancellation.

PutWithOptions sends Content-Type, Content-Encoding, always Content-MD5, and Object Lock retention (x-amz-object-lock-mode GOVERNANCE/COMPLIANCE with x-amz-object-lock-retain-until-date) — Amazon S3 API Reference, PutObject: "The Content-MD5 … header is required for any request to upload an object with a retention period configured using Amazon S3 Object Lock". ObjectLockConfiguration reads GET /?object-lock (GetObjectLockConfiguration); a bucket without one (404/400 ObjectLockConfigurationNotFoundError, 501) reads as not enabled. MinIO and Ceph RGW both implement the bucket call; Object Lock must be enabled when the bucket is created and needs versioning. The two XML parsers are fuzzed (FuzzParseObjectLock, FuzzParseS3Error). | s3 (root) | Store implementing pool.Backing (ReadAt, WriteAt, Truncate, Sync, Close) over fixed-size chunk objects + a write-back cache + a persisted metadata object. |

Design

bucket layout (all under a configurable prefix):

  <prefix>/__meta              metadata: { logical size, chunk size }
  <prefix>/0000000000000000    chunk 0   (default 4 MiB)
  <prefix>/0000000000000001    chunk 1
  ...

• Chunked address space. The logical size lives in __meta, not in the set of present objects, so the store is sparse: a chunk that was never written reads back as zeros.
• Write-back cache. WriteAt does read-modify-write of the touched chunks in memory and marks them dirty; nothing hits S3 until Sync. Sync Puts every dirty chunk then the metadata object — this is the durability point.
• Truncate. Grows sparsely; on shrink it deletes chunk objects fully beyond the new size and zeroes the tail of the partially-kept final chunk.
• Reopen. Open reads __meta if present and adopts its size and chunk size, so a reopened store knows its geometry.
• Concurrency-safe via a mutex (the pool already calls under its own lock; the store is defensive anyway).

Example

cli, _ := client.New(client.Config{
    Endpoint: "https://s3.us-east-1.amazonaws.com",
    Region:   "us-east-1",
    Bucket:   "my-bucket",
    AccessKey: os.Getenv("AWS_ACCESS_KEY_ID"),
    SecretKey: os.Getenv("AWS_SECRET_ACCESS_KEY"),
})
store, _ := s3.Open(s3.Config{Client: cli, Prefix: "vol1"})

// Run a copy-on-write pool directly on S3:
p, _ := pool.CreateWith(store, 8<<20, 4096)
vol, _ := p.CreateVolume("data", 1<<20)
vol.WriteAt([]byte("hello"), 0)
p.Close() // Sync + Close — durably committed to S3

For MinIO / garage set PathStyle: true.

Tests

CGO_ENABLED=0, 100% statement coverage, gated in CI:

• sigv4: the official AWS test-suite vectors (get-vanilla, get-vanilla-query-order-key-case, get-header-value-trim, post-x-www-form-urlencoded, get-utf8) asserted byte-for-byte on the canonical request, string-to-sign, and Authorization; the get-vanilla signature and signing key pinned to AWS's published golden values.
• client: driven against an httptest.Server emulating GET / ranged-GET / PUT / DELETE / HEAD plus S3-style XML errors; ctx-cancel, non-2xx, 404 → ErrNotFound, malformed XML.
• Store: the real consumer — pool.CreateWith + volume write/read round-trip, cross-chunk and partial-chunk read-modify-write, sparse zero reads, EOF past size, Truncate up/down with chunk GC, Sync flushes dirty + meta, reopen-reads-meta, and client error propagation.

CI runs native (amd64 + arm64) and emulated (riscv64, loong64, ppc64le, s390x — the big-endian check for SigV4 byte handling) under QEMU.

License

BSD-3-Clause. See LICENSE.

Documentation

Overview

Package s3 implements an S3-backed block store: a flat byte address space mapped onto fixed-size chunk objects in an S3 bucket, with an in-memory write-back cache. The *Store type satisfies the go-volumes/pool Backing interface (io.ReaderAt, io.WriterAt, Truncate, Sync, io.Closer), so a copy-on-write volume pool — or any block consumer — can run on S3 with no local file.

Production code here keeps no dependency on go-volumes/pool; the Backing conformance is asserted only in the test file s3_conformance_test.go.

Layout in the bucket (all under a configurable key prefix):

<prefix>/__meta              little metadata object: logical size + chunk size
<prefix>/<016d chunk index>  one object per 4 MiB (default) data chunk

A chunk that was never written reads back as zeros (the store is sparse): the logical size in the metadata object — not the presence of chunk objects — defines the address space. Writes are buffered in the cache and flushed to S3 only on Sync (or Close), which is the durability point.

Index

• Constants
• type Config
• type Store
  • func Open(cfg Config) (*Store, error)
  • func (s *Store) ChunkSize() int64
  • func (s *Store) Close() error
  • func (s *Store) ReadAt(p []byte, off int64) (int, error)
  • func (s *Store) Size() int64
  • func (s *Store) Sync() error
  • func (s *Store) Truncate(size int64) error
  • func (s *Store) WriteAt(p []byte, off int64) (int, error)

Constants

const DefaultChunkSize = 4 << 20

DefaultChunkSize is the default chunk-object size (4 MiB).

Types

type Config

type Config struct {
	// Client is the S3 client (or any objectStore). Required.
	Client objectStore
	// Prefix is the key prefix under which chunk objects and metadata live.
	Prefix string
	// ChunkSize is the chunk-object size in bytes; must be a power of two >= 512.
	// Zero means DefaultChunkSize.
	ChunkSize int64
	// Context is used for all S3 operations. Zero means context.Background().
	Context context.Context
}

Config configures Open.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store is an S3-backed block store satisfying the pool.Backing shape.

func Open

func Open(cfg Config) (*Store, error)

Open opens (or, if no metadata object exists, initialises) a Store. If the metadata object is present its logical size and chunk size are adopted and the supplied ChunkSize, when non-zero, must match. Otherwise the store starts empty at size 0 with the requested (or default) chunk size.

func (*Store) ChunkSize

func (s *Store) ChunkSize() int64

ChunkSize returns the store's chunk size in bytes.

func (*Store) Close

func (s *Store) Close() error

Close syncs and releases the store. Subsequent operations return an error.

func (*Store) ReadAt

func (s *Store) ReadAt(p []byte, off int64) (int, error)

ReadAt implements io.ReaderAt. Reads past the logical size return io.EOF with the bytes read so far, matching os.File semantics.

func (*Store) Size

func (s *Store) Size() int64

Size returns the current logical size in bytes.

func (*Store) Sync

func (s *Store) Sync() error

Sync flushes every dirty chunk and then the metadata object to S3, clearing dirty flags. This is the store's durability point.

func (*Store) Truncate

func (s *Store) Truncate(size int64) error

Truncate sets the logical size. Growing is sparse (no objects written until a later WriteAt+Sync). Shrinking drops cached chunks and deletes S3 chunk objects that lie entirely beyond the new size; a partially-truncated final chunk has its tail zeroed and is marked dirty.

func (*Store) WriteAt

func (s *Store) WriteAt(p []byte, off int64) (int, error)

WriteAt implements io.WriterAt with read-modify-write of the touched chunks. Written bytes beyond the current logical size grow the size. Writes only mutate the cache; durability happens at Sync.