s3rp

package module
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 31, 2026 License: MIT Imports: 40 Imported by: 0

README

s3rp

s3rp is an S3 API reverse proxy with SigV4 re-signing.

[!WARNING] This is a proof of concept toward a multi-tenant S3-compatible object storage service, built to validate the architecture (SigV4 verification, operation reconstruction via aws-sdk-go-v2, tenant/user model, bucket policies). Do not use it for any other purpose. It is not production-ready: tenants and backends are defined in a static YAML file (a database-backed control plane is planned), the config format may change without notice, and no security review has been done.

Clients access s3rp using the S3 API (path-style) with access keys issued per tenant user. s3rp verifies the SigV4 signature of incoming requests, then executes the operations against per-bucket backends (any S3-compatible server: Ceph, versitygw, Amazon S3, etc.) using the backend's own credentials.

S3 client --(SigV4, front keys)--> s3rp --(SigV4, backend keys)--> S3-compatible backend

Use cases:

  • Issue per-tenant access keys in front of backends that make key management hard.
  • Hide the backend credentials and endpoints from clients.
  • Serve buckets belonging to different tenants and backends from a single endpoint.

Install

Binary

Download the binaries (s3rp and s3rp-admin) from Releases.

go install
$ go install github.com/fujiwara/s3rp/cmd/s3rp@latest
$ go install github.com/fujiwara/s3rp/cmd/s3rp-admin@latest

Usage

Usage: s3rp [flags]

S3 API reverse proxy with SigV4 re-signing

Flags:
  -h, --help                  Show context-sensitive help.
      --config="s3rp.yaml"    config file path ($S3RP_CONFIG)
      --listen=STRING         listen address (overrides config) ($S3RP_LISTEN)
      --log-level="info"      log level ($S3RP_LOG_LEVEL)
      --version               show version

Configuration

The config file is YAML. Environment variables in the file are expanded (${VAR} or $VAR).

A tenant owns one or more buckets and users. A user is the stable identity within a tenant (name: [a-z][a-z0-9_-]+); access keys are issued per user and rotate under it — add a new key, switch clients, then remove the old one. Every key of a tenant can access all of the tenant's buckets, unless restricted by a bucket policy.

listen: ":8080"
tenants:
  - name: acme                       # tenant identifier
    users:
      - name: app1                   # stable user identity
        keys:                        # access keys of the user (multiple for rotation)
          - access_key_id: S3RPKEY001
            secret_access_key: ${ACME_APP1_SECRET_001}
          - access_key_id: S3RPKEY002
            secret_access_key: ${ACME_APP1_SECRET_002}
      - name: batch
        keys:
          - access_key_id: S3RPKEY003
            secret_access_key: ${ACME_BATCH_SECRET_001}
    buckets:                         # buckets owned by this tenant
      - name: photos                 # bucket name on the front side
        backend:
          endpoint: http://ceph.internal:7480
          region: us-east-1          # default "us-east-1"
          bucket: photos-prod        # bucket name on the backend (default: same as name)
          access_key_id: ${CEPH_ACCESS_KEY_ID}
          secret_access_key: ${CEPH_SECRET_ACCESS_KEY}
          use_path_style: true       # default true
      - name: logs
        backend:
          # no endpoint: Amazon S3, resolved by the SDK from the region
          region: ap-northeast-1
          access_key_id: ${AWS_ACCESS_KEY_ID_FOR_LOGS}
          secret_access_key: ${AWS_SECRET_ACCESS_KEY_FOR_LOGS}

Notes:

  • Bucket names and access key ids must be unique across all tenants (path-style URLs carry no tenant discriminator). User names must be unique within a tenant.
  • When backend.endpoint is omitted, the backend is Amazon S3: the SDK resolves the endpoint from region, and use_path_style defaults to false (it defaults to true when an endpoint is set).
  • When backend.access_key_id and backend.secret_access_key are omitted, the SDK default credential chain is used (environment variables, shared config, IAM roles, etc.).
  • GET / (ListBuckets) returns the buckets of the key's tenant, with the tenant name as the owner.
  • Copying (CopyObject / UploadPartCopy) resolves the source within the requesting key's tenant, so cross-tenant copying is impossible.
Definition store

By default, tenants and buckets are defined directly in the YAML config as above. They can instead be read from a sqlite database:

listen: ":8080"
store:
  driver: sqlite
  dsn: s3rp.db
# tenants: must not be present with the sqlite driver

The proxy always opens the database read-only (a mode= parameter in the DSN is rejected). All writes go through the separate s3rp-admin binary, so the proxy deployment carries no write code or credentials:

$ s3rp-admin --dsn s3rp.db migrate                       # apply the schema (idempotent)
$ s3rp-admin --dsn s3rp.db import --config tenants.yaml  # load a tenants-form YAML into the DB

The schema lives in db/schema.sql, shared by the read side (proxy) and write side (admin) via sqlc-generated packages.

Client usage

Point any S3 client at s3rp with path-style addressing and a front-side key.

$ export AWS_ACCESS_KEY_ID=S3RPKEY001
$ export AWS_SECRET_ACCESS_KEY=...
$ aws --endpoint-url http://localhost:8080 s3api put-object --bucket photos --key foo.jpg --body foo.jpg
$ aws --endpoint-url http://localhost:8080 s3api get-object --bucket photos --key foo.jpg out.jpg
$ aws --endpoint-url http://localhost:8080 s3api list-objects-v2 --bucket photos

Supported operations

  • GetObject
  • PutObject
  • HeadObject
  • DeleteObject
  • DeleteObjects
  • CopyObject
  • ListObjects
  • ListObjectsV2
  • HeadBucket
  • GetBucketLocation
  • ListBuckets
  • GetObjectTagging
  • PutObjectTagging
  • DeleteObjectTagging
  • GetBucketVersioning
  • PutBucketVersioning
  • ListObjectVersions
  • GetBucketAcl
  • GetObjectAcl
  • GetBucketPolicy
  • GetBucketCors
  • CreateMultipartUpload
  • UploadPart
  • UploadPartCopy
  • CompleteMultipartUpload
  • AbortMultipartUpload
  • ListParts
  • ListMultipartUploads

Other operations return a NotImplemented error.

CopyObject and UploadPartCopy work between buckets served by the same backend (same endpoint, region and credentials); copying across different backends returns NotImplemented. The copy source bucket must belong to the requester's tenant.

The versionId query parameter is passed through on GetObject, HeadObject, DeleteObject, GetObjectAcl and the object tagging operations. Versioning requires a backend that supports it.

aws-chunked request bodies (STREAMING-AWS4-HMAC-SHA256-PAYLOAD and the trailer variants), which the AWS CLI and SDKs use for uploads over plain http endpoints, are decoded and their chunk signatures are verified.

Checksums

x-amz-checksum-* checksums (CRC32, CRC32C, CRC64NVME, SHA1, SHA256) flow end-to-end:

  • Precomputed checksum headers on uploads pass through to the backend, which validates and stores them.
  • Trailing checksums in aws-chunked bodies (the SDK default) are verified by the proxy against the decoded payload (BadDigest on mismatch), and the algorithm is forwarded so the backend recomputes and stores the checksum.
  • Downloads pass x-amz-checksum-mode: ENABLED through and return the backend's checksum headers, so client SDKs can validate response payloads. Multipart part checksums are carried through UploadPart / CompleteMultipartUpload as well.

Whether a checksum is actually stored and returned depends on the backend (versitygw and Amazon S3 do; some Ceph RGW builds do not).

Bucket policies

A bucket may carry an AWS-style policy document, written as JSON text in the config (buckets[].policy) or the database. GetBucketPolicy returns it; PutBucketPolicy / DeleteBucketPolicy are not supported (policies are defined in the store, not via the S3 API).

Two simplifications against AWS: principals are plain user names of the tenant under the S3RP key (no ARNs), and resources are plain "bucket" / "bucket/prefix*" strings (no ARNs). * in Action / Resource matches any characters including /.

buckets:
  - name: photos
    backend: { ... }
    policy: |
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Sid": "BatchIsReadOnly",
            "Effect": "Deny",
            "Principal": {"S3RP": ["batch"]},
            "Action": ["s3:PutObject", "s3:DeleteObject"],
            "Resource": ["photos/*"]
          }
        ]
      }

Evaluation model: every user of a tenant has full access to the tenant's buckets by default, and explicit Deny statements restrict it. Allow statements are accepted but have no effect yet (everything is already allowed); they will become meaningful when anonymous and cross-tenant access are introduced.

Principal forms:

  • {"S3RP": ["name", ...]} — the listed users of the tenant.
  • "*" — all users, including ones added later. Note that the scope of "*" will widen when anonymous / cross-tenant access arrives (an Allow with "*" will then mean public access).
  • NotPrincipal (exclusive with Principal) — everyone except the listed users. Deny + NotPrincipal expresses "only these users may ..." so that newly added users are denied by default.

Limitations: policies only cover users of the owning tenant; versioned operations use the same action names as unversioned ones (no s3:GetObjectVersion distinction). DeleteObjects is evaluated per object: denied keys are reported in the Error entries of the response. Copying evaluates s3:GetObject on the source and s3:PutObject on the destination.

CORS

CORS is handled by the proxy itself (it is a contract between the browser and the server the browser talks to, so backend CORS settings are not passed through). Rules are defined per bucket:

buckets:
  - name: photos
    backend: { ... }
    cors:
      - allowed_origins: ["https://app.example.com", "https://*.preview.example.com"]
        allowed_methods: [GET, PUT]   # GET, PUT, POST, DELETE, HEAD
        allowed_headers: ["*"]
        expose_headers: [ETag]
        max_age_seconds: 3600

Preflight OPTIONS requests are answered without authentication based on these rules, which makes browser-direct uploads via presigned URLs work. Actual responses carry Access-Control-Allow-Origin / Access-Control-Allow-Credentials / Access-Control-Expose-Headers when the request's Origin matches a rule. * in allowed_origins matches any characters (e.g. https://*.example.com).

GetBucketCors returns the configuration (NoSuchCORSConfiguration when absent); PutBucketCors / DeleteBucketCors are not supported (rules are defined in the store, not via the S3 API).

ACLs

s3rp behaves like a bucket with ACLs disabled (Object Ownership = bucket owner enforced, the AWS default since 2023). GetBucketAcl / GetObjectAcl return a fixed policy granting FULL_CONTROL to the tenant; PutBucketAcl / PutObjectAcl return AccessControlListNotSupported, and canned ACLs other than private / bucket-owner-full-control are rejected on uploads. Use bucket policies for access control instead.

Presigned URLs

Presigned URLs (SigV4 query string authentication) generated with front-side keys against the s3rp endpoint are supported for the operations above. Expiry (X-Amz-Expires, up to 7 days) is enforced.

$ aws --endpoint-url http://localhost:8080 s3 presign s3://photos/foo.jpg
http://localhost:8080/photos/foo.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&...

Limitations

  • The payload SHA-256 declared in x-amz-content-sha256 is not independently verified against the request body (the signature covers the declared hash; verifying the body would require buffering it). Chunk signatures and trailing checksums of aws-chunked bodies are verified.
  • Requests that sign the user-agent or other headers the AWS SDK signer ignores will fail verification. Real AWS SDK/CLI clients do not do this.

Development

Unit tests run without any backend:

$ go test -race ./...

The integration test suite runs against a real S3-compatible backend, selected by environment variables. Two backends are provided in compose.yml:

# versitygw (lightweight, default)
$ docker compose up -d --wait versitygw
$ S3RP_TEST_BACKEND_ENDPOINT=http://localhost:7070 go test -race -run TestIntegration ./...

# Ceph RGW (heavyweight, compatibility check)
$ docker compose up -d --wait ceph
$ S3RP_TEST_BACKEND_ENDPOINT=http://127.0.0.1:7480 go test -race -run TestIntegration ./...

Note: access Ceph RGW via 127.0.0.1, not localhost — RGW resolves Host names that do not match its rgw dns name as virtual-hosted bucket names. CI runs the integration suite against both backends as a matrix.

LICENSE

MIT

Author

fujiwara

Documentation

Index

Constants

View Source 
const (
	DefaultListen = ":8080"
	DefaultRegion = "us-east-1"
)

Variables

View Source 
var Version = "v0.0.1"

Functions

func NewConfigStore 

func NewConfigStore(cfg *Config) store.Store

NewConfigStore builds a store.Store from a validated config.

func Run 

func Run(ctx context.Context) error

Run parses the command line, loads the config and serves until ctx is done.

Types

type AccessControlPolicy 

type AccessControlPolicy struct {
	XMLName           xml.Name `xml:"AccessControlPolicy"`
	XMLNS             string   `xml:"xmlns,attr"`
	Owner             Owner    `xml:"Owner"`
	AccessControlList struct {
		Grants []Grant `xml:"Grant"`
	} `xml:"AccessControlList"`
}

AccessControlPolicy is the response of GetBucketAcl / GetObjectAcl.

type BackendClient 

type BackendClient interface {
	GetObject(ctx context.Context, in *s3.GetObjectInput, optFns ...func(*s3.Options)) (*s3.GetObjectOutput, error)
	PutObject(ctx context.Context, in *s3.PutObjectInput, optFns ...func(*s3.Options)) (*s3.PutObjectOutput, error)
	HeadObject(ctx context.Context, in *s3.HeadObjectInput, optFns ...func(*s3.Options)) (*s3.HeadObjectOutput, error)
	DeleteObject(ctx context.Context, in *s3.DeleteObjectInput, optFns ...func(*s3.Options)) (*s3.DeleteObjectOutput, error)
	ListObjectsV2(ctx context.Context, in *s3.ListObjectsV2Input, optFns ...func(*s3.Options)) (*s3.ListObjectsV2Output, error)
	HeadBucket(ctx context.Context, in *s3.HeadBucketInput, optFns ...func(*s3.Options)) (*s3.HeadBucketOutput, error)
	CreateMultipartUpload(ctx context.Context, in *s3.CreateMultipartUploadInput, optFns ...func(*s3.Options)) (*s3.CreateMultipartUploadOutput, error)
	UploadPart(ctx context.Context, in *s3.UploadPartInput, optFns ...func(*s3.Options)) (*s3.UploadPartOutput, error)
	CompleteMultipartUpload(ctx context.Context, in *s3.CompleteMultipartUploadInput, optFns ...func(*s3.Options)) (*s3.CompleteMultipartUploadOutput, error)
	AbortMultipartUpload(ctx context.Context, in *s3.AbortMultipartUploadInput, optFns ...func(*s3.Options)) (*s3.AbortMultipartUploadOutput, error)
	ListParts(ctx context.Context, in *s3.ListPartsInput, optFns ...func(*s3.Options)) (*s3.ListPartsOutput, error)
	ListMultipartUploads(ctx context.Context, in *s3.ListMultipartUploadsInput, optFns ...func(*s3.Options)) (*s3.ListMultipartUploadsOutput, error)
	ListObjects(ctx context.Context, in *s3.ListObjectsInput, optFns ...func(*s3.Options)) (*s3.ListObjectsOutput, error)
	DeleteObjects(ctx context.Context, in *s3.DeleteObjectsInput, optFns ...func(*s3.Options)) (*s3.DeleteObjectsOutput, error)
	CopyObject(ctx context.Context, in *s3.CopyObjectInput, optFns ...func(*s3.Options)) (*s3.CopyObjectOutput, error)
	UploadPartCopy(ctx context.Context, in *s3.UploadPartCopyInput, optFns ...func(*s3.Options)) (*s3.UploadPartCopyOutput, error)
	GetBucketVersioning(ctx context.Context, in *s3.GetBucketVersioningInput, optFns ...func(*s3.Options)) (*s3.GetBucketVersioningOutput, error)
	PutBucketVersioning(ctx context.Context, in *s3.PutBucketVersioningInput, optFns ...func(*s3.Options)) (*s3.PutBucketVersioningOutput, error)
	ListObjectVersions(ctx context.Context, in *s3.ListObjectVersionsInput, optFns ...func(*s3.Options)) (*s3.ListObjectVersionsOutput, error)
	GetObjectTagging(ctx context.Context, in *s3.GetObjectTaggingInput, optFns ...func(*s3.Options)) (*s3.GetObjectTaggingOutput, error)
	PutObjectTagging(ctx context.Context, in *s3.PutObjectTaggingInput, optFns ...func(*s3.Options)) (*s3.PutObjectTaggingOutput, error)
	DeleteObjectTagging(ctx context.Context, in *s3.DeleteObjectTaggingInput, optFns ...func(*s3.Options)) (*s3.DeleteObjectTaggingOutput, error)
}

BackendClient is the narrow interface of the S3 client methods s3rp uses, for injecting stubs in tests.

type BackendConfig 

type BackendConfig = store.Backend

Password and BackendConfig are defined in the store package; the aliases keep the config schema in one place with the rest of the config types.

type BucketConfig 

type BucketConfig struct {
	Name    string            `yaml:"name" json:"name"`
	Backend *BackendConfig    `yaml:"backend" json:"backend"`
	Policy  string            `yaml:"policy,omitempty" json:"policy,omitempty"` // bucket policy JSON text
	CORS    []*store.CORSRule `yaml:"cors,omitempty" json:"cors,omitempty"`
}

type BucketEntry 

type BucketEntry struct {
	Name         string `xml:"Name"`
	CreationDate string `xml:"CreationDate"`
}

type CLI 

type CLI struct {
	Config   string           `help:"config file path" default:"s3rp.yaml" env:"S3RP_CONFIG"`
	Listen   string           `help:"listen address (overrides config)" env:"S3RP_LISTEN"`
	LogLevel string           `help:"log level" default:"info" enum:"debug,info,warn,error" env:"S3RP_LOG_LEVEL"`
	Version  kong.VersionFlag `help:"show version"`
}

type CORSConfiguration 

type CORSConfiguration struct {
	XMLName xml.Name      `xml:"CORSConfiguration"`
	XMLNS   string        `xml:"xmlns,attr"`
	Rules   []CORSRuleXML `xml:"CORSRule"`
}

CORSConfiguration is the response of GetBucketCors.

type CORSRuleXML 

type CORSRuleXML struct {
	AllowedOrigin []string `xml:"AllowedOrigin"`
	AllowedMethod []string `xml:"AllowedMethod"`
	AllowedHeader []string `xml:"AllowedHeader,omitempty"`
	ExposeHeader  []string `xml:"ExposeHeader,omitempty"`
	MaxAgeSeconds int      `xml:"MaxAgeSeconds,omitempty"`
}

type CommonPrefix 

type CommonPrefix struct {
	Prefix string `xml:"Prefix"`
}

type CompleteMultipartUploadResult 

type CompleteMultipartUploadResult struct {
	XMLName           xml.Name `xml:"CompleteMultipartUploadResult"`
	XMLNS             string   `xml:"xmlns,attr"`
	Location          string   `xml:"Location"`
	Bucket            string   `xml:"Bucket"`
	Key               string   `xml:"Key"`
	ETag              string   `xml:"ETag"`
	ChecksumCRC32     string   `xml:"ChecksumCRC32,omitempty"`
	ChecksumCRC32C    string   `xml:"ChecksumCRC32C,omitempty"`
	ChecksumCRC64NVME string   `xml:"ChecksumCRC64NVME,omitempty"`
	ChecksumSHA1      string   `xml:"ChecksumSHA1,omitempty"`
	ChecksumSHA256    string   `xml:"ChecksumSHA256,omitempty"`
	ChecksumType      string   `xml:"ChecksumType,omitempty"`
}

CompleteMultipartUploadResult is the response of CompleteMultipartUpload.

type Config 

type Config struct {
	Listen  string          `yaml:"listen" json:"listen"`
	Store   *StoreConfig    `yaml:"store,omitempty" json:"store,omitempty"`
	Tenants []*TenantConfig `yaml:"tenants,omitempty" json:"tenants,omitempty"`
}

func LoadConfig 

func LoadConfig(path string) (*Config, error)

LoadConfig reads a YAML config file, expanding environment variables in the content.

func (*Config) SetDefaults 

func (c *Config) SetDefaults()

func (*Config) StoreDriver 

func (c *Config) StoreDriver() string

StoreDriver returns the effective store driver.

func (*Config) Validate 

func (c *Config) Validate() error

type CopyObjectResult 

type CopyObjectResult struct {
	XMLName      xml.Name `xml:"CopyObjectResult"`
	XMLNS        string   `xml:"xmlns,attr"`
	ETag         string   `xml:"ETag"`
	LastModified string   `xml:"LastModified,omitempty"`
}

CopyObjectResult is the response of CopyObject.

type CopyPartResult 

type CopyPartResult struct {
	XMLName      xml.Name `xml:"CopyPartResult"`
	XMLNS        string   `xml:"xmlns,attr"`
	ETag         string   `xml:"ETag"`
	LastModified string   `xml:"LastModified,omitempty"`
}

CopyPartResult is the response of UploadPartCopy.

type DeleteError 

type DeleteError struct {
	Key       string `xml:"Key"`
	VersionID string `xml:"VersionId,omitempty"`
	Code      string `xml:"Code"`
	Message   string `xml:"Message"`
}

type DeleteMarkerEntry 

type DeleteMarkerEntry struct {
	Key          string `xml:"Key"`
	VersionID    string `xml:"VersionId"`
	IsLatest     bool   `xml:"IsLatest"`
	LastModified string `xml:"LastModified,omitempty"`
	Owner        *Owner `xml:"Owner,omitempty"`
}

type DeleteResult 

type DeleteResult struct {
	XMLName xml.Name        `xml:"DeleteResult"`
	XMLNS   string          `xml:"xmlns,attr"`
	Deleted []DeletedObject `xml:"Deleted"`
	Errors  []DeleteError   `xml:"Error"`
}

DeleteResult is the response of DeleteObjects.

type DeletedObject 

type DeletedObject struct {
	Key                   string `xml:"Key"`
	VersionID             string `xml:"VersionId,omitempty"`
	DeleteMarker          bool   `xml:"DeleteMarker,omitempty"`
	DeleteMarkerVersionID string `xml:"DeleteMarkerVersionId,omitempty"`
}

type Grant 

type Grant struct {
	Grantee    Grantee `xml:"Grantee"`
	Permission string  `xml:"Permission"`
}

type Grantee 

type Grantee struct {
	XMLNSXSI    string `xml:"xmlns:xsi,attr"`
	Type        string `xml:"xsi:type,attr"`
	ID          string `xml:"ID"`
	DisplayName string `xml:"DisplayName,omitempty"`
}

type InitiateMultipartUploadResult 

type InitiateMultipartUploadResult struct {
	XMLName  xml.Name `xml:"InitiateMultipartUploadResult"`
	XMLNS    string   `xml:"xmlns,attr"`
	Bucket   string   `xml:"Bucket"`
	Key      string   `xml:"Key"`
	UploadID string   `xml:"UploadId"`
}

InitiateMultipartUploadResult is the response of CreateMultipartUpload.

type KeyConfig 

type KeyConfig struct {
	AccessKeyID     string   `yaml:"access_key_id" json:"access_key_id"`
	SecretAccessKey Password `yaml:"secret_access_key" json:"secret_access_key"`
}

type ListAllMyBucketsResult 

type ListAllMyBucketsResult struct {
	XMLName xml.Name `xml:"ListAllMyBucketsResult"`
	XMLNS   string   `xml:"xmlns,attr"`
	Owner   Owner    `xml:"Owner"`
	Buckets struct {
		Bucket []BucketEntry `xml:"Bucket"`
	} `xml:"Buckets"`
}

ListAllMyBucketsResult is the response of ListBuckets.

type ListBucketResult 

type ListBucketResult struct {
	XMLName               xml.Name       `xml:"ListBucketResult"`
	XMLNS                 string         `xml:"xmlns,attr"`
	Name                  string         `xml:"Name"`
	Prefix                string         `xml:"Prefix"`
	StartAfter            string         `xml:"StartAfter,omitempty"`
	ContinuationToken     string         `xml:"ContinuationToken,omitempty"`
	NextContinuationToken string         `xml:"NextContinuationToken,omitempty"`
	KeyCount              int32          `xml:"KeyCount"`
	MaxKeys               int32          `xml:"MaxKeys"`
	Delimiter             string         `xml:"Delimiter,omitempty"`
	EncodingType          string         `xml:"EncodingType,omitempty"`
	IsTruncated           bool           `xml:"IsTruncated"`
	Contents              []Object       `xml:"Contents"`
	CommonPrefixes        []CommonPrefix `xml:"CommonPrefixes"`
}

ListBucketResult is the response of ListObjectsV2.

type ListBucketResultV1 

type ListBucketResultV1 struct {
	XMLName        xml.Name       `xml:"ListBucketResult"`
	XMLNS          string         `xml:"xmlns,attr"`
	Name           string         `xml:"Name"`
	Prefix         string         `xml:"Prefix"`
	Marker         string         `xml:"Marker"`
	NextMarker     string         `xml:"NextMarker,omitempty"`
	MaxKeys        int32          `xml:"MaxKeys"`
	Delimiter      string         `xml:"Delimiter,omitempty"`
	EncodingType   string         `xml:"EncodingType,omitempty"`
	IsTruncated    bool           `xml:"IsTruncated"`
	Contents       []Object       `xml:"Contents"`
	CommonPrefixes []CommonPrefix `xml:"CommonPrefixes"`
}

ListBucketResultV1 is the response of ListObjects (version 1).

type ListMultipartUploadsResult 

type ListMultipartUploadsResult struct {
	XMLName            xml.Name       `xml:"ListMultipartUploadsResult"`
	XMLNS              string         `xml:"xmlns,attr"`
	Bucket             string         `xml:"Bucket"`
	KeyMarker          string         `xml:"KeyMarker,omitempty"`
	UploadIDMarker     string         `xml:"UploadIdMarker,omitempty"`
	NextKeyMarker      string         `xml:"NextKeyMarker,omitempty"`
	NextUploadIDMarker string         `xml:"NextUploadIdMarker,omitempty"`
	Delimiter          string         `xml:"Delimiter,omitempty"`
	Prefix             string         `xml:"Prefix,omitempty"`
	MaxUploads         int32          `xml:"MaxUploads"`
	IsTruncated        bool           `xml:"IsTruncated"`
	Uploads            []Upload       `xml:"Upload"`
	CommonPrefixes     []CommonPrefix `xml:"CommonPrefixes"`
}

ListMultipartUploadsResult is the response of ListMultipartUploads.

type ListPartsResult 

type ListPartsResult struct {
	XMLName              xml.Name `xml:"ListPartsResult"`
	XMLNS                string   `xml:"xmlns,attr"`
	Bucket               string   `xml:"Bucket"`
	Key                  string   `xml:"Key"`
	UploadID             string   `xml:"UploadId"`
	PartNumberMarker     string   `xml:"PartNumberMarker,omitempty"`
	NextPartNumberMarker string   `xml:"NextPartNumberMarker,omitempty"`
	MaxParts             int32    `xml:"MaxParts"`
	IsTruncated          bool     `xml:"IsTruncated"`
	Parts                []Part   `xml:"Part"`
	Initiator            *Owner   `xml:"Initiator,omitempty"`
	Owner                *Owner   `xml:"Owner,omitempty"`
	StorageClass         string   `xml:"StorageClass,omitempty"`
}

ListPartsResult is the response of ListParts.

type ListVersionsResult 

type ListVersionsResult struct {
	XMLName             xml.Name            `xml:"ListVersionsResult"`
	XMLNS               string              `xml:"xmlns,attr"`
	Name                string              `xml:"Name"`
	Prefix              string              `xml:"Prefix"`
	KeyMarker           string              `xml:"KeyMarker"`
	VersionIDMarker     string              `xml:"VersionIdMarker"`
	NextKeyMarker       string              `xml:"NextKeyMarker,omitempty"`
	NextVersionIDMarker string              `xml:"NextVersionIdMarker,omitempty"`
	MaxKeys             int32               `xml:"MaxKeys"`
	Delimiter           string              `xml:"Delimiter,omitempty"`
	EncodingType        string              `xml:"EncodingType,omitempty"`
	IsTruncated         bool                `xml:"IsTruncated"`
	Versions            []ObjectVersion     `xml:"Version"`
	DeleteMarkers       []DeleteMarkerEntry `xml:"DeleteMarker"`
	CommonPrefixes      []CommonPrefix      `xml:"CommonPrefixes"`
}

ListVersionsResult is the response of ListObjectVersions.

type LocationConstraint 

type LocationConstraint struct {
	XMLName xml.Name `xml:"LocationConstraint"`
	XMLNS   string   `xml:"xmlns,attr"`
	Value   string   `xml:",chardata"`
}

LocationConstraint is the response of GetBucketLocation.

type Object 

type Object struct {
	Key          string `xml:"Key"`
	LastModified string `xml:"LastModified"`
	ETag         string `xml:"ETag"`
	Size         int64  `xml:"Size"`
	StorageClass string `xml:"StorageClass,omitempty"`
	Owner        *Owner `xml:"Owner,omitempty"`
}

type ObjectVersion 

type ObjectVersion struct {
	Key          string `xml:"Key"`
	VersionID    string `xml:"VersionId"`
	IsLatest     bool   `xml:"IsLatest"`
	LastModified string `xml:"LastModified,omitempty"`
	ETag         string `xml:"ETag"`
	Size         int64  `xml:"Size"`
	StorageClass string `xml:"StorageClass,omitempty"`
	Owner        *Owner `xml:"Owner,omitempty"`
}

type Owner 

type Owner struct {
	ID          string `xml:"ID"`
	DisplayName string `xml:"DisplayName"`
}

type Part 

type Part struct {
	PartNumber   int32  `xml:"PartNumber"`
	LastModified string `xml:"LastModified,omitempty"`
	ETag         string `xml:"ETag"`
	Size         int64  `xml:"Size"`
}

type Password 

type Password = store.Password

Password and BackendConfig are defined in the store package; the aliases keep the config schema in one place with the rest of the config types.

type S3Error 

type S3Error struct {
	XMLName   xml.Name `xml:"Error"`
	Code      string   `xml:"Code"`
	Message   string   `xml:"Message"`
	Resource  string   `xml:"Resource,omitempty"`
	RequestID string   `xml:"RequestId,omitempty"`
	// contains filtered or unexported fields
}

S3Error is an S3 API error response. https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html

func (*S3Error) Error 

func (e *S3Error) Error() string

func (*S3Error) Status 

func (e *S3Error) Status() int

type S3RP 

type S3RP struct {
	// contains filtered or unexported fields
}

S3RP is an S3 API reverse proxy that verifies SigV4 signatures with front-side access keys and forwards operations to per-bucket backends.

func New 

func New(ctx context.Context, cfg *Config) (*S3RP, error)

New creates an S3RP from a config, selecting the definition store by the store.driver setting.

func NewWithStore 

func NewWithStore(_ context.Context, cfg *Config, store store.Store) (*S3RP, error)

NewWithStore creates an S3RP using the given Store for tenant, key and bucket definitions.

func (*S3RP) Handler 

func (app *S3RP) Handler() http.Handler

Handler returns the http.Handler of the proxy.

A single catch-all route is used instead of ServeMux patterns because the mux cleans paths (collapsing // and resolving dot segments) and redirects, which breaks S3 keys and signature verification.

func (*S3RP) Serve 

func (app *S3RP) Serve(ctx context.Context) error

Serve runs the HTTP server until ctx is done, then shuts down gracefully.

type StoreConfig 

type StoreConfig struct {
	Driver string `yaml:"driver" json:"driver"` // "yaml" (default) or "sqlite"
	DSN    string `yaml:"dsn,omitempty" json:"dsn,omitempty"`
}

StoreConfig selects where tenant/user/bucket definitions come from.

type Tag 

type Tag struct {
	Key   string `xml:"Key"`
	Value string `xml:"Value"`
}

type TagSet 

type TagSet struct {
	Tags []Tag `xml:"Tag"`
}

type Tagging 

type Tagging struct {
	XMLName xml.Name `xml:"Tagging"`
	XMLNS   string   `xml:"xmlns,attr,omitempty"`
	TagSet  TagSet   `xml:"TagSet"`
}

Tagging is the request and response body of PutObjectTagging / GetObjectTagging.

type TenantConfig 

type TenantConfig struct {
	Name    string          `yaml:"name" json:"name"`
	Users   []*UserConfig   `yaml:"users" json:"users"`
	Buckets []*BucketConfig `yaml:"buckets" json:"buckets"`
}

TenantConfig defines a tenant: its users and the buckets it owns.

type Upload 

type Upload struct {
	Key          string `xml:"Key"`
	UploadID     string `xml:"UploadId"`
	Initiator    *Owner `xml:"Initiator,omitempty"`
	Owner        *Owner `xml:"Owner,omitempty"`
	StorageClass string `xml:"StorageClass,omitempty"`
	Initiated    string `xml:"Initiated,omitempty"`
}

type UserConfig 

type UserConfig struct {
	Name string       `yaml:"name" json:"name"`
	Keys []*KeyConfig `yaml:"keys" json:"keys"`
}

UserConfig defines a user of a tenant. The user name is the stable identity (e.g. for policy principals); access keys rotate under it.

type VersioningConfiguration 

type VersioningConfiguration struct {
	XMLName xml.Name `xml:"VersioningConfiguration"`
	XMLNS   string   `xml:"xmlns,attr,omitempty"`
	Status  string   `xml:"Status,omitempty"`
}

VersioningConfiguration is the request and response body of PutBucketVersioning / GetBucketVersioning.

Source Files

View all Source files

Directories

Path Synopsis
cmd
s3rp command
s3rp-admin command
s3rp-admin is the write-side tooling for the s3rp database: schema migration and importing a YAML config.
 s3rp-admin is the write-side tooling for the s3rp database: schema migration and importing a YAML config.
db
Package db holds the shared schema and the write-side operations (migration and importing a YAML config).
 Package db holds the shared schema and the write-side operations (migration and importing a YAML config).
writedb
Package policy implements AWS-style bucket policy documents for s3rp.
 Package policy implements AWS-style bucket policy documents for s3rp.
Package store defines the read-only contract for tenant, key and bucket definitions used by s3rp.
 Package store defines the read-only contract for tenant, key and bucket definitions used by s3rp.
rdb
Package rdb is a read-only store.Store implementation backed by a relational database (sqlite for the PoC), using the sqlc-generated queries in the readdb package.
 Package rdb is a read-only store.Store implementation backed by a relational database (sqlite for the PoC), using the sqlc-generated queries in the readdb package.
rdb/readdb

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.