s3

package
v0.13.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 9, 2026 License: Apache-2.0 Imports: 41 Imported by: 0

Documentation

Overview

Package s3 provides an AWS S3 backend with storage tier management.

This package implements the core object storage functionality for ObjectFS: S3 integration with multi-tier storage support and cost accounting.

Throughput figures are deliberately absent, here and everywhere else in this repository. This doc comment used to open by claiming "up to 4.6x performance improvements over standard S3 operations", repeated twice more below. Nothing in this repository measured that number, and no benchmark here can produce it — it came from CargoShip's own reporting on CargoShip's own workload, and was restated as a property of ObjectFS. A reader had no way to tell it apart from something this project had measured. See benchmarks/ for what can actually be run against a named bucket and object size. The transporter that figure was attributed to is gone (#362); when it was finally measured here it was 35% slower at 4 KiB and a wash at 8 MiB.

Architecture Overview

The S3 backend provides multiple layers of functionality: 

┌─────────────────────────────────────────────────────────────┐
│                   ObjectFS Interface                       │
│              (types.Backend Implementation)                │
└─────────────────────────────────────────────────────────────┘
                          │
┌─────────────────────────────────────────────────────────────┐
│                    S3 Backend Layer                        │
│  ┌─────────────────┐ ┌──────────────┐ ┌─────────────────┐  │
│  │  Cost Optimizer │ │ Tier Manager │ │ Pricing Manager │  │
│  └─────────────────┘ └──────────────┘ └─────────────────┘  │
└─────────────────────────────────────────────────────────────┘
                          │
┌─────────────────────────────────────────────────────────────┐
│                 AWS S3 Service                             │
│    Connection Pool  │  Multiple Regions  │  Storage Tiers  │
└─────────────────────────────────────────────────────────────┘

One upload path

PutObject has a single implementation. Until v0.15.0 a config flag routed it through CargoShip's transporter instead of the direct SDK call, for that library's BBR/CUBIC congestion control; #362 removed both the flag and the branch.

The reason is worth recording, because it is the failure mode this package is most exposed to. A cargoships3.Archive cannot express what a PutObjectInput expresses — there is no field for a Content-Encoding, for the configured encryption headers, or for a per-object storage class — and the branch had grown three bypasses saying so. The fourth of the same kind was not bypassed: Content-Type was written into Archive.Metadata, which is S3 user metadata rather than the header, so every object under MultipartThreshold was stored as application/octet-stream while detectContentType had computed the right value one line earlier. Each of those failures leaves a readable object, which is why none of them surfaced without an assertion made at the endpoint.

There was also nothing on the other side. The transporter was only reachable *below* MultipartThreshold, since an object at or above it returned into this package's own multipart path, which never consulted a transporter — so the 64 MiB multipart buffer it installed served an upload shape a mount cannot produce, and sync.Pool released that buffer at every GC cycle. See upload_path_test.go for the assertions that now hold at the boundary regardless of what runs behind it.

Storage Tier Management

Comprehensive support for all AWS S3 storage classes.

Rates are deliberately not restated here. This section used to carry a per-GB figure for each tier and a Cost/GB column in the summary table below, which made it two more copies of the S3 rate card — and a rate in a doc comment has no way to be told it is stale, so the only question is when it starts lying rather than whether. internal/awsrates holds every rate, in one place, checked against the live AWS Pricing API by a test. Read it there, or call PricingManager.GetTierPricing, which serves from it.

What stays here is the part that is S3 *behavior* rather than S3 *price*: minimum billable size, minimum storage duration, and retrieval latency. Those change when AWS changes the product, not when AWS changes a number.

Standard Tier (STANDARD): - Instant access, no retrieval costs - Recommended for frequently accessed data - No minimum object size or storage duration

Standard-IA (STANDARD_IA): - Instant access with retrieval costs - 128KB minimum object size - 30-day minimum storage duration - Retrieval is charged per GB

One Zone-IA (ONEZONE_IA): - Single availability zone storage - Lower cost than Standard-IA - Same constraints as Standard-IA - Retrieval is charged per GB

Glacier Instant Retrieval (GLACIER_IR): - Instant access for archive data - 128KB minimum object size - 90-day minimum storage duration - Retrieval is charged per GB, at the highest rate of the instant-access tiers

Glacier Flexible Retrieval (GLACIER): - Minutes to hours retrieval time - 40KB minimum object size - 90-day minimum storage duration - Retrieval is charged per GB and varies by requested speed

Deep Archive (DEEP_ARCHIVE): - Lowest cost, hours for retrieval - 40KB minimum object size - 180-day minimum storage duration - Retrieval is charged per GB and varies by requested speed

Intelligent Tiering (INTELLIGENT_TIERING): - Automatic tier optimization - No retrieval charges - Priced as Standard for the frequent-access tier, plus per-object monitoring charges

Cost Optimization

Advanced cost optimization capabilities:

Intelligent Tier Selection: The system analyzes access patterns and automatically recommends optimal storage tiers:

Access patterns are recorded by GetObject as reads happen; the report is derived from what has been observed, so a freshly-opened backend has nothing to say yet. 

report := backend.GetCostOptimizationReport()

for _, o := range report.OptimizationResults {
	fmt.Printf("%s: %s → %s (%s), $%.2f/month, confidence %.0f%%\n",
		o.ObjectKey, o.FromTier, o.ToTier, o.Reason,
		o.EstimatedMonthlySavings, o.ConfidenceLevel*100)
}

fmt.Printf("%d objects, $%.2f/month total\n",
	report.TotalObjects, report.TotalPotentialSavings)

Note that recording is gated on Config.MonitorAccessPatterns, which defaults false — with it off, the report is empty rather than wrong.

Enterprise Pricing Support: - Volume discount calculation - Reserved capacity pricing - Custom enterprise rates - Multi-region cost analysis

Configuration

Flexible configuration options: 

config := &s3.Config{
	Region:   "us-west-2",
	Endpoint: "", // empty: the AWS endpoint for the region

	// Connections and timeouts
	PoolSize:       8,
	ConnectTimeout: 10 * time.Second,
	RequestTimeout: 30 * time.Second,

	// Storage tier
	StorageTier: s3.TierStandard,
}

Every field may be omitted. NewBackend fills in a default for each one whose zero value is not a usable setting, so &s3.Config{Region: "us-west-2"} is a complete configuration — see NewBackend. The exception is ParallelReadThreshold, where zero means "off" rather than "unset".

Usage Examples

Basic backend initialization: 

backend, err := s3.NewBackend(ctx, "my-bucket", config)
if err != nil {
	log.Fatal(err)
}
defer backend.Close()

Object operations with automatic optimization: 

// Put object with automatic tier selection. The final argument is user metadata, stored as
// x-amz-meta-* headers; nil is fine. ObjectFS writes its own integrity keys there
// (objectfs-sha256, objectfs-original-size) last, so those two names win over anything you
// supply under them — they describe the bytes actually uploaded, which after compression are
// not the bytes you handed in.
err := backend.PutObject(ctx, "data/file.txt", data, nil)

// Get object; -1 reads to the end
data, err := backend.GetObject(ctx, "data/file.txt", 0, -1)

// Head object for metadata
info, err := backend.HeadObject(ctx, "data/file.txt")

Batch operations for improved performance: 

// Batch get operations
keys := []string{"file1.txt", "file2.txt", "file3.txt"}
results, err := backend.GetObjects(ctx, keys)

// Batch put operations
objects := map[string][]byte{
	"file1.txt": data1,
	"file2.txt": data2,
}
err = backend.PutObjects(ctx, objects)

Performance Optimization

Multi-level performance optimizations:

Connection Pooling: - Configurable pool size (default: 8 connections) - Health monitoring and replacement - Load balancing across connections - Connection lifetime management

Tier-Aware Operations: - Automatic tier detection - Optimized operations based on storage class - Retrieval cost prediction - Access pattern learning

Enterprise Features

Advanced enterprise capabilities:

Cost Management: - Real-time cost tracking - Budget alerts and controls - Cost attribution by application/team - Reserved capacity optimization

Multi-Region Support: - Cross-region replication - Regional failover - Latency-based routing - Cost-optimized regional storage

Security Integration: - IAM role integration - KMS encryption support - VPC endpoint compatibility - Access logging and monitoring

Monitoring and Observability

Comprehensive monitoring integration:

Metrics Collection: - Operation latency and throughput - Error rates and retry statistics - Cost tracking and attribution - Storage tier utilization

Health Monitoring: - Connection pool health - Service availability checks - Performance degradation detection - Automatic recovery triggers

Alerting: - Cost threshold violations - Performance anomaly detection - Tier optimization opportunities - Error rate escalations

Error Handling

Robust error handling and recovery:

Transient Error Recovery: - Exponential backoff retry logic - Circuit breaker patterns - Connection pool failover - Graceful degradation

Permanent Error Handling: - Clear error categorization - Detailed error context - Recovery recommendations - Operational guidance

Thread Safety

The backend is designed for concurrent access:

- All public methods are thread-safe - Internal state is protected with appropriate synchronization - Connection pool handles concurrent requests - Statistics collection is atomic

Storage Classes Summary

Quick reference for S3 storage classes:

| Tier | Access | Min Size | Min Duration | Use Case | |------|--------|----------|--------------|----------| | Standard | Instant | None | None | Frequent | | Standard-IA | Instant | 128KB | 30 days | Infrequent | | One Zone-IA | Instant | 128KB | 30 days | Non-critical | | Glacier IR | Instant | 128KB | 90 days | Archive + instant | | Glacier | Minutes-Hours | 40KB | 90 days | Long-term archive | | Deep Archive | Hours | 40KB | 180 days | Very long-term | | Intelligent | Variable | 128KB | None | Auto-optimize |

This package provides enterprise-grade S3 integration with advanced optimization, comprehensive cost management, and high-performance operation capabilities.

Index

Examples

Constants

View Source 
const (
	EncryptionModeOff = awsname.SSEModeOff
	EncryptionModeS3  = awsname.SSEModeS3
	EncryptionModeKMS = awsname.SSEModeKMS
)

Server-side encryption modes, as the values the `mode` config key accepts.

Aliases of the awsname constants, exactly as the Tier* constants in tiers.go are: the mode is read by internal/config and acted on here, and config cannot import this package. One authority for the set of modes that exist, in a package both sides can reach. See awsname.SSEModeOff and its siblings for what each mode does and costs.

View Source 
const (
	AccessNever = "never"
	AccessCold  = "cold"
)

Access Frequency Constants

View Source 
const (
	TierStandard          = awsname.StorageClassStandard
	TierStandardIA        = awsname.StorageClassStandardIA
	TierOneZoneIA         = awsname.StorageClassOneZoneIA
	TierReducedRedundancy = awsname.StorageClassReducedRedundancy
	TierGlacierIR         = awsname.StorageClassGlacierIR
	TierGlacier           = awsname.StorageClassGlacier
	TierDeepArchive       = awsname.StorageClassDeepArchive
	TierIntelligent       = awsname.StorageClassIntelligent
)

S3 Storage Tier Constants.

These are aliases of internal/awsname's storage classes rather than independent string literals. The tier is named in configuration, validated at load by internal/config, and acted on here — and internal/config cannot import this package (see the awsname package comment for the cycle). So the names have to live somewhere both sides reach, and if they were spelled twice the two spellings could disagree: a tier this table knows about but the loader rejects, or worse, one the loader accepts and this table has no entry for. TestStorageTiersCoversEveryStorageClass pins the other direction — that every class awsname admits has a billing entry here.

View Source 
const (
	AccessFrequent   = "frequent"
	AccessInfrequent = "infrequent"
	AccessArchive    = "archive"
)

Access Pattern Constants

View Source 
const (
	DefaultCurrency = "USD"
)

Currency Constants

Variables

View Source 
var StorageTiers = map[string]StorageTierInfo{
	TierStandard: {
		Name:               "Standard",
		MinObjectSize:      0,
		DeletionEmbargo:    0,
		RetrievalLatency:   "instant",
		RetrievalCost:      false,
		MinimumStorageDays: 0,
		RecommendedUseCase: "Frequently accessed data",
	},
	TierStandardIA: {
		Name:               "Standard-Infrequent Access",
		MinObjectSize:      minBillableSize128KB,
		DeletionEmbargo:    30 * 24 * time.Hour,
		RetrievalLatency:   "instant",
		RetrievalCost:      true,
		MinimumStorageDays: 30,
		RecommendedUseCase: "Infrequently accessed data that needs instant access",
	},
	TierOneZoneIA: {
		Name:               "One Zone-Infrequent Access",
		MinObjectSize:      minBillableSize128KB,
		DeletionEmbargo:    30 * 24 * time.Hour,
		RetrievalLatency:   "instant",
		RetrievalCost:      true,
		MinimumStorageDays: 30,
		RecommendedUseCase: "Infrequently accessed data in single AZ",
	},
	TierReducedRedundancy: {
		Name:               "Reduced Redundancy",
		MinObjectSize:      0,
		DeletionEmbargo:    0,
		RetrievalLatency:   "instant",
		RetrievalCost:      false,
		MinimumStorageDays: 0,
		RecommendedUseCase: "Non-critical, reproducible data (deprecated)",
	},
	TierGlacierIR: {
		Name:               "Glacier Instant Retrieval",
		MinObjectSize:      minBillableSize128KB,
		DeletionEmbargo:    90 * 24 * time.Hour,
		RetrievalLatency:   "instant",
		RetrievalCost:      true,
		MinimumStorageDays: 90,
		RecommendedUseCase: "Archive data needing instant access",
	},
	TierGlacier: {
		Name: "Glacier Flexible Retrieval",

		MinObjectSize:          0,
		PerObjectOverheadBytes: archiveOverheadGlacierKB + archiveOverheadStandardKB,
		DeletionEmbargo:        90 * 24 * time.Hour,
		RetrievalLatency:       "minutes-hours",
		RetrievalCost:          true,
		MinimumStorageDays:     90,
		RecommendedUseCase:     "Long-term archive with flexible retrieval",
	},
	TierDeepArchive: {
		Name: "Glacier Deep Archive",

		MinObjectSize:          0,
		PerObjectOverheadBytes: archiveOverheadGlacierKB + archiveOverheadStandardKB,
		DeletionEmbargo:        180 * 24 * time.Hour,
		RetrievalLatency:       "hours",
		RetrievalCost:          true,
		MinimumStorageDays:     180,
		RecommendedUseCase:     "Long-term archive rarely accessed",
	},
	TierIntelligent: {
		Name: "Intelligent Tiering",

		MinObjectSize:              0,
		MonitoringEligibilityBytes: monitoringEligibility128KB,
		DeletionEmbargo:            0,
		RetrievalLatency:           "variable",
		RetrievalCost:              false,
		MinimumStorageDays:         0,
		RecommendedUseCase:         "Automatic cost optimization for changing access patterns",
	},
}

StorageTiers holds the per-tier constraints for every S3 storage class.

Constraints only — minimum size, embargo, retrieval latency — which are S3 behavior and are the same everywhere AWS runs. It carries no rate, for two reasons that arrived a release apart. Rates were the one thing this table used to state for itself, and stating them made it the third of five copies of the S3 rate card in this repo; two of the five disagreed by a factor of ten, so what a write cost depended on which package the caller reached for. Then, once the rates were read from awsrates by a withRates helper, the remaining problem was that a package-level map is built before any configuration exists — so the rate in it was always us-east-1's. Both are the same mistake at different scopes: a price stored where nothing can say which region or which discount schedule produced it. Ask PricingManager for money; ask this for behavior.

Functions

func ArchiveOverhead added in v0.10.3 

func ArchiveOverhead(tier string) (archiveBytes, standardBytes int64)

ArchiveOverhead returns the per-object overhead for a tier, split by the rate each part is billed at: archiveBytes at the tier's own rate, standardBytes at the S3 Standard rate.

Both are zero for every class but GLACIER and DEEP_ARCHIVE. The split is exposed because pricing the 40 KB at one rate is wrong by about a factor of six on the 8 KB portion — Standard is $0.023 per GB-month against Deep Archive's $0.00099 — so a caller that sums first and prices second gets the cheaper answer for the more expensive part.

func CalculateOptimalChunkSize 

func CalculateOptimalChunkSize(fileSize int64, multipartThreshold int64, baseChunkSize int64) int64

CalculateOptimalChunkSize calculates the optimal chunk size based on file size and network conditions

func CalculatePartCount 

func CalculatePartCount(fileSize int64, chunkSize int64) int

CalculatePartCount calculates the number of parts for a multipart upload

func ConvertTierToCargoShipStorageClass 

func ConvertTierToCargoShipStorageClass(tier string) config.StorageClass

ConvertTierToCargoShipStorageClass converts our tier constants to CargoShip storage class types

func ConvertTierToStorageClass 

func ConvertTierToStorageClass(tier string) types.StorageClass

ConvertTierToStorageClass converts our tier constants to AWS SDK storage class types

Types

type AccelerationStats added in v0.13.0 

type AccelerationStats struct {
	Configured  bool          `json:"configured"`
	Active      bool          `json:"active"`
	GateState   circuit.State `json:"gate_state"`
	Requests    int64         `json:"requests"`
	Bytes       int64         `json:"bytes"`
	Fallbacks   int64         `json:"fallbacks"`
	AvgLatency  time.Duration `json:"avg_latency"`
	RetryPeriod time.Duration `json:"retry_period"`
}

AccelerationStats is the Transfer Acceleration state as an exporter sees it.

Configured and Active are both here and they are different questions — which is the distinction #204 turned on. Configured is what the operator asked for and never changes; Active is whether requests are going to the accelerate endpoint right now. Configured true with Active false is the fallback in effect, and it is the state that used to be unreportable.

type AccessPattern 

type AccessPattern struct {
	ObjectKey       string        `json:"object_key"`
	AccessCount     int64         `json:"access_count"`
	LastAccessTime  time.Time     `json:"last_access_time"`
	FirstAccessTime time.Time     `json:"first_access_time"`
	AvgAccessGap    time.Duration `json:"avg_access_gap"`
	ObjectSize      int64         `json:"object_size"`
	CurrentTier     string        `json:"current_tier"`
	EstimatedCost   float64       `json:"estimated_cost"`
}

AccessPattern tracks object access patterns for cost optimization

type AdditionalCosts 

type AdditionalCosts struct {
	DataTransferOut   DataTransferPricing `yaml:"data_transfer_out"`  // Data transfer out costs
	ReplicationCosts  ReplicationPricing  `yaml:"replication_costs"`  // Cross-region replication
	CloudWatchMetrics float64             `yaml:"cloudwatch_metrics"` // CloudWatch metrics cost per metric
	InventoryReports  float64             `yaml:"inventory_reports"`  // S3 Inventory cost per object
	AccessLogging     float64             `yaml:"access_logging"`     // Access logging cost per request
}

AdditionalCosts defines additional cost factors

type Backend 

type Backend struct {
	// contains filtered or unexported fields
}

Backend implements the S3 storage backend with CargoShip optimization

Example (ExecuteWithAccelerationFallback)

Example usage demonstrating the fallback pattern 

// This is how the fallback would be used in GetObject:
//
// err := b.executeWithAccelerationFallback(ctx, "GetObject", func(client *s3.Client) error {
// 	input := &s3.GetObjectInput{
// 		Bucket: aws.String(b.bucket),
// 		Key:    aws.String(key),
// 	}
// 	result, err := client.GetObject(ctx, input)
// 	if err != nil {
// 		return err
// 	}
// 	// Process result...
// 	return nil
// })

// Placeholder to make this a valid example
_ = (*s3.Client)(nil)

func NewBackend 

func NewBackend(ctx context.Context, bucket string, cfg *Config) (*Backend, error)

NewBackend creates a new S3 backend instance

func (*Backend) AccelerationStats added in v0.13.0 

func (b *Backend) AccelerationStats() AccelerationStats

AccelerationStats reports the Transfer Acceleration state and its counters, for whoever is exporting them.

This exists because GetMetrics had no caller outside this package, which was the other half of #204: the fallback state was tracked accurately and reachable by nothing, so an operator whose mount had silently dropped to the standard endpoint had no scrape, log line, or health check that said so. The throughput loss was invisible.

A purpose-built struct rather than returning BackendMetrics, because the consumer is the metrics surface and the mapping it needs is a set of numbers with settled names. BackendMetrics carries twenty-odd fields on four unrelated subjects and a time.Duration that a gauge cannot take; handing that to internal/adapter would put the choice of what to export, and the unit conversions, in the caller.

func (*Backend) Bucket 

func (b *Backend) Bucket() string

Bucket returns the bucket this backend operates on.

The bucket is fixed at construction and already appears in every log line and error this backend produces, so exposing it reveals nothing new. It is here because a caller holding a backend otherwise has no way to name the bucket it is using — which a second backend over the same objects, configured differently, needs in order to be constructed at all.

func (*Backend) CalculateCostWithVolume 

func (b *Backend) CalculateCostWithVolume(tier string, sizeGB float64) (float64, error)

CalculateCostWithVolume calculates cost for a specific volume and tier

func (*Backend) Capabilities added in v0.12.0 

func (b *Backend) Capabilities() types.BackendCapabilities

Capabilities implements [types.CapabilityReporter], reporting what the endpoint in front of this process actually implements.

Probed once, lazily, and cached — not re-probed per call. A conditional write is on a coordination path where an extra round trip per attempt is a real cost, and the answer cannot change under a running process: the endpoint is fixed at construction.

Lazily rather than in NewBackend deliberately. Every backend pays construction, including the ones in tests and the ones that will never issue a conditional write, and a probe that ran there would put a request on the wire before the caller had asked for anything — turning a wrong endpoint into a startup failure for a feature nobody in that process is using.

func (*Backend) Close 

func (b *Backend) Close() error

Close closes the backend and releases resources

func (*Backend) CopyObject added in v0.11.0 

func (b *Backend) CopyObject(ctx context.Context, src, dst string) error

CopyObject copies src to dst server-side, preserving the properties that make the destination readable and correctly billed.

Why the source is headed first

Two reasons, and both are requirements rather than optimizations. S3's CopyObject fails outright above 5 GiB, so the size decides whether this is one request or a multipart copy — and discovering that from an InvalidRequest response would mean guessing, since InvalidRequest is also what several unrelated mistakes return. And the properties this must restate are only available from the source object: a copy does not inherit Content-Encoding, Content-Type, storage class, or metadata unless the request carries them.

What must survive the copy, and why each one

Content-Encoding, because the read path dispatches decoding on the stored encoding and fails closed on one it cannot handle. A rename that dropped it would leave a compressed object permanently unreadable — its bytes intact and no code able to interpret them.

Storage class, because the default is STANDARD. Omitting it silently promotes the object out of the tier the user is paying for, which is audit finding L26, observed where CopyObject was already used for tier transitions.

User metadata, because POSIX mode, ownership, and mtime live there and nowhere else. A rename that lost them would reset a file's permissions, which is not a thing rename does.

Encryption is applied from this backend's configuration rather than copied from the source, for the reason Backend.SetObjectMetadata states at length: a copy does not inherit the source's encryption, S3 encrypts the destination per the request, and a request that says nothing gets the bucket default. Using the configured value means a key rotation takes effect on renamed objects; preserving the source's key would mean it never did.

func (*Backend) CostStats added in v0.13.0 

func (b *Backend) CostStats() CostStats

CostStats returns what this mount has spent at AWS, priced at the current tier's rates.

What the figures are

Costs incurred by *this process since it started*, at list prices for the first volume band with any configured discounts applied. Not a bill, and not a reconciliation of one: nothing here knows about the bucket's existing contents, other mounts, cross-region transfer, or the free tier. What it is good for is the question an operator actually asks — is this mount's access pattern expensive, and which part of it — and for that a figure that moves with the workload matters more than one that ties out to the invoice.

Why requests are counted at the SDK layer

See [costTally]. Briefly: a 5 GB write is one PutObject to this package and 641 billable requests to AWS, so a count taken at the wrapper layer understates exactly the operations that cost the most.

Every dollar figure is monotonic, because the counts it derives from are. That is deliberate: a cost series that can decrease cannot have a rate-of-change query written against it, and rate-of-change is the form every useful alert on this takes.

func (*Backend) DeleteObject 

func (b *Backend) DeleteObject(ctx context.Context, key string) error

DeleteObject removes an object from S3

func (*Backend) EstimateStandardTierOverhead 

func (b *Backend) EstimateStandardTierOverhead(objectSize int64, targetTier string) float64

EstimateStandardTierOverhead calculates potential overhead from Standard tier usage

func (*Backend) GetAccessPatternCount 

func (b *Backend) GetAccessPatternCount() int

GetAccessPatternCount returns the number of objects with a tracked access pattern.

It delegates rather than taking len() of the map directly: the map is written from every reader goroutine when MonitorAccessPatterns is on, and a bare len() of a map being written concurrently is a race the runtime can abort the process for.

func (*Backend) GetAllComponentsHealth 

func (b *Backend) GetAllComponentsHealth() map[string]*health.ComponentHealth

GetAllComponentsHealth returns health status for all S3 operation components

func (*Backend) GetAllTiers 

func (b *Backend) GetAllTiers() map[string]StorageTierInfo

GetAllTiers returns information about all available storage tiers

func (*Backend) GetComponentHealth 

func (b *Backend) GetComponentHealth(component string) (*health.ComponentHealth, error)

GetComponentHealth returns health status for a specific S3 operation component

func (*Backend) GetCostOptimizationReport 

func (b *Backend) GetCostOptimizationReport() OptimizationReport

GetCostOptimizationReport generates a cost optimization analysis report

func (*Backend) GetCurrentTier 

func (b *Backend) GetCurrentTier() StorageTierInfo

GetCurrentTier returns the current storage tier information

func (*Backend) GetHealthStatus 

func (b *Backend) GetHealthStatus() health.HealthState

GetHealthStatus returns the overall health status of the S3 backend

func (*Backend) GetMetrics 

func (b *Backend) GetMetrics() BackendMetrics

GetMetrics returns current backend metrics

func (*Backend) GetObject 

func (b *Backend) GetObject(ctx context.Context, key string, offset, size int64) ([]byte, error)

GetObject retrieves an object or part of an object from S3 with CargoShip optimization

func (*Backend) GetObjects 

func (b *Backend) GetObjects(ctx context.Context, keys []string) (map[string][]byte, error)

GetObjects fetches the named objects concurrently, up to batchConcurrency at a time.

It returns every object it fetched together with an error naming every one it did not, so a non-nil error alongside a non-empty map is the normal way a partial batch is reported. See the contract on [types.Backend] for what a caller may assume.

func (*Backend) GetPricingSummary 

func (b *Backend) GetPricingSummary() PricingSummary

GetPricingSummary returns current pricing configuration and rates

func (*Backend) GetTierConstraints 

func (b *Backend) GetTierConstraints() TierConstraints

GetTierConstraints returns the current tier constraints

func (*Backend) GetTierCostEstimate 

func (b *Backend) GetTierCostEstimate(sizeGB float64) float64

GetTierCostEstimate estimates the monthly storage cost of sizeGB on the current tier, in the configured pricing region.

List price for the first volume band, with no discounts applied — Backend.CalculateCostWithVolume is the form that applies them. sizeGB is decimal GB, matching what AWS bills in; see awsrates.GBFromBytes for converting a byte count.

func (*Backend) GetTierPricingWithDiscounts 

func (b *Backend) GetTierPricingWithDiscounts(tier string) (TierPricing, error)

GetTierPricingWithDiscounts returns pricing for a tier with all discounts applied

func (*Backend) GetTierRecommendations 

func (b *Backend) GetTierRecommendations(objectSize int64, accessFrequency string) []string

GetTierRecommendations returns tier recommendations for an object

func (*Backend) HeadObject 

func (b *Backend) HeadObject(ctx context.Context, key string) (*types.ObjectInfo, error)

HeadObject retrieves metadata about an object

func (*Backend) HealthCheck 

func (b *Backend) HealthCheck(ctx context.Context) error

HealthCheck verifies the backend connection

func (*Backend) IsFullyHealthy 

func (b *Backend) IsFullyHealthy() bool

IsFullyHealthy checks if all components are in healthy state

func (*Backend) IsReadAvailable 

func (b *Backend) IsReadAvailable() bool

IsReadAvailable checks if read operations are currently available

func (*Backend) IsWriteAvailable 

func (b *Backend) IsWriteAvailable() bool

IsWriteAvailable checks if write operations are currently available

func (*Backend) ListObjects 

func (b *Backend) ListObjects(ctx context.Context, prefix string, limit int) ([]types.ObjectInfo, error)

ListObjects lists objects in the bucket with the given prefix, following continuation tokens until the limit is met or the prefix is exhausted. A limit of zero or less means every object.

Pagination is not an optimization. S3 caps a single ListObjectsV2 response at 1000 keys regardless of MaxKeys, and v0.10.0 issued exactly one request — so a directory with more than 1000 entries was silently truncated, and a truncated listing is not a cosmetic problem: the missing entries do not exist as far as readdir, and therefore as far as `cp -r` or `rm -r`, are concerned.

func (*Backend) OptimizeStorageCosts 

func (b *Backend) OptimizeStorageCosts(ctx context.Context) error

OptimizeStorageCosts analyzes and applies cost optimizations

func (*Backend) PutObject 

func (b *Backend) PutObject(ctx context.Context, key string, data []byte, meta map[string]string) error

PutObject stores an object in S3 with CargoShip optimization.

meta is merged into the object's user metadata. The integrity keys this method computes — objectfs-sha256 and objectfs-original-size — are written last and win over anything meta supplies: they describe the bytes being uploaded, which the caller has not seen after compression.

func (*Backend) PutObjectIf added in v0.12.0 

func (b *Backend) PutObjectIf(ctx context.Context, key string, data []byte, meta map[string]string,
	cond types.Precondition,
) (string, error)

PutObjectIf implements [types.Backend].

The contract is on the interface; what is worth recording here is the machinery this deliberately does *not* reuse from PutObject, and why.

It is not wrapped in b.retryer. The retryer decides what to retry from an error code, and the two outcomes a conditional write exists to produce sit on opposite sides of that decision: a precondition failure must never be retried, and a conflict may be. Leaving the retry to the caller is the honest arrangement, because a CAS caller cannot retry a *write* anyway — it has to re-read the state and recompute the bytes first, which is a loop only it can run. A retryer resending the same body against the same asserted ETag would be spending requests to be told the same thing.

It does run inside the circuit breaker and does feed the health tracker, on success and on genuine failures. Those two mechanisms are about whether S3 is reachable, and a conditional write is evidence about that like any other request. What they must not see is a lost race, and they do not: ErrCodePreconditionFailed is a non-failure per errors.IsServiceFailure, which both consult. Without that, N contenders for one lease would take writes offline for all of them — ErrorThreshold is 3 — under exactly the contention the precondition is there to arbitrate.

func (*Backend) PutObjects 

func (b *Backend) PutObjects(ctx context.Context, objects map[string][]byte) error

PutObjects stores multiple objects in batch with CargoShip optimization

func (*Backend) RefreshPricing 

func (b *Backend) RefreshPricing(ctx context.Context) error

RefreshPricing forces a refresh of pricing data from AWS API

func (*Backend) SetObjectMetadata 

func (b *Backend) SetObjectMetadata(ctx context.Context, key string, meta map[string]string) error

SetObjectMetadata replaces key's user metadata in place, without rewriting its contents.

S3 has no metadata-update operation, so this is a CopyObject onto the same key with MetadataDirective=REPLACE. The object's bytes are never transferred — the copy happens server-side — which is the whole reason a chmod does not read and rewrite a 10 GiB file.

Every other stored property has to be restated, because REPLACE discards all of them and not just the metadata map. Content-Encoding is the one that matters for integrity: the read path dispatches decoding on the stored encoding and fails closed on an encoding it cannot handle, so a chmod that dropped the header would leave a compressed object permanently unreadable. Storage class is restated because the default is STANDARD, so omitting it would silently promote an object out of the tier the user is paying for — the same defect shape as L26.

func (*Backend) SetStorageTier 

func (b *Backend) SetStorageTier(tier string, constraints TierConstraints) error

SetStorageTier changes the storage tier (requires restarting backend for full effect)

func (*Backend) ValidateObjectForTier 

func (b *Backend) ValidateObjectForTier(key string, size int64) error

ValidateObjectForTier validates if an object meets current tier requirements

type BackendMetrics 

type BackendMetrics struct {
	Requests        int64         `json:"requests"`
	Errors          int64         `json:"errors"`
	BytesUploaded   int64         `json:"bytes_uploaded"`
	BytesDownloaded int64         `json:"bytes_downloaded"`
	AverageLatency  time.Duration `json:"average_latency"`
	LastError       string        `json:"last_error"`
	LastErrorTime   time.Time     `json:"last_error_time"`

	// Transfer Acceleration metrics
	AcceleratedRequests int64         `json:"accelerated_requests"`
	AcceleratedBytes    int64         `json:"accelerated_bytes"`
	FallbackEvents      int64         `json:"fallback_events"`
	AccelerationEnabled bool          `json:"acceleration_enabled"`
	AccelerationLatency time.Duration `json:"acceleration_latency"`

	// Multipart upload metrics
	MultipartUploads          int64         `json:"multipart_uploads"`           // Total multipart uploads initiated
	MultipartUploadsParts     int64         `json:"multipart_uploads_parts"`     // Total parts uploaded
	MultipartUploadsCompleted int64         `json:"multipart_uploads_completed"` // Completed multipart uploads
	MultipartUploadsFailed    int64         `json:"multipart_uploads_failed"`    // Failed multipart uploads
	MultipartBytes            int64         `json:"multipart_bytes"`             // Total bytes uploaded via multipart
	AveragePartSize           int64         `json:"average_part_size"`           // Average part size in bytes
	MultipartLatency          time.Duration `json:"multipart_latency"`           // Average multipart upload latency
}

BackendMetrics tracks S3 backend performance metrics

type CircuitBreakerConfig 

type CircuitBreakerConfig struct {
	// Enabled false means the breaker never opens. It stays in the call path counting and reporting
	// state; it just never rejects. That is not the same as removing it, and removing it is not an
	// option this config offers — a bypass would be a second code path through every S3 operation
	// with no test coverage.
	Enabled bool `yaml:"enabled"`

	// FailureThreshold is the number of failures within one Interval that opens the breaker. Zero
	// means the package default, which is proportional rather than absolute: at least 20 requests in
	// the interval with half of them failing.
	//
	// A failure here is what circuit.defaultIsSuccessful calls one — a service failure, per
	// errors.IsServiceFailure. A missing object is an answer, not an outage, and does not count.
	//
	// Values above maxFailureThreshold are clamped rather than honored; see readyToTrip.
	FailureThreshold int `yaml:"failure_threshold"`

	// Timeout is how long the breaker stays open before admitting probe requests. Zero means 30s.
	Timeout time.Duration `yaml:"timeout"`
}

CircuitBreakerConfig defines the breaker that fronts S3 operations.

Plain data rather than a circuit.Config, deliberately. circuit.Config expresses the trip decision as a ReadyToTrip predicate — a func field, which is the right shape for the breaker and the wrong shape for configuration: it cannot be compared, printed usefully, round-tripped through YAML, or carried through the config fuzzer's %#v dedup key. NewBackend turns these three values into that predicate, so the translation lives in one place and the config stays a value.

type ClientManager 

type ClientManager struct {
	// contains filtered or unexported fields
}

ClientManager handles S3 client creation and management

func NewClientManager 

func NewClientManager(ctx context.Context, bucket string, cfg *Config, logger *slog.Logger) (*ClientManager, error)

NewClientManager creates a new S3 client manager

func (*ClientManager) Close 

func (cm *ClientManager) Close() error

Close closes all client resources Close releases the manager's pooled SDK clients and the TCP sockets its transport holds idle.

Both halves are needed, and only the first used to happen. The ConnectionPool holds *s3.Client values, which are cheap structs sharing one transport; draining it frees no sockets at all. The sockets are the transport's idle connections — up to MaxIdleConns of them, kept for IdleConnTimeout, which is 90 seconds. A process that builds and closes backends in a loop therefore accumulated file descriptors until it ran out: measured at 2 leaked sockets per cycle against a local endpoint, and reported by a fuzz run as "can't assign requested address" after the ephemeral port range filled.

CloseIdleConnections rather than anything more forceful: it closes connections not currently in use and leaves an in-flight request alone to finish. A caller closing a backend while still using it has a bug this cannot fix, and cutting the request short would turn it into a confusing I/O error instead.

func (*ClientManager) DisableAcceleration 

func (cm *ClientManager) DisableAcceleration(reason string)

DisableAcceleration temporarily disables Transfer Acceleration and falls back to standard client.

Idempotent under concurrency: the flag is re-checked under the write lock, so a burst of acceleration errors across many goroutines logs once rather than once per request. This is the common case — a bucket without the acceleration configuration fails every request — and the unsynchronized version could log a hundred identical warnings for one condition.

func (*ClientManager) EnableAcceleration 

func (cm *ClientManager) EnableAcceleration()

EnableAcceleration re-enables Transfer Acceleration if configured.

Called by accelerationGate when its breaker goes half-open, which is the one path back from a fallback (#204). Until then nothing called it and the fallback was one-way for the life of the mount: one acceleration error, whatever its cause, sent every subsequent request to the standard endpoint until restart. A thirty-second DNS failure reaching the accelerate endpoint therefore cost a long-lived mount its acceleration permanently, and nothing reported it had happened.

The gate, not this method, decides *when*. Re-enabling on any schedule of its own would put this method in the business of retry policy, and the retry has to be capped to one in-flight probe — which is the breaker's MaxRequests, not something a mutex around two fields can express.

func (*ClientManager) GetAcceleratedClient 

func (cm *ClientManager) GetAcceleratedClient() *s3.Client

GetAcceleratedClient returns the accelerated client if acceleration is active, and nil otherwise.

Prefer this over checking IsAccelerationActive and then reading the client: between those two calls the fallback path may have run, and only the combined check under one lock can report "active, and here is the client" as a single fact. Callers must still handle nil.

func (*ClientManager) GetClient 

func (cm *ClientManager) GetClient() *s3.Client

GetClient returns the main S3 client — whichever of the accelerated and standard clients is currently in use, which the fallback path can change at any time.

func (*ClientManager) GetNetworkMonitor 

func (cm *ClientManager) GetNetworkMonitor() *network.Monitor

GetNetworkMonitor returns the network monitor that tracks bytes and connection counts for all S3 connections made through this client.

func (*ClientManager) GetPool 

func (cm *ClientManager) GetPool() *ConnectionPool

GetPool returns the connection pool for statistics

func (*ClientManager) GetPooledClient 

func (cm *ClientManager) GetPooledClient() (*s3.Client, error)

GetPooledClient gets a client from the connection pool.

It returns an error rather than a nil client: callers dereference the result immediately, and a nil here previously panicked and unmounted the filesystem once the pool was saturated.

func (*ClientManager) GetStandardClient 

func (cm *ClientManager) GetStandardClient() *s3.Client

GetStandardClient returns the standard (non-accelerated) client.

Immutable after construction, so it needs no lock — unlike GetClient, which returns whichever client is currently selected.

func (*ClientManager) GetStats 

func (cm *ClientManager) GetStats() PoolStats

GetStats returns connection pool statistics

func (*ClientManager) HealthCheck 

func (cm *ClientManager) HealthCheck(ctx context.Context, bucket string) error

HealthCheck verifies the client connection

func (*ClientManager) IsAccelerationActive 

func (cm *ClientManager) IsAccelerationActive() bool

IsAccelerationActive returns whether Transfer Acceleration is currently active.

A point-in-time answer: the fallback path can flip it on any request. Treat it as a metric, not as a precondition for a later read of the client.

func (*ClientManager) RequestCounts added in v0.13.0 

func (cm *ClientManager) RequestCounts() costCounts

RequestCounts returns the billable AWS requests this manager's clients have made, by pricing group, and the bytes they have retrieved.

Monotonic for the life of the manager and never reset — a cost figure that can go down is one no rate-of-change query can be written against. Counting starts at construction, so a mount restarted mid-month reports its own requests and not the month's; pricing that as a monthly projection is the caller's decision, not this method's.

func (*ClientManager) ReturnPooledClient 

func (cm *ClientManager) ReturnPooledClient(client *s3.Client)

ReturnPooledClient returns a client to the connection pool

type CompressionConfig 

type CompressionConfig struct {
	// Enabled turns transparent S3 compression on or off.
	Enabled bool `yaml:"enabled"`
	// Algorithm selects the codec: "none", "zstd" (recommended), "lz4", or "gzip".
	// The authoritative list is pkg/compression.SupportedAlgorithms; this comment
	// is a convenience, and a value it disagrees with is a bug in the comment.
	Algorithm string `yaml:"algorithm"`
	// Level is the codec-specific compression level (0 = the codec's default).
	// The valid range differs per algorithm: zstd accepts 0-22 (3 is a good
	// default), gzip only 0-9. A level valid for one is often invalid for the
	// other, so changing Algorithm may require changing Level.
	Level int `yaml:"level"`
	// MinSize is the minimum object size to compress (e.g. "4KB").
	// Objects smaller than MinSize are stored uncompressed.
	MinSize string `yaml:"min_size"`
}

CompressionConfig defines transparent compression settings for S3 objects. When Enabled is true, objects are compressed on upload (PutObject) and decompressed on download (GetObject) using the configured algorithm.

type Config 

type Config struct {
	Region          string `yaml:"region"`
	Endpoint        string `yaml:"endpoint"`
	AccessKeyID     string `yaml:"access_key_id"`
	SecretAccessKey string `yaml:"secret_access_key"`
	SessionToken    string `yaml:"session_token"`
	ForcePathStyle  bool   `yaml:"force_path_style"`

	// Performance settings
	MaxRetries     int           `yaml:"max_retries"`
	ConnectTimeout time.Duration `yaml:"connect_timeout"`
	RequestTimeout time.Duration `yaml:"request_timeout"`
	PoolSize       int           `yaml:"pool_size"`

	// Retry configuration
	RetryConfig retry.Config `yaml:"retry_config"`

	// CircuitBreaker controls the breaker that fronts S3 operations.
	CircuitBreaker CircuitBreakerConfig `yaml:"circuit_breaker"`

	// Advanced settings
	UseAccelerate bool `yaml:"use_accelerate"`
	UseDualStack  bool `yaml:"use_dual_stack"`

	// AccelerationRetry is how long the fallback stays in effect before one request is allowed to
	// try the accelerate endpoint again. Zero means the default below.
	//
	// The fallback used to be permanent — one acceleration error and every later request in the
	// mount's life used the standard endpoint (#204). The argument for that was sound as far as it
	// went: the usual trigger is a bucket without the Transfer Acceleration configuration, which
	// does not resolve on its own, so retrying it per request would pay a failed round trip forever.
	// What it missed is that the trigger set is not only that one. A DNS or TLS failure reaching
	// <bucket>.s3-accelerate.amazonaws.com matches too, and so does a transient InvalidRequest; those
	// do resolve, and a mount that lost acceleration to a thirty-second network fault kept paying for
	// it for weeks.
	//
	// A period rather than a per-request retry is what makes both cases affordable. At the default,
	// the unrecoverable case costs one failed request every five minutes — a rounding error against
	// the request rate of any mount doing enough traffic to want acceleration — and the recoverable
	// case comes back on its own within one period. See newAccelerationGate for the state machine;
	// it is [circuit.CircuitBreaker], not a second bespoke one.
	AccelerationRetry time.Duration `yaml:"acceleration_retry"`

	// Multipart upload configuration
	MultipartThreshold   int64 `yaml:"multipart_threshold"`   // Size threshold for multipart uploads (bytes)
	MultipartChunkSize   int64 `yaml:"multipart_chunk_size"`  // Chunk size for multipart uploads (bytes)
	MultipartConcurrency int   `yaml:"multipart_concurrency"` // Number of concurrent part uploads

	// Parallel read configuration — fan out large object reads into concurrent range GETs.
	ParallelReadThreshold   int64 `yaml:"parallel_read_threshold"`   // bytes; 0 = disabled
	ReadChunkSize           int64 `yaml:"read_chunk_size"`           // bytes per range GET
	ParallelReadConcurrency int   `yaml:"parallel_read_concurrency"` // 0 = MultipartConcurrency

	// S3 Storage Tier Configuration
	StorageTier      string           `yaml:"storage_tier"`      // "STANDARD", "STANDARD_IA", "ONEZONE_IA", etc.
	TierConstraints  TierConstraints  `yaml:"tier_constraints"`  // Tier-specific constraints
	CostOptimization CostOptimization `yaml:"cost_optimization"` // Cost optimization settings
	PricingConfig    PricingConfig    `yaml:"pricing_config"`    // Custom pricing configuration

	// Transparent object compression configuration
	Compression CompressionConfig `yaml:"compression"`

	// Server-side encryption applied to every object this backend writes.
	Encryption EncryptionConfig `yaml:"encryption"`

	// CongestionAlgorithm is the TCP congestion control algorithm to request
	// for each S3 connection: "auto" (detect best), "bbr", "cubic", "reno".
	// On non-Linux platforms the value is silently ignored.
	// Default: "auto".
	CongestionAlgorithm string `yaml:"congestion_algorithm"`
}

Config represents S3 backend configuration

func NewDefaultConfig 

func NewDefaultConfig() *Config

NewDefaultConfig returns a configuration with sensible defaults

func (*Config) GetOptimalChunkSize 

func (c *Config) GetOptimalChunkSize(fileSize int64) int64

GetOptimalChunkSize returns the optimal chunk size for a given file size

func (*Config) ShouldUseMultipart 

func (c *Config) ShouldUseMultipart(fileSize int64) bool

ShouldUseMultipart determines if a file should use multipart upload

type ConnectionPool 

type ConnectionPool struct {
	// contains filtered or unexported fields
}

ConnectionPool manages a pool of S3 client connections

func NewConnectionPool 

func NewConnectionPool(ctx context.Context, maxSize int, bucket string, factory func() (*s3.Client, error)) (*ConnectionPool, error)

NewConnectionPool creates a pool of at most maxSize clients from factory.

ctx is the parent of every health probe the pool's background checker makes, and bucket is what it probes. Both were absent: the checker built each probe context from context.Background(), so nothing a caller attached reached it and shutdown relied entirely on stopCh — and it probed with ListBuckets, which is the wrong call. See testConnection.

func (*ConnectionPool) Close 

func (p *ConnectionPool) Close() error

Close shuts the pool down. It is idempotent.

The channel is drained rather than closed. Closing it would be a data race against any Put still in flight — and since Put performs its closed check and its send under the same write lock this takes, draining is sufficient: after Close returns, no Put can succeed. An S3 client needs no explicit teardown, so the drained connections are simply discarded.

func (*ConnectionPool) Get 

func (p *ConnectionPool) Get() (*s3.Client, error)

Get retrieves a connection from the pool, waiting up to 30 seconds for one to become available.

It returns an error rather than a nil client. The previous signature returned *s3.Client alone and yielded nil once currentSize reached maxSize, which every one of its six call sites dereferenced unchecked — including the path taken by every GET and PUT. The ninth concurrent operation on a default 8-connection pool panicked and unmounted the filesystem under every open descriptor.

func (*ConnectionPool) GetWithTimeout 

func (p *ConnectionPool) GetWithTimeout(timeout time.Duration) (*s3.Client, error)

GetWithTimeout retrieves a connection, waiting up to timeout for one.

The order is: take an idle connection, else create one if the pool is below its limit, else wait for a connection to be returned. That last arm is the one that was missing — the old implementation had a `default` clause, so `select` never reached its `time.After` case and a full pool failed instantly instead of waiting for the connection that was about to come back.

func (*ConnectionPool) Put 

func (p *ConnectionPool) Put(conn *s3.Client)

Put returns a connection to the pool. A nil connection is ignored, so a caller may defer Put unconditionally alongside a Get that failed.

The check of closed and the send are one critical section under the write lock. Close takes the same lock, so a Put either observes closed and drops the connection or completes its send while Close waits — it can never send to a pool that has finished draining. Splitting the two, as this once did, is a check-then-act race that panicked on shutdown.

func (*ConnectionPool) Resize 

func (p *ConnectionPool) Resize(newSize int) error

Resize changes the maximum number of connections the pool will hold.

It cannot grow past the size the pool was built with: the buffer of the idle channel is fixed at construction, and every reservation made by GetWithTimeout and Warmup relies on that buffer having room for the connection it is about to produce. Raising maxSize above the buffer would make a return block while holding the write lock — a deadlock, not a bigger pool. Growing for real means a new channel, so this refuses instead of pretending, and the caller is told to raise performance.connection_pool_size and restart.

Shrinking discards idle connections until the total is within the new limit. Connections that are currently checked out are left alone; they are dropped by Put once they come back.

func (*ConnectionPool) Stats 

func (p *ConnectionPool) Stats() PoolStats

Stats returns current pool statistics

func (*ConnectionPool) Warmup 

func (p *ConnectionPool) Warmup(ctx context.Context, count int) error

Warmup pre-fills the pool so the first requests find idle connections instead of constructing them. Connections it adds are idle, not active: a caller accounts for one only when it draws it.

A count above the pool's size warms the whole pool; a count of zero or less means the whole pool.

type CostOptimization 

type CostOptimization struct {
	// SmallObjectsOnStandard stores an object on STANDARD when the configured tier would bill it as
	// larger than it is. Read on the PutObject path, and the one field in this struct a mount
	// configuration maps (`storage.s3.cost_optimization.small_objects_on_standard`).
	//
	// It changes the storage class of stored objects, which is why it is its own field and its own
	// key. This behavior used to be gated by MonitorAccessPatterns — a name that says the process
	// observes something, attached to a switch that silently rewrote the storage class of every
	// object under 128 KiB. See [CostOptimizer.HandleStandardTierOverhead] for which tiers it applies
	// to and why "under 128 KiB" was the wrong test.
	SmallObjectsOnStandard bool `yaml:"small_objects_on_standard"`

	// EnableAutoTiering lets [CostOptimizer.AnalyzeAndOptimize] issue the CopyObject calls that move
	// objects between tiers. Off by default and deliberately not mapped from a mount configuration:
	// nothing on the mount path calls AnalyzeAndOptimize, so a YAML key for it would configure a
	// feature no mount can invoke. It is reachable through [Backend.OptimizeStorageCosts] for callers
	// using this package as a library.
	EnableAutoTiering bool `yaml:"enable_auto_tiering"`

	// CostThreshold is the monthly saving in dollars below which a suggested transition is not worth
	// making. Read only by the analyzer above, so it is unmapped for the same reason.
	CostThreshold float64 `yaml:"cost_threshold"`

	// MonitorAccessPatterns records an access pattern per object key on every read, which is what
	// gives [Backend.GetCostOptimizationReport] something to report.
	//
	// Also unmapped, and this one for a reason that is not just reachability: the map it fills holds
	// one entry per distinct key read and nothing ever evicts from it, so on a bucket with many
	// objects it grows for the life of the mount. A key that trades unbounded memory for a report no
	// mount can display is not a key worth offering. Library callers that want the report accept that
	// trade knowingly.
	MonitorAccessPatterns bool `yaml:"monitor_access_patterns"`
}

CostOptimization defines cost optimization settings.

Three fields, where there were six. `TransitionRules []TransitionRule`, `LifecycleManagement` and `IntelligentTiering` were removed rather than plumbed: no code read any of the three, and the two bools were additionally misleading — S3 lifecycle rules and Intelligent-Tiering configuration are bucket-level settings applied with PutBucketLifecycleConfiguration and PutBucketIntelligentTieringConfiguration, neither of which this backend calls, so a `true` there described something ObjectFS has no code to do. `TransitionRules` was the reason internal/config.S3CostOptimization could not be mapped onto this type at all: there is no reading of `transition_to_ia: 30` that produces a []TransitionRule without inventing the rest of a rule (#203).

What survives is separated by who can reach it, because that turned out to be the whole question: one field is read on the write path of every mount, and two gate an analyzer that only a caller holding a *Backend can invoke.

type CostOptimizer 

type CostOptimizer struct {
	// contains filtered or unexported fields
}

CostOptimizer handles cost optimization decisions and Standard tier overhead management

func NewCostOptimizer 

func NewCostOptimizer(backend *Backend, config CostOptimization, logger *slog.Logger) *CostOptimizer

NewCostOptimizer creates a new cost optimizer

func (*CostOptimizer) AnalyzeAndOptimize 

func (co *CostOptimizer) AnalyzeAndOptimize(ctx context.Context) error

AnalyzeAndOptimize analyzes access patterns and suggests/applies optimizations

func (*CostOptimizer) EstimateStandardTierOverhead 

func (co *CostOptimizer) EstimateStandardTierOverhead(objectSize int64, targetTier string) float64

EstimateStandardTierOverhead calculates potential overhead from using Standard tier

func (*CostOptimizer) GetOptimizationReport 

func (co *CostOptimizer) GetOptimizationReport() OptimizationReport

GetOptimizationReport generates a cost optimization report

func (*CostOptimizer) HandleStandardTierOverhead 

func (co *CostOptimizer) HandleStandardTierOverhead(objectKey string, objectSize int64) string

HandleStandardTierOverhead returns the storage class an object of this size should be written with: the configured tier, or STANDARD where the configured tier would bill the object as larger than it is and STANDARD is therefore cheaper.

Three conditions, all required, where there used to be one:

  • The configured tier publishes a minimum billable size. Only STANDARD_IA, ONEZONE_IA and GLACIER_IR do. The old test was `objectSize < minBillableSize128KB` with no reference to the configured tier at all, so with `storage_tier: STANDARD` it returned STANDARD — harmless — and with `storage_tier: DEEP_ARCHIVE` it moved every object under 128 KiB onto STANDARD, which is 23× the storage rate. The tier that has no floor cannot be rounding anything up.
  • The object is below that minimum. Above it there is nothing to avoid.
  • STANDARD actually costs less for this object. Computed, not assumed, and this is the condition that does the most work: being below the floor does not make STANDARD cheaper. The crossover is at minBillable × rateTier / rateStandard, which at list prices is 69.6 KiB for STANDARD_IA, 55.7 KiB for ONEZONE_IA and 22.3 KiB for GLACIER_IR — so a 32 KiB object is cheaper on GLACIER_IR *billed as 128 KiB* than on STANDARD at its own size, by nearly 3×. The old size-only test moved all three to STANDARD anywhere under 128 KiB and raised the bill across most of that range. [CostOptimizer.calculateObjectCost] also applies any volume or enterprise discount and any CustomPricing override, so the comparison is against the prices this deployment pays rather than list.

The archive classes are outside this entirely, by the first condition, and that is deliberate beyond the arithmetic: GLACIER and DEEP_ARCHIVE objects cannot be read without a restore, so silently writing one to STANDARD instead would change what a read of that object does, not only what it costs. A cost heuristic must not decide retrieval semantics.

func (*CostOptimizer) PatternCount 

func (co *CostOptimizer) PatternCount() int

PatternCount reports how many objects currently have a tracked access pattern.

func (*CostOptimizer) RecordAccess 

func (co *CostOptimizer) RecordAccess(objectKey string, objectSize int64)

RecordAccess records an access pattern for cost optimization analysis.

This is called from GetObject, on both the serial and the parallel read paths, so it runs on every reader goroutine — see the mu field for why that requires a lock rather than merely benefiting from one.

type CostStats added in v0.13.0 

type CostStats struct {
	// Region is the region the rates were read for, which is not always the configured region. Publish
	// this rather than PricingConfig.Region: when the configured region has no published rates the
	// manager falls back to [awsrates.DefaultRegion], and a dollar figure labeled with the region that
	// was asked for instead of the one it was priced in is a figure that cannot be checked.
	Region string

	// Tier is the storage class this mount writes to, which decides every rate below. Two mounts of the
	// same bucket at different tiers have different per-request costs, so a cost series without this is
	// not interpretable across a fleet.
	Tier string

	// Request counts by pricing group, since construction. Free is the requests AWS bills nothing for.
	WriteRequests int64
	ListRequests  int64
	ReadRequests  int64
	FreeRequests  int64

	// BytesRetrieved is bytes off the wire, the quantity a retrieval fee is charged on.
	BytesRetrieved int64

	// RequestCost is the dollars the counted requests have cost, at this tier's per-request rates.
	RequestCost float64

	// RetrievalCost is the dollars BytesRetrieved has cost. Zero on STANDARD and INTELLIGENT_TIERING,
	// which have no retrieval fee — a zero here is a real answer and not a missing one.
	RetrievalCost float64

	// StoredBytes is the object bytes this mount has written, and StorageCostPerMonth what holding them
	// for a month costs at this tier's rate.
	//
	// Not the bucket's size. Nothing here lists the bucket — that would be a request per tick, billed,
	// to publish a metric — so this is what this process has uploaded since it started, which on a
	// long-lived mount of a large bucket is a small fraction of what is stored. It answers "what is this
	// mount adding to the bill", not "what does the bucket cost".
	StoredBytes         int64
	StorageCostPerMonth float64

	// RatePerWriteRequest and the two below it are the rates the costs above were computed at, published
	// so a dashboard can show the arithmetic rather than only its result. They are dollars per single
	// request, not per thousand: #209 was a per-1,000 figure stored as if it were per-request, and a
	// mount that publishes the rate it used makes that class of error visible in a scrape instead of
	// only in a bill.
	RatePerWriteRequest float64
	RatePerListRequest  float64
	RatePerReadRequest  float64
	RatePerGBRetrieved  float64
	RatePerGBMonth      float64
}

CostStats is what this mount has spent at AWS, as an exporter sees it.

A purpose-built struct for the same reason AccelerationStats is one: the consumer is the metrics surface, and the choice of what to publish — along with every unit conversion — belongs here beside the rates rather than in internal/adapter. See Backend.CostStats.

type DataTransferPricing 

type DataTransferPricing struct {
	FirstTBPerGB  float64 `yaml:"first_tb_per_gb"`  // First TB pricing
	Next9TBPerGB  float64 `yaml:"next_9tb_per_gb"`  // 1-10 TB pricing
	Next40TBPerGB float64 `yaml:"next_40tb_per_gb"` // 10-50 TB pricing
	Over50TBPerGB float64 `yaml:"over_50tb_per_gb"` // >50 TB pricing
}

DataTransferPricing defines data transfer cost structure

type DiscountConfig 

type DiscountConfig struct {
	EnableVolumeDiscounts    bool               `yaml:"enable_volume_discounts"`    // Enable volume-based discounts
	VolumeTiers              []VolumeTier       `yaml:"volume_tiers"`               // Volume discount tiers
	EnterpriseDiscount       float64            `yaml:"enterprise_discount"`        // Overall enterprise discount (%)
	ReservedCapacityDiscount float64            `yaml:"reserved_capacity_discount"` // Reserved capacity discount (%)
	SpotDiscount             float64            `yaml:"spot_discount"`              // Spot pricing discount (%)
	CustomDiscounts          map[string]float64 `yaml:"custom_discounts"`           // Custom discounts per tier
}

DiscountConfig defines volume discounts and enterprise pricing

type EncryptionConfig 

type EncryptionConfig struct {
	// Mode selects the encryption to request: "off", "sse-s3", or "sse-kms". Empty means "off".
	Mode string `yaml:"mode"`

	// KMSKeyID is the key SSE-KMS encrypts with — a key ID, an alias, or a full ARN. Required when
	// Mode is "sse-kms" and rejected otherwise, rather than ignored: a key set beside a mode that
	// does not use it means the two disagree about what is being asked for, and silently honoring
	// the mode is how a configuration comes to name a KMS key and encrypt with something else.
	KMSKeyID string `yaml:"kms_key_id"`

	// BucketKeys requests S3 Bucket Keys, which reduce SSE-KMS's per-object KMS calls by up to 99%
	// by deriving a bucket-level key. Recommended with "sse-kms" and meaningless without it.
	//
	// This is a cost and throughput control rather than a security one, and it is the difference
	// between SSE-KMS being usable for a filesystem workload and not: without it, every object read
	// is a billed KMS Decrypt against a per-region rate limit, so a directory traversal can be
	// throttled by KMS while S3 is entirely idle.
	BucketKeys bool `yaml:"bucket_keys"`
}

EncryptionConfig defines the server-side encryption ObjectFS requests on every object it writes.

It exists because v0.10.0 shipped a `security.encryption.at_rest` key that defaulted to **true** and was read by nothing: a grep for ServerSideEncryption, SSEKMS, or aws:kms across the tree returned zero non-test hits, while OBJECTFS.md documented a `kms_key:` ARN (audit finding P-7). A configuration key that claims a security property and sets no header is worse than an absent feature, because an operator who reads it stops looking — and the thing they stopped looking for is the one an auditor will ask about.

The mode is the whole of the decision and there is no separate boolean, deliberately. Two switches where one will do is how `at_rest: true` came to coexist with no header: a bool cannot say which of the three things a reader might mean, so it says the one that sounds safest.

func (EncryptionConfig) Enabled 

func (e EncryptionConfig) Enabled() bool

Enabled reports whether any encryption header should be sent.

type HealthChecker 

type HealthChecker struct {
	// contains filtered or unexported fields
}

HealthChecker monitors connection health

type MetricsCollector 

type MetricsCollector struct {
	// contains filtered or unexported fields
}

MetricsCollector handles metrics collection and aggregation for S3 backend

func NewMetricsCollector 

func NewMetricsCollector() *MetricsCollector

NewMetricsCollector creates a new metrics collector

func (*MetricsCollector) GetAccelerationRate 

func (mc *MetricsCollector) GetAccelerationRate() float64

GetAccelerationRate calculates the percentage of requests using acceleration

func (*MetricsCollector) GetAveragePartsPerUpload 

func (mc *MetricsCollector) GetAveragePartsPerUpload() float64

GetAveragePartsPerUpload calculates the average number of parts per multipart upload

func (*MetricsCollector) GetErrorRate 

func (mc *MetricsCollector) GetErrorRate() float64

GetErrorRate calculates the current error rate

func (*MetricsCollector) GetFallbackRate 

func (mc *MetricsCollector) GetFallbackRate() float64

GetFallbackRate calculates the fallback rate

func (*MetricsCollector) GetMetrics 

func (mc *MetricsCollector) GetMetrics() BackendMetrics

GetMetrics returns current backend metrics

func (*MetricsCollector) GetMultipartSuccessRate 

func (mc *MetricsCollector) GetMultipartSuccessRate() float64

GetMultipartSuccessRate calculates the success rate of multipart uploads

func (*MetricsCollector) GetMultipartUsageRate 

func (mc *MetricsCollector) GetMultipartUsageRate() float64

GetMultipartUsageRate calculates the percentage of uploads using multipart

func (*MetricsCollector) GetThroughput 

func (mc *MetricsCollector) GetThroughput() (uploadMBps, downloadMBps float64)

GetThroughput calculates upload and download throughput

func (*MetricsCollector) RecordAcceleratedRequest 

func (mc *MetricsCollector) RecordAcceleratedRequest(bytes int64, duration time.Duration)

RecordAcceleratedRequest records a request that used Transfer Acceleration

func (*MetricsCollector) RecordBytesDownloaded 

func (mc *MetricsCollector) RecordBytesDownloaded(bytes int64)

RecordBytesDownloaded records downloaded bytes

func (*MetricsCollector) RecordBytesUploaded 

func (mc *MetricsCollector) RecordBytesUploaded(bytes int64)

RecordBytesUploaded records uploaded bytes

func (*MetricsCollector) RecordError 

func (mc *MetricsCollector) RecordError(err error)

RecordError records an error occurrence

func (*MetricsCollector) RecordFallbackEvent 

func (mc *MetricsCollector) RecordFallbackEvent()

RecordFallbackEvent records when acceleration fallback occurs

func (*MetricsCollector) RecordMetrics 

func (mc *MetricsCollector) RecordMetrics(duration time.Duration, isError bool)

RecordMetrics records operation metrics with duration and error status

func (*MetricsCollector) RecordMultipartUploadComplete 

func (mc *MetricsCollector) RecordMultipartUploadComplete(totalBytes int64, duration time.Duration)

RecordMultipartUploadComplete records successful completion of a multipart upload

func (*MetricsCollector) RecordMultipartUploadFailed 

func (mc *MetricsCollector) RecordMultipartUploadFailed()

RecordMultipartUploadFailed records when a multipart upload fails

func (*MetricsCollector) RecordMultipartUploadPart 

func (mc *MetricsCollector) RecordMultipartUploadPart(partSize int64)

RecordMultipartUploadPart records when a part is uploaded

func (*MetricsCollector) RecordMultipartUploadStart 

func (mc *MetricsCollector) RecordMultipartUploadStart()

RecordMultipartUploadStart records when a multipart upload is initiated

func (*MetricsCollector) Reset 

func (mc *MetricsCollector) Reset()

Reset resets all metrics to zero

func (*MetricsCollector) SetAccelerationEnabled 

func (mc *MetricsCollector) SetAccelerationEnabled(enabled bool)

SetAccelerationEnabled records whether acceleration is currently in effect.

"Currently in effect", not "configured". Until #204 this was called once, from NewBackend, with cfg.UseAccelerate — so the field reported the operator's request and never the outcome: a mount could say acceleration was enabled, have fallen back on its first request, and have used the standard endpoint ever since. The gate's OnStateChange now calls this on every transition, which is what makes AccelerationEnabled a fact about the mount.

type MultipartStateManager 

type MultipartStateManager struct {
	// contains filtered or unexported fields
}

MultipartStateManager manages the state of multiple concurrent multipart uploads

func NewMultipartStateManager 

func NewMultipartStateManager() *MultipartStateManager

NewMultipartStateManager creates a new multipart state manager

func (*MultipartStateManager) CleanupOldUploads 

func (m *MultipartStateManager) CleanupOldUploads(maxAge time.Duration) int

CleanupOldUploads removes uploads that have been in a terminal state for longer than the specified duration

func (*MultipartStateManager) GetAllUploads 

func (m *MultipartStateManager) GetAllUploads() []*MultipartUploadState

GetAllUploads returns all tracked uploads

func (*MultipartStateManager) GetInProgressUploads 

func (m *MultipartStateManager) GetInProgressUploads() []*MultipartUploadState

GetInProgressUploads returns all uploads that are currently in progress

func (*MultipartStateManager) GetUploadCount 

func (m *MultipartStateManager) GetUploadCount() int

GetUploadCount returns the total number of tracked uploads

func (*MultipartStateManager) GetUploadState 

func (m *MultipartStateManager) GetUploadState(uploadID string) (*MultipartUploadState, bool)

GetUploadState retrieves the state of a tracked upload

func (*MultipartStateManager) MarkUploadCompleted 

func (m *MultipartStateManager) MarkUploadCompleted(uploadID string)

MarkUploadCompleted marks an upload as completed

func (*MultipartStateManager) MarkUploadFailed 

func (m *MultipartStateManager) MarkUploadFailed(uploadID string)

MarkUploadFailed marks an upload as failed

func (*MultipartStateManager) RemoveUpload 

func (m *MultipartStateManager) RemoveUpload(uploadID string)

RemoveUpload removes a tracked upload from the manager

func (*MultipartStateManager) TrackUpload 

func (m *MultipartStateManager) TrackUpload(state *MultipartUploadState)

TrackUpload starts tracking a new multipart upload

func (*MultipartStateManager) UpdatePartStatus 

func (m *MultipartStateManager) UpdatePartStatus(uploadID string, partNumber int, size int64, etag string, err error)

UpdatePartStatus updates the status of a specific part. The manager lock is released before calling MarkPartCompleted / MarkPartFailed because those methods acquire the state's own mutex (state.mu). Holding the manager write lock while calling them creates a two-lock nesting chain that can deadlock if any concurrent goroutine holds state.mu and then tries to acquire m.mu (e.g. via GetInProgressUploads).

type MultipartUploadState 

type MultipartUploadState struct {
	UploadID       string                `json:"upload_id"`
	Bucket         string                `json:"bucket"`
	Key            string                `json:"key"`
	TotalSize      int64                 `json:"total_size"`
	ChunkSize      int64                 `json:"chunk_size"`
	Parts          map[int]*UploadPart   `json:"parts"` // Key is part number
	StartedAt      time.Time             `json:"started_at"`
	LastUpdatedAt  time.Time             `json:"last_updated_at"`
	CompletedParts int                   `json:"completed_parts"`
	TotalParts     int                   `json:"total_parts"`
	BytesUploaded  int64                 `json:"bytes_uploaded"`
	Status         MultipartUploadStatus `json:"status"`
	Metadata       map[string]string     `json:"metadata,omitempty"`
	// contains filtered or unexported fields
}

MultipartUploadState tracks the state of an in-progress multipart upload. All exported methods are safe for concurrent use.

func NewMultipartUploadState 

func NewMultipartUploadState(uploadID, bucket, key string, totalSize, chunkSize int64) *MultipartUploadState

NewMultipartUploadState creates a new multipart upload state tracker

func (*MultipartUploadState) GetCompletedParts 

func (s *MultipartUploadState) GetCompletedParts() []*UploadPart

GetCompletedParts returns a list of successfully uploaded parts.

func (*MultipartUploadState) GetProgress 

func (s *MultipartUploadState) GetProgress() float64

GetProgress returns the upload progress as a percentage (0-100).

func (*MultipartUploadState) GetRemainingParts 

func (s *MultipartUploadState) GetRemainingParts() []int

GetRemainingParts returns a list of part numbers that still need to be uploaded.

func (*MultipartUploadState) IsComplete 

func (s *MultipartUploadState) IsComplete() bool

IsComplete returns true if all parts have been uploaded.

func (*MultipartUploadState) MarkPartCompleted 

func (s *MultipartUploadState) MarkPartCompleted(partNumber int, size int64, etag string)

MarkPartCompleted marks a part as successfully uploaded.

func (*MultipartUploadState) MarkPartFailed 

func (s *MultipartUploadState) MarkPartFailed(partNumber int, err error)

MarkPartFailed marks a part as failed.

type MultipartUploadStatus 

type MultipartUploadStatus string

MultipartUploadStatus represents the status of a multipart upload 

const (
	UploadStatusInitiated  MultipartUploadStatus = "initiated"
	UploadStatusInProgress MultipartUploadStatus = "in_progress"
	UploadStatusCompleted  MultipartUploadStatus = "completed"
	UploadStatusFailed     MultipartUploadStatus = "failed"
	UploadStatusAborted    MultipartUploadStatus = "aborted"
)

func (MultipartUploadStatus) IsCompleted 

func (s MultipartUploadStatus) IsCompleted() bool

IsCompleted returns true if the upload is in a terminal state

type OptimizationReport 

type OptimizationReport struct {
	TotalObjects          int                `json:"total_objects"`
	OptimizationResults   []TierOptimization `json:"optimization_results"`
	TotalPotentialSavings float64            `json:"total_potential_savings"`
	GeneratedAt           time.Time          `json:"generated_at"`
}

OptimizationReport contains cost optimization analysis results

type PoolStats 

type PoolStats struct {
	Active      int       `json:"active"`
	Idle        int       `json:"idle"`
	Total       int       `json:"total"`
	MaxSize     int       `json:"max_size"`
	Hits        int64     `json:"hits"`
	Misses      int64     `json:"misses"`
	Timeouts    int64     `json:"timeouts"`
	Errors      int64     `json:"errors"`
	Created     int64     `json:"created"`
	Destroyed   int64     `json:"destroyed"`
	LastCreated time.Time `json:"last_created"`
	LastError   string    `json:"last_error"`
	LastErrorAt time.Time `json:"last_error_at"`
}

PoolStats tracks connection pool statistics

type PricingConfig 

type PricingConfig struct {
	// Deprecated: the AWS Pricing API integration was removed in v0.10.1 — it
	// downloaded the ~100 MB S3 offer index and then returned hardcoded
	// us-east-1 constants for every tier. Setting this now logs a warning and
	// has no other effect. Use CustomPricing for exact or negotiated rates.
	UsePricingAPI      bool                   `yaml:"use_pricing_api"`
	Region             string                 `yaml:"region"`               // Pricing region (may differ from bucket region)
	CustomPricing      map[string]TierPricing `yaml:"custom_pricing"`       // Override pricing per tier
	DiscountConfig     DiscountConfig         `yaml:"discount_config"`      // Volume discounts and enterprise rates
	DiscountConfigFile string                 `yaml:"discount_config_file"` // Path to external discount config file
	AdditionalCosts    AdditionalCosts        `yaml:"additional_costs"`     // Request costs, data transfer, etc.
	LastUpdated        string                 `yaml:"last_updated"`         // When pricing was last updated
	Currency           string                 `yaml:"currency"`             // USD, EUR, etc.
}

PricingConfig defines custom pricing configuration for S3 costs

type PricingManager 

type PricingManager struct {
	// contains filtered or unexported fields
}

PricingManager resolves S3 tier pricing from a built-in static rate table, applying operator-supplied overrides and discounts.

Rates are list prices read from awsrates, generated from AWS's published price list, for the region in PricingConfig.Region. They suit comparing tiers, not billing reconciliation: they are first volume band and they assume Standard retrieval speed on the Glacier classes. Set PricingConfig.CustomPricing for exact or negotiated rates.

region

Resolved once, in NewPricingManager, and stored separately from config.Region — which stays as the operator wrote it so PricingManager.GetPricingSummary can report both what was asked for and what was used. That distinction is the whole point of the field: a summary labeled with the operator's region while carrying another region's numbers is worse than one that names the region it used, because it looks correct.

Resolved once rather than per lookup because GetTierPricing is reachable from GetObject via RecordAccess, so it runs on the read path. A warning emitted there would be the loudest line in the log; a map lookup that misses on every call would be a silent cost.

func NewPricingManager 

func NewPricingManager(config PricingConfig, logger *slog.Logger) *PricingManager

NewPricingManager creates a new pricing manager.

An unset PricingConfig.Region means awsrates.DefaultRegion, quietly — that is the documented default rather than a mistake. A region that is set but has no published rates gets a warning naming both it and the region actually used, because that case is an operator asking for something specific and not getting it. #161's acceptance criterion is exactly this: "an unknown region falls back to us-east-1 with a warning rather than returning zero".

func (*PricingManager) CalculateVolumeDiscount 

func (pm *PricingManager) CalculateVolumeDiscount(tier string, sizeGB float64, baseCost float64) float64

CalculateVolumeDiscount calculates volume-based discounts

func (*PricingManager) GetPricingSummary 

func (pm *PricingManager) GetPricingSummary() PricingSummary

GetPricingSummary returns a summary of current pricing configuration.

Region is the region the rates in it are from, which is what a reader needs in order to know what the numbers mean. ConfiguredRegion is what the operator asked for, and the two differ exactly when the configured region has no published rates — see NewPricingManager.

func (*PricingManager) GetTierPricing 

func (pm *PricingManager) GetTierPricing(tier string) (TierPricing, error)

GetTierPricing returns pricing for a specific tier with discounts applied

func (*PricingManager) RefreshPricing 

func (pm *PricingManager) RefreshPricing(_ context.Context) error

RefreshPricing is retained for API compatibility and is a no-op.

Pricing is served from a built-in static rate table, so there is nothing to refresh. The AWS Pricing API integration this method used to drive was removed: it downloaded the ~100 MB S3 offer index and then discarded the parse, returning two hardcoded us-east-1 constants for every tier — strictly worse than reading the static table directly.

func (*PricingManager) Region added in v0.11.0 

func (pm *PricingManager) Region() string

Region returns the region rates are read for, which is not always PricingConfig.Region.

It is awsrates.DefaultRegion when the configured region is unset or has no published rates. A caller rendering a cost figure should label it with this rather than with the configured value.

func (*PricingManager) StorageRate added in v0.11.0 

func (pm *PricingManager) StorageRate(tier string) float64

StorageRate returns the list storage rate for a tier in this manager's region, before discounts.

For callers that need the storage rate alone and would otherwise reach for a package-level table. It exists because that is what the removed StorageTierInfo.CostPerGBMonth field was doing: a map built at package init cannot see a region, so every reader of it got us-east-1's price no matter what the configuration said. Use PricingManager.GetTierPricing for anything that should include discounts and overrides.

type PricingSummary 

type PricingSummary struct {
	UsePricingAPI bool `json:"use_pricing_api"`

	// Region is the region the rates in TierPricing come from.
	//
	// Read this, not ConfiguredRegion, when labeling a figure. The field used to hold the
	// configured value while the rates were unconditionally us-east-1's, which made this summary the
	// one place in the repo that actively asserted something false: `region: us-west-2` above
	// us-east-1 numbers.
	Region string `json:"region"`

	// ConfiguredRegion is pricing_config.region as the operator set it, empty if unset.
	//
	// Differs from Region only when the configured region has no published rates. Carried so that
	// case is visible in a report rather than only in a startup warning someone may not have kept.
	ConfiguredRegion string `json:"configured_region,omitempty"`

	Currency           string                        `json:"currency"`
	LastUpdated        time.Time                     `json:"last_updated"`
	EnterpriseDiscount float64                       `json:"enterprise_discount"`
	TierPricing        map[string]TierPricingSummary `json:"tier_pricing"`
}

PricingSummary provides a summary of pricing configuration

type ReplicationPricing 

type ReplicationPricing struct {
	ReplicationPerGB       float64 `yaml:"replication_per_gb"`       // Cost per GB replicated
	DestinationPutRequests float64 `yaml:"destination_put_requests"` // PUT request cost at destination
}

ReplicationPricing defines cross-region replication costs

type RequestCosts 

type RequestCosts struct {
	PutRequestCost    float64 `yaml:"put_request_cost"`    // Cost per PUT request
	GetRequestCost    float64 `yaml:"get_request_cost"`    // Cost per GET request
	DeleteRequestCost float64 `yaml:"delete_request_cost"` // Cost per DELETE request
	ListRequestCost   float64 `yaml:"list_request_cost"`   // Cost per LIST request
	HeadRequestCost   float64 `yaml:"head_request_cost"`   // Cost per HEAD request
}

RequestCosts defines per-request pricing

type StorageTierInfo 

type StorageTierInfo struct {
	Name string `json:"name"`

	// MinObjectSize is AWS's minimum *billable* object size: an object smaller than this is billed
	// as though it were this size. Zero where AWS publishes none, which is five of the eight classes.
	//
	// Only three classes have one — STANDARD_IA, ONEZONE_IA, and GLACIER_IR, all at 128 KB. This
	// field previously also carried 128 KB for INTELLIGENT_TIERING and 40 KB for GLACIER and
	// DEEP_ARCHIVE, and neither of those is a minimum billable size; see PerObjectOverheadBytes and
	// MonitoringEligibilityBytes below for what those two numbers actually are. AWS's storage class
	// comparison table lists "Min billable object size" as None for Intelligent-Tiering and NA for
	// both Glacier Flexible Retrieval and Deep Archive:
	// https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html#sc-compare
	//
	// A floor and an overhead point opposite ways for anything that reasons about small objects,
	// which is why they cannot share a field. Under a floor, compressing a 30 KB object to 10 KB
	// saves nothing. Under an overhead, it saves 20 KB and the surcharge is unaffected.
	MinObjectSize int64 `json:"min_object_size"`

	// PerObjectOverheadBytes is storage AWS bills per object in addition to the object itself, zero
	// where there is none.
	//
	// Only the two archive classes have it, at 40 KB, and it is not billed at one rate: 32 KB is
	// charged at the archive class's own rate for the index and metadata Glacier maintains, and 8 KB
	// at the S3 Standard rate for the name and metadata S3 keeps so the object can be listed. The
	// split is why this is a poor fit for a single number, and callers that only need a size can use
	// the sum; ArchiveOverhead returns the two parts for callers that price them.
	//
	// It dominates for small objects: a 10 KB object on DEEP_ARCHIVE is billed for 10 KB of payload
	// plus 40 KB of overhead, so roughly 23× the payload's cost once the two rates are applied.
	// https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html#sc-glacier
	PerObjectOverheadBytes int64 `json:"per_object_overhead_bytes"`

	// MonitoringEligibilityBytes is the size below which INTELLIGENT_TIERING does not monitor an
	// object, zero for every other class.
	//
	// 128 KB, the same number as the IA classes' billable minimum, which is very likely how it came
	// to be stored in MinObjectSize with the comment "128 KB minimum for optimization". It is not a
	// billing floor in either direction: an object below it is billed for its real size, is not
	// charged the per-object monitoring and automation fee, and stays in the Frequent Access tier
	// permanently rather than being auto-tiered. So it is a statement about what the tier will *do*
	// with an object, not about what the object costs.
	MonitoringEligibilityBytes int64 `json:"monitoring_eligibility_bytes"`

	DeletionEmbargo    time.Duration `json:"deletion_embargo"`
	RetrievalLatency   string        `json:"retrieval_latency"`
	RetrievalCost      bool          `json:"retrieval_cost"`
	MinimumStorageDays int           `json:"minimum_storage_days"`
	RecommendedUseCase string        `json:"recommended_use_case"`
}

StorageTierInfo contains tier-specific information and constraints

type TierConstraints 

type TierConstraints struct {
	MinObjectSize      int64         `yaml:"min_object_size"`      // Minimum object size in bytes
	DeletionEmbargo    time.Duration `yaml:"deletion_embargo"`     // Minimum storage duration before deletion
	RetrievalLatency   string        `yaml:"retrieval_latency"`    // Expected retrieval latency ("instant", "minutes", "hours")
	RetrievalCost      bool          `yaml:"retrieval_cost"`       // Whether retrieval incurs additional charges
	MinimumStorageDays int           `yaml:"minimum_storage_days"` // Minimum billable storage period
	TransitionDelay    time.Duration `yaml:"transition_delay"`     // Delay before transitioning to this tier
}

TierConstraints defines tier-specific constraints and limitations

type TierOptimization 

type TierOptimization struct {
	ObjectKey               string  `json:"object_key"`
	FromTier                string  `json:"from_tier"`
	ToTier                  string  `json:"to_tier"`
	Reason                  string  `json:"reason"`
	EstimatedMonthlySavings float64 `json:"estimated_monthly_savings"`
	ConfidenceLevel         float64 `json:"confidence_level"`
	ObjectSize              int64   `json:"object_size"`
	AccessFrequency         string  `json:"access_frequency"`
}

TierOptimization represents a suggested tier optimization

type TierPricing 

type TierPricing struct {
	StorageCostPerGBMonth float64      `yaml:"storage_cost_per_gb_month"` // $/GB/month for storage
	RetrievalCostPerGB    float64      `yaml:"retrieval_cost_per_gb"`     // $/GB for retrieval
	RequestCosts          RequestCosts `yaml:"request_costs"`             // Per-request pricing

	// MinimumBillableSize is the size an object smaller than it is billed as. Zero for the five
	// classes AWS publishes no minimum for — see [StorageTierInfo.MinObjectSize], which this mirrors.
	MinimumBillableSize int64 `yaml:"minimum_billable_size"`

	// PerObjectOverheadBytes is billed *in addition* to the object, not instead of a smaller size.
	// Only the two archive classes have it. It is a separate field from MinimumBillableSize because
	// the two are arithmetically opposite: a minimum replaces the size, an overhead adds to it. Both
	// were previously carried in MinimumBillableSize, which made the cost of a 10 KB DEEP_ARCHIVE
	// object 40 KB when it is 50 KB, and made compressing it look free of benefit when it is not.
	PerObjectOverheadBytes int64 `yaml:"per_object_overhead_bytes"`

	// OverheadStandardRateBytes is the portion of PerObjectOverheadBytes billed at the S3 Standard
	// rate rather than at this tier's rate — 8 KB of the archive classes' 40 KB. Priced at the tier
	// rate it would be understated about 23-fold on DEEP_ARCHIVE.
	OverheadStandardRateBytes int64 `yaml:"overhead_standard_rate_bytes"`

	MinimumBillableDays int                `yaml:"minimum_billable_days"` // Minimum billable period
	TransitionCosts     map[string]float64 `yaml:"transition_costs"`      // Cost to transition to other tiers
}

TierPricing defines pricing for a specific storage tier

type TierPricingSummary 

type TierPricingSummary struct {
	StorageCostPerGBMonth float64 `json:"storage_cost_per_gb_month"`
	RetrievalCostPerGB    float64 `json:"retrieval_cost_per_gb"`
	PutRequestCost        float64 `json:"put_request_cost"`
	GetRequestCost        float64 `json:"get_request_cost"`
}

TierPricingSummary provides a summary of tier pricing

type TierValidator 

type TierValidator struct {
	// contains filtered or unexported fields
}

TierValidator validates operations against storage tier constraints

func NewTierValidator 

func NewTierValidator(region, tier string, constraints TierConstraints, logger *slog.Logger) *TierValidator

NewTierValidator creates a new tier validator for a tier in a pricing region.

region is the pricing region — PricingConfig.Region, not necessarily the bucket's region — and it is only read by TierValidator.GetRecommendations, which compares this tier's storage rate against Standard's. An empty region, or one AWS publishes no rates for, falls back to awsrates.DefaultRegion with a warning: the crossover size at which STANDARD becomes cheaper than a tier's billing minimum moves with the ratio of the two rates, and that ratio is not the same everywhere. It is 0.543 in us-east-1 and 0.309 in sa-east-1 for STANDARD_IA, so a recommendation computed against the wrong region's rates changes at the wrong size.

func (*TierValidator) GetRecommendations 

func (tv *TierValidator) GetRecommendations(objectSize int64, accessFrequency string) []string

GetRecommendations returns tier recommendations based on access patterns.

The size-based recommendation is the same judgement CostOptimizer.HandleStandardTierOverhead makes on the write path, and it used to be wrong in the same way: `objectSize < 128 KiB` with no reference to the configured tier, so it advised moving to STANDARD from tiers that have no billing minimum at all, and from GLACIER_IR at sizes where GLACIER_IR billed at 128 KiB is nearly 3× cheaper than STANDARD at the object's own size. Being below a floor does not make STANDARD cheaper; the crossover is at minBillable × rateTier / rateStandard. Advice nobody can act on wrongly is still advice someone will act on.

This is the recommendation form, so it compares list rates from awsrates for the validator's pricing region rather than the discounted prices a deployment pays — the CostOptimizer holds the discounts and the validator does not have one. That makes it slightly conservative where an operator has a negotiated rate, which is the safe direction for a suggestion.

Both rates come from the same region, which is the property that matters here. The comparison is a ratio, so mixing regions would be worse than using the wrong one consistently: the crossover size would then be computed from a numerator and denominator that no operator anywhere pays.

func (*TierValidator) GetTierInfo 

func (tv *TierValidator) GetTierInfo() StorageTierInfo

GetTierInfo returns information about the current tier

func (*TierValidator) ValidateDelete 

func (tv *TierValidator) ValidateDelete(key string, objectAge time.Duration) error

ValidateDelete validates a delete operation against tier constraints

func (*TierValidator) ValidateWrite 

func (tv *TierValidator) ValidateWrite(key string, dataSize int64) error

ValidateWrite validates a write operation against tier constraints.

Two different things are checked against a size here, and only one of them can refuse the write.

AWS's per-tier minimum is a *billing* floor: S3 accepts a 1-byte STANDARD_IA object and bills it as 128 KiB. It never rejects the write. Refusing it here was therefore ObjectFS policy dressed as S3 behavior, and it made the filesystem unusable rather than expensive — `internal/fuse` creates both directory markers and new files by PUTting zero bytes, so on any tier with a minimum every `mkdir` and every `touch` failed, including the ones an IA-tier test needs to set itself up. It is a warning now, reporting the size that will be billed alongside the size that was written (#154).

`TierConstraints.MinObjectSize` is the one that still errors. An operator who sets it has asked for a floor that is not AWS's, and a policy nobody configured is the only kind worth removing. Note the consequence of that split, because it is easy to configure by accident: setting the constraint to the tier's own published minimum restores the old behavior in full, zero-byte directory markers included.

What #229 fixed is the input to all of this: the gate previously refused writes under 40 KB to GLACIER and DEEP_ARCHIVE and under 128 KB to INTELLIGENT_TIERING, and AWS publishes no minimum billable size for any of those three — so it was rejecting writes on the strength of numbers that were not minimums at all. Those three now have no minimum, and the two archive classes warn about their per-object overhead instead, which is the real cost and points the other way.

func (*TierValidator) ValidateWriteToTier added in v0.11.0 

func (tv *TierValidator) ValidateWriteToTier(key string, dataSize int64, tier string) error

ValidateWriteToTier is ValidateWrite for an object that is not going to the configured tier.

The cost-optimization setting `small_objects_on_standard` diverts an individual object to STANDARD when the configured tier would bill it as larger than it is, so the class an object is written with is a per-object decision while the validator is constructed once per mount. Validating against the configured tier regardless meant an operator who enabled that setting *because of* the warning below still got the warning: "billed_size 131072" on a 16 KiB object that was about to be stored on STANDARD and billed as 16 KiB. The diversion itself logs at Debug, so at the default level the only thing visible was the part that was wrong.

The split between what follows the effective tier and what does not is deliberate. The two billing warnings describe what AWS will charge, so they follow the tier the object is actually stored on. TierConstraints.MinObjectSize does not: it is a floor the operator set for this mount, and an ObjectFS-internal cost diversion is not a reason to stop enforcing a policy someone configured.

type UploadPart 

type UploadPart struct {
	PartNumber   int       `json:"part_number"`
	Size         int64     `json:"size"`
	ETag         string    `json:"etag"`
	Completed    bool      `json:"completed"`
	LastModified time.Time `json:"last_modified"`
	Offset       int64     `json:"offset"`          // Byte offset in the file
	RetryCount   int       `json:"retry_count"`     // Number of retry attempts
	Error        string    `json:"error,omitempty"` // Last error if any
}

UploadPart represents a single part of a multipart upload

type VolumeTier 

type VolumeTier struct {
	MinSizeGB       float64  `yaml:"min_size_gb"`      // Minimum size for this tier
	MaxSizeGB       float64  `yaml:"max_size_gb"`      // Maximum size for this tier (-1 = unlimited)
	DiscountPercent float64  `yaml:"discount_percent"` // Discount percentage for this tier
	AppliesTo       []string `yaml:"applies_to"`       // Which storage tiers this applies to
}

VolumeTier defines volume-based discount tiers

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.