Documentation
¶
Overview ¶
Package httputil provides HTTP utilities for package registry clients.
Overview ¶
This package provides infrastructure used by all registry API clients:
Caching ¶
Cache stores HTTP responses in the filesystem (~/.cache/stacktower/) with configurable TTL. This dramatically speeds up repeated operations and reduces load on package registries.
Usage:
cache, err := httputil.NewCache(24 * time.Hour)
data, ok := cache.Get("pypi:fastapi") // Check cache
if !ok {
data = fetchFromAPI()
cache.Set("pypi:fastapi", data) // Store for later
}
Cache keys should be namespaced by registry to avoid collisions.
Retry ¶
Retry wraps HTTP requests with automatic retry for transient failures:
- Network errors
- 5xx server errors
- 429 rate limit responses
It uses exponential backoff with jitter to avoid thundering herd:
resp, err := httputil.Retry(func() (*http.Response, error) {
return http.Get(url)
})
Configuration ¶
Default settings are suitable for most use cases:
- Cache directory: ~/.cache/stacktower/
- Default TTL: 24 hours
- Max retries: 3
- Base backoff: 1 second
The cache can be cleared via `stacktower cache clear` or by deleting the cache directory.
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ErrExpired = errors.New("cache entry expired")
ErrExpired is returned by Cache.Get when a cached entry exists but has exceeded its time-to-live (TTL).
When you receive ErrExpired, the cached data still exists on disk but is considered stale. Callers should fetch fresh data from the source and update the cache with Cache.Set.
Use errors.Is to check for this error:
ok, err := cache.Get("key", &value)
if errors.Is(err, httputil.ErrExpired) {
// Fetch fresh data and update cache
}
Functions ¶
func Retry ¶
Retry executes fn up to attempts times with exponential backoff.
Only errors wrapped with RetryableError trigger a retry; all other errors are returned immediately. Between retries, Retry waits for delay, then doubles the delay for the next attempt (1s, 2s, 4s, etc.). If ctx is cancelled during a retry delay, Retry returns ctx.Err() immediately.
Parameters:
- ctx: Context for cancellation. If cancelled during backoff, returns ctx.Err().
- attempts: Maximum number of attempts (minimum 1). Zero or negative values default to 1.
- delay: Initial backoff duration. Doubled after each failed attempt.
- fn: Function to execute. Wrap errors in RetryableError to enable retries.
Returns the result of fn on success, the last error if all attempts fail, or ctx.Err() if the context is cancelled during backoff.
Retry is safe to call from multiple goroutines. However, fn itself must handle any concurrency concerns for the operation it performs.
Example ¶
package main
import (
"context"
"fmt"
"time"
"github.com/matzehuels/stacktower/pkg/httputil"
)
func main() {
ctx := context.Background()
attempts := 0
// Simulate an operation that fails twice then succeeds
err := httputil.Retry(ctx, 3, 10*time.Millisecond, func() error {
attempts++
if attempts < 3 {
// Wrap transient errors to enable retry
return &httputil.RetryableError{
Err: fmt.Errorf("temporary failure (attempt %d)", attempts),
}
}
return nil // Success
})
if err != nil {
fmt.Println("Failed:", err)
} else {
fmt.Println("Success after", attempts, "attempts")
}
}
Output: Success after 3 attempts
func RetryWithBackoff ¶
RetryWithBackoff is a convenience wrapper around Retry with sensible defaults.
It performs up to 3 attempts with exponential backoff starting at 1 second: attempt 1 (immediate), wait 1s, attempt 2, wait 2s, attempt 3. Total maximum wait time is 3 seconds across all retries.
Use this when you need retry logic but don't need custom retry parameters. For more control over attempts or delay, call Retry directly.
Example ¶
package main
import (
"context"
"fmt"
"github.com/matzehuels/stacktower/pkg/httputil"
)
func main() {
ctx := context.Background()
// Fetch data with automatic retry on transient failures
err := httputil.RetryWithBackoff(ctx, func() error {
// Your HTTP request or other operation here
// Return &httputil.RetryableError{...} for transient failures
// Return regular errors for permanent failures
return nil
})
if err != nil {
fmt.Println("Error:", err)
} else {
fmt.Println("Success")
}
}
Output: Success
func Retryable ¶ added in v0.2.2
Retryable wraps an error as a RetryableError, signaling to Retry that this failure should trigger a retry attempt.
This is a convenience helper that avoids verbose struct literal syntax. Returns nil if err is nil, allowing safe use in error returns:
if err := doSomething(); err != nil {
return httputil.Retryable(err)
}
Example ¶
package main
import (
"context"
"errors"
"fmt"
"github.com/matzehuels/stacktower/pkg/httputil"
)
func main() {
ctx := context.Background()
attempts := 0
// Using the Retryable helper for cleaner code
err := httputil.RetryWithBackoff(ctx, func() error {
attempts++
if attempts < 2 {
// Wrap errors concisely with Retryable()
return httputil.Retryable(errors.New("temporary failure"))
}
return nil
})
if err == nil {
fmt.Println("Success")
}
}
Output: Success
Types ¶
type Cache ¶
type Cache struct {
// contains filtered or unexported fields
}
Cache provides file-based caching of arbitrary JSON-marshalable data.
Each cache entry is stored as a JSON file in the cache directory, with the filename derived from a SHA-256 hash of the cache key. This design ensures safe key names (no filesystem special characters) and prevents key collisions across different namespaces.
Cache operations are not goroutine-safe. If multiple goroutines access the same Cache instance, the caller must synchronize access. However, multiple Cache instances (even in different processes) can safely share the same directory, as the filesystem provides atomic file operations.
Cache entries have a time-to-live (TTL) based on file modification time. A TTL of 0 means entries never expire.
Use Cache.Namespace to create scoped views that automatically prefix keys, avoiding collisions between different data sources:
pypi := cache.Namespace("pypi:")
npm := cache.Namespace("npm:")
pypi.Set("requests", data) // key becomes "pypi:requests"
Example ¶
package main
import (
"fmt"
"os"
"path/filepath"
"time"
"github.com/matzehuels/stacktower/pkg/httputil"
)
func main() {
// Create a cache with 24-hour TTL in a temp directory
dir := filepath.Join(os.TempDir(), "stacktower-example")
cache, err := httputil.NewCache(dir, 24*time.Hour)
if err != nil {
fmt.Println("Error:", err)
return
}
// Store a value
data := map[string]string{"name": "example", "version": "1.0.0"}
if err := cache.Set("mykey", data); err != nil {
fmt.Println("Error:", err)
return
}
// Retrieve the value
var result map[string]string
if ok, err := cache.Get("mykey", &result); ok && err == nil {
fmt.Println("Name:", result["name"])
fmt.Println("Version:", result["version"])
}
// Clean up
os.RemoveAll(dir)
}
Output: Name: example Version: 1.0.0
Example (Miss) ¶
package main
import (
"fmt"
"os"
"path/filepath"
"time"
"github.com/matzehuels/stacktower/pkg/httputil"
)
func main() {
dir := filepath.Join(os.TempDir(), "stacktower-example-miss")
cache, _ := httputil.NewCache(dir, time.Hour)
defer os.RemoveAll(dir)
// Try to get a non-existent key
var result string
ok, err := cache.Get("nonexistent", &result)
fmt.Println("Found:", ok)
fmt.Println("Error:", err)
}
Output: Found: false Error: <nil>
func NewCache ¶
NewCache creates a Cache that stores entries in dir with the given TTL.
If dir is empty, NewCache uses the default directory ~/.cache/stacktower/. The directory is created with mode 0755 if it doesn't exist. If directory creation fails (e.g., due to permissions), NewCache returns an error.
Parameters:
- dir: Cache directory path. Use "" for default (~/.cache/stacktower/).
- ttl: Time-to-live for cache entries. Use 0 for no expiration.
The returned Cache is ready to use. Directory creation errors are the only possible source of failure.
Example (DefaultDir) ¶
package main
import (
"fmt"
"time"
"github.com/matzehuels/stacktower/pkg/httputil"
)
func main() {
// Pass empty string to use default directory (~/.cache/stacktower/)
cache, err := httputil.NewCache("", 24*time.Hour)
if err != nil {
fmt.Println("Error:", err)
return
}
fmt.Println("Cache TTL:", cache.TTL())
}
Output: Cache TTL: 24h0m0s
func (*Cache) Get ¶
Get retrieves a cached value by key and unmarshals it into v.
Return values indicate three distinct outcomes:
- (true, nil): Cache hit. The value was found, is fresh, and unmarshaled into v.
- (false, nil): Cache miss. No entry exists for this key. v is unchanged.
- (false, ErrExpired): Entry exists but exceeded its TTL. v is unchanged.
- (false, other error): I/O error, JSON unmarshal error, etc. v may be partially modified.
The key can be any string. Consider namespacing keys to avoid collisions (e.g., "pypi:requests", "npm:react"). The key is hashed with SHA-256, so long keys are acceptable.
The value v must be a pointer to a type compatible with json.Unmarshal. Common types include *string, *[]byte, *map[string]any, and pointers to custom structs with JSON tags.
Get does not modify the cache or update modification times; reads are non-mutating operations.
func (*Cache) Namespace ¶ added in v0.2.2
Namespace returns a new Cache that automatically prefixes all keys with prefix.
This creates a scoped view of the cache, useful for avoiding key collisions between different data sources or components. The returned Cache shares the same underlying directory and TTL as the parent.
Example:
cache, _ := httputil.NewCache("", 24*time.Hour)
pypiCache := cache.Namespace("pypi:")
npmCache := cache.Namespace("npm:")
pypiCache.Set("requests", pypiData) // Stored as "pypi:requests"
npmCache.Set("express", npmData) // Stored as "npm:express"
Namespace calls can be chained to create hierarchical key spaces:
cache.Namespace("python:").Namespace("pypi:") // prefix: "python:pypi:"
The prefix is applied transparently to all Get and Set operations. An empty prefix is valid and results in no key transformation.
Example ¶
package main
import (
"fmt"
"os"
"path/filepath"
"time"
"github.com/matzehuels/stacktower/pkg/httputil"
)
func main() {
dir := filepath.Join(os.TempDir(), "stacktower-namespace-example")
cache, _ := httputil.NewCache(dir, 24*time.Hour)
defer os.RemoveAll(dir)
// Create namespaced caches for different registries
pypiCache := cache.Namespace("pypi:")
npmCache := cache.Namespace("npm:")
// Store values in different namespaces
pypiCache.Set("requests", map[string]string{"version": "2.31.0"})
npmCache.Set("express", map[string]string{"version": "4.18.2"})
// Retrieve from appropriate namespace
var pypiData map[string]string
pypiCache.Get("requests", &pypiData)
fmt.Println("PyPI requests:", pypiData["version"])
var npmData map[string]string
npmCache.Get("express", &npmData)
fmt.Println("npm express:", npmData["version"])
}
Output: PyPI requests: 2.31.0 npm express: 4.18.2
func (*Cache) Set ¶
Set stores a value in the cache under the given key.
The value v is marshaled to JSON using encoding/json and written to disk. If v cannot be marshaled (e.g., contains channels or functions), Set returns a json.MarshalError. If the write fails (e.g., disk full, permission denied), Set returns the underlying I/O error.
Set overwrites any existing entry for key, resetting its modification time to the current time. This effectively refreshes the TTL.
The value v is not modified by Set; marshaling operates on a copy.
type RetryableError ¶
type RetryableError struct{ Err error }
RetryableError wraps an error to indicate it should trigger a retry. Use this type to signal transient failures like network timeouts, temporary DNS resolution failures, or HTTP 5xx server errors. Errors not wrapped in RetryableError are treated as permanent failures and cause Retry to return immediately without further attempts.
Prefer using the Retryable helper function for convenience:
if resp.StatusCode >= 500 {
return httputil.Retryable(fmt.Errorf("server error: %d", resp.StatusCode))
}
RetryableError implements error unwrapping, so errors.Is and errors.As work correctly with the wrapped error.
Example ¶
package main
import (
"context"
"errors"
"fmt"
"time"
"github.com/matzehuels/stacktower/pkg/httputil"
)
func main() {
ctx := context.Background()
networkErr := errors.New("connection refused")
err := httputil.Retry(ctx, 2, 10*time.Millisecond, func() error {
// Permanent error - no retry
if false {
return errors.New("invalid request")
}
// Transient error - will retry
return &httputil.RetryableError{Err: networkErr}
})
// Check if the underlying error is our network error
if errors.Is(err, networkErr) {
fmt.Println("Failed due to network error")
}
}
Output: Failed due to network error
func (*RetryableError) Error ¶
func (e *RetryableError) Error() string
Error returns the error message of the wrapped error.
func (*RetryableError) Unwrap ¶
func (e *RetryableError) Unwrap() error
Unwrap returns the wrapped error, enabling errors.Is and errors.As to inspect the underlying cause.
Source Files