httpcache

README

httpcache

Package httpcache provides an http.RoundTripper implementation that works as a mostly RFC 9111 (HTTP Caching) compliant cache for HTTP responses. It improves application performance by reducing redundant HTTP requests and supports various backends for use cases such as API caching, web scraping, and microservices.

RFC Compliance: This implementation follows RFC 9111 (2022), which obsoletes RFC 7234 (2014). See the compliance features for details on supported directives and behaviors.

Note: This is a maintained fork of gregjones/httpcache, which is no longer actively maintained. This fork aims to modernize the codebase while maintaining backward compatibility, fix bugs, and add new features.

Use Cases

• API Caching: Reduce latency and server load by caching API responses.
• Web Scraping: Avoid repeated requests to the same endpoints.
• Microservices: Cache responses between services for better performance.
• Web Applications: Improve user experience by caching dynamic content.
• Resource Caching: Store static or frequently accessed resources locally.

Table of Contents

• httpcache
  • Use Cases
  • Table of Contents
  • Features
  • Quick Start
  • Installation
  • Documentation
    • 📚 Core Documentation
    • 🔍 Quick Navigation
  • Practical Examples
  • Limitations
    • Private Directive for Public Caches
    • Authorization Header in Shared Caches
  • Performance
  • Testing
  • Contributing
  • Acknowledgments
  • License
  • Support

Features

• ✅ RFC 9111 Compliant (~95% compliance) - Implements HTTP Caching standard (obsoletes RFC 7234)
  • ✅ Age header calculation with full Section 4.2.3 algorithm (request_time, response_time, response_delay tracking)
  • ✅ Age header validation per Section 5.1 (handles multiple values, invalid values with logging)
  • ✅ Cache-Control directive validation per Section 4.2.1 (duplicate detection, conflict resolution, value validation)
  • ✅ Warning headers for stale responses (Section 5.5 - deprecated but supported for compatibility)
  • ✅ must-revalidate directive enforcement (Section 5.2.2.1)
  • ✅ Pragma: no-cache support (Section 5.4 - HTTP/1.0 backward compatibility)
  • ✅ Cache invalidation on unsafe methods (Section 4.4)
  • ✅ Content-Location and Location header invalidation (Section 4.4)
  • ✅ Same-origin policy enforcement for cache invalidation
  • ✅ Cache-Control: private directive support (Section 5.2.2.6)
  • ✅ Cache-Control: must-understand directive support (Section 5.2.2.3)
  • ✅ Vary header matching per Section 4.1 (wildcard, whitespace normalization, case-insensitive)
  • ✅ Vary header separation - Optional separate cache entries for response variants (Section 4.1)
  • ✅ Authorization header handling per Section 3.5 (secure caching in shared caches)
• ✅ Multiple Backends - Memory, Disk, Redis, LevelDB, Memcache, PostgreSQL, MongoDB, NATS K/V, Hazelcast, Cloud Storage (S3/GCS/Azure)
• ✅ Multi-Tier Caching - Combine multiple backends with automatic fallback and promotion
• ✅ Compression Wrapper - Automatic Gzip, Brotli, or Snappy compression for cached data
• ✅ Security Wrapper - Optional SHA-256 key hashing and AES-256 encryption
• ✅ Thread-Safe - Safe for concurrent use
• ✅ Zero Dependencies - Core package uses only Go standard library
• ✅ Easy Integration - Drop-in replacement for http.Client
• ✅ ETag & Validation - Automatic cache revalidation
• ✅ Stale-If-Error - Resilient caching with RFC 5861 support
• ✅ Stale-While-Revalidate - Async cache updates for better performance (RFC 5861)
• ✅ Configurable Cache Mode - Use as private cache (default) or shared/public cache

Quick Start

package main

import (
    "fmt"
    "io"
    "net/http"
    
    "github.com/sandrolain/httpcache"
)

func main() {
    // Create a cached HTTP client
    transport := httpcache.NewMemoryCacheTransport()
    client := transport.Client()
    
    // Make requests - second request will be cached!
    resp, _ := client.Get("https://example.com")
    io.Copy(io.Discard, resp.Body)
    resp.Body.Close()
    
    // Check if response came from cache
    if resp.Header.Get(httpcache.XFromCache) == "1" {
        fmt.Println("Response was cached!")
    }
}

Installation

go get github.com/sandrolain/httpcache

Documentation

📚 Core Documentation

• Cache Backends - Choose and configure storage backends (Memory, Redis, PostgreSQL, etc.)
• How It Works - RFC 7234 implementation details, cache headers, and detecting cache hits
• Advanced Features - Transport configuration, stale-if-error, cache key headers, custom cache control
• Security Considerations - Multi-user applications, secure cache wrapper, best practices
• Monitoring with Prometheus - Optional metrics integration for production monitoring

🔍 Quick Navigation

Getting Started:

• Installation (this page)
• Quick Start (this page)
• Cache Backends - See all backend examples

Common Tasks:

• Detecting cache hits
• User-specific caching
• Authorization header handling
• Securing sensitive data
• Monitoring performance

Advanced Topics:

• RFC 9111 compliance
• Stale-while-revalidate
• Multi-tier caching strategies
• Compression wrapper - Gzip, Brotli, Snappy compression
• Custom cache implementation
• Multi-user considerations

Practical Examples

See the examples/ directory for complete, runnable examples:

• Basic - Simple in-memory caching
• Disk Cache - Persistent filesystem cache
• Redis - Distributed caching with Redis
• LevelDB - High-performance persistent cache
• PostgreSQL - SQL-based persistent cache
• NATS K/V - NATS JetStream Key/Value cache
• Hazelcast - Enterprise distributed cache
• FreeCache - High-performance in-memory with zero GC
• Security Best Practices - Secure cache with encryption and key hashing
• Compress Cache - Automatic Gzip/Brotli/Snappy compression
• Multi-Tier Cache - Multi-tiered caching with automatic fallback and promotion
• Custom Backend - Build your own cache backend
• Prometheus Metrics - Monitoring cache performance
• Cache Key Headers - User-specific caching with headers

Each example includes:

• Complete working code
• Detailed README
• Use case explanations
• Best practices

Limitations

Private Directive for Public Caches

⚠️ Note: When configured as a public cache (IsPublicCache: true), responses with the Cache-Control: private directive are not cached.

Default Behavior: By default, httpcache operates as a private cache, which allows caching of responses marked as private.

Public Cache Mode: When IsPublicCache is set to true, the cache behaves as a shared cache and respects the private directive by not caching such responses.

See Security Considerations and Advanced Features - Private vs Public Cache for details.

Authorization Header in Shared Caches

⚠️ Note: When configured as a shared/public cache (IsPublicCache: true), requests with an Authorization header are NOT cached unless the response contains one of these directives:

• Cache-Control: public
• Cache-Control: must-revalidate
• Cache-Control: s-maxage=<seconds>

This implements RFC 9111 Section 3.5 to prevent unauthorized access to cached authenticated responses in shared caches.

Default Behavior: Private caches (default) can cache Authorization responses without restrictions.

Shared Cache Mode: Requires explicit server permission via the directives above. Additionally, use CacheKeyHeaders to separate cache entries per user:

transport.IsPublicCache = true
transport.CacheKeyHeaders = []string{"Authorization"}  // Separate cache per user

See Authorization Header and Shared Caches for detailed examples and security considerations.

Performance

• Memory cache: ~100ns per operation
• Disk cache: ~1-5ms per operation (depends on filesystem)
• Redis cache: ~1-3ms per operation (network latency dependent)
• Overhead vs no-cache: < 5% for cache hits

Benchmarks are available in each backend's *_bench_test.go file.

Testing

# Run all tests
go test ./...

# Run with coverage
go test -cover ./...

# Run integration tests (requires Docker)
go test -tags=integration ./...

# Run benchmarks
go test -bench=. ./...

Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes with tests
4. Run golangci-lint run and govulncheck ./...
5. Commit your changes (git commit -m 'feat: add amazing feature')
6. Push to the branch (git push origin feature/amazing-feature)
7. Open a Pull Request

Acknowledgments

This project is a fork of gregjones/httpcache by Greg Jones, which was archived in 2023. We're grateful for the original work and continue its development with modern Go practices.

Additional acknowledgments:

• RFC 9111 - HTTP Caching (obsoletes RFC 7234)
• RFC 7234 - HTTP Caching (obsoleted by RFC 9111, still referenced for historical context)
• RFC 5861 - HTTP Cache-Control Extensions for Stale Content
• All contributors to the original and forked projects

License

MIT License - see LICENSE.txt for details.

Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Documentation: This README and the docs/ directory
• Examples: See examples/ for practical use cases

Documentation

Overview

Package httpcache provides a http.RoundTripper implementation that works as a mostly RFC 9111 compliant cache for HTTP responses.

By default, it operates as a 'private' cache (suitable for web browsers or API clients). It can also be configured as a 'shared/public' cache by setting IsPublicCache to true, which enforces stricter caching rules for multi-user scenarios (e.g., CDNs, reverse proxies).

RFC 9111 (HTTP Caching) obsoletes RFC 7234 and is the current HTTP caching standard.

Index

• Constants
• Variables
• func CachedResponse(c Cache, req *http.Request) (resp *http.Response, err error)
• func Date(respHeaders http.Header) (date time.Time, err error)
• func GetLogger() *slog.Logger
• func SetLogger(l *slog.Logger)
• type Cache
• type MemoryCache
  • func NewMemoryCache() *MemoryCache
  • func (c *MemoryCache) Delete(key string)
  • func (c *MemoryCache) Get(key string) (resp []byte, ok bool)
  • func (c *MemoryCache) Set(key string, resp []byte)
• type Transport
  • func NewMemoryCacheTransport() *Transport
  • func NewTransport(c Cache) *Transport
  • func (t *Transport) Client() *http.Client
  • func (t *Transport) RoundTrip(req *http.Request) (resp *http.Response, err error)

Constants

const (

	// XFromCache is the header added to responses that are returned from the cache
	XFromCache = "X-From-Cache"
	// XRevalidated is the header added to responses that got revalidated
	XRevalidated = "X-Revalidated"
	// XStale is the header added to responses that are stale
	XStale = "X-Stale"
	// XFreshness is the header added to responses indicating the freshness state
	XFreshness = "X-Cache-Freshness"
	// XCachedTime is the internal header used to store when a response was cached
	XCachedTime = "X-Cached-Time"
	// XRequestTime stores when the HTTP request was initiated (for Age calculation per RFC 9111)
	XRequestTime = "X-Request-Time"
	// XResponseTime stores when the HTTP response was received (for Age calculation per RFC 9111)
	XResponseTime = "X-Response-Time"
)

Variables

var ErrNoDateHeader = errors.New("no Date header")

ErrNoDateHeader indicates that the HTTP headers contained no Date header.

Functions

func CachedResponse

func CachedResponse(c Cache, req *http.Request) (resp *http.Response, err error)

CachedResponse returns the cached http.Response for req if present, and nil otherwise.

func Date

func Date(respHeaders http.Header) (date time.Time, err error)

Date parses and returns the value of the Date header.

func GetLogger

func GetLogger() *slog.Logger

GetLogger returns the configured logger or the default slog logger.

func SetLogger

func SetLogger(l *slog.Logger)

SetLogger sets a custom slog.Logger instance to be used by the httpcache package. If not set, the default slog logger will be used.

Types

type Cache

type Cache interface {
	// Get returns the []byte representation of a cached response and a bool
	// set to true if the value isn't empty
	Get(key string) (responseBytes []byte, ok bool)
	// Set stores the []byte representation of a response against a key
	Set(key string, responseBytes []byte)
	// Delete removes the value associated with the key
	Delete(key string)
}

A Cache interface is used by the Transport to store and retrieve responses.

type MemoryCache

type MemoryCache struct {
	// contains filtered or unexported fields
}

MemoryCache is an implemtation of Cache that stores responses in an in-memory map.

func NewMemoryCache

func NewMemoryCache() *MemoryCache

NewMemoryCache returns a new Cache that will store items in an in-memory map

func (*MemoryCache) Delete

func (c *MemoryCache) Delete(key string)

Delete removes key from the cache

func (*MemoryCache) Get

func (c *MemoryCache) Get(key string) (resp []byte, ok bool)

Get returns the []byte representation of the response and true if present, false if not

func (*MemoryCache) Set

func (c *MemoryCache) Set(key string, resp []byte)

Set saves response resp to the cache with key

type Transport

type Transport struct {
	// The RoundTripper interface actually used to make requests
	// If nil, http.DefaultTransport is used
	Transport http.RoundTripper
	Cache     Cache
	// If true, responses returned from the cache will be given an extra header, X-From-Cache
	MarkCachedResponses bool
	// If true, server errors (5xx status codes) will not be served from cache
	// even if they are fresh. This forces a new request to the server.
	// Default is false to maintain backward compatibility.
	SkipServerErrorsFromCache bool
	// AsyncRevalidateTimeout is the context timeout for async requests triggered by stale-while-revalidate.
	// If zero, no timeout is applied to async revalidation requests.
	AsyncRevalidateTimeout time.Duration
	// IsPublicCache enables public cache mode (default: false for private cache).
	// When true, the cache will NOT store responses with Cache-Control: private directive.
	// When false (default), the cache acts as a private cache and CAN store private responses.
	// RFC 9111: Private caches (browsers, API clients) can cache private responses.
	// Shared caches (CDNs, proxies) must NOT cache private responses.
	// Set to true only if using httpcache as a shared/public cache (CDN, reverse proxy).
	IsPublicCache bool
	// EnableVarySeparation enables RFC 9111 compliant Vary header separation (default: false).
	// When true, responses with Vary headers create separate cache entries for each variant.
	// When false (default), the previous behavior is maintained where variants overwrite each other.
	// RFC 9111 Section 4.1: Caches should maintain separate entries for different variants.
	// Enable this for full RFC 9111 compliance with content negotiation (Accept-Language, Accept, etc.).
	// Note: Enabling this may increase cache storage usage as each variant is stored separately.
	EnableVarySeparation bool
	// ShouldCache allows configuring non-standard caching behaviour based on the response.
	// If set, this function is called to determine whether a non-200 response should be cached.
	// This enables caching of responses like 404 Not Found, 301 Moved Permanently, etc.
	// If nil, only 200 OK responses are cached (standard behavior).
	// The function receives the http.Response and should return true to cache it.
	// Note: This only bypasses the status code check; Cache-Control headers are still respected.
	ShouldCache func(*http.Response) bool
	// CacheKeyHeaders specifies additional request headers to include in the cache key generation.
	// This allows creating separate cache entries based on request header values.
	// Common use cases include "Authorization" for user-specific caches or "Accept-Language"
	// for locale-specific responses.
	// Header names are case-insensitive and will be canonicalized.
	// Example: []string{"Authorization", "Accept-Language"}
	// Note: This is different from the HTTP Vary response header mechanism, which is handled separately.
	CacheKeyHeaders []string
	// DisableWarningHeader disables the deprecated Warning header (RFC 7234) in responses.
	// RFC 9111 has obsoleted the Warning header field, making it no longer part of the standard.
	// When true, Warning headers (110, 111, etc.) will not be added to cached responses.
	// Default is false (Warning headers are enabled for backward compatibility).
	// Set to true to comply with RFC 9111 and avoid deprecated headers.
	DisableWarningHeader bool
}

Transport is an implementation of http.RoundTripper that will return values from a cache where possible (avoiding a network request) and will additionally add validators (etag/if-modified-since) to repeated requests allowing servers to return 304 / Not Modified

func NewMemoryCacheTransport

func NewMemoryCacheTransport() *Transport

NewMemoryCacheTransport returns a new Transport using the in-memory cache implementation

func NewTransport

func NewTransport(c Cache) *Transport

NewTransport returns a new Transport with the provided Cache implementation and MarkCachedResponses set to true

func (*Transport) Client

func (t *Transport) Client() *http.Client

Client returns an *http.Client that caches responses.

func (*Transport) RoundTrip

func (t *Transport) RoundTrip(req *http.Request) (resp *http.Response, err error)

RoundTrip takes a Request and returns a Response

If there is a fresh Response already in cache, then it will be returned without connecting to the server.

If there is a stale Response, then any validators it contains will be set on the new request to give the server a chance to respond with NotModified. If this happens, then the cached Response will be returned.