filecache

README

📦 FileCache v2

Store data from io.Reader or bytes to cache files with TTL and metadata.

Installation

go get github.com/kukymbr/filecache/v2 

Usage

Initializing the cache instance

// With target dir specified
fc, err := filecache.New("/path/to/cache/dir")

// With temp dir as a target
fc, err := filecache.NewInTemp()

// With options
fc, err := filecache.New(
    "/path/to/cache/dir",
    filecache.InstanceOptions{
        PathGenerator: filecache.FilteredKeyPath,
        DefaultTTL:    time.Hour,
        GC:            filecache.NewIntervalGarbageCollector("/path/to/cache/dir", time.Hour),
    },
)

See the InstanceOptions godoc for the instance configuration values.

Saving data to the cache

// From the io.Reader
_, err := fc.Write(context.Background(), "key1", strings.NewReader("value1"))

// From the byte array
_, err := fc.WriteData(context.Background(), "key2", []byte("value2"))

// With the item options
_, err := fc.Write(
    context.Background(), 
    "key3", 
    strings.NewReader("value3"),
    filecache.ItemOptions{
        Name:   "Key 3",
        TTL:    time.Hour * 24,
        Fields: filecache.NewValues("field1", "val1", "field2", "val2"),
    },
)

See the ItemOptions godoc for the instance configuration values.

Reading from cache

// Opening the cache file reader
res, err := fc.Open(context.Background(), "key1")
if err != nil { 
    // Handle the error...
}

if res.Hit() {
    reader := res.Reader()
    // Read the data...
}

// Read all the data
res, err := fc.Read(context.Background(), "key2")
if err == nil && res.Hit() {
    data := res.Data()
}

// Read options
res, err := fc.Read(context.Background(), "key3")
if err == nil && res.Hit() {
    name := res.Options().Name
}

The Open() and Read() functions return an error only if context is canceled or if the file open operation has failed. If there is no error, this doesn't mean the result is found, the res.Hit() function should be called.

Iterate through the cached items

To iterate through the cached items, use the Scanner tool:

scanner := filecache.NewScanner("/path/to/cache/dir")

err := scanner.Scan(func(entry filecache.ScanEntry) error {
    // Do something with the found entry...
    return nil
})
if err := nil {
    // Handle the error...
}

Removing the expired items

The expired cache items are removed by the GarbageCollector, assigned to the FileCache instance.

There are three predefined realizations of the GarbageCollector:

• filecache.NewNopGarbageCollector() — the GarbageCollector doing nothing, all the files are kept;
• filecache.NewProbabilityGarbageCollector() — the GarbageCollector running with the defined probability, used by default;
• filecache.NewIntervalGarbageCollector() — the GarbageCollector running by the time interval.

See the gc.go's godocs for more info.

License

MIT.

Documentation

Index

• Constants
• func FilteredKeyPath(key string) string
• func HashedKeyPath(key string) string
• func HashedKeySplitPath(key string) string
• type FileCache
  • func New(targetDir string, options ...InstanceOptions) (FileCache, error)
  • func NewInTemp(options ...InstanceOptions) (FileCache, error)
  • func NewNop() FileCache
• type GarbageCollector
  • func NewIntervalGarbageCollector(dir string, interval time.Duration) GarbageCollector
  • func NewNopGarbageCollector() GarbageCollector
  • func NewProbabilityGarbageCollector(dir string, onInitDivisor uint, onOpDivisor uint) GarbageCollector
• type InstanceOptions
• type ItemOptions
• type OpenResult
  • func (r *OpenResult) Hit() bool
  • func (r *OpenResult) Options() *ItemOptions
  • func (r *OpenResult) Reader() io.ReadCloser
• type PathGeneratorFn
  • func WithExt(fn PathGeneratorFn, ext string) PathGeneratorFn
• type ReadResult
  • func (r *ReadResult) Data() []byte
  • func (r *ReadResult) Hit() bool
  • func (r *ReadResult) Options() *ItemOptions
• type ScanEntry
• type Scanner
  • func NewScanner(dir string) Scanner
• type ScannerHitFn
• type Values
  • func NewValues(keysAndValues ...any) Values

Constants

const (
	// TTLEternal is a TTL value for eternal cache.
	TTLEternal = util.TTLEternal
)

Functions

func FilteredKeyPath

func FilteredKeyPath(key string) string

FilteredKeyPath uses a key without path separators as a path.

func HashedKeyPath

func HashedKeyPath(key string) string

HashedKeyPath return hashed key and uses it as a file name.

func HashedKeySplitPath

func HashedKeySplitPath(key string) string

HashedKeySplitPath return hashes key, splits it on the parts which are directories and a file name.

Types

type FileCache

type FileCache interface {
	// GetPath returns the target path of the FileCache instance.
	GetPath() string

	// Write writes data from the reader to the cache file.
	Write(ctx context.Context, key string, reader io.Reader, options ...ItemOptions) (written int64, err error)

	// WriteData writes data to the cache file.
	WriteData(ctx context.Context, key string, data []byte, options ...ItemOptions) (written int64, err error)

	// Open opens the reader with cached data.
	//
	// Returns no error on successful cache hit, on no hit, on invalid cache files.
	// Returns an error if failed to open an existing cache file or if context is done.
	Open(ctx context.Context, key string) (result *OpenResult, err error)

	// Read reads data from the cache file.
	//
	// Returns no error on successful cache hit, on no hit, on invalid cache files.
	// Returns an error if failed to open or read an existing cache file or if context is done.
	Read(ctx context.Context, key string) (result *ReadResult, err error)

	// Invalidate removes data associated with a key from a cache.
	Invalidate(ctx context.Context, key string) error

	// Close closes the FileCache instance.
	Close() error
}

FileCache is a tool to cache data from any io.Reader to the file.

func New

func New(targetDir string, options ...InstanceOptions) (FileCache, error)

New creates a new FileCache instance with a specified target dir & options.

func NewInTemp

func NewInTemp(options ...InstanceOptions) (FileCache, error)

NewInTemp creates a new FileCache instance with files stored in the system's temp dir.

func NewNop

func NewNop() FileCache

NewNop creates no-operation file cache instance.

type GarbageCollector

type GarbageCollector interface {
	// OnInstanceInit is executed on the initialization of the FileCache instance.
	OnInstanceInit()

	// OnOperation is executed on the every item's operation in the FileCache instance.
	OnOperation()

	// Close closes the GarbageCollector.
	Close() error
}

GarbageCollector is a tool to remove expired cache items.

func NewIntervalGarbageCollector

func NewIntervalGarbageCollector(dir string, interval time.Duration) GarbageCollector

NewIntervalGarbageCollector returns the GarbageCollector running by the interval.

Function arguments:

• dir: the directory with the FileCache's instance files;
• interval: the GC interval duration.

func NewNopGarbageCollector

func NewNopGarbageCollector() GarbageCollector

NewNopGarbageCollector returns the GarbageCollector doing nothing.

func NewProbabilityGarbageCollector

func NewProbabilityGarbageCollector(dir string, onInitDivisor uint, onOpDivisor uint) GarbageCollector

NewProbabilityGarbageCollector returns the GarbageCollector running with the defined probability.

Function arguments:

• dir: the directory with the FileCache's instance files;
• onInitDivisor: divisor for the probability on the OnInstanceInit() function call;
• onOpDivisor: divisor for the probability on the OnOperation() function call.

Divisor is a run probability divisor (e.g., divisor equals 100 is a 1/100 probability).

type InstanceOptions

type InstanceOptions struct {
	// PathGenerator is a function to generate cache item's file path.
	// Receives the key of the cache item and returns the path of the item relative to the cache instance's dir.
	//
	// There are some built-in path generators:
	// 	 - FilteredKeyPath: removes path separators from the key and uses it as a file name;
	//   - HashedKeyPath: hashes the key and uses the result as a file name;
	//   - HashedKeySplitPath: hashes the key, splits result to the parts and uses them as a directories and file names.
	//
	// Also, there is a WithExt wrapper adding the extension to any path generator result.
	PathGenerator PathGeneratorFn

	// DefaultTTL is a TTL value for the items without it.
	DefaultTTL time.Duration

	// GC is a GarbageCollector instance for the cache instance.
	//
	// May be initialized with any GarbageCollector instance or using one of the predefined GC constructors:
	//   - NewNopGarbageCollector: the GarbageCollector doing nothing;
	//   - NewProbabilityGarbageCollector: the GarbageCollector running with the defined probability;
	//   - NewIntervalGarbageCollector: the GarbageCollector running by the interval.
	GC GarbageCollector

	// GCDivisor is a garbage collector run probability divisor
	// (e.g., 100 is 1/100 probability).
	//
	// Deprecated: use the GC property instead.
	GCDivisor uint
}

InstanceOptions are a cache instance options.

type ItemOptions

type ItemOptions struct {
	// Name is a human-readable item name.
	Name string

	// TTL is an item's time-to-live value.
	TTL time.Duration

	// Fields is a map of any other metadata fields.
	Fields Values
}

ItemOptions are a cache item options.

type OpenResult

type OpenResult struct {
	// contains filtered or unexported fields
}

OpenResult is a result of the file cache's Open operation.

func (*OpenResult) Hit

func (r *OpenResult) Hit() bool

Hit returns true, if requested key found in cache.

func (*OpenResult) Options

func (r *OpenResult) Options() *ItemOptions

Options returns a found cache item options.

func (*OpenResult) Reader

func (r *OpenResult) Reader() io.ReadCloser

Reader returns the cached data reader.

type PathGeneratorFn

type PathGeneratorFn util.PathGeneratorFn

PathGeneratorFn is a function to generate cache item's file path. Receives the key of the cache item and returns the path of the item relative to the cache instance's dir.

func WithExt

func WithExt(fn PathGeneratorFn, ext string) PathGeneratorFn

WithExt returns new PathGeneratorWithExt instance.

type ReadResult

type ReadResult struct {
	// contains filtered or unexported fields
}

ReadResult is a result of the file cache's Read operation.

func (*ReadResult) Data

func (r *ReadResult) Data() []byte

Data returns the cached data.

func (*ReadResult) Hit

func (r *ReadResult) Hit() bool

Hit returns true, if requested key found in cache.

func (*ReadResult) Options

func (r *ReadResult) Options() *ItemOptions

Options returns a found cache item options.

type ScanEntry

type ScanEntry struct {
	// Key is a cache item key.
	Key string

	// CreatedAt is a cache item created-at timestamp.
	CreatedAt time.Time

	// Options are the options of the item stored in the cache.
	Options *ItemOptions
	// contains filtered or unexported fields
}

ScanEntry is a scanner hit entry.

type Scanner

type Scanner interface {
	Scan(onHit ScannerHitFn) error
}

Scanner is a tool to scan cache items inside the specified directory.

func NewScanner

func NewScanner(dir string) Scanner

NewScanner creates a Scanner looking for the valid cache items.

type ScannerHitFn

type ScannerHitFn func(entry ScanEntry) error

ScannerHitFn is a function called on every scanner's hit. Function receives the ScanEntry, describing the found cache item. If the function returns an error, the iteration will be stopped.

type Values

type Values map[string]any

Values are a custom values map

func NewValues

func NewValues(keysAndValues ...any) Values

NewValues creates a new custom Values map. The keysAndValues parameters expect a list of keys and values in format: key1, value1, key2, value2...