filecache

README

go-filecache

Overview

Package filecache provides a LRU file caching mechanism to cache resources to the local disk that take a long time to generate or download from the network.

The creation process runs only once even if multiple go-routines concurrently request for a key that does not exist in the cache.

Documentation

• Read the documentation.

License

go-filecache is available under the MIT license. See the LICENSE file for more information.

Documentation

Overview

Package filecache provides a LRU file caching mechanism to cache resources to the local disk that take a long time to generate or download from the network.

Index

• Constants
• Variables
• type ByteSliceKey
  • func (k ByteSliceKey) Hash() Hash
  • func (k ByteSliceKey) String() string
• type Cache
  • func New[K Key](dir string, create CreateFunc[K]) (*Cache[K], error)
  • func NewWithConfig[K Key](conf *Config[K]) (*Cache[K], error)
  • func (c *Cache[K]) Get(key K) (*File[K], bool, error)
  • func (c *Cache[K]) Serve(ctx context.Context) error
  • func (c *Cache[_]) Status() *Status
• type Config
• type CreateFunc
• type File
  • func (f *File[_]) Close() error
  • func (f *File[_]) Name() string
  • func (f *File[_]) OSFile() *os.File
  • func (f *File[_]) Read(b []byte) (int, error)
  • func (f *File[_]) ReadAt(b []byte, off int64) (int, error)
  • func (f *File[_]) Seek(offset int64, whence int) (int64, error)
  • func (f *File[K]) Stat() (os.FileInfo, error)
• type FileInfo
  • func (*FileInfo[_]) IsDir() bool
  • func (i *FileInfo[_]) ModTime() time.Time
  • func (i *FileInfo[_]) Mode() fs.FileMode
  • func (i *FileInfo[K]) Name() string
  • func (i *FileInfo[_]) Size() int64
  • func (i *FileInfo[_]) Sys() any
• type Hash
• type Key
• type Logger
• type Status
  • func (s Status) String() string
• type StringKey
  • func (k StringKey) Hash() Hash
  • func (k StringKey) String() string
• type Uint32Key
  • func (k Uint32Key) Hash() (b Hash)
  • func (k Uint32Key) String() string
• type Uint64Key
  • func (k Uint64Key) Hash() (b Hash)
  • func (k Uint64Key) String() string

Constants

const HashSize = 32

HashSize is the size, in bytes, of the key hash.

Variables

var ErrInternal = errors.New("internal error")

ErrInternal is the error thrown when an internal error occurred.

var ErrInvalidConfig = errors.New("invalid config")

ErrInvalidConfig is the error thrown when the passed configuration parameter is not valid.

Types

type ByteSliceKey

type ByteSliceKey []byte

ByteSliceKey is a wrapper type to attach the Hash method to []byte.

func (ByteSliceKey) Hash

func (k ByteSliceKey) Hash() Hash

Hash calculates and returns a hash value of the []byte using SHA-512/256.

func (ByteSliceKey) String

func (k ByteSliceKey) String() string

String returns the string representation of the []byte key.

type Cache

type Cache[K Key] struct {
	// contains filtered or unexported fields
}

Cache represents the file cache directory.

func New

func New[K Key](dir string, create CreateFunc[K]) (*Cache[K], error)

New create a cache with the default configuration.

func NewWithConfig

func NewWithConfig[K Key](conf *Config[K]) (*Cache[K], error)

NewWithConfig create a cache using the given configuration parameters.

func (*Cache[K]) Get

func (c *Cache[K]) Get(key K) (*File[K], bool, error)

Get gets the file for the key from the cache. If the file for the specified key does not exist in the cache, it will call the CreateFunc to create the new file. It returns the file opened for read, cached or not.

The returned file is guaranteed to remain referenced until it is closed and not removed from the cache during that time. After using the file, it is the caller's responsibility to call the File.Close() method of the returned file. Otherwise the file will remain in the cache, and the reference will remain in the memory.

func (*Cache[K]) Serve

func (c *Cache[K]) Serve(ctx context.Context) error

Serve serves the Cache instance. It performs find and delete old cache files.

func (*Cache[_]) Status

func (c *Cache[_]) Status() *Status

Status returns the current cache status and statistics.

type Config

type Config[K Key] struct {
	// The path to the directory for cache files. It should be a dedicated
	// directory used exclusively for this cache. The directory will be
	// automatically created if it does not exist. Both absolute and
	// relative paths are allowed. A relative path is treated as relative
	// from the user-specific cache directory returned by os.UserCacheDir().
	// If it is empty, use the program name directory.
	Dir string

	// The callback function that is called when a not-cached resource is
	// requested.
	Create CreateFunc[K]

	// The upper limit on the number of files that can be cached. Zero
	// value means unlimited. When more than this number of files are
	// cached, the oldest files will be removed. Note that more than this
	// number of files may be cached temporarily.
	MaxFiles uint64

	// The limit on the total size of files that can be cached. Zero
	// value means unlimited. When more than this total size of files are
	// cached, the oldest files will be removed. Note that more than this
	// size of files may be cached temporarily. There is no guarantee that
	// more disk space than this will not be used.
	MaxSize infounit.ByteCount

	// The maximum age of cache files. Note that it is the time since
	// last access, not the time since creation. Also the cache is not
	// removed immediately after this age. It is still possible that an
	// aged cache file will continue to be hit and reused. Zero value
	// means unlimited.
	MaxAge time.Duration

	// The interval between GC processing to find and remove old cache
	// files that exceed the configured limits.
	GCInterval time.Duration

	// If not nil, Cache outputs log messages to this Logger object.
	Logger Logger

	// If true, Cache outputs debug log messages. Only effective if
	// Logger is not nil.
	DebugLog bool
}

Config represents the parameters to configure Cache creation.

type CreateFunc

type CreateFunc[K Key] func(K, *os.File) error

CreateFunc represents a function for file creation. It will be called when Get is called for a new key that does not exist in the cache. It should create the file using the pre-opened file argument passed. It does not have to close the file, while it will be closed automatically after return.

type File

type File[K Key] struct {
	// contains filtered or unexported fields
}

File represents the cached file returned by Get() method.

func (*File[_]) Close

func (f *File[_]) Close() error

Close implements io.Closer interface.

func (*File[_]) Name

func (f *File[_]) Name() string

Name returns the string representation of the associated key.

func (*File[_]) OSFile

func (f *File[_]) OSFile() *os.File

OSFile returns the underlying os.File object. Use with caution, as all writes will fail and operations such as deleting, renaming or closing the file will cause unexpected results. Only use if it is required to pass the file to a package that accepts only os.File or the file descriptor. Also use File.Close instead of os.File.Close even if this method is called.

func (*File[_]) Read

func (f *File[_]) Read(b []byte) (int, error)

Read implements io.Reader interface.

func (*File[_]) ReadAt

func (f *File[_]) ReadAt(b []byte, off int64) (int, error)

ReadAt implements io.ReaderAt interface.

func (*File[_]) Seek

func (f *File[_]) Seek(offset int64, whence int) (int64, error)

Seek implements io.Seeker interface.

func (*File[K]) Stat

func (f *File[K]) Stat() (os.FileInfo, error)

Stat returns the file information.

type FileInfo

type FileInfo[K Key] struct {
	// contains filtered or unexported fields
}

FileInfo implements fs.FileInfo interface.

func (*FileInfo[_]) IsDir

func (*FileInfo[_]) IsDir() bool

IsDir always returns false, since a directory can not be cached.

func (*FileInfo[_]) ModTime

func (i *FileInfo[_]) ModTime() time.Time

ModTime returns the last access time or created time of the cache entry.

func (*FileInfo[_]) Mode

func (i *FileInfo[_]) Mode() fs.FileMode

Mode always returns 0400.

func (*FileInfo[K]) Name

func (i *FileInfo[K]) Name() string

Name returns the string representation of the key. Note that it is not the underlying file path.

func (*FileInfo[_]) Size

func (i *FileInfo[_]) Size() int64

Size returns the file size in byte.

func (*FileInfo[_]) Sys

func (i *FileInfo[_]) Sys() any

Sys returns the associated key.

type Hash

type Hash [HashSize]byte

Hash represents a 32-bytes hash value.

type Key

type Key interface {
	fmt.Stringer

	Hash() Hash
}

Key is the interface implemented by the key type to identify a file cache entry.

type Logger

type Logger interface {
	FileCacheLog(string)
}

Logger is the interface implemented to receive log messages from the running Cache instance.

type Status

type Status struct {
	NumFiles     uint64             // number of files currently in cache.
	TotalSize    infounit.ByteCount // total size of files currently in cache.
	NumRequested uint64             // total number of files requested.
	NumHit       uint64             // total number of cache hits.
	NumCreated   uint64             // total number of newly created cache files.
	NumFailed    uint64             // total number of operation failures.
	NumRemoved   uint64             // total number of removed cache files.
	NumOps       int                // number of operations currently being processed.
	NumRefs      int                // number of currently referenced cache files.
}

Status represents the cache status and statistics.

func (Status) String

func (s Status) String() string

String returns the string representation of Status.

type StringKey

type StringKey string

StringKey is a wrapper type to attach the Hash method to string.

func (StringKey) Hash

func (k StringKey) Hash() Hash

Hash calculates and returns a hash value of the string using SHA-512/256.

func (StringKey) String

func (k StringKey) String() string

String returns the string value itself of StringKey.

type Uint32Key

type Uint32Key uint32

Uint32Key is a wrapper type to attach the Hash method to uint32.

func (Uint32Key) Hash

func (k Uint32Key) Hash() (b Hash)

Hash returns a byte sequence in which the LSBs represent the uint32 value itself as a hash value.

func (Uint32Key) String

func (k Uint32Key) String() string

String returns the string representation of the uint32 value.

type Uint64Key

type Uint64Key uint64

Uint64Key is a wrapper type to attach the Hash method to uint64.

func (Uint64Key) Hash

func (k Uint64Key) Hash() (b Hash)

Hash returns a byte sequence in which the LSBs represent the uint64 value itself as a hash value.

func (Uint64Key) String

func (k Uint64Key) String() string

String returns the string representation of the uint64 value.