modelcache

Documentation

Overview

Package modelcache provides an LRU model file cache for pre-caching GGUF models on Kubernetes nodes via a DaemonSet.

Experimental — this package is not yet wired into the main framework.

Package modelcache provides an LRU model file cache for pre-caching GGUF models on Kubernetes nodes via a DaemonSet.

Index

• type Cache
  • func NewCache(dir string, maxSize int64) *Cache
  • func (c *Cache) Evict()
  • func (c *Cache) Get(ref string) (string, bool)
  • func (c *Cache) Prefetch(refs []string, pullFn func(ref string, dest string) error) error
  • func (c *Cache) Put(ref string, srcPath string) error

Types

type Cache

type Cache struct {
	// contains filtered or unexported fields
}

Cache manages local model files with LRU eviction.

func NewCache

func NewCache(dir string, maxSize int64) *Cache

NewCache creates a cache that stores files under dir with a maximum aggregate size of maxSize bytes.

func (*Cache) Evict

func (c *Cache) Evict()

Evict removes least-recently-used entries until the total cache size is at or below maxSize.

func (*Cache) Get

func (c *Cache) Get(ref string) (string, bool)

Get returns the local file path for the given model ref if it is cached.

func (*Cache) Prefetch

func (c *Cache) Prefetch(refs []string, pullFn func(ref string, dest string) error) error

Prefetch downloads models for the given refs using pullFn, which should fetch the model identified by ref and write it to dest. Already-cached refs are skipped.

func (*Cache) Put

func (c *Cache) Put(ref string, srcPath string) error

Put copies the file at srcPath into the cache under the given ref.