cache

README

File cache for multiple processes

• based on file locks
• will release lock after object creation

warning: result object is protected by time only

• assumes created object will be used before max-age

config: max age of item, typically 10 days => refresh after maxAge/10 => delete scan every maxAge/10 or upon request

objects that are older than max age will be deleted

file layout

$cacheDir/gorun/config.json
$cacheDir/gorun/config.lock
$cacheDir/gorun/trim.txt
$cacheDir/gorun/trim.lock
$cacheDir/gorun/README

$cacheDir/gorun/xx-t/lockfile
$cacheDir/gorun/xx-t/xxyyy/lockfile
$cacheDir/gorun/xx-t/xxyyy/info
$cacheDir/gorun/xx-t/xxyyy/     = folder owned by lockfile
$cacheDir/gorun/xx-t/xxyyy/zzzz = object creation folder, always new and uniq

xx/yy/zz regexp [0-9a-f]

requirements

• if two or more P race to the same key and one process has started to create entry but fails before completion, another P will take over the task
• protect against user error: if cache-dir is set to root '/', delete operation should delete zero or very few files
• if user creates symlinks in cache dir, delete should only delete symlinks
• out-of-disk space should not corrupt the cache, only fail it => need validation of entry data, e.g. guard against truncation
• graceful failure: if locks are no-op, cache should still work mostly ok

Documentation

Index

• func Lockedfile(lockfile string, lockType LockType, f func() error) error
• func Lookup(input string, create func(outDir string) error) (string, error)
• func UpdateMultiprocess(lockfile string, lockType LockType, datafile string, ...) error
• type Config
  • func DefaultConfig() (*Config, error)
  • func NewConfig(dir string, maxAge time.Duration) (*Config, error)
  • func (config *Config) Dir() string
  • func (config *Config) GetInfo() (Stat, error)
  • func (config *Config) GetPartInfo(stat *Stat, part int)
  • func (config *Config) Lookup(input string, create func(outDir string) error) (string, error)
  • func (config *Config) Lookup2(input string, userCreate func(outDir string) error, useCache bool) (string, error)
  • func (config *Config) TrimNow() error
  • func (config *Config) TrimPeriodically() error
• type Item
• type LockType
• type Lockpair
  • func NewLockPair(dir, lockfile, datafile string) Lockpair
• type Stat

Functions

func Lockedfile

func Lockedfile(lockfile string, lockType LockType, f func() error) error

func Lookup

func Lookup(input string, create func(outDir string) error) (string, error)

input should be a string that represents a complete description of a repeatable computation (command line equivalent, environment variables, input file contents, executable contents). returns outdir

func UpdateMultiprocess

func UpdateMultiprocess(lockfile string, lockType LockType, datafile string, updateContent func(old string, writeString func(new string) error) error) error

Types

type Config

type Config struct {
	// contains filtered or unexported fields
}

func DefaultConfig

func DefaultConfig() (*Config, error)

func NewConfig

func NewConfig(dir string, maxAge time.Duration) (*Config, error)

func (*Config) Dir

func (config *Config) Dir() string

func (*Config) GetInfo

func (config *Config) GetInfo() (Stat, error)

func (*Config) GetPartInfo

func (config *Config) GetPartInfo(stat *Stat, part int)

func (*Config) Lookup

func (config *Config) Lookup(input string, create func(outDir string) error) (string, error)

func (*Config) Lookup2

func (config *Config) Lookup2(input string, userCreate func(outDir string) error, useCache bool) (string, error)

func (*Config) TrimNow

func (config *Config) TrimNow() error

func (*Config) TrimPeriodically

func (config *Config) TrimPeriodically() error

type Item

type Item struct {
	// contains filtered or unexported fields
}

type LockType

type LockType int

const (
	SHARED_LOCK    LockType = 64
	EXCLUSIVE_LOCK LockType = 128
)

type Lockpair

type Lockpair struct {
	// contains filtered or unexported fields
}

func NewLockPair

func NewLockPair(dir, lockfile, datafile string) Lockpair

type Stat

type Stat struct {
	Count     int
	SizeBytes int64
	Dir       string
}