vfscache

Documentation

Overview

Package vfscache deals with caching of files locally for the VFS layer

Index

• type AddVirtualFn
• type Cache
  • func New(ctx context.Context, fremote fs.Fs, opt *vfscommon.Options, avFn AddVirtualFn) (*Cache, error)
  • func (c *Cache) AddVirtual(remote string, size int64, isDir bool) error
  • func (c *Cache) CleanUp() error
  • func (c *Cache) DirExists(name string) bool
  • func (c *Cache) DirRename(oldDirName string, newDirName string) (err error)
  • func (c *Cache) DirtyItem(name string) (item *Item)
  • func (c *Cache) Dump() string
  • func (c *Cache) Exists(name string) bool
  • func (c *Cache) InUse(name string) bool
  • func (c *Cache) Item(name string) (item *Item)
  • func (c *Cache) KickCleaner()
  • func (c *Cache) Remove(name string) (wasWriting bool)
  • func (c *Cache) Rename(name string, newName string, newObj fs.Object) (err error)
  • func (c *Cache) SetModTime(name string, modTime time.Time)
  • func (c *Cache) TotalInUse() (n int)
• type Info
• type Item
  • func (item *Item) Close(storeFn StoreFn) (err error)
  • func (item *Item) Dirty()
  • func (item *Item) Exists() bool
  • func (item *Item) FindMissing(r ranges.Range) (outr ranges.Range)
  • func (item *Item) GetModTime() (modTime time.Time, err error)
  • func (item *Item) GetName() (name string)
  • func (item *Item) GetSize() (size int64, err error)
  • func (item *Item) HasRange(r ranges.Range) bool
  • func (item *Item) IsDataDirty() bool
  • func (item *Item) IsDirty() bool
  • func (item *Item) Open(o fs.Object) (err error)
  • func (item *Item) ReadAt(b []byte, off int64) (n int, err error)
  • func (item *Item) RemoveNotInUse(maxAge time.Duration, emptyOnly bool) (removed bool, spaceFreed int64)
  • func (item *Item) Reset() (rr ResetResult, spaceFreed int64, err error)
  • func (item *Item) Sync() (err error)
  • func (item *Item) Truncate(size int64) (err error)
  • func (item *Item) WriteAt(b []byte, off int64) (n int, err error)
  • func (item *Item) WriteAtNoOverwrite(b []byte, off int64) (n int, skipped int, err error)
  • func (item *Item) WrittenBack() bool
• type Items
  • func (v Items) Len() int
  • func (v Items) Less(i, j int) bool
  • func (v Items) Swap(i, j int)
• type ResetResult
  • func (rr ResetResult) String() string
• type StoreFn

Types

type AddVirtualFn

type AddVirtualFn func(remote string, size int64, isDir bool) error

AddVirtualFn if registered by the WithAddVirtual method, can be called to register the object or directory at remote as a virtual entry in directory listings.

This is used when reloading the Cache and uploading items need to go into the directory tree.

type Cache

type Cache struct {
	// contains filtered or unexported fields
}

Cache opened files

func New

func New(ctx context.Context, fremote fs.Fs, opt *vfscommon.Options, avFn AddVirtualFn) (*Cache, error)

New creates a new cache hierarchy for fremote

This starts background goroutines which can be cancelled with the context passed in.

func (*Cache) AddVirtual

func (c *Cache) AddVirtual(remote string, size int64, isDir bool) error

AddVirtual adds a virtual directory entry by calling the addVirtual callback if one has been registered.

func (*Cache) CleanUp

func (c *Cache) CleanUp() error

CleanUp empties the cache of everything

func (*Cache) DirExists

func (c *Cache) DirExists(name string) bool

DirExists checks to see if the directory exists in the cache or not.

func (*Cache) DirRename

func (c *Cache) DirRename(oldDirName string, newDirName string) (err error)

DirRename the dir in cache

func (*Cache) DirtyItem

func (c *Cache) DirtyItem(name string) (item *Item)

DirtyItem returns the Item if it exists in the cache **and** is dirty otherwise it returns nil.

name should be a remote path not an osPath

func (*Cache) Dump

func (c *Cache) Dump() string

Dump the cache into a string for debugging purposes

func (*Cache) Exists

func (c *Cache) Exists(name string) bool

Exists checks to see if the file exists in the cache or not.

This is done by bringing the item into the cache which will validate the backing file and metadata and then asking if the Item exists or not.

func (*Cache) InUse

func (c *Cache) InUse(name string) bool

InUse returns whether the name is in use in the cache

name should be a remote path not an osPath

func (*Cache) Item

func (c *Cache) Item(name string) (item *Item)

Item gets a cache item for name

To use it item.Open will need to be called

name should be a remote path not an osPath

func (*Cache) KickCleaner

func (c *Cache) KickCleaner()

KickCleaner kicks cache cleaner upon out of space situation

func (*Cache) Remove

func (c *Cache) Remove(name string) (wasWriting bool)

Remove should be called if name is deleted

This returns true if the file was in the transfer queue so may not have completely uploaded yet.

func (*Cache) Rename

func (c *Cache) Rename(name string, newName string, newObj fs.Object) (err error)

Rename the item in cache

func (*Cache) SetModTime

func (c *Cache) SetModTime(name string, modTime time.Time)

SetModTime should be called to set the modification time of the cache file

func (*Cache) TotalInUse

func (c *Cache) TotalInUse() (n int)

TotalInUse returns the number of items in the cache which are InUse

type Info

type Info struct {
	ModTime     time.Time     // last time file was modified
	ATime       time.Time     // last time file was accessed
	Size        int64         // size of the file
	Rs          ranges.Ranges // which parts of the file are present
	Fingerprint string        // fingerprint of remote object
	Dirty       bool          // set if the backing file has been modified
}

Info is persisted to backing store

type Item

type Item struct {
	// contains filtered or unexported fields
}

Item is stored in the item map

The Info field is written to the backing store to store status

func (*Item) Close

func (item *Item) Close(storeFn StoreFn) (err error)

Close the cache file

func (*Item) Dirty

func (item *Item) Dirty()

Dirty marks the item as changed and needing writeback

func (*Item) Exists

func (item *Item) Exists() bool

Exists returns whether the backing file for the item exists or not

func (*Item) FindMissing

func (item *Item) FindMissing(r ranges.Range) (outr ranges.Range)

FindMissing adjusts r returning a new ranges.Range which only contains the range which needs to be downloaded. This could be empty - check with IsEmpty. It also adjust this to make sure it is not larger than the file.

func (*Item) GetModTime

func (item *Item) GetModTime() (modTime time.Time, err error)

GetModTime of the cache file

func (*Item) GetName

func (item *Item) GetName() (name string)

GetName gets the vfs name of the item

func (*Item) GetSize

func (item *Item) GetSize() (size int64, err error)

GetSize gets the current size of the item

func (*Item) HasRange

func (item *Item) HasRange(r ranges.Range) bool

HasRange returns true if the current ranges entirely include range

func (*Item) IsDataDirty

func (item *Item) IsDataDirty() bool

IsDataDirty returns true if the item's data is dirty

func (*Item) IsDirty

func (item *Item) IsDirty() bool

IsDirty returns true if the item is dirty

func (*Item) Open

func (item *Item) Open(o fs.Object) (err error)

Open the local file from the object passed in. Wraps open() to provide recovery from out of space error.

func (*Item) ReadAt

func (item *Item) ReadAt(b []byte, off int64) (n int, err error)

ReadAt bytes from the file at off

func (*Item) RemoveNotInUse

func (item *Item) RemoveNotInUse(maxAge time.Duration, emptyOnly bool) (removed bool, spaceFreed int64)

RemoveNotInUse is called to remove cache file that has not been accessed recently It may also be called for removing empty cache files too when the quota is already reached.

func (*Item) Reset

func (item *Item) Reset() (rr ResetResult, spaceFreed int64, err error)

Reset is called by the cache purge functions only to reset (empty the contents) cache files that are not dirty. It is used when cache space runs out and we see some ENOSPC error.

func (*Item) Sync

func (item *Item) Sync() (err error)

Sync commits the current contents of the file to stable storage. Typically, this means flushing the file system's in-memory copy of recently written data to disk.

func (*Item) Truncate

func (item *Item) Truncate(size int64) (err error)

Truncate the item to the given size, creating it if necessary

If the new size is shorter than the existing size then the object will be shortened and marked as dirty.

If the new size is longer than the old size then the object will be extended and the extended data will be filled with zeros. The object will be marked as dirty in this case also.

func (*Item) WriteAt

func (item *Item) WriteAt(b []byte, off int64) (n int, err error)

WriteAt bytes to the file at off

func (*Item) WriteAtNoOverwrite

func (item *Item) WriteAtNoOverwrite(b []byte, off int64) (n int, skipped int, err error)

WriteAtNoOverwrite writes b to the file, but will not overwrite already present ranges.

This is used by the downloader to write bytes to the file

It returns n the total bytes processed and skipped the number of bytes which were processed but not actually written to the file.

func (*Item) WrittenBack

func (item *Item) WrittenBack() bool

WrittenBack checks to see if the item has been written back or not

type Items

type Items []*Item

Items are a slice of *Item ordered by ATime

func (Items) Len

func (v Items) Len() int

func (Items) Less

func (v Items) Less(i, j int) bool

func (Items) Swap

func (v Items) Swap(i, j int)

type ResetResult

type ResetResult int

ResetResult reports the actual action taken in the Reset function and reason

const (
	SkippedDirty         ResetResult = iota // Dirty item cannot be reset
	SkippedPendingAccess                    // Reset pending access can lead to deadlock
	SkippedEmpty                            // Reset empty item does not save space
	RemovedNotInUse                         // Item not used. Remove instead of reset
	ResetFailed                             // Reset failed with an error
	ResetComplete                           // Reset completed successfully
)

Constants used to report actual action taken in the Reset function and reason

func (ResetResult) String

func (rr ResetResult) String() string

type StoreFn

type StoreFn func(fs.Object)

StoreFn is called back with an object after it has been uploaded