datablock

README

Datablock

Module for reading and caching files into a fixed size cache buffer.

Can also cache calls to os.Stat, for cases of extreme optimization.

Coding style

• The code shall always be formatted with go fmt.

Online API Documentation

godoc.org

General information

• Version: 1.2.0
• License: BSD-3
• Alexander F Rødseth <xyproto@archlinux.org>

Documentation

Index

• Variables
• type DataBlock
  • func NewDataBlock(data []byte, compressionSpeed bool) *DataBlock
  • func (b *DataBlock) Bytes() []byte
  • func (b *DataBlock) Compress() error
  • func (b *DataBlock) Decompress() error
  • func (b *DataBlock) Gzipped() ([]byte, int, error)
  • func (b *DataBlock) HasData() bool
  • func (b *DataBlock) IsCompressed() bool
  • func (b *DataBlock) Length() int
  • func (b *DataBlock) MustData() []byte
  • func (b *DataBlock) String() string
  • func (b *DataBlock) StringLength() string
  • func (b *DataBlock) ToClient(w http.ResponseWriter, req *http.Request, name string, canGzip bool, ...)
  • func (b *DataBlock) UncompressedData() ([]byte, int, error)
• type FileCache
  • func NewFileCache(cacheSize uint64, compress bool, maxEntitySize uint64, compressionSpeed bool, ...) *FileCache
  • func (cache *FileCache) Clear()
  • func (cache *FileCache) IsEmpty() bool
  • func (cache *FileCache) Read(filename string, cached bool) (*DataBlock, error)
  • func (cache *FileCache) Stats() string
• type FileStat
  • func NewFileStat(useCache bool, clearStatCacheDelay time.Duration) *FileStat
  • func (fs *FileStat) Exists(path string) bool
  • func (fs *FileStat) IsDir(path string) bool
  • func (fs *FileStat) Sleep(extraSleep time.Duration)

Variables

var (
	// ErrRemoval is used if a filename that does not exist is attempted to be removed
	ErrRemoval = errors.New("can't remove a file ID that does not exist")

	// ErrNoData is used if no data is attempted to be stored in the cache
	ErrNoData = errors.New("no data")

	// ErrAlreadyStored is used if a given filename has already been stored in the cache
	ErrAlreadyStored = errors.New("file ID is already stored")

	// ErrLargerThanCache is used if the given data is larger than the total cache size
	ErrLargerThanCache = errors.New("data is larger than the the total cache size")

	// ErrEntityTooLarge is used if a maximum size per entity has been set
	ErrEntityTooLarge = errors.New("data is larger than the allowed size")

	// ErrGivenDataSizeTooLarge is returned if the uncompressed size of the given data is too large
	ErrGivenDataSizeTooLarge = errors.New("size of given data is larger than allowed")
)

var (
	// EmptyDataBlock is an empty data block
	EmptyDataBlock = &DataBlock{[]byte{}, false, 0, true}
)

Types

type DataBlock

type DataBlock struct {
	// contains filtered or unexported fields
}

DataBlock represents a block of data that may be compressed

func NewDataBlock

func NewDataBlock(data []byte, compressionSpeed bool) *DataBlock

NewDataBlock creates a new uncompressed data block. compressionSpeed is if speedy compression should be used over compact compression

func (*DataBlock) Bytes

func (b *DataBlock) Bytes() []byte

Bytes returns the uncompressed data or an empty byte slice

func (*DataBlock) Compress

func (b *DataBlock) Compress() error

Compress this data block

func (*DataBlock) Decompress

func (b *DataBlock) Decompress() error

Decompress this data block

func (*DataBlock) Gzipped

func (b *DataBlock) Gzipped() ([]byte, int, error)

Gzipped returns the compressed data, length and an error. Will compress if needed.

func (*DataBlock) HasData

func (b *DataBlock) HasData() bool

HasData returns true if there is data present

func (*DataBlock) IsCompressed

func (b *DataBlock) IsCompressed() bool

IsCompressed checks if this data block is compressed

func (*DataBlock) Length

func (b *DataBlock) Length() int

Length returns the lentgth of the current data (not the length of the original data, but in the current state)

func (*DataBlock) MustData

func (b *DataBlock) MustData() []byte

MustData returns the uncompressed data or an empty byte slice

func (*DataBlock) String

func (b *DataBlock) String() string

String returns the uncompressed data as a string or as an empty string. Same as MustData, but converted to a string.

func (*DataBlock) StringLength

func (b *DataBlock) StringLength() string

StringLength returns the length of the data, represented as a string

func (*DataBlock) ToClient

func (b *DataBlock) ToClient(w http.ResponseWriter, req *http.Request, name string, canGzip bool, gzipThreshold int)

ToClient writes the data to the client. Also sets the right headers and compresses the data with gzip if needed. Set canGzip to true if the http client can handle gzipped data. gzipThreshold is the threshold (in bytes) for when it makes sense to compress the data with gzip

func (*DataBlock) UncompressedData

func (b *DataBlock) UncompressedData() ([]byte, int, error)

UncompressedData returns the the original, uncompressed data, the length of the data and an error. Will decompress if needed.

type FileCache

type FileCache struct {
	// contains filtered or unexported fields
}

FileCache manages a set of bytes as a cache

func NewFileCache

func NewFileCache(cacheSize uint64, compress bool, maxEntitySize uint64, compressionSpeed bool, maxGivenDataSize uint64) *FileCache

NewFileCache creates a new FileCache struct. cacheSize is the total cache size, in bytes. compress is for enabling compression of cache data. maxEntitySize is for setting a per-file maximum size. (0 to disable) compressionSpeed is if speedy compression should be used over compact compression. maxGivenDataSize is the maximum amount of bytes that can be given at once. (0 to disable, 1 MiB is recommended)

func (*FileCache) Clear

func (cache *FileCache) Clear()

Clear the entire cache

func (*FileCache) IsEmpty

func (cache *FileCache) IsEmpty() bool

IsEmpty checks if the cache is empty

func (*FileCache) Read

func (cache *FileCache) Read(filename string, cached bool) (*DataBlock, error)

For reading files, with optional caching.

func (*FileCache) Stats

func (cache *FileCache) Stats() string

Stats returns formatted cache statistics

type FileStat

type FileStat struct {
	// contains filtered or unexported fields
}

FileStat can cache calls to os.Stat. This requires that the user wants to assume that no files are removed from the server directory while the server is running, to gain some additional speed (and a tiny bit of memory use for the cache).

func NewFileStat

func NewFileStat(useCache bool, clearStatCacheDelay time.Duration) *FileStat

NewFileStat creates a new FileStat struct, with optional caching. Only use the caching if it is not critical that os.Stat is always correct.

func (*FileStat) Exists

func (fs *FileStat) Exists(path string) bool

Exists check if a given path exists

func (*FileStat) IsDir

func (fs *FileStat) IsDir(path string) bool

IsDir checks if a given path is a directory

func (*FileStat) Sleep

func (fs *FileStat) Sleep(extraSleep time.Duration)

Sleep for an entire stat cache clear cycle + optional extra sleep time