README
¶
LazyFS
Provides transparent byte-level caching of files pulled from a remote source (currently either another filesystem or an HTTP server that supports Content-Range).
Tried to make a modular system where different Sources (e.g., HTTP, local file) and Stores (e.g., a local flat file, a database) can be paired up. The resulting Source implements the Go ReaderAt interface.
Currently has the these sources:
- Local file
- HTTP
And these stores:
- Local files (makes a copy of the remote file to a local path)
- Sparse file (creates an empty file of the same length as the source, fills in bytes as they are pulled from the source). I think this will be very space efficient on compressed filesystems. Not sure about the performance consequences. Right now this store isn't persistent (it doesn't store the map of which bytes have been cached between executions, that's on the TODO list).
In the long-run I think one or more database-backed Stores will be the preferred solution for active deployment.
My first Golang library, so excuse any non-idiomatic patterns or other tomfoolery.
The repo go-lazyfs-benchmarking contains a number of benchmarking tests which are computationally (or network bandwidth) more expensive (not appropriate as frequently-run testcases).
TODO
License
This library released is under the MIT License
Documentation
¶
Index ¶
- Constants
- func InitializeSparsefile(sparsefile string, sz int64) (*os.File, error)
- type BlockStore
- func (fs *BlockStore) FileSize() (int64, error)
- func (fs *BlockStore) HasAt(p []byte, off int64) (n int, err error)
- func (fs *BlockStore) Path() string
- func (fs *BlockStore) ReadAt(p []byte, off int64) (n int, err error)
- func (fs *BlockStore) Reader() io.Reader
- func (fs *BlockStore) WriteAt(p []byte, off int64) (n int, err error)
- type BlockStoreError
- type FileSource
- type HttpSource
- type LocalFileSource
- type LocalFileStore
- type LocalFileStoreError
- type SparseFileStore
- func (fs *SparseFileStore) FileSize() (int64, error)
- func (fs *SparseFileStore) HasAt(p []byte, off int64) (n int, err error)
- func (fs *SparseFileStore) Path() string
- func (fs *SparseFileStore) ReadAt(p []byte, off int64) (n int, err error)
- func (fs *SparseFileStore) Reader() io.Reader
- func (fs *SparseFileStore) WriteAt(p []byte, off int64) (n int, err error)
- type SparseFileStoreError
- type ZeroReader
Constants ¶
const Version = "v0.1.0"
Variables ¶
This section is empty.
Functions ¶
Types ¶
type BlockStore ¶
type BlockStore struct {
// contains filtered or unexported fields
}
=====================================
func OpenBlockStore ¶
func OpenBlockStore(source FileSource, bits uint) (*BlockStore, error)
func (*BlockStore) FileSize ¶
func (fs *BlockStore) FileSize() (int64, error)
func (*BlockStore) Path ¶
func (fs *BlockStore) Path() string
func (*BlockStore) ReadAt ¶
func (fs *BlockStore) ReadAt(p []byte, off int64) (n int, err error)
=====================================
func (*BlockStore) Reader ¶
func (fs *BlockStore) Reader() io.Reader
type BlockStoreError ¶
type BlockStoreError struct {
Err string
}
=====================================
func (BlockStoreError) Error ¶
func (e BlockStoreError) Error() string
type FileSource ¶
type FileSource interface {
io.ReaderAt
FileSize() (int64, error)
Reader() io.Reader
Path() string
}
func SourceFromPath ¶
func SourceFromPath(path string) (FileSource, error)
Source from path will automatically build an appropriate FileSource depending on whether path is an URL os a local path.
type HttpSource ¶
type HttpSource struct {
// contains filtered or unexported fields
}
func OpenHttpSource ¶
func OpenHttpSource(url url.URL) (hsrc *HttpSource, err error)
func (*HttpSource) FileSize ¶
func (fs *HttpSource) FileSize() (int64, error)
func (*HttpSource) Path ¶
func (fs *HttpSource) Path() string
func (*HttpSource) Reader ¶
func (fs *HttpSource) Reader() io.Reader
type LocalFileSource ¶
type LocalFileSource struct {
// contains filtered or unexported fields
}
func OpenLocalFile ¶
func OpenLocalFile(fpath string) (fsrc *LocalFileSource, err error)
OpenLocalFile takes a local path and produces a pointer to a LocalFileSource, which implements the FileSource interface.
func OpenLocalFileSource ¶
func OpenLocalFileSource(root string, ff string) (fsrc *LocalFileSource, err error)
OpenLocalFileSource takes a two-term path (a "root" and an relative path) and produces a pointer to a LocalFileSource, which implements the FileSource interface.
func (*LocalFileSource) FileSize ¶
func (fs *LocalFileSource) FileSize() (int64, error)
FileSize returns the size of file underlying the LocalFileSource
func (*LocalFileSource) Path ¶
func (fs *LocalFileSource) Path() string
Returns the path to the LocalFileSource
func (*LocalFileSource) ReadAt ¶
func (fs *LocalFileSource) ReadAt(p []byte, off int64) (n int, err error)
ReadAt implements the ReaderAt interface for LocalFileSource. Attempts to read len(p) bytes from offset off within the LocalFile. Returns the number of bytes read. ( it's really a wrapper around fs.file.ReadAt() )
func (*LocalFileSource) Reader ¶
func (fs *LocalFileSource) Reader() io.Reader
Generate a Reader for the file pointed to by the LocalFileSource
type LocalFileStore ¶
type LocalFileStore struct {
// contains filtered or unexported fields
}
func OpenLocalFileStore ¶
func OpenLocalFileStore(source FileSource, root string) (*LocalFileStore, error)
OpenLocalFileStore makes a LocalFileStore which wraps around a FileSource
root specifies the local location for local file store cache.
Returns a pointer to the LocalFileStore
func (*LocalFileStore) FileSize ¶
func (fs *LocalFileStore) FileSize() (int64, error)
Returns the size of the file underlying the LocalFileStore
func (*LocalFileStore) HasAt ¶
func (fs *LocalFileStore) HasAt(p []byte, off int64) (n int, err error)
LocalFileStore implements HasAt by first lazy-loading the file using load(). Once the file is available locally, the question is not whether the cache has the bytes, but whether the bytes exist in the file at all ... or are e.g., off the end of the file. Returns the number of bytes available, which may be <= len(p) if successful. Or 0 if the bytes are not available.
func (*LocalFileStore) Path ¶
func (fs *LocalFileStore) Path() string
Returns the path to the LocalFileStore
func (*LocalFileStore) ReadAt ¶
func (fs *LocalFileStore) ReadAt(p []byte, off int64) (n int, err error)
ReadAt is first lazy-loads the wrapped FileSource to the local disk using load(), then reads from the local disk
func (*LocalFileStore) Reader ¶
func (fs *LocalFileStore) Reader() io.Reader
Returns a Reader to the LocalFileStore
type LocalFileStoreError ¶
type LocalFileStoreError struct {
Err string
}
func (LocalFileStoreError) Error ¶
func (e LocalFileStoreError) Error() string
type SparseFileStore ¶
type SparseFileStore struct {
// contains filtered or unexported fields
}
=====================================
func OpenSparseFileStore ¶
func OpenSparseFileStore(source FileSource, root string) (*SparseFileStore, error)
func (*SparseFileStore) FileSize ¶
func (fs *SparseFileStore) FileSize() (int64, error)
func (*SparseFileStore) HasAt ¶
func (fs *SparseFileStore) HasAt(p []byte, off int64) (n int, err error)
func (*SparseFileStore) Path ¶
func (fs *SparseFileStore) Path() string
func (*SparseFileStore) ReadAt ¶
func (fs *SparseFileStore) ReadAt(p []byte, off int64) (n int, err error)
=====================================
func (*SparseFileStore) Reader ¶
func (fs *SparseFileStore) Reader() io.Reader
type SparseFileStoreError ¶
type SparseFileStoreError struct {
Err string
}
=====================================
func (SparseFileStoreError) Error ¶
func (e SparseFileStoreError) Error() string
type ZeroReader ¶
type ZeroReader struct {
// contains filtered or unexported fields
}
Source Files