vfs

README

Filesystem Abstraction in Go

Yet another one, created because none of the bazillon existing Golang VFS matched the needs we have.

Specifically, many tend to be either oriented toward local filesystems or cloud storage providers. Typically, both are very different. Local filesystems allow files to be modified, while cloud providers typically require a whole file to be re-uploaded for any change. As such, cloud-oriented libraries may support local filesystem but have no API for partial writes locally.

Here, cloud storage solutions such as AWS S3 are considered "keyvals", similar to databases where changing a byte in a value requires rewriting the whole value. The goal is to be able to offer converter interfaces that expose such backends as proper filesystems supporting partial writes.

Focus

This implementation focuses on the following goals:

• Stay as close as possible to filesystem concepts
• Be as compatible as possible with Golang's interfaces
• Be as simple as possible to extend
• Allow working with limited key/value backends

Features

• Filesystem Backends:
  • localfs filesystem
  • memfs
  • zipfs (read only)
• Keyval Backends:
  • memkv
  • boltkv using boltdb
• Converters:
  • vdirfs: provides directory indexation/listing for backends which do not have this feature (such as zipfs)

Planned

• Support for a wide range of backends (AWS S3, etc)
• Support for frontends (fuse, http, etc)
• Middlewares (keyval→filesystem adapters, encryption, etc)

Documentation

Index

• Variables
• func Fingerprint(f fs.FileInfo) string
• func MkdirAll(fsbase FileSystem, path string, perm fs.FileMode) error
• func NewStat(name string, size int64, mode fs.FileMode, modTime time.Time, sys interface{}) fs.FileInfo
• func ReadDir(fsbase FileSystem, path string) ([]fs.DirEntry, error)
• func Walk(fsbase FileSystem, path string, walkFn fs.WalkDirFunc) error
• type File
• type FileSystem
• type Fingerprintable
• type KVEntry
• type Keyval

Variables

var (
	ErrNotDirectory = errors.New("not a directory")
	ErrIsDirectory  = errors.New("is a directory")
	ErrNotEmpty     = errors.New("not empty")
)

Functions

func Fingerprint

func Fingerprint(f fs.FileInfo) string

Fingerprint will return a fingerprint for a given file, silently falling back on using size+timestamp in case of backend failure.

func MkdirAll

func MkdirAll(fsbase FileSystem, path string, perm fs.FileMode) error

func NewStat

func NewStat(name string, size int64, mode fs.FileMode, modTime time.Time, sys interface{}) fs.FileInfo

func ReadDir

func ReadDir(fsbase FileSystem, path string) ([]fs.DirEntry, error)

func Walk

func Walk(fsbase FileSystem, path string, walkFn fs.WalkDirFunc) error

Walk is the equivalent of filepath.Walk

Types

type File

type File interface {
	io.Closer
	io.Reader
	io.ReaderAt
	io.Writer
	io.WriterAt
	io.Seeker
	fs.ReadDirFile
	Stat() (fs.FileInfo, error)
}

type FileSystem

type FileSystem interface {
	fs.FS
	fs.StatFS
	OpenFile(path string, flag int, perm fs.FileMode) (fs.File, error)
	// Lstat returns the fs.FileInfo for the given path, without
	// following symlinks.
	Lstat(path string) (fs.FileInfo, error)
	// Mkdir creates a directory at the given path. If the directory
	// already exists or its parent directory does not exist or
	// the permissions don't allow it, an error will be returned. See
	// also the shorthand function MkdirAll.
	Mkdir(path string, perm fs.FileMode) error
	// Remove removes the item at the given path. If the path does
	// not exists or the permissions don't allow removing it or it's
	// a non-empty directory, an error will be returned. See also
	// the shorthand function RemoveAll.
	Remove(path string) error
}

type Fingerprintable

type Fingerprintable interface {
	Fingerprint() (string, error)
}

Fingerprintable is a type of object returned by Sys() that allows obtaining a string identifying a file at a current status. In case the file is modified the fingerprint should be different, and its value can be used in HTTP etag for example. If the backend storage allows it, the value should be a hash (ie. CRC32 for zipfs), and if not, size and timestamp may do.

type KVEntry

type KVEntry interface {
	Data() ([]byte, error)
}

type Keyval

type Keyval interface {
	Get(key string) (KVEntry, error)
	Put(key string, value KVEntry) error
	Delete(key string) error

	// List will call callback for each entry stored in the keyval which prefix
	// matches. If the callback returns cont=false or err != nil, the function
	// will stop.
	// There is no guarantee entries will be listed in any order.
	List(prefix string, callback func(key string, value KVEntry) (cont bool, err error)) error
}