absfs

package module
v0.0.0-...-6f31c6a Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 18, 2023 License: MIT Imports: 8 Imported by: 14

README

absfs - Abstract File System for go

absfs is a go package that defines an abstract filesystem interface.

The design goal of absfs is to support a system of composable filesystem implementations for various uses from testing to complex storage management.

Implementors can create stand alone filesystems that implement the abfs.Filer interface providing new components that can easily be add to existing compositions, and data pipelines.

absfs will provide a testing suite to help implementers produce packages that function and error consistently with all other abstract filesystems implementations.

Install

$ go get github.com/absfs/absfs

The Interfaces

absfs defines 4 interfaces.

  1. Filer - The minimum set of methods that an implementation must define.
  2. FileSystem - The complete set of Abstract FileSystem methods, including convenience functions.
  3. SymLinker - Methods for implementing symbolic links.
  4. SymlinkFileSystem - A FileSystem that supports symbolic links.
Filer - Interface

The minimum set of methods that an implementation must define.

type Filer interface {

    Mkdir(name string, perm os.FileMode) error
    OpenFile(name string, flag int, perm os.FileMode) File, error
    Remove(name string) error
    Rename(oldname, newname string) error
    Stat(name string) os.FileInfo, error
    Chmod(name string, mode os.FileMode) error
    Chtimes(name string, atime time.Time, mtime time.Time) error

}
FileSystem - Interface

The complete set of Abstract FileSystem methods, including convenience functions.

type FileSystem interface {

    // Filer interface
    OpenFile(name string, flag int, perm os.FileMode) (File, error)
    Mkdir(name string, perm os.FileMode) error
    Remove(name string) error
    Stat(name string) (os.FileInfo, error)
    Chmod(name string, mode os.FileMode) error
    Chtimes(name string, atime time.Time, mtime time.Time) error
    Chown(name string, uid, gid int) error

    Separator() uint8
    ListSeparator() uint8
    Chdir(dir string) error
    Getwd() (dir string, err error)
    TempDir() string
    Open(name string) (File, error)
    Create(name string) (File, error)
    MkdirAll(name string, perm os.FileMode) error
    RemoveAll(path string) (err error)
    Truncate(name string, size int64) error

}
SymLinker - Interface

Additional methods for implementing symbolic links.

type SymLinker interface {

    Lstat(fi1, fi2 os.FileInfo) bool
    Lchown(name string, uid, gid int) error
    Readlink(name string) (string, error)
    Symlink(oldname, newname string) error

}
SymlinkFileSystem - Interface

A FileSystem that supports symbolic links.

type SymlinkFileSystem interface {

    // Filer interface
    OpenFile(name string, flag int, perm os.FileMode) (File, error)
    Mkdir(name string, perm os.FileMode) error
    Remove(name string) error
    Stat(name string) (os.FileInfo, error)
    Chmod(name string, mode os.FileMode) error
    Chtimes(name string, atime time.Time, mtime time.Time) error
    Chown(name string, uid, gid int) error

    // FileSystem interface
    Separator() uint8
    ListSeparator() uint8
    Chdir(dir string) error
    Getwd() (dir string, err error)
    TempDir() string
    Open(name string) (File, error)
    Create(name string) (File, error)
    MkdirAll(name string, perm os.FileMode) error
    RemoveAll(path string) (err error)
    Truncate(name string, size int64) error

    // SymLinker interface
    Lstat(fi1, fi2 os.FileInfo) bool
    Lchown(name string, uid, gid int) error
    Readlink(name string) (string, error)
    Symlink(oldname, newname string) error
}

File Systems

OsFs

An absfs.FileSystem implementation that wraps the equivalent os standard library filesystem functions. Using osfs is essentially identical to using the os package except that it implements a per object current working directory.

MemFs

The memfs filesystem implements a memory only filesystem.

NilFs

The Nil FileSystem is a no-op implementation. Methods have no effect and most return no errors. The Read and ReadAt methods always return io.EOF to avoid infinite loops.

PTFS

The Pass Through FileSystem (ptfs) takes any object that implements absfs.FileSystem and passes all methods through unaltered to that object.

ROFS

The rofs is a read-only filesystem that can wrap any other filesystem implementation and make it read only.

BaseFS

The basefs filesystem takes a absfs.FileSystem and an absolute path within that filesystem, and provides a absfs.FileSystem interface that is rooted at that path. Relative paths are handled correctly such that it is not possible to 'escape' the base path constraint (for example by using a path like "/.."). In addition errors are rewritten to accurately obscure the base path.

HttpFS

A filesystem that can convert a absfs.FileSystem to a http.FileSystem interface for use with http.FileServer.

CorFS (cache on read fs)

An absfs.Filer that wraps two absfs.Filers when data is read if it doesn't exist in the second tier filesytem the filer copies data from the underlying file systems into the second tier filesystem causing it to function like a cache.

CowFS (copy on write fs)

An absfs.Filer that wraps two absfs.Filers the first of which may be a read only filesystem. Any attempt to modify or write data to a CowFS causes the Filer to write those changes to the secondary fs leaving the underlying filesystem unmodified.

BoltFS

An absfs.Filer that provides and absfs.Filer interface for the popular embeddable key/value store called bolt.

S3FS

An absfs.Filer that provides and absfs. Filer interface to any S3 compatible object storage API.

SftpFs

A filesystem which reads and writes securely between computers across networks using the SFTP interface.

Implementing a FileSystem

AbsFs compatible filesystems can and should be implemented in their own package and repo. You may want to start with nilfs, ptfs, or osfs as a template. It is not necessary to implement all FileSystem methods on your custom FileSystem since you can use the ExtendFiler function to convert a Filer implementation to a full FileSystem implementation.

Step 1 - Filer the minimum FileSystem interface

These methods are OpenFile, Mkdir, Remove, Stat, Chmod, Chtimes, Chown

// template for a Filer implementation
type MyFiler struct {
    // ... 
}

func New() *MyFiler {
    return &MyFiler{}
}

func (fs *MyFiler) OpenFile(name string, flag int, perm os.FileMode) (File, error) {
    // ...
}

func (fs *MyFiler) Mkdir(name string, perm os.FileMode) error  {
    // ...
}

func (fs *MyFiler) Remove(name string) error  {
    // ...
}

func (fs *MyFiler) Stat(name string) (os.FileInfo, error)  {
    // ...
}

func (fs *MyFiler) Chmod(name string, mode os.FileMode) error  {
    // ...
}

func (fs *MyFiler) Chtimes(name string, atime time.Time, mtime time.Time) error  {
    // ...
}

func (fs *MyFiler) Chown(name string, uid, gid int) error  {
    // ...
}

Optionally you can also implement any of the FileSystem interface methods if you need for performance or other reasons. The object returned by ExtendFiler only adds the missing methods FileSystem methods. The FileSystem interface adds the following additional methods: Separator, ListSeparator, Chdir, Getwd, TempDir, Open, Create, MkdirAll , RemoveAll, Truncate.

Step 2 - Extend A Filer to make a FileSystem

After implementing any of the additional FileSystem methods create a NewFS function that uses ExtendFiler to add the missing methods.

package myfs

import "github.com/absfs/absfs"

type MyFiler struct {
    // ...
}

// MyFiler implements the absfs.Filer interface

func NewFS() absfs.FileSystem {
    return absfs.ExtendFiler(&MyFiler{})
}

The implementation provided by ExtendFiler will first check for an existing method of the same signature on the underlying Filer. If found it will call that method, if not it provides a default implementation.

An extended Filer implements the FileSystem interface as follows. If the FileSystem method is one of the convenience functions like Open, Create, or MkdirAll the default implementation simply uses the Filer methods (i.e. OpenFile and Mkdir) to implement the convenience function on top of the Filer interface. If the missing methods is one of Separator, ListSeparator, or TempDir, then the local operating system values are returned. Path navigation as provided by Chdir, and Getwd are provided as a complete path management implementation that resolves both absolute and relative paths much the say way as the os package. The extended filer will resolve paths into absolute paths and maintains a unique current working directory for each FileSystem interface object. A Filer is not required to implement relative paths.

Contributing

We strongly encourage contributions, please fork and submit Pull Requests, and publish any FileSystems you implement.

New FileSystem types do not need to be added to this repo, but we'd be happy to link to yours so please open an issue or better yet add a Pull Request with the updated Readme.

LICENSE

This project is governed by the MIT License. See LICENSE

Documentation

Overview

Package absfs provides file and file system interfaces for abstract file system primitives. It's purpose is to define a filesystem abstraction that supports common file system operations.

Index

Constants

View Source 
const (
	OS_READ        = 04
	OS_WRITE       = 02
	OS_EX          = 01
	OS_USER_SHIFT  = 6
	OS_GROUP_SHIFT = 3
	OS_OTH_SHIFT   = 0

	OS_USER_R   = OS_READ << OS_USER_SHIFT
	OS_USER_W   = OS_WRITE << OS_USER_SHIFT
	OS_USER_X   = OS_EX << OS_USER_SHIFT
	OS_USER_RW  = OS_USER_R | OS_USER_W
	OS_USER_RWX = OS_USER_RW | OS_USER_X

	OS_GROUP_R   = OS_READ << OS_GROUP_SHIFT
	OS_GROUP_W   = OS_WRITE << OS_GROUP_SHIFT
	OS_GROUP_X   = OS_EX << OS_GROUP_SHIFT
	OS_GROUP_RW  = OS_GROUP_R | OS_GROUP_W
	OS_GROUP_RWX = OS_GROUP_RW | OS_GROUP_X

	OS_OTH_R   = OS_READ << OS_OTH_SHIFT
	OS_OTH_W   = OS_WRITE << OS_OTH_SHIFT
	OS_OTH_X   = OS_EX << OS_OTH_SHIFT
	OS_OTH_RW  = OS_OTH_R | OS_OTH_W
	OS_OTH_RWX = OS_OTH_RW | OS_OTH_X

	OS_ALL_R   = OS_USER_R | OS_GROUP_R | OS_OTH_R
	OS_ALL_W   = OS_USER_W | OS_GROUP_W | OS_OTH_W
	OS_ALL_X   = OS_USER_X | OS_GROUP_X | OS_OTH_X
	OS_ALL_RW  = OS_ALL_R | OS_ALL_W
	OS_ALL_RWX = OS_ALL_RW | OS_GROUP_X
)

Permission flags not provided by the standard library.

View Source 
const (
	O_ACCESS = 0x3 // masks the access mode (O_RDONLY, O_WRONLY, or O_RDWR)

	// Exactly one of O_RDONLY, O_WRONLY, or O_RDWR must be specified.
	O_RDONLY = os.O_RDONLY // open the file read-only.
	O_WRONLY = os.O_WRONLY // open the file write-only.
	O_RDWR   = os.O_RDWR   // open the file read-write.

	// The remaining values may be or'ed in to control behavior.
	O_APPEND = os.O_APPEND // append data to the file when writing.
	O_CREATE = os.O_CREATE // create a new file if none exists.
	O_EXCL   = os.O_EXCL   // used with O_CREATE, file must not exist.
	O_SYNC   = os.O_SYNC   // open for synchronous I/O.
	O_TRUNC  = os.O_TRUNC  // if possible, truncate file when opened.
)

Variables

View Source 
var ErrNotImplemented = errors.New("not implemented")

Functions

func ParseFileMode 

func ParseFileMode(input string) (os.FileMode, error)

ParseFileMode - parses a unix style file mode string and returns an `os.FileMode` and an error.

Types

type FastWalkFunc 

type FastWalkFunc func(string, os.FileMode) error

type File 

type File interface {
	Seekable

	// ReadAt reads len(b) bytes from the File starting at byte offset off. It
	// returns the number of bytes read and the error, if any. ReadAt always
	// returns a non-nil error when n < len(b). At end of file, that error is
	// io.EOF.
	ReadAt(b []byte, off int64) (n int, err error)

	// WriteAt writes len(b) bytes to the File starting at byte offset off. It
	// returns the number of bytes written and an error, if any. WriteAt returns
	// a non-nil error when n != len(b).
	WriteAt(b []byte, off int64) (n int, err error)

	// WriteString is like Write, but writes the contents of string s rather than
	// a slice of bytes.
	WriteString(s string) (n int, err error)

	// Truncate - changes the size of the file. It does not change the I/O offset.
	// If there is an error, it should be of type `*os.PathError`.
	Truncate(size int64) error

	// Readdirnames - reads the contents of the directory associated with file and
	// returns a slice of up to n names. Subsequent calls on the same
	// file will yield further names.
	//
	// If n > 0, Readdirnames returns at most n names. In this case, if
	// Readdirnames returns an empty slice, it will return a non-nil error
	// explaining why. At the end of a directory, the error is io.EOF.
	//
	// If n <= 0, Readdirnames returns all the names from the directory in a single
	// slice. In this case, if Readdirnames succeeds (reads all the way to the end of
	// the directory), it returns the slice and a nil error. If it encounters an
	// error before the end of the directory, Readdirnames returns the names read
	// until that point and a non-nil error.
	Readdirnames(n int) (names []string, err error)
}

File - is an interface for file handles that supports the most common file operations. Filesystem interfaces in this package have functions that return values of type `File`, but be aware that some implementations may only support a subset of these functions given the limitations particular file systems.

func ExtendSeekable 

func ExtendSeekable(sf Seekable) File

ExtendSeekable - extends a `Seekable` interface implementation to a File interface implementation. First type assertion is used to check if the argument already supports the `File` interface.

If the argument's native type is not recognized, and cannot be type asserted to an implementation of `File` `ExtendSeekable` wraps the argument in a type that implements the additional `File` functions using only the functions in the `Seekable` interface. Functions that are identical between `File` and `Seekable` are passed through unaltered. This should would fine, but may have have performance implications.

type FileSystem 

type FileSystem interface {
	Filer

	Separator() uint8
	ListSeparator() uint8
	Chdir(dir string) error
	Getwd() (dir string, err error)
	TempDir() string
	Open(name string) (File, error)
	Create(name string) (File, error)
	MkdirAll(name string, perm os.FileMode) error
	RemoveAll(path string) (err error)
	Truncate(name string, size int64) error
}

func ExtendFiler 

func ExtendFiler(filer Filer) FileSystem

ExtendFiler adds the FileSystem convenience functions to any Filer implementation.

type Filer 

type Filer interface {

	// OpenFile opens a file using the given flags and the given mode.
	OpenFile(name string, flag int, perm os.FileMode) (File, error)

	// Mkdir creates a directory in the filesystem, return an error if any
	// happens.
	Mkdir(name string, perm os.FileMode) error

	// Remove removes a file identified by name, returning an error, if any
	// happens.
	Remove(name string) error

	// Rename renames (moves) oldpath to newpath. If newpath already exists and
	// is not a directory, Rename replaces it. OS-specific restrictions may apply
	// when oldpath and newpath are in different directories. If there is an
	// error, it will be of type *LinkError.
	Rename(oldpath, newpath string) error

	// Stat returns the FileInfo structure describing file. If there is an error,
	// it will be of type *PathError.
	Stat(name string) (os.FileInfo, error)

	//Chmod changes the mode of the named file to mode.
	Chmod(name string, mode os.FileMode) error

	//Chtimes changes the access and modification times of the named file
	Chtimes(name string, atime time.Time, mtime time.Time) error

	//Chown changes the owner and group ids of the named file
	Chown(name string, uid, gid int) error
}

type Flags 

type Flags int

Flags - represents access and permission flags for use with file opening functions.

func ParseFlags 

func ParseFlags(input string) (Flags, error)

ParseFlags - parses a string of flags separated by "|" and returns a `Flags` value and an error.

Following is the list of recognized flags, "O_RDONLY", "O_RDWR", "O_WRONLY", "O_APPEND", "O_CREATE", "O_EXCL", "O_SYNC", "O_TRUNC". The access mode values ("O_RDONLY", "O_RDWR", "O_WRONLY") are mutually exclusive, only one may be specified. If no access mode is specified "O_RDONLY" is the default. All other flags may appear more than once, but subsequent occurrences have no effect.

func (Flags) String 

func (f Flags) String() string

String - returns the list of values set in a `Flag` separated by "|".

type InvalidFile 

type InvalidFile struct {

	// Path should be set to the file path provided to the function opening the
	// file (i.e. OpenFile, Open, Create).
	Path string
}

InvalidFile is a no-op implementation of File that should be returned from any file open methods when an error occurs. InvalidFile mimics the behavior of file handles returnd by the `os` package when there is an error.

func (*InvalidFile) Close 

func (f *InvalidFile) Close() error

Close - does nothing and does not return an error.

func (*InvalidFile) Name 

func (f *InvalidFile) Name() string

Name - returns the name of the file path provided to the function that attempted to open the file.

func (*InvalidFile) Read 

func (f *InvalidFile) Read(p []byte) (int, error)

Read - returns an *os.PathError indicating a bad file handle.

func (*InvalidFile) ReadAt 

func (f *InvalidFile) ReadAt(b []byte, off int64) (n int, err error)

ReadAt - returns an *os.PathError indicating a bad file handle.

func (*InvalidFile) Readdir 

func (f *InvalidFile) Readdir(int) ([]os.FileInfo, error)

Readdir - returns an *os.PathError indicating a bad file handle.

func (*InvalidFile) Readdirnames 

func (f *InvalidFile) Readdirnames(n int) (names []string, err error)

Readdirnames - returns an *os.PathError indicating a bad file handle.

func (*InvalidFile) Seek 

func (f *InvalidFile) Seek(offset int64, whence int) (ret int64, err error)

Seek - returns an *os.PathError indicating a bad file handle.

func (*InvalidFile) Stat 

func (f *InvalidFile) Stat() (os.FileInfo, error)

Stat - returns an *os.PathError indicating a bad file handle.

func (*InvalidFile) Sync 

func (f *InvalidFile) Sync() error

Sync - returns an *os.PathError indicating a bad file handle.

func (*InvalidFile) Truncate 

func (f *InvalidFile) Truncate(size int64) error

Truncate - returns an *os.PathError indicating a bad file handle.

func (*InvalidFile) Write 

func (f *InvalidFile) Write(p []byte) (int, error)

Write - returns an *os.PathError indicating a bad file handle.

func (*InvalidFile) WriteAt 

func (f *InvalidFile) WriteAt(b []byte, off int64) (n int, err error)

WriteAt - returns an *os.PathError indicating a bad file handle.

func (*InvalidFile) WriteString 

func (f *InvalidFile) WriteString(s string) (n int, err error)

WriteString - returns an *os.PathError indicating a bad file handle.

type ReadOnlyFiler 

type ReadOnlyFiler interface {
	Open(name string) (io.ReadCloser, error)
}

type Seekable 

type Seekable interface {
	UnSeekable
	io.Seeker
}

Seekable - is an interface for file handles that can perform reads and/or writes and can seek to specific locations within a file. A Seekable is also an io.ReadWriteCloser, and an io.Seeker.

type SymLinker 

type SymLinker interface {

	// Lstat returns a FileInfo describing the named file. If the file is a
	// symbolic link, the returned FileInfo describes the symbolic link. Lstat
	// makes no attempt to follow the link. If there is an error, it will be of type *PathError.
	Lstat(name string) (os.FileInfo, error)

	// Lchown changes the numeric uid and gid of the named file. If the file is a
	// symbolic link, it changes the uid and gid of the link itself. If there is
	// an error, it will be of type *PathError.
	//
	// On Windows, it always returns the syscall.EWINDOWS error, wrapped in
	// *PathError.
	Lchown(name string, uid, gid int) error

	// Readlink returns the destination of the named symbolic link. If there is an
	// error, it will be of type *PathError.
	Readlink(name string) (string, error)

	// Symlink creates newname as a symbolic link to oldname. If there is an
	// error, it will be of type *LinkError.
	Symlink(oldname, newname string) error
}

type SymlinkFileSystem 

type SymlinkFileSystem interface {
	FileSystem
	SymLinker
}

type UnSeekable 

type UnSeekable interface {

	// Name returns the name of the file as presented to Open.
	Name() string

	// Read reads up to len(b) bytes from the File. It returns the number of bytes
	// read and any error encountered. At end of file, Read returns 0, io.EOF.
	Read(b []byte) (int, error)

	// Write writes len(b) bytes to the File. It returns the number of bytes
	// written and an error, if any. Write returns a non-nil error when
	// n != len(b).
	Write(b []byte) (int, error)

	// Close - closes the File, rendering it unusable for I/O. It returns an error,
	// if any.
	Close() error

	// Sync - commits the current contents of the file to stable storage.
	Sync() error

	// Stat - returns the FileInfo structure describing file. If there is an
	// error, it should be of type `*os.PathError`.
	Stat() (os.FileInfo, error)

	// Readdir - reads the contents of the directory associated with file and
	// returns a slice of up to n `os.FileInfo` values, as would be returned by
	// Stat or Lstat in directory order. Subsequent calls on the same file will
	// yield further `os.FileInfos`.
	//
	// If n > 0, Readdir returns at most n FileInfo structures. In this case, if
	// Readdir returns an empty slice, it will return a non-nil error explaining
	// why. At the end of a directory, the error is io.EOF.
	//
	// If n <= 0, Readdir returns all the FileInfo from the directory in a single
	// slice. In this case, if Readdir succeeds (reads all the way to the end of
	// the directory), it returns the slice and a nil error. If it encounters an
	// error before the end of the directory, Readdir returns the FileInfo read
	// until that point and a non-nil error.
	Readdir(int) ([]os.FileInfo, error)
}

UnSeekable - is an interface for file handles that can perform reads and/or writes but cannot perform seek operations. An UnSeekable is also an io.ReadWriteCloser.

type WriteOnlyFiler 

type WriteOnlyFiler interface {
	Open(name string) (io.WriteCloser, error)
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.