hackpadfs

package module
v0.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 22, 2023 License: Apache-2.0 Imports: 7 Imported by: 24

README

hackpadfs Go Reference CI Coverage Status

File systems, interfaces, and helpers for Go.

Want to get started? Check out the guides below.

File systems

hackpadfs includes several implemented file systems, ready for use in a wide variety of applications:

  • os.FS - The familiar os package. Implements all of the familiar behavior from the standard library using new interface design.
  • mem.FS - In-memory file system.
  • indexeddb.FS - WebAssembly compatible file system, uses IndexedDB under the hood.
  • tar.ReaderFS - A streaming tar FS for memory and time-constrained programs.
  • mount.FS - Composable file system. Capable of mounting file systems on top of each other.
  • keyvalue.FS - Generic key-value file system. Excellent for quickly writing your own file system. mem.FS and indexeddb.FS are built upon it.

Looking for custom file system inspiration? Examples include:

Each of these file systems runs through the rigorous hackpadfs/fstest suite to ensure both correctness and compliance with the standard library's os package behavior. If you're implementing your own FS, we recommend using fstest in your own tests as well.

Interfaces

Based upon the groundwork laid in Go 1.16's io/fs package, hackpadfs defines many essential file system interfaces.

Here's a few of the interfaces defined by hackpadfs:

type FS interface {
    Open(name string) (File, error)
}

type CreateFS interface {
    FS
    Create(name string) (File, error)
}

type MkdirFS interface {
    FS
    Mkdir(name string, perm FileMode) error
}

type StatFS interface {
    FS
    Stat(name string) (FileInfo, error)
}

See the reference docs for full documentation.

Using these interfaces, you can create and compose your own file systems. The interfaces are small, enabling custom file systems to implement only the required pieces.

Getting started

There's many ways to use hackpadfs. Jump to one of these guides:

Quick start

If you've used the standard library's os package, you probably understand most of how os.FS works!

In this example, we create a new os.FS and print the contents of /tmp/hello.txt.

import (
    "fmt"

    "github.com/hack-pad/hackpadfs"
    "github.com/hack-pad/hackpadfs/os"
)

filePath := "tmp/hello.txt"
fs, _ := os.NewFS()
file, _ := fs.Open(filePath)
defer file.Close()

buffer := make([]byte, 1024)
n, _ := file.Read(buffer)
fmt.Println("Contents of hello.txt:", string(buffer[:n]))
Relative file paths

Relative paths are not allowed in Go's io/fs specification, so we must use absolute paths (without the first /). To simulate relative paths, use the SubFS interface to create a new "rooted" FS like this:

import (
    goOS "os"
    "github.com/hack-pad/hackpadfs/os"
)

fs := os.NewFS()
workingDirectory, _ := goOS.Getwd()                    // Get current working directory
workingDirectory, _ = fs.FromOSPath(workingDirectory)  // Convert to an FS path
workingDirFS, _ := fs.Sub(workingDirectory)            // Run all file system operations rooted at the current working directory
Path separators (slashes)

Following the io/fs specification:

Path names passed to open are UTF-8-encoded, unrooted, slash-separated sequences of path elements, like “x/y/z”. Path names must not contain an element that is “.” or “..” or the empty string, except for the special case that the root directory is named “.”. Paths must not start or end with a slash: “/x” and “x/” are invalid.

Note that paths are slash-separated on all systems, even Windows. Paths containing other characters such as backslash and colon are accepted as valid, but those characters must never be interpreted by an FS implementation as path element separators.

In hackpadfs, this means:

  • All path separators are "forward slashes" or /. Even on Windows, slashes are converted under the hood.
  • A path starting or ending with / is invalid
  • To reference the root file path, use .
  • All paths are unrooted (not relative paths)
  • Paths are not necessarily cleaned when containing relative-path elements (e.g. mypath/.././myotherpath). Some FS implementations resolve these, but it is not guaranteed. File systems should reject these paths via io/fs.ValidPath().
Working with interfaces

It's a good idea to use interfaces -- file systems should be no different. Swappable file systems enable powerful combinations.

However, directly handling interface values can be difficult to deal with. Luckily, we have several helpers available.

In the below example, we use hackpadfs's package-level functions to do interface checks and call the appropriate methods for us:

import (
    "github.com/hack-pad/hackpadfs"
    "github.com/hack-pad/hackpadfs/mem"
)

func main() {
    fs, _ := mem.NewFS()
    helloWorld(fs)
}

// helloWorld uses a generic hackpadfs.FS to create a file containing "world".
// Returns an error if 'fs' does not support creating or writing to files.
func helloWorld(fs hackpadfs.FS) error {
    // hackpadfs.Create(...) checks to see if 'fs' implements Create, with a few fallback interfaces as well.
    file, err := hackpadfs.Create(fs, "hello.txt")
    if err != nil {
	return err
    }
    // Same here for hackpadfs.WriteFile(...). If the file doesn't support writing, a "not implemented" error is returned.
    _, err = hackpadfs.WriteFile(file, []byte("world"))
    return err
}

Notice the package-level function calls to hackpadfs.Create(...) and hackpadfs.WriteFile(...). Since the interface we're using doesn't know about those methods, we use these helpers to detect support and run those operations in one call.

Now whenever we need to reuse helloWorld() with a completely different file system, it's ready to go!

Documentation

Overview

Package hackpadfs defines many essential file and file system interfaces as well as helpers for use with the standard library's 'io/fs' package.

Index

Constants

View Source 
const (
	FlagReadOnly  int = syscall.O_RDONLY
	FlagWriteOnly int = syscall.O_WRONLY
	FlagReadWrite int = syscall.O_RDWR

	FlagAppend    int = syscall.O_APPEND
	FlagCreate    int = syscall.O_CREAT
	FlagExclusive int = syscall.O_EXCL
	FlagSync      int = syscall.O_SYNC
	FlagTruncate  int = syscall.O_TRUNC
)

Flags are bit-wise OR'd with each other in fs.OpenFile(). Exactly one of Read/Write flags must be specified, and any other flags can be OR'd together.

View Source 
const (
	ModeDir        = gofs.ModeDir
	ModeAppend     = gofs.ModeAppend
	ModeExclusive  = gofs.ModeExclusive
	ModeTemporary  = gofs.ModeTemporary
	ModeSymlink    = gofs.ModeSymlink
	ModeDevice     = gofs.ModeDevice
	ModeNamedPipe  = gofs.ModeNamedPipe
	ModeSocket     = gofs.ModeSocket
	ModeSetuid     = gofs.ModeSetuid
	ModeSetgid     = gofs.ModeSetgid
	ModeCharDevice = gofs.ModeCharDevice
	ModeSticky     = gofs.ModeSticky
	ModeIrregular  = gofs.ModeIrregular

	ModeType = gofs.ModeType
	ModePerm = gofs.ModePerm
)

Mode values are bit-wise OR'd with a file's permissions to form the FileMode. Mirror io/fs.Mode... values.

Variables

View Source 
var (
	ErrInvalid    = syscall.EINVAL // TODO update to fs.ErrInvalid, once errors.Is supports it
	ErrPermission = fs.ErrPermission
	ErrExist      = fs.ErrExist
	ErrNotExist   = fs.ErrNotExist
	ErrClosed     = fs.ErrClosed

	ErrIsDir          = syscall.EISDIR
	ErrNotDir         = syscall.ENOTDIR
	ErrNotEmpty       = syscall.ENOTEMPTY
	ErrNotImplemented = syscall.ENOSYS

	SkipDir = fs.SkipDir
)

Errors commonly returned by file systems. Mirror their equivalents in the syscall and io/fs packages.

Functions

func Chmod 

func Chmod(fs FS, name string, mode FileMode) error

Chmod attempts to call an optimized fs.Chmod(), falls back to opening the file and running file.Chmod().

func ChmodFile 

func ChmodFile(file File, mode FileMode) error

ChmodFile runs file.Chmod() is available, fails with a not implemented error otherwise.

func Chown 

func Chown(fs FS, name string, uid, gid int) error

Chown attempts to call an optimized fs.Chown(), falls back to opening the file and running file.Chown().

func ChownFile 

func ChownFile(file File, uid, gid int) error

ChownFile runs file.Chown() is available, fails with a not implemented error otherwise.

func Chtimes 

func Chtimes(fs FS, name string, atime time.Time, mtime time.Time) error

Chtimes attempts to call an optimized fs.Chtimes(), falls back to opening the file and running file.Chtimes().

func ChtimesFile 

func ChtimesFile(file File, atime, mtime time.Time) error

ChtimesFile runs file.Chtimes() is available, fails with a not implemented error otherwise.

func Mkdir 

func Mkdir(fs FS, name string, perm FileMode) error

Mkdir creates a directory. Fails with a not implemented error if it's not a MkdirFS.

func MkdirAll 

func MkdirAll(fs FS, path string, perm FileMode) error

MkdirAll attempts to call an optimized fs.MkdirAll(), falls back to multiple fs.Mkdir() calls.

func ReadAtFile 

func ReadAtFile(file File, p []byte, off int64) (n int, err error)

ReadAtFile runs file.ReadAt() is available, fails with a not implemented error otherwise.

func ReadFile 

func ReadFile(fs FS, name string) ([]byte, error)

ReadFile attempts to call an optimized fs.ReadFile(), falls back to io/fs.ReadFile().

func Remove 

func Remove(fs FS, name string) error

Remove removes a file with fs.Remove(). Fails with a not implemented error if it's not a RemoveFS.

func RemoveAll 

func RemoveAll(fs FS, path string) error

RemoveAll attempts to call an optimized fs.RemoveAll(), falls back to removing files and directories recursively.

func Rename 

func Rename(fs FS, oldName, newName string) error

Rename moves files with fs.Rename(). Fails with a not implemented error if it's not a RenameFS.

func SeekFile 

func SeekFile(file File, offset int64, whence int) (int64, error)

SeekFile runs file.Seek() is available, fails with a not implemented error otherwise. 

func Symlink(fs FS, oldname, newname string) error

Symlink creates a symlink. Fails with a not implemented error if it's not a SymlinkFS.

func SyncFile 

func SyncFile(file File) error

SyncFile runs file.Sync() is available, fails with a not implemented error otherwise.

func TruncateFile 

func TruncateFile(file File, size int64) error

TruncateFile runs file.Truncate() is available, fails with a not implemented error otherwise.

func ValidPath 

func ValidPath(path string) bool

ValidPath returns true if 'path' is a valid FS path. See io/fs.ValidPath() for details on FS-safe paths.

func WalkDir 

func WalkDir(fs FS, root string, fn WalkDirFunc) error

WalkDir recursively scans 'fs' starting at path 'root', calling 'fn' every time a new file or directory is visited.

func WriteAtFile 

func WriteAtFile(file File, p []byte, off int64) (n int, err error)

WriteAtFile runs file.WriteAt() is available, fails with a not implemented error otherwise.

func WriteFile 

func WriteFile(file File, p []byte) (n int, err error)

WriteFile runs file.Write() is available, fails with a not implemented error otherwise.

func WriteFullFile added in v0.1.9 

func WriteFullFile(fs FS, name string, data []byte, perm FileMode) error

WriteFullFile attempts to call an optimized fs.WriteFile(), falls back to fs.OpenFile() with file.Write().

Types

type ChmodFS 

type ChmodFS interface {
	FS
	Chmod(name string, mode FileMode) error
}

ChmodFS is an FS that can change file or directory permissions. Should match the behavior of os.Chmod().

type ChmoderFile 

type ChmoderFile interface {
	File
	Chmod(mode FileMode) error
}

ChmoderFile is a File that supports Chmod() operations.

type ChownFS 

type ChownFS interface {
	FS
	Chown(name string, uid, gid int) error
}

ChownFS is an FS that can change file or directory ownership. Should match the behavior of os.Chown().

type ChownerFile 

type ChownerFile interface {
	File
	Chown(uid, gid int) error
}

ChownerFile is a File that supports Chown() operations.

type ChtimesFS 

type ChtimesFS interface {
	FS
	Chtimes(name string, atime time.Time, mtime time.Time) error
}

ChtimesFS is an FS that can change a file's access and modified timestamps. Should match the behavior of os.Chtimes().

type ChtimeserFile 

type ChtimeserFile interface {
	File
	Chtimes(atime time.Time, mtime time.Time) error
}

ChtimeserFile is a File that supports Chtimes() operations.

type CreateFS 

type CreateFS interface {
	FS
	Create(name string) (File, error)
}

CreateFS is an FS that can create files. Should match the behavior of os.Create().

type DirEntry 

type DirEntry = gofs.DirEntry

DirEntry is an entry read from a directory. Mirrors io/fs.DirEntry.

func ReadDir 

func ReadDir(fs FS, name string) ([]DirEntry, error)

ReadDir attempts to call an optimized fs.ReadDir(), falls back to io/fs.ReadDir().

func ReadDirFile 

func ReadDirFile(file File, n int) ([]DirEntry, error)

ReadDirFile runs file.ReadDir() is available, fails with a not implemented error otherwise.

type DirReaderFile 

type DirReaderFile interface {
	File
	ReadDir(n int) ([]DirEntry, error)
}

DirReaderFile is a File that supports ReadDir() operations. Mirrors io/fs.ReadDirFile.

type FS 

type FS = gofs.FS

FS provides access to a file system and its files. It is the minimum functionality required for a file system, and mirrors Go's io/fs.FS interface.

func Sub 

func Sub(fs FS, dir string) (FS, error)

Sub attempts to call an optimized fs.Sub() if available. Falls back to a small MountFS implementation.

type File 

type File = gofs.File

File provides access to a file. Mirrors io/fs.File.

func Create 

func Create(fs FS, name string) (File, error)

Create attempts to call an optimized fs.Create() if available, falls back to OpenFile() with create flags.

func OpenFile 

func OpenFile(fs FS, name string, flag int, perm FileMode) (File, error)

OpenFile attempts to call fs.Open() or fs.OpenFile() if available. Fails with a not implemented error otherwise.

type FileInfo 

type FileInfo = gofs.FileInfo

FileInfo describes a file and is returned by Stat(). Mirrors io/fs.FileInfo.

func Lstat 

func Lstat(fs FS, name string) (FileInfo, error)

Lstat stats files and does not follow symlinks. Fails with a not implemented error if it's not a LstatFS.

func LstatOrStat 

func LstatOrStat(fs FS, name string) (FileInfo, error)

LstatOrStat attempts to call an optimized fs.LstatOrStat(), fs.Lstat(), or fs.Stat() - whichever is supported first.

func Stat 

func Stat(fs FS, name string) (FileInfo, error)

Stat attempts to call an optimized fs.Stat(), falls back to fs.Open() and file.Stat().

type FileMode 

type FileMode = gofs.FileMode

FileMode represents a file's mode and permission bits. Mirrors io/fs.FileMode.

type LinkError 

type LinkError struct {
	Op  string
	Old string
	New string
	Err error
}

LinkError records a file system rename error and the paths that caused it. Mirrors os.LinkError

NOTE: Is not identical to os.LinkError to avoid importing "os". Still resolves errors.Is() calls correctly.

func (*LinkError) Error 

func (e *LinkError) Error() string

func (*LinkError) Unwrap 

func (e *LinkError) Unwrap() error

Unwrap supports errors.Unwrap().

type LstatFS 

type LstatFS interface {
	FS
	Lstat(name string) (FileInfo, error)
}

LstatFS is an FS that can lstat files. Same as Stat, but returns file info of symlinks instead of their target. Should match the behavior of os.Lstat().

type MkdirAllFS 

type MkdirAllFS interface {
	FS
	MkdirAll(path string, perm FileMode) error
}

MkdirAllFS is an FS that can make all missing directories in a given path. Should match the behavior of os.MkdirAll().

type MkdirFS 

type MkdirFS interface {
	FS
	Mkdir(name string, perm FileMode) error
}

MkdirFS is an FS that can make directories. Should match the behavior of os.Mkdir().

type MountFS 

type MountFS interface {
	FS
	Mount(name string) (mountFS FS, subPath string)
}

MountFS is an FS that meshes one or more FS's together. Returns the FS for a file located at 'name' and its 'subPath' inside that FS.

type OpenFileFS 

type OpenFileFS interface {
	FS
	OpenFile(name string, flag int, perm FileMode) (File, error)
}

OpenFileFS is an FS that can open files with the given flags and can create with the given permission. Should matche the behavior of os.OpenFile().

type PathError 

type PathError = fs.PathError

PathError records a file system or file operation error and the path that caused it. Mirrors io/fs.PathError

type ReadDirFS 

type ReadDirFS interface {
	FS
	ReadDir(name string) ([]DirEntry, error)
}

ReadDirFS is an FS that can read a directory and return its DirEntry's. Should match the behavior of os.ReadDir().

type ReadFileFS 

type ReadFileFS interface {
	FS
	ReadFile(name string) ([]byte, error)
}

ReadFileFS is an FS that can read an entire file in one pass. Should match the behavior of os.ReadFile().

type ReadWriterFile 

type ReadWriterFile interface {
	File
	io.Writer
}

ReadWriterFile is a File that supports Write() operations.

type ReaderAtFile 

type ReaderAtFile interface {
	File
	io.ReaderAt
}

ReaderAtFile is a File that supports ReadAt() operations.

type RemoveAllFS 

type RemoveAllFS interface {
	FS
	RemoveAll(name string) error
}

RemoveAllFS is an FS that can remove files or directories recursively. Should match the behavior of os.RemoveAll().

type RemoveFS 

type RemoveFS interface {
	FS
	Remove(name string) error
}

RemoveFS is an FS that can remove files or empty directories. Should match the behavior of os.Remove().

type RenameFS 

type RenameFS interface {
	FS
	Rename(oldname, newname string) error
}

RenameFS is an FS that can move files or directories. Should match the behavior of os.Rename().

type SeekerFile 

type SeekerFile interface {
	File
	io.Seeker
}

SeekerFile is a File that supports Seek() operations.

type StatFS 

type StatFS interface {
	FS
	Stat(name string) (FileInfo, error)
}

StatFS is an FS that can stat files or directories. Should match the behavior of os.Stat().

type SubFS 

type SubFS interface {
	FS
	Sub(dir string) (FS, error)
}

SubFS is an FS that can return a subset of the current FS. The same effect as `chroot` in a program.

type SymlinkFS 

type SymlinkFS interface {
	FS
	Symlink(oldname, newname string) error
}

SymlinkFS is an FS that can create symlinks. Should match the behavior of os.Symlink().

type SyncerFile 

type SyncerFile interface {
	File
	Sync() error
}

SyncerFile is a File that supports Sync() operations.

type TruncaterFile 

type TruncaterFile interface {
	File
	Truncate(size int64) error
}

TruncaterFile is a File that supports Truncate() operations.

type WalkDirFunc 

type WalkDirFunc = gofs.WalkDirFunc

WalkDirFunc is the type of function called in WalkDir().

type WriteFileFS added in v0.1.9 

type WriteFileFS interface {
	FS
	WriteFile(name string, data []byte, perm FileMode) error
}

WriteFileFS is an FS that can write an entire file in one pass. Should match the behavior of os.WriteFile().

type WriterAtFile 

type WriterAtFile interface {
	File
	io.WriterAt
}

WriterAtFile is a File that supports WriteAt() operations.

Source Files

View all Source files

Directories

Path Synopsis
Package cache contains a read-only cache file system.
 Package cache contains a read-only cache file system.
examples module
Package fstest runs test suites against a target FS.
 Package fstest runs test suites against a target FS.
Package indexeddb contains a WebAssembly compatible file system.
 Package indexeddb contains a WebAssembly compatible file system.
idbblob
Package idbblob contains a JavaScript implementation of blob.Blob.
 Package idbblob contains a JavaScript implementation of blob.Blob.
internal
assert
Package assert contains common assertions on builtin types.
 Package assert contains common assertions on builtin types.
fserrors
Package fserrors contains error wrappers.
 Package fserrors contains error wrappers.
jswrapper
Package jswrapper contains a Wrapper for interoperating with JavaScript.
 Package jswrapper contains a Wrapper for interoperating with JavaScript.
mounttest
Package mounttest contains mount.FS wrappers for more thorough testing.
 Package mounttest contains mount.FS wrappers for more thorough testing.
pathlock
Package pathlock contains Mutex, which locks and unlocks using file paths as keys.
 Package pathlock contains Mutex, which locks and unlocks using file paths as keys.
Package keyvalue contains a key-value based FS for easy, custom FS implementations.
 Package keyvalue contains a key-value based FS for easy, custom FS implementations.
blob
Package blob defines a common data interchange type for keyvalue FS's.
 Package blob defines a common data interchange type for keyvalue FS's.
Package mem contains an in-memory FS.
 Package mem contains an in-memory FS.
Package mount contains an implementation of hackpadfs.MountFS.
 Package mount contains an implementation of hackpadfs.MountFS.
Package os implements all of the familiar behavior from the standard library using hackpadfs's interfaces.
 Package os implements all of the familiar behavior from the standard library using hackpadfs's interfaces.
Package tar contains a tar file based file system.
 Package tar contains a tar file based file system.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.