Documentation
¶
Overview ¶
Package fs implements a very simple and limited file system on top of etcd.
Index ¶
- Constants
- Variables
- func NewEtcdFs(client interfaces.Client, metadata string) afero.Fs
- func PathSplit(p string) []string
- type File
- func (obj *File) Close() error
- func (obj *File) Name() string
- func (obj *File) Read(b []byte) (n int, err error)
- func (obj *File) ReadAt(b []byte, off int64) (n int, err error)
- func (obj *File) Readdir(count int) ([]os.FileInfo, error)
- func (obj *File) Readdirnames(n int) (names []string, _ error)
- func (obj *File) Seek(offset int64, whence int) (int64, error)
- func (obj *File) Stat() (os.FileInfo, error)
- func (obj *File) Sync() error
- func (obj *File) Truncate(size int64) error
- func (obj *File) Write(b []byte) (n int, err error)
- func (obj *File) WriteAt(b []byte, off int64) (n int, err error)
- func (obj *File) WriteString(s string) (n int, err error)
- type FileInfo
- type Fs
- func (obj *Fs) Chmod(name string, mode os.FileMode) error
- func (obj *Fs) Chown(name string, uid, gid int) error
- func (obj *Fs) Chtimes(name string, atime time.Time, mtime time.Time) error
- func (obj *Fs) Create(name string) (afero.File, error)
- func (obj *Fs) Lchown(name string, uid, gid int) error
- func (obj *Fs) Lstat(name string) (os.FileInfo, error)
- func (obj *Fs) Mkdir(name string, perm os.FileMode) error
- func (obj *Fs) MkdirAll(path string, perm os.FileMode) error
- func (obj *Fs) Name() string
- func (obj *Fs) Open(name string) (afero.File, error)
- func (obj *Fs) OpenFile(name string, flag int, perm os.FileMode) (afero.File, error)
- func (obj *Fs) ReadDir(dirname string) ([]os.FileInfo, error)
- func (obj *Fs) ReadFile(filename string) ([]byte, error)
- func (obj *Fs) Remove(name string) error
- func (obj *Fs) RemoveAll(path string) error
- func (obj *Fs) Rename(oldname, newname string) error
- func (obj *Fs) Stat(name string) (os.FileInfo, error)
- func (obj *Fs) TempDir(dir, prefix string) (name string, err error)
- func (obj *Fs) TempFile(dir, prefix string) (f afero.File, err error)
- func (obj *Fs) URI() string
- func (obj *Fs) Walk(root string, walkFn filepath.WalkFunc) error
- func (obj *Fs) WriteFile(filename string, data []byte, perm os.FileMode) error
Constants ¶
const ( // Scheme is the unique name for this filesystem type. Scheme = "etcdfs" // EtcdTimeout is the timeout to wait before erroring. EtcdTimeout = 5 * time.Second // FIXME: chosen arbitrarily // DefaultDataPrefix is the default path for data storage in etcd. DefaultDataPrefix = "/_etcdfs/data" // DefaultHash is the default hashing algorithm to use. DefaultHash = "sha256" // PathSeparator is the path separator to use on this filesystem. PathSeparator = os.PathSeparator // usually the slash character )
Variables ¶
var ( IsPathSeparator = os.IsPathSeparator // ErrNotImplemented is returned when something is not implemented by design. ErrNotImplemented = errors.New("not implemented") // ErrExist is returned when requested path already exists. ErrExist = os.ErrExist // ErrNotExist is returned when we can't find the requested path. ErrNotExist = os.ErrNotExist ErrFileClosed = errors.New("file is closed") ErrFileReadOnly = errors.New("file handle is read only") ErrOutOfRange = errors.New("out of range") )
Functions ¶
Types ¶
type File ¶
type File struct {
Path string // relative path to file, trailing slash if it's a directory
Mode os.FileMode
ModTime time.Time
Children []*File // dir's use this
Hash string // string not []byte so it's readable, matches data
// contains filtered or unexported fields
}
File represents a file node. This is the node of our tree structure. This is not thread safe, and you can have at most one open file handle at a time.
func (*File) Read ¶
Read reads up to len(b) bytes from the File. It returns the number of bytes read and any error encountered. At end of file, Read returns 0, io.EOF. NOTE: This reads into the byte input. It's a side effect!
func (*File) ReadAt ¶
ReadAt reads len(b) bytes from the File starting at byte offset off. It returns the number of bytes read and the error, if any. ReadAt always returns a non-nil error when n < len(b). At end of file, that error is io.EOF.
func (*File) Readdir ¶
Readdir lists the contents of the directory and returns a list of file info objects for each entry.
func (*File) Readdirnames ¶
Readdirnames returns a list of name is the current file handle's directory. TODO: this implementation shares the dirCursor with Readdir, is this okay? TODO: should Readdirnames even use a dirCursor at all?
func (*File) Seek ¶
Seek sets the offset for the next Read or Write on file to offset, interpreted according to whence: 0 means relative to the origin of the file, 1 means relative to the current offset, and 2 means relative to the end. It returns the new offset and an error, if any. The behavior of Seek on a file opened with O_APPEND is not specified.
func (*File) Sync ¶
Sync flushes the file contents to the server and calls the filesystem metadata sync as well. FIXME: instead of a txn, run a get and then a put in two separate stages. if the get already found the data up there, then we don't need to push it all in the put phase. with the txn it is always all sent up even if the put is never needed. the get should just be a "key exists" test, and not a download of the whole file. if we *do* do the download, we can byte-by-byte check for hash collisions and panic if we find one :)
func (*File) Truncate ¶
Truncate trims the file to the requested size. Since our file system can only read and write data, but never edit existing data blocks, doing this will not cause more space to be available.
type FileInfo ¶
type FileInfo struct {
// contains filtered or unexported fields
}
FileInfo is a struct which provides some information about a file handle.
type Fs ¶
type Fs struct {
Client interfaces.Client
Metadata string // location of "superblock" for this filesystem
DataPrefix string // prefix of data storage (no trailing slashes)
Hash string // eg: sha256
Debug bool
Logf func(format string, v ...interface{})
// contains filtered or unexported fields
}
Fs is a specialized afero.Fs implementation for etcd. It implements a small subset of the features, and has some special properties. In particular, file data is stored with it's unique reference being a hash of the data. In this way, you cannot actually edit a file, but rather you create a new one, and update the metadata pointer to point to the new blob. This might seem slow, but it has the unique advantage of being relatively straight forward to implement, and repeated uploads of the same file cost almost nothing. Since etcd isn't meant for large file systems, this fits the desired use case. This implementation is designed to have a single writer for each superblock, but as many readers as you like. FIXME: this is not currently thread-safe, nor is it clear if it needs to be. XXX: we probably aren't updating the modification time everywhere we should! XXX: because we never delete data blocks, we need to occasionally "vacuum". XXX: this is harder because we need to list of *all* metadata paths, if we want them to be able to share storage backends. (we do)
func (*Fs) Chtimes ¶
Chtimes changes the access and modification times of the named file, similar to the Unix utime() or utimes() functions. The underlying filesystem may truncate or round the values to a less precise time unit. If there is an error, it will be of type *PathError. FIXME: make sure everything we error is a *PathError TODO: atime is not currently implement and so it is silently ignored.
func (*Fs) Lstat ¶
Lstat does exactly the same as Stat because we currently do not support symbolic links.
func (*Fs) MkdirAll ¶
MkdirAll creates a directory named path, along with any necessary parents, and returns nil, or else returns an error. The permission bits perm are used for all directories that MkdirAll creates. If path is already a directory, MkdirAll does nothing and returns nil.
func (*Fs) ReadDir ¶
ReadDir reads the directory named by dirname and returns a list of sorted directory entries.
func (*Fs) ReadFile ¶
ReadFile reads the file named by filename and returns the contents. A successful call returns err == nil, not err == EOF. Because ReadFile reads the whole file, it does not treat an EOF from Read as an error to be reported.
func (*Fs) RemoveAll ¶
RemoveAll removes path and any children it contains. It removes everything it can but returns the first error it encounters. If the path does not exist, RemoveAll returns nil (no error).
func (*Fs) Rename ¶
Rename moves or renames a file or directory. TODO: seems it's okay to move files or directories, but you can't clobber dirs but you can clobber single files. a dir can't clobber a file and a file can't clobber a dir. but a file can clobber another file but a dir can't clobber another dir. you can also transplant dirs or files into other dirs.
func (*Fs) TempDir ¶
TempDir creates a new temporary directory in the directory dir with a name beginning with prefix and returns the path of the new directory. If dir is the empty string, TempDir uses the default directory for temporary files (see os.TempDir). Multiple programs calling TempDir simultaneously will not choose the same directory. It is the caller's responsibility to remove the directory when no longer needed.
func (*Fs) TempFile ¶
TempFile creates a new temporary file in the directory dir with a name beginning with prefix, opens the file for reading and writing, and returns the resulting *File. If dir is the empty string, TempFile uses the default directory for temporary files (see os.TempDir). Multiple programs calling TempFile simultaneously will not choose the same file. The caller can use f.Name() to find the pathname of the file. It is the caller's responsibility to remove the file when no longer needed.
func (*Fs) Walk ¶
Walk walks the file tree rooted at root, calling walkFn for each file or directory in the tree, including root. All errors that arise visiting files and directories are filtered by walkFn. The files are walked in lexical order, which makes the output deterministic but means that for very large directories Walk can be inefficient. Walk does not follow symbolic links.
Source Files