muxfys

package module
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 11, 2017 License: GPL-3.0 Imports: 27 Imported by: 0

README

muxfys

GoDoc Go Report Card Build Status Coverage Status

muxfys is a pure Go library for temporarily in-process mounting multiple different remote file systems or object stores on to the same mount point as a "filey" system. Currently only support for S3-like systems has been implemented.

It has high performance, and is easy to use with nothing else to install, and no root permissions needed (except to initially install/configure fuse: on old linux you may need to install fuse-utils, and for macOS you'll need to install osxfuse; for both you must ensure that 'user_allow_other' is set in /etc/fuse.conf or equivalent).

It has good S3 compatibility, working with AWS Signature Version 4 (Amazon S3, Minio, et al.) and AWS Signature Version 2 (Google Cloud Storage, Openstack Swift, Ceph Object Gateway, Riak CS, et al).

It allows "multiplexing": you can mount multiple different S3 buckets (or sub directories of the same bucket) on the same local directory. This makes commands you want to run against the files in your buckets much simpler, eg. instead of mounting s3://publicbucket, s3://myinputbucket and s3://myoutputbucket to separate mount points and running:

$ myexe -ref /mnt/publicbucket/refs/human/ref.fa -i /mnt/myinbucket/xyz/123/
  input.file > /mnt/myoutputbucket/xyz/123/output.file

You could multiplex the 3 buckets (at the desired paths) on to the directory you will work from and just run:

$ myexe -ref ref.fa -i input.file > output.file

It is a "filey" system ('fys' instead of 'fs') in that it cares about performance and efficiency first, and POSIX second. It is designed around a particular use-case:

Non-interactively read a small handful of files who's paths you already know, probably a few times for small files and only once for large files, then upload a few large files. Eg. we want to mount S3 buckets that contain thousands of unchanging cache files, and a few big input files that we process using those cache files, and finally generate some results.

In particular this means we hold on to directory and file attributes forever and assume they don't change externally. Permissions are ignored and only you get read/write access.

When using muxfys, you 1) mount, 2) do something that needs the files in your S3 bucket(s), 3) unmount. Then repeat 1-3 for other things that need data in your S3 buckets.

Performance

To get a basic sense of performance, a 1GB file in a Ceph Object Gateway S3 bucket was read, twice in a row for tools with caching, using the methods that worked for me (I had to hack minfs to get it to work); units are seconds (average of 3 attempts) needed to read the whole file:

| method         | fresh | cached |
|----------------|-------|--------|
| s3cmd          | 5.9   | n/a    |
| mc             | 7.9   | n/a    |
| minfs          | 40    | n/a    |
| s3fs           | 12.1  | n/a    |
| s3fs caching   | 12.2  | 1.0    |
| muxfys         | 5.7   | n/a    |
| muxfys caching | 5.8   | 0.7    |

Ie. minfs is very slow, and muxfys is about 2x faster than s3fs, with no noticeable performance penalty for fuse mounting vs simply downloading the files you need to local disk. (You also get the benefit of being able to seek and read only small parts of the remote file, without having to download the whole thing.)

The same story holds true when performing the above test 100 times ~simultaneously; while some reads take much longer due to Ceph/network overload, muxfys remains on average twice as fast as s3fs. The only significant change is that s3cmd starts to fail.

For a real-world test, some data processing and analysis was done with samtools, a tool that can end up reading small parts of very large files. www.htslib.org/workflow was partially followed to map fastqs with 441 read pairs (extracted from an old human chr20 mapping). Mapping, sorting and calling was carried out, in addition to creating and viewing a cram. The different caching strategies used were: cup == reference-related files cached, fastq files uncached, working in a normal POSIX directory; cuf == as cup, but working in a fuse mounted writable directory; uuf == as cuf, but with no caching for the reference-related files. The local(mc) method involved downloading all files with mc first, with the cached result being the maximum possible performance: that of running bwa and samtools when all required files are accessed from the local POSIX filesystem. Units are seconds (average of 3 attempts):

| method     | fresh | cached |
|------------|-------|--------|
| local(mc)  | 157   | 40     |
| s3fs.cup   | 175   | 50     |
| muxfys.cup | 80    | 45     |
| muxfys.cuf | 79    | 44     |
| muxfys.uuf | 88    | n/a    |

Ie. muxfys is about 2x faster than just downloading all required files manually, and over 2x faster than using s3fs. There isn't much performance loss when the data is cached vs maximum possible performance. There's no noticeable penalty (indeed it's a little faster) for working directly in a muxfys-mounted directory.

Finally, to compare to a highly optimised tool written in C that has built-in support (via libcurl) for reading from S3, samtools was once again used, this time to read 100bp (the equivalent of a few lines) from an 8GB indexed cram file. The builtin(mc) method involved downloading the single required cram cache file from S3 first using mc, then relying on samtools' built-in S3 support by giving it the s3:// path to the cram file; the cached result involves samtools reading this cache file and the cram's index files from the local POSIX filesystem, but it still reads cram data itself from the remote S3 system. The other methods used samtools normally, giving it paths within the fuse mount(s) created. The different caching strategies used were: cu == reference-related files cached, cram-related files uncached; cc == everything cached; uu == nothing cached. Units are seconds (average of 3 attempts):

| method      | fresh | cached |
|-------------|-------|--------|
| builtin(mc) | 1.3   | 0.5    |
| s3fs.cu     | 4.3   | 1.7    |
| s3fs.cc     | 4.4   | 0.5    |
| s3fs.uu     | 4.4   | 2.2    |
| muxfys.cu   | 0.3   | 0.1    |
| muxfys.cc   | 0.3   | 0.06   |
| muxfys.uu   | 0.3   | 0.1    |

Ie. muxfys is much faster than s3fs (more than 2x faster probably due to much faster and more efficient stating of files), and using it also gives a significant benefit over using a tools' built-in support for S3.

Status & Limitations

The only "Remote" type implemented so far is for S3-like systems.

In cached mode, random reads and writes have been implemented.

In non-cached mode, only random reads have been implemented so far.

Coming soon: serial writes in non-cached mode.

Non-POSIX behaviours:

  • does not store file mode/owner/group
  • does not support hardlinks
  • symlinks are only supported temporarily in a cached writeable mount: they can be created and used, but do not get uploaded
  • atime (and typically ctime) is always the same as mtime
  • mtime of files is not stored remotely (remote file mtimes are of their upload time, and muxfys only guarantees that files are uploaded in the order of their mtimes)
  • does not upload empty directories, can't rename remote directories
  • fsync is ignored, files are only flushed on close

Guidance

CacheData: true will usually give you the best performance. Not setting an explicit CacheDir will also give the best performance, as if you read a small part of a large file, only the part you read will be downloaded and cached in the unique CacheDir.

Only turn on Write mode if you have to write.

Use CacheData: false if you will read more data than can be stored on local disk.

If you know that you will definitely end up reading the same data multiple times (either during a mount, or from different mounts) on the same machine, and have sufficient local disk space, use CacheData: true and set an explicit CacheDir (with a constant absolute path, eg. starting in /tmp). Doing this results in any file read downloading the whole remote file to cache it, which can be wasteful if you only need to read a small part of a large file. (But this is the only way that muxfys can coordinate the cache amongst independent processes.)

Usage

import "github.com/VertebrateResequencing/wr/muxfys"

// fully manual target configuration
target1 := &muxfys.Target{
    Target:     "https://s3.amazonaws.com/mybucket/subdir",
    Region:     "us-east-1",
    AccessKey:  os.Getenv("AWS_ACCESS_KEY_ID"),
    SecretKey:  os.Getenv("AWS_SECRET_ACCESS_KEY"),
    CacheDir:   "/tmp/muxfys/cache",
    Write:      true,
}

// or read some configuration from standard AWS S3 config files and
// environment variables
target2 := &muxfys.Target{
    CacheData: true,
}
target2.ReadEnvironment("default", "myotherbucket/another/subdir")

cfg := &muxfys.Config{
    Mount: "/tmp/muxfys/mount",
    CacheBase: "/tmp",
    Retries:    3,
    Verbose:    true,
    Targets:    []*muxfys.Target{target, target2},
}

fs, err := muxfys.New(cfg)
if err != nil {
    log.Fatalf("bad configuration: %s\n", err)
}

err = fs.Mount()
if err != nil {
    log.Fatalf("could not mount: %s\n", err)
}
fs.UnmountOnDeath()

// read from & write to files in /tmp/muxfys/mount, which contains the
// contents of mybucket/subdir and myotherbucket/another/subdir; writes will
// get uploaded to mybucket/subdir when you Unmount()

err = fs.Unmount()
if err != nil {
    log.Fatalf("could not unmount: %s\n", err)
}

logs := fs.Logs()

Provenance

There are many ways of accessing data in S3 buckets. Common tools include s3cmd for direct up/download of particular files, and s3fs for fuse-mounting a bucket. But these are not written in Go.

Amazon provide aws-sdk-go for interacting with S3, but this does not work with (my) Ceph Object Gateway and possibly other implementations of S3.

minio-go is an alternative Go library that provides good compatibility with a wide variety of S3-like systems.

There are at least 3 Go libraries for creating fuse-mounted file-systems. github.com/jacobsa/fuse was based on bazil.org/fuse, claiming higher performance. Also claiming high performance is github.com/hanwen/go-fuse.

There are at least 2 projects that implement fuse-mounting of S3 buckets:

  • github.com/minio/minfs is implemented using minio-go and bazil, but in my hands was very slow. It is designed to be run as root, requiring file-based configuration.
  • github.com/kahing/goofys is implemented using aws-sdk-go and jacobsa/fuse, making it incompatible with (my) Ceph Object Gateway.

Both are designed to be run as daemons as opposed to being used in-process.

muxfys is implemented using minio-go for compatibility, and hanwen/go-fuse for speed. (In my testing, hanwen/go-fuse and jacobsa/fuse did not have noticeably difference performance characteristics, but go-fuse was easier to write for.) However, some of its read code is inspired by goofys. Thanks to minimising remote calls to the remote S3 system, and only implementing what S3 is generally capable of, it shares and adds to goofys' non-POSIX behaviours.

Documentation

Overview

Package muxfys is a pure Go library that lets you in-process temporarily fuse-mount remote file systems or object stores as a "filey" system. Currently only support for S3-like systems has been implemented.

It has high performance, and is easy to use with nothing else to install, and no root permissions needed (except to initially install/configure fuse: on old linux you may need to install fuse-utils, and for macOS you'll need to install osxfuse; for both you must ensure that 'user_allow_other' is set in /etc/fuse.conf or equivalent).

It allows "multiplexing": you can mount multiple different buckets (or sub directories of the same bucket) on the same local directory. This makes commands you want to run against the files in your buckets much simpler, eg. instead of mounting s3://publicbucket, s3://myinputbucket and s3://myoutputbucket to separate mount points and running: $ myexe -ref /mnt/publicbucket/refs/human/ref.fa -i /mnt/myinputbucket/xyz/123/ 

input.file > /mnt/myoutputbucket/xyz/123/output.file

You could multiplex the 3 buckets (at the desired paths) on to the directory you will work from and just run: $ myexe -ref ref.fa -i input.file > output.file

When using muxfys, you 1) mount, 2) do something that needs the files in your S3 bucket(s), 3) unmount. Then repeat 1-3 for other things that need data in your S3 buckets.

Usage 

import "github.com/VertebrateResequencing/wr/muxfys"

// fully manual target configuration
target1 := &muxfys.Target{
    Target:     "https://s3.amazonaws.com/mybucket/subdir",
    Region:     "us-east-1",
    AccessKey:  os.Getenv("AWS_ACCESS_KEY_ID"),
    SecretKey:  os.Getenv("AWS_SECRET_ACCESS_KEY"),
    CacheDir:   "/tmp/muxfys/cache",
    Write:      true,
}

// or read some configuration from standard AWS S3 config files and
// environment variables
target2 := &muxfys.Target{
    CacheData: true,
}
target2.ReadEnvironment("default", "myotherbucket/another/subdir")

cfg := &muxfys.Config{
    Mount: "/tmp/muxfys/mount",
    CacheBase: "/tmp",
    Retries:    3,
    Verbose:    true,
    Targets:    []*muxfys.Target{target, target2},
}

fs, err := muxfys.New(cfg)
if err != nil {
    log.Fatalf("bad configuration: %s\n", err)
}

err = fs.Mount()
if err != nil {
    log.Fatalf("could not mount: %s\n", err)
}
fs.UnmountOnDeath()

// read from & write to files in /tmp/muxfys/mount, which contains the
// contents of mybucket/subdir and myotherbucket/another/subdir; writes will
// get uploaded to mybucket/subdir when you Unmount()

err = fs.Unmount()
if err != nil {
    log.Fatalf("could not unmount: %s\n", err)
}

logs := fs.Logs()

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func NewCachedFile 

func NewCachedFile(r *remote, remotePath, localPath string, attr *fuse.Attr, flags uint32, logger log15.Logger) nodefs.File

NewCachedFile makes a CachedFile that reads each byte from remotePath only once, returning subsequent reads from and writing to localPath.

func NewS3File 

func NewS3File(r *remote, path string, size uint64) nodefs.File

NewS3File creates a new S3File. For all the methods not yet implemented, fuse will get a not yet implemented error.

func SetLogHandler 

func SetLogHandler(h log15.Handler)

SetLogHandler defines how log messages (globally for this package) are logged. Logs are always retrievable as strings from individual MuxFys instances using MuxFys.Logs(), but otherwise by default are discarded. To have them logged somewhere as they are emitted, supply a github.com/inconshreveable/log15 Handler, eg. log15.StderrHandler to log everything to STDERR.

Types

type CacheTracker 

type CacheTracker struct {
	sync.Mutex
	// contains filtered or unexported fields
}

CacheTracker struct is used to track what parts of which files have been cached.

func NewCacheTracker 

func NewCacheTracker() *CacheTracker

NewCacheTracker creates a new *CacheTracker.

func (*CacheTracker) CacheDelete 

func (c *CacheTracker) CacheDelete(path string)

CacheDelete should be used if you delete a cache file.

func (*CacheTracker) CacheOverride 

func (c *CacheTracker) CacheOverride(path string, iv Interval)

CacheOverride should be used if you do something like delete a cache file and then recreate it and cache some data inside it. This is the slightly more efficient alternative to calling Delete(path) followed by Cached(path, iv).

func (*CacheTracker) CacheRename 

func (c *CacheTracker) CacheRename(oldPath, newPath string)

CacheRename should be used if you rename a cache file on disk.

func (*CacheTracker) CacheTruncate 

func (c *CacheTracker) CacheTruncate(path string, offset int64)

CacheTruncate should be used to update the tracker if you truncate a cache file. The internal knowledge of what you have cached for that file will then be updated to exclude anything beyond the truncation point.

func (*CacheTracker) CacheWipe 

func (c *CacheTracker) CacheWipe()

CacheWipe should be used if you delete all your cache files.

func (*CacheTracker) Cached 

func (c *CacheTracker) Cached(path string, iv Interval)

Cached updates the tracker with what you have now cached. Once you have stored bytes 0..9 in /abs/path/to/sparse.file, you would call: Cached("/abs/path/to/sparse.file", NewInterval(0, 10)).

func (*CacheTracker) Uncached 

func (c *CacheTracker) Uncached(path string, iv Interval) Intervals

Uncached tells you what parts of a file in the given interval you haven't already cached (based on your prior Cached() calls). You would want to then cache the data in each of the returned intervals and call Cached() on each one afterwards.

type CachedFile 

type CachedFile struct {
	nodefs.File

	log15.Logger
	// contains filtered or unexported fields
}

CachedFile is used as a wrapper around a nodefs.loopbackFile, the only difference being that on Write it updates the given attr's Size, Mtime and Atime, and on Read it copies data from remote to local disk if not requested before.

func (*CachedFile) InnerFile 

func (f *CachedFile) InnerFile() nodefs.File

InnerFile returns the loopbackFile that deals with local files on disk.

func (*CachedFile) Read 

func (f *CachedFile) Read(buf []byte, offset int64) (fuse.ReadResult, fuse.Status)

Read checks to see if we've previously stored these bytes in our local cached file, and if so just defers to our InnerFile(). If not, gets the data from the remote object and stores it in the cache file.

func (*CachedFile) Utimens 

func (f *CachedFile) Utimens(Atime *time.Time, Mtime *time.Time) (status fuse.Status)

Utimens gets called by things like `touch -d "2006-01-02 15:04:05" filename`, and we need to update our cached attr as well as the local file.

func (*CachedFile) Write 

func (f *CachedFile) Write(data []byte, offset int64) (uint32, fuse.Status)

Write passes the real work to our InnerFile(), also updating our cached attr.

type Config 

type Config struct {
	// Mount is the local directory to mount on top of (muxfys will try to
	// create this if it doesn't exist). If not supplied, defaults to the
	// subdirectory "mnt" in the current working directory. Note that mounting
	// will only succeed if the Mount directory either doesn't exist or is
	// empty.
	Mount string

	// Retries is the number of times to automatically retry failed remote S3
	// system requests. The default of 0 means don't retry; at least 3 is
	// recommended.
	Retries int

	// CacheBase is the base directory that will be used to create any Target
	// cache directories, when those Targets have CacheData true but CacheDir
	// undefined. Defaults to the current working directory.
	CacheBase string

	// Verbose results in every remote request getting an entry in the output of
	// Logs(). Errors always appear there.
	Verbose bool

	// Targets is a slice of Target, describing what you want to mount and
	// allowing you to multiplex more than one bucket/ sub directory on to
	// Mount. Only 1 of these Target can be writeable.
	Targets []*Target
}

Config struct provides the configuration of a MuxFys.

type Interval 

type Interval struct {
	Start int64
	End   int64
}

Interval struct is used to describe something with a start and end. End must be greater than start.

func NewInterval 

func NewInterval(start, length int64) Interval

NewInterval is a convenience for creating a new Interval when you have a length instead of an end.

func (*Interval) Length 

func (i *Interval) Length() int64

Length returns the length of this interval.

func (*Interval) Merge 

func (i *Interval) Merge(j Interval) bool

Merge merges the supplied interval with this interval if they overlap or are adjacent. Returns true if a merge actually occurred.

func (*Interval) Overlaps 

func (i *Interval) Overlaps(j Interval) bool

Overlaps returns true if this interval overlaps with the supplied one.

func (*Interval) OverlapsOrAdjacent 

func (i *Interval) OverlapsOrAdjacent(j Interval) bool

OverlapsOrAdjacent returns true if this interval overlaps with or is adjacent to the supplied one.

type Intervals 

type Intervals []Interval

Intervals type is a slice of Interval.

func (Intervals) Difference 

func (ivs Intervals) Difference(iv Interval) Intervals

Difference returns any portions of iv that do not overlap with any of our intervals. Assumes that all of our intervals have been Merge()d in.

func (Intervals) Merge 

func (ivs Intervals) Merge(iv Interval) Intervals

Merge adds another interval to this slice of intervals, merging with any prior intervals if it overlaps with or is adjacent to them. Returns the new slice of intervals, which have the property of not overlapping with or being adjacent to each other. They are also sorted by Start if Merge() was used to add all of them.

func (Intervals) Truncate 

func (ivs Intervals) Truncate(pos int64) Intervals

Truncate removes all intervals that start after the given position, and truncates any intervals that overlap with the position. Assumes that all of our intervals have been Merge()d in.

type MuxFys 

type MuxFys struct {
	pathfs.FileSystem

	log15.Logger
	// contains filtered or unexported fields
}

MuxFys struct is the main filey system object.

func New 

func New(config *Config) (fs *MuxFys, err error)

New returns a MuxFys that you'll use to Mount() your S3 bucket(s), ensure you un-mount if killed by calling UnmountOnDeath(), then Unmount() when you're done. You might check Logs() afterwards. The other methods of MuxFys can be ignored in most cases.

func (*MuxFys) Access 

func (fs *MuxFys) Access(name string, mode uint32, context *fuse.Context) fuse.Status

Access is ignored.

func (*MuxFys) Chmod 

func (fs *MuxFys) Chmod(name string, mode uint32, context *fuse.Context) fuse.Status

Chmod is ignored.

func (*MuxFys) Chown 

func (fs *MuxFys) Chown(name string, uid uint32, gid uint32, context *fuse.Context) fuse.Status

Chown is ignored.

func (*MuxFys) Create 

func (fs *MuxFys) Create(name string, flags uint32, mode uint32, context *fuse.Context) (nodefs.File, fuse.Status)

Create creates a new file. mode and context are not currently used. Only currently implemented for when configured with CacheData; the contents of the created file are only uploaded at Unmount() time.

func (*MuxFys) GetAttr 

func (fs *MuxFys) GetAttr(name string, context *fuse.Context) (attr *fuse.Attr, status fuse.Status)

GetAttr finds out about a given object, returning information from a permanent cache if possible. context is not currently used.

func (*MuxFys) Logs 

func (fs *MuxFys) Logs() []string

Logs returns messages generated while mounted; you might call it after Unmount() to see how things went. By default these will only be errors that occurred, but if this MuxFys was configured with Verbose on, it will also contain informational and warning messages. If the muxfys package was configured with a log Handler (see SetLogHandler()), these same messages would have been logged as they occurred.

func (*MuxFys) Mkdir 

func (fs *MuxFys) Mkdir(name string, mode uint32, context *fuse.Context) fuse.Status

Mkdir for a directory that doesn't exist yet only works whilst mounted in CacheData mode. neither mode nor context are currently used.

func (*MuxFys) Mount 

func (fs *MuxFys) Mount() (err error)

Mount carries out the mounting of your configured S3 bucket to your configured mount point. On return, the files in your bucket will be accessible. Once mounted, you can't mount again until you Unmount().

func (*MuxFys) OnMount 

func (fs *MuxFys) OnMount(nodeFs *pathfs.PathNodeFs)

OnMount prepares MuxFys for use once Mount() has been called.

func (*MuxFys) Open 

func (fs *MuxFys) Open(name string, flags uint32, context *fuse.Context) (file nodefs.File, status fuse.Status)

Open is what is called when any request to read a file is made. The file must already have been stat'ed (eg. with a GetAttr() call), or we report the file doesn't exist. context is not currently used. If CacheData has been configured, we defer to openCached(). Otherwise the real implementation is in S3File.

func (*MuxFys) OpenDir 

func (fs *MuxFys) OpenDir(name string, context *fuse.Context) ([]fuse.DirEntry, fuse.Status)

OpenDir gets the contents of the given directory for eg. `ls` purposes. It also caches the attributes of all the files within. context is not currently used. 

func (fs *MuxFys) Readlink(name string, context *fuse.Context) (out string, status fuse.Status)

Readlink returns the destination of a symbolic link that was created with Symlink(). context is not currently used.

func (*MuxFys) Rename 

func (fs *MuxFys) Rename(oldPath string, newPath string, context *fuse.Context) fuse.Status

Rename only works where oldPath is found in the writeable remote. For files, first remotely copies oldPath to newPath (ignoring any local changes to oldPath), renames any local cached (and possibly modified) copy of oldPath to newPath, and finally deletes the remote oldPath; if oldPath had been modified, its changes will only be uploaded to newPath at Unmount() time. For directories, is only capable of renaming directories you have created whilst mounted. context is not currently used.

func (*MuxFys) Rmdir 

func (fs *MuxFys) Rmdir(name string, context *fuse.Context) fuse.Status

Rmdir only works for directories you have created whilst mounted in CacheData mode. context is not currently used.

func (*MuxFys) SetXAttr 

func (fs *MuxFys) SetXAttr(name string, attr string, data []byte, flags int, context *fuse.Context) fuse.Status

SetXAttr is ignored.

func (*MuxFys) StatFs 

func (fs *MuxFys) StatFs(name string) *fuse.StatfsOut

StatFs returns a constant (faked) set of details describing a very large file system. 

func (fs *MuxFys) Symlink(source string, dest string, context *fuse.Context) (status fuse.Status)

Symlink creates a symbolic link. Only implemented for temporary use when configured with CacheData: you can create and use symlinks but they don't get uploaded. context is not currently used.

func (*MuxFys) Truncate 

func (fs *MuxFys) Truncate(name string, offset uint64, context *fuse.Context) fuse.Status

Truncate truncates any local cached copy of the file. Only currently implemented for when configured with CacheData; the results of the Truncate are only uploaded at Unmount() time. If offset is > size of file, does nothing and returns OK. context is not currently used. 

func (fs *MuxFys) Unlink(name string, context *fuse.Context) fuse.Status

Unlink deletes a file from the remote S3 bucket, as well as any locally cached copy. context is not currently used.

func (*MuxFys) Unmount 

func (fs *MuxFys) Unmount(doNotUpload ...bool) (err error)

Unmount must be called when you're done reading from/ writing to your bucket. Be sure to close any open filehandles before hand! It's a good idea to defer this after calling Mount(), and possibly also call UnmountOnDeath(). In CacheData mode, it is only at Unmount() that any files you created or altered get uploaded, so this may take some time. You can optionally supply a bool which if true prevents any uploads. If a target was not configured with a specific CacheDir but CacheData was true, the CacheDir will be deleted.

func (*MuxFys) UnmountOnDeath 

func (fs *MuxFys) UnmountOnDeath()

UnmountOnDeath captures SIGINT (ctrl-c) and SIGTERM (kill) signals, then calls Unmount() before calling os.Exit(1 if the unmount worked, 2 otherwise) to terminate your program. Manually calling Unmount() after this cancels the signal capture. This does NOT block.

func (*MuxFys) Utimens 

func (fs *MuxFys) Utimens(name string, Atime *time.Time, Mtime *time.Time, context *fuse.Context) (status fuse.Status)

Utimens only functions when configured with CacheData and the file is already in the cache; otherwise ignored. This only gets called by direct operations like os.Chtimes() (that don't first Open()/Create() the file). context is not currently used.

type S3File 

type S3File struct {
	nodefs.File
	// contains filtered or unexported fields
}

S3File struct is muxfys' implementation of pathfs.File, providing read/write operations on files in the remote bucket (when we're not doing any local caching of data).

func (*S3File) Fsync 

func (f *S3File) Fsync(flags int) fuse.Status

Fsync always returns OK as opposed to "not imlemented" so that write-sync- write works.

func (*S3File) Read 

func (f *S3File) Read(buf []byte, offset int64) (fuse.ReadResult, fuse.Status)

Read supports random reading of data from the file. This gets called as many times as are needed to get through all the desired data len(buf) bytes at a time.

func (*S3File) Release 

func (f *S3File) Release()

Release makes sure we close our readers.

type Target 

type Target struct {
	// The full URL of your bucket and possible sub-path, eg.
	// https://cog.domain.com/bucket/subpath. For performance reasons, you
	// should specify the deepest subpath that holds all your files. This will
	// be set for you by a call to ReadEnvironment().
	Target string

	// Region is optional if you need to use a specific region. This can be set
	// for you by a call to ReadEnvironment().
	Region string

	// AccessKey and SecretKey can be set for you by calling ReadEnvironment().
	AccessKey string
	SecretKey string

	// CacheData enables caching of remote files that you read locally on disk.
	// Writes will also be staged on local disk prior to upload.
	CacheData bool

	// CacheDir is the directory used to cache data if CacheData is true.
	// (muxfys will try to create this if it doesn't exist). If not supplied
	// when CacheData is true, muxfys will create a unique temporary directory
	// in the CacheBase directory of the containing Config (these get
	// automatically deleted on Unmount() - specified CacheDirs do not).
	// Defining this makes CacheData be treated as true.
	CacheDir string

	// Write enables write operations in the mount. Only set true if you know
	// you really need to write. Since writing currently requires caching of
	// data, CacheData will be treated as true.
	Write bool
}

Target struct provides details of the remote target (S3 bucket) you wish to mount, and particulars about caching and writing for this target.

func (*Target) CreateRemote 

func (t *Target) CreateRemote(cacheBase string, maxAttempts int, logger log15.Logger) (r *remote, err error)

CreateRemote uses the configured details of the Target to create a *remote, used internally by MuxFys.New().

func (*Target) ReadEnvironment 

func (t *Target) ReadEnvironment(profile, path string) error

ReadEnvironment sets Target, AccessKey and SecretKey and possibly Region. It determines these by looking primarily at the given profile section of ~/.s3cfg (s3cmd's config file). If profile is an empty string, it comes from $AWS_DEFAULT_PROFILE or $AWS_PROFILE or defaults to "default". If ~/.s3cfg doesn't exist or isn't fully specified, missing values will be taken from the file pointed to by $AWS_SHARED_CREDENTIALS_FILE, or ~/.aws/credentials (in the AWS CLI format) if that is not set. If this file also doesn't exist, ~/.awssecret (in the format used by s3fs) is used instead. AccessKey and SecretKey values will always preferably come from $AWS_ACCESS_KEY_ID and $AWS_SECRET_ACCESS_KEY respectively, if those are set. If no config file specified host_base, the default domain used is s3.amazonaws.com. Region is set by the $AWS_DEFAULT_REGION environment variable, or if that is not set, by checking the file pointed to by $AWS_CONFIG_FILE (~/.aws/config if unset). To allow the use of a single configuration file, users can create a non- standard file that specifies all relevant options: use_https, host_base, region, access_key (or aws_access_key_id) and secret_key (or aws_secret_access_key) (saved in any of the files except ~/.awssecret). The path argument should at least be the bucket name, but ideally should also specify the deepest subpath that holds all the files that need to be accessed. Because reading from a public s3.amazonaws.com bucket requires no credentials, no error is raised on failure to find any values in the environment when profile is supplied as an empty string.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.