goofys

README

Goofys is a high-performance, POSIX-ish Amazon S3 file system written in Go

Overview

Goofys allows you to mount an S3 bucket as a filey system.

It's a Filey System instead of a File System because goofys strives for performance first and POSIX second. Particularly things that are difficult to support on S3 or would translate into more than one round-trip would either fail (random writes) or faked (no per-file permission). Goofys does not have an on disk data cache (checkout catfs), and consistency model is close-to-open.

Installation

• On Linux, install via pre-built binaries. You may also need to install fuse too if you want to mount it on startup.

• On macOS, install via Homebrew:

$ brew cask install osxfuse
$ brew install goofys

• Or build from source with Go 1.10 or later:

$ export GOPATH=$HOME/work
$ go get github.com/voyvodov/goofys
$ go install github.com/voyvodov/goofys

Usage

$ cat ~/.aws/credentials
[default]
aws_access_key_id = AKID1234567890
aws_secret_access_key = MY-SECRET-KEY
$ $GOPATH/bin/goofys <bucket> <mountpoint>
$ $GOPATH/bin/goofys <bucket:prefix> <mountpoint> # if you only want to mount objects under a prefix

Users can also configure credentials via the AWS CLI or the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

To mount an S3 bucket on startup, make sure the credential is configured for root, and can add this to /etc/fstab:

goofys#bucket   /mnt/mountpoint        fuse     _netdev,allow_other,--file-mode=0666,--dir-mode=0777    0       0

See also: Instruction for Azure Blob Storage, Azure Data Lake Gen1, and Azure Data Lake Gen2.

Got more questions? Check out questions other people asked

Optional FUSE capabilities

Use --async-dio to enable asynchronous direct I/O, --async-reads for asynchronous buffered reads, and --readdirplus to return inode attributes with directory entries. Each option is disabled by default and can be enabled independently:

goofys --async-dio --async-reads --readdirplus bucket /mountpoint

Inode cache eviction

By default, goofys checks once a minute for unreferenced cached inodes that have been idle for five minutes. Set --inode-cache-ttl to change the idle timeout, or use --inode-cache-ttl 0 to disable eviction. This timeout is independent of --stat-cache-ttl and --type-cache-ttl.

Eviction removes cached files and empty directories without deleting objects from storage. Later lookups reload objects that still exist and return a missing-file error for objects deleted outside goofys. Referenced inodes, open files, and directories being listed are protected. An active directory listing also protects its children to preserve listing offsets. Busy entries are retried on later sweeps, so the timeout is not a strict memory limit. Referenced inodes remain until the kernel releases them.

Send kill -HUP <pid> to the mounted goofys process to write a JSON state dump. The log reports the path to goofys-state-*.json in the process's temporary directory (TMPDIR, or usually /tmp). The file is readable only by its owner. It contains inode IDs, parent/child relationships, reference counts, idle timestamps, open handles, eviction settings, and memory counts. The snapshot is best effort: busy inodes are marked busy and have partial details, and entries removed during collection are marked accordingly. SIGHUP keeps the mount running. SIGUSR1 still reports inode counts and requests memory reclamation.

Applications using NewGoofys directly can enable eviction through FlagStorage.InodeCacheTTL and must call Destroy when finished. The FUSE server calls Destroy on unmount; canceling the constructor context also stops the eviction worker.

Benchmark

Using --stat-cache-ttl 1s --type-cache-ttl 1s for goofys -ostat_cache_expire=1 for s3fs to simulate cold runs. Detail for the benchmark can be found in bench.sh. Raw data is available as well. The test was run on an EC2 m5.4xlarge in us-west-2a connected to a bucket in us-west-2. Units are seconds.

To run the benchmark, configure EC2's instance role to be able to write to $TESTBUCKET, and then do:

$ sudo docker run -e BUCKET=$TESTBUCKET -e CACHE=false --rm --privileged --net=host -v /tmp/cache:/tmp/cache voyvodov/goofys-bench
# result will be written to $TESTBUCKET

See also: cached benchmark result and result on Azure.

License

Copyright (C) 2015 - 2019 Ka-Hing Cheung

Licensed under the Apache License, Version 2.0

Current Status

goofys has been tested under Linux and macOS.

List of non-POSIX behaviors/limitations:

• only sequential writes supported
• does not store file mode/owner/group
  • use --(dir|file)-mode or --(uid|gid) options
• does not support symlink or hardlink
• ctime, atime is always the same as mtime
• cannot rename directories with more than 1000 children
• unlink returns success even if file is not present
• fsync is ignored, files are only flushed on close

Running tests

Run make test with Go, Docker, curl, and FUSE installed. The runner starts an isolated rustfs/rustfs:1.0.4 container, waits for readiness, runs the tests, and removes the container when it exits. Set RUSTFS_IMAGE to test another RustFS image.

To use an existing RustFS server instead of starting a container:

TEST_ENDPOINT_URL=http://127.0.0.1:9000 make test

The test credentials default to rustfsadmin / rustfsadmin. Set RUSTFS_ACCESS_KEY and RUSTFS_SECRET_KEY when using different credentials. Use a dedicated test server: the suite creates and deletes buckets and objects. Existing cloud test modes selected with CLOUD, AWS, or GCS remain available.

./test/run-tests.local.sh runs the suite in Docker Compose with FUSE access and requires Docker Compose v2. For the manual parallel-read test, first start RustFS with docker compose -f test/docker-compose.tests.yml up -d --wait rustfs, then run ./test/prepare_parallel_read.sh (requires AWS CLI). Mount its test bucket at mnt with the RustFS endpoint and credentials before running ./test/run_parallel_read.sh. Stop the server afterwards with docker compose -f test/docker-compose.tests.yml down --volumes.

Compatibility with non-AWS S3

goofys has been tested with the following non-AWS S3 providers:

• Amplidata / WD ActiveScale
• Ceph (ex: Digital Ocean Spaces, DreamObjects, gridscale)
• EdgeFS
• EMC Atmos
• Google Cloud Storage
• Minio (limited)
• OpenStack Swift
• RustFS
• S3Proxy
• Scaleway
• Wasabi

Additionally, goofys also works with the following non-S3 object stores:

• Azure Blob Storage
• Azure Data Lake Gen1
• Azure Data Lake Gen2

Integration with meta-fuse-csi-plugin

This repository also contains files from the the meta-fuse-csi-plugin repository with more information being found here. The initial commit to add them is here with subsequent commits refining the process. The purpose of this was to simplify the process described here.

References

• Data is stored on Amazon S3
• Amazon SDK for Go
• Other related fuse filesystems
  • catfs: caching layer that can be used with goofys
  • s3fs: another popular filesystem for S3
  • gcsfuse: filesystem for Google Cloud Storage. Goofys borrowed some skeleton code from this project.
• RustFS is used for local and CI S3 tests
• fuse binding, also used by gcsfuse