rufs

package module
v1.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 26, 2019 License: MIT Imports: 24 Imported by: 0

README

Rufs

Rufs is a remote union filesystem which aims to provide a lightweight and secure solution for distributed file storage. Rufs uses a client-server system where servers expose branches and clients mount one or several branches into a tree structure. The client can overlay branches providing a union view.

Code coverage Go Report Card Go Doc

Features

  • Client-Server model using RPC call over SSL.
  • Single executable for client and server.
  • Communication is secured via a secret token which is never transferred over the network and certificate pinning once a client has connected successfully.
  • Clients can provide a unified view with files from different locations.
  • Default client provides CLI, REST API and a web interface.
  • Branches can be read-only.
  • A read-only version of the file system can be exported via FUSE and mounted.

Getting Started

You can download a pre-compiled package for Windows (win64) or Linux (amd64) here.

The archive contains a single executable which contains the server and client code for Rufs.

You can also pull the latest docker image of Rufs from Dockerhub:

docker pull krotik/rufs

Create an empty directory, change into it and run the following to start the server:

docker run --rm --user $(id -u):$(id -g) -v $PWD:/data -p 9020:9020 krotik/rufs server

This exposes port 9020 from the container on the local machine. All runtime related files are written to the current directory as the current user/group.

Run the client by running:

docker run --rm --network="host" -it -v $PWD:/data --user $(id -u):$(id -g) -v $PWD:/data krotik/rufs client

The client will also use the runtime related files from the current directory.

Tutorial:

To get an idea of what Rufs is about have a look at the tutorial.

REST API:

The terminal uses a REST API to communicate with the backend. The REST API can be browsed using a dynamically generated swagger.json definition (https://localhost:9090/fs/swagger.json). You can browse the API of Rufs's latest version here.

Command line options

The main Rufs executable has two main tools:

Rufs 1.0.0

Usage of ./rufs [tool]

The tools are:

    server    Run as a server
    client    Run as a client

Use ./rufs [tool] --help for more information about a tool.

The most important one is server which starts the file server. The server has several options:

Rufs 1.0.0

Usage of ./rufs server [options]

  -config string
    	Server configuration file (default "rufs.server.json")
  -help
    	Show this help message
  -secret string
    	Secret file containing the secret token (default "rufs.secret")
  -ssl-dir string
    	Directory containing the ssl key.pem and cert.pem files (default "ssl")

The server will automatically create a default config file and
default directories if nothing is specified.

Once the server is started the client tool can be used to interact with the server. The options of the client tool are:

Rufs 1.0.0

Usage of ./rufs client [mapping file]

  -fuse-mount string
    	Mount tree as FUSE filesystem at specified path (read-only)
  -help
    	Show this help message
  -secret string
    	Secret file containing the secret token (default "rufs.secret")
  -ssl-dir string
    	Directory containing the ssl key.pem and cert.pem files (default "ssl")
  -web string
    	Export the tree through a https interface on the specified host:port

The mapping file assignes remote branches to the local tree.
The client tries to load rufs.mapping.json if no mapping file is defined.
It starts empty if no mapping file exists. The mapping file
should have the following json format:

{
  "branches" : [
    {
      "branch"      : <branch name>,
      "rpc"         : <rpc interface>,
      "fingerprint" : <fingerprint>
    },
    ...
  ],
  "tree" : [
    {
      "path"      : <path>,
      "branch"    : <branch name>,
      "writeable" : <writable flag>
    },
    ...
  ]
}

On the console type q to exit and help to get an overview of available commands:

Available commands:
----
branch [branch name] [rpc] [fingerprint] : List all known branches or add a new branch to the tree
cat <file>                               : Read and print the contents of a file
cd [path]                                : Show or change the current directory
checksum [path] [glob]                   : Show a directory listing and file checksums
cp <src file/dir> <dst dir>              : Copy a file or directory
dir [path] [glob]                        : Show a directory listing
get <src file> [dst local file]          : Retrieve a file and store it locally (in the current directory)
help [cmd]                               : Show general or command specific help
mkdir <dir>                              : Create a new directory
mount [path] [branch name] [ro]          : List all mount points or add a new mount point to the tree
ping <branch name> [rpc]                 : Ping a remote branch
put [src local file] [dst file]          : Read a local file and store it
refresh                                  : Refreshes all known branches and reconnect if possible.
ren <file> <newfile>                     : Rename a file or directory
reset [mounts|brances]                   : Remove all mounts or all mounts and all branches
rm <file>                                : Delete a file or directory (* all files; ** all files/recursive)
storeconfig [local file]                 : Store the current tree mapping in a local file
sync <src dir> <dst dir>                 : Make sure dst has the same files and directories as src
tree [path] [glob]                       : Show the listing of a directory and its subdirectories
Configuration

The Rufs client and server use each their own configuration file and require a shared rufs.secret file to be able to talk to each other. The server configuration is called rufs.server.json. After starting the server for the first time it should create a default configuration file. Available configurations are:

Configuration Option Description
BranchName Branch name which the server will export.
EnableReadOnly Export the branch only for read operations.
LocalFolder Local physical folder which is exported.
RPCHost RPC host for communication with clients.
RPCPort RPC port for communication with clients.

Note: It is not (and will never be) possible to access the REST API via HTTP.

Building Rufs

To build Rufs from source you need to have Go installed (go >= 1.12):

Create a directory, change into it and run:

git clone https://devt.de/krotik/rufs/ .

You can build Rufs's executable with:

go build -o rufs ./cli/...

Rufs also has a web interface which should be bundled with the executable. The bundled web interface in web.zip can be attached by running:

./attach_webzip.sh

This assumes that the rufs executable is in the same folder as the script.

Building Rufs as Docker image

Rufs can be build as a secure and compact Docker image.

  • Create a directory, change into it and run:
git clone https://devt.de/krotik/rufs/ .
  • You can now build the Docker image with:
docker build --tag krotik/rufs .

License

Rufs source code is available under the MIT License.

Documentation

Overview

Package rufs contains the main API to Rufs.

Rufs is organized as a collection of branches. Each branch represents a physical file system structure which can be queried and updated by an authorized client.

On the client side one or several branches are organized into a tree. The single branches can overlay each other. For example:

Branch A /foo/A /foo/B /bar/C

Branch B /foo/C /test/D

Tree 1 /myspace => Branch A, Branch B

Accessing tree with: /myspace/foo/A gets file /foo/A from Branch A while /myspace/foo/C gets file /foo/C from Branch B

Write operations go only to branches which are mapped as writing branches and who accept them (i.e. are not set to readonly on the side of the branch).

Index

Constants

View Source 
const (
	ItemOpAction  = "itemop_action"  // ItemOp action
	ItemOpName    = "itemop_name"    // Item name
	ItemOpNewName = "itemop_newname" // New item name
)

ItemOp parameter

View Source 
const (
	ItemOpActRename = "rename" // Rename a file or directory
	ItemOpActDelete = "delete" // Delete a file or directory
	ItemOpActMkDir  = "mkdir"  // Create a directory
)

ItemOp actions

View Source 
const (
	ParamAction    = "a" // Requested action
	ParamPath      = "p" // Path string
	ParamPattern   = "x" // Pattern string
	ParamRecursive = "r" // Recursive flag
	ParamChecksums = "c" // Checksum flag
	ParamOffset    = "o" // Offset parameter
	ParamSize      = "s" // Size parameter
)

Meta parameter

View Source 
const (
	OpDir    = "dir"    // Read the contents of a path
	OpRead   = "read"   // Read the contents of a file
	OpWrite  = "write"  // Read the contents of a file
	OpItemOp = "itemop" // File or directory operation
)

Possible actions

View Source 
const (
	SyncCreateDirectory = "Create directory"
	SyncCopyFile        = "Copy file"
	SyncRemoveDirectory = "Remove directory"
	SyncRemoveFile      = "Remove file"
)

Sync operations

Variables

View Source 
var DefaultReadBufferSize = 1024 * 16

DefaultReadBufferSize is the default size for file reading.

Functions

func DirResultToString 

func DirResultToString(paths []string, infos [][]os.FileInfo) string

DirResultToString formats a given Dir result into a human-readable string.

func IsEOF 

func IsEOF(err error) bool

IsEOF tests if the given error is an EOF error.

func WrapFileInfo 

func WrapFileInfo(path string, i os.FileInfo) os.FileInfo

WrapFileInfo wraps a single os.FileInfo object in a serializable FileInfo.

func WrapFileInfos 

func WrapFileInfos(path string, is []os.FileInfo) []os.FileInfo

WrapFileInfos wraps a list of os.FileInfo objects into a list of serializable FileInfo objects. This function will modify the given list.

Types

type Branch 

type Branch struct {
	// contains filtered or unexported fields
}

Branch models a single exported branch in Rufs.

func NewBranch 

func NewBranch(cfg map[string]interface{}, cert *tls.Certificate) (*Branch, error)

NewBranch returns a new exported branch.

func (*Branch) Dir 

func (b *Branch) Dir(spath string, pattern string, recursive bool, checksums bool) ([]string, [][]os.FileInfo, error)

Dir returns file listings matching a given pattern of one or more directories. The contents of the given path is returned along with checksums if the checksum flag is specified. Optionally, also the contents of all subdirectories can be returned if the recursive flag is set. The return values is a list of traversed directories (platform-agnostic) and their corresponding contents.

func (*Branch) IsReadOnly 

func (b *Branch) IsReadOnly() bool

IsReadOnly returns if this branch is read-only.

func (*Branch) ItemOp 

func (b *Branch) ItemOp(spath string, opdata map[string]string) (bool, error)

ItemOp executes a file or directory specific operation which can either succeed or fail (e.g. rename or delete). Actions and parameters should be given in the opdata map.

func (*Branch) Name 

func (b *Branch) Name() string

Name returns the name of the branch.

func (*Branch) ReadFile 

func (b *Branch) ReadFile(spath string, p []byte, offset int64) (int, error)

ReadFile reads up to len(p) bytes into p from the given offset. It returns the number of bytes read (0 <= n <= len(p)) and any error encountered.

func (*Branch) ReadFileToBuffer 

func (b *Branch) ReadFileToBuffer(spath string, buf io.Writer) error

ReadFileToBuffer reads a complete file into a given buffer which implements io.Writer.

func (*Branch) SSLFingerprint 

func (b *Branch) SSLFingerprint() string

SSLFingerprint returns the SSL fingerprint of the branch.

func (*Branch) Shutdown 

func (b *Branch) Shutdown() error

Shutdown shuts the branch down.

func (*Branch) WriteFile 

func (b *Branch) WriteFile(spath string, p []byte, offset int64) (int, error)

WriteFile writes p into the given file from the given offset. It returns the number of written bytes and any error encountered.

func (*Branch) WriteFileFromBuffer 

func (b *Branch) WriteFileFromBuffer(spath string, buf io.Reader) error

WriteFileFromBuffer writes a complete file from a given buffer which implements io.Reader.

type FileInfo 

type FileInfo struct {
	FiName     string      // Base name
	FiSize     int64       // Size in bytes
	FiMode     os.FileMode // File mode bits
	FiModTime  time.Time   // Modification time
	FiChecksum string      // Checksum of files
	// contains filtered or unexported fields
}

FileInfo implements os.FileInfo in an platform-agnostic way

func (*FileInfo) Checksum 

func (rfi *FileInfo) Checksum() string

Checksum returns the checksum of this file. May be an empty string if it was not calculated.

func (*FileInfo) IsDir 

func (rfi *FileInfo) IsDir() bool

IsDir returns if this is a directory.

func (*FileInfo) ModTime 

func (rfi *FileInfo) ModTime() time.Time

ModTime returns the modification time.

func (*FileInfo) Mode 

func (rfi *FileInfo) Mode() os.FileMode

Mode returns the file mode bits.

func (*FileInfo) Name 

func (rfi *FileInfo) Name() string

Name returns the base name.

func (*FileInfo) Size 

func (rfi *FileInfo) Size() int64

Size returns the length in bytes.

func (*FileInfo) String 

func (rfi *FileInfo) String() string

func (*FileInfo) Sys 

func (rfi *FileInfo) Sys() interface{}

Sys should return the underlying data source but will always return nil for FileInfo nodes.

type Tree 

type Tree struct {
	// contains filtered or unexported fields
}

Tree models a Rufs client which combines several branches.

func NewTree 

func NewTree(cfg map[string]interface{}, cert *tls.Certificate) (*Tree, error)

NewTree creates a new tree.

func (*Tree) ActiveBranches 

func (t *Tree) ActiveBranches() ([]string, []string)

ActiveBranches returns a list of all known active branches and their fingerprints.

func (*Tree) AddBranch 

func (t *Tree) AddBranch(branchName string, branchRPC string, branchFingerprint string) error

AddBranch adds a branch to the tree.

func (*Tree) AddMapping 

func (t *Tree) AddMapping(dir, branchName string, writable bool) error

AddMapping adds a mapping from tree path to a branch.

func (*Tree) Config 

func (t *Tree) Config() string

Config returns the current tree configuration as a JSON string.

func (*Tree) Copy 

func (t *Tree) Copy(src []string, dst string,
	updFunc func(file string, writtenBytes, totalBytes, currentFile, totalFiles int64)) error

Copy is a general purpose copy function which creates files and directories. Destination must be a directory. A non-existing destination directory will be created.

func (*Tree) CopyFile 

func (t *Tree) CopyFile(srcPath, dstPath string, updFunc func(writtenBytes int)) error

CopyFile copies a given file using a simple io.Pipe.

func (*Tree) Dir 

func (t *Tree) Dir(dir string, pattern string, recursive bool, checksums bool) ([]string, [][]os.FileInfo, error)

Dir returns file listings matching a given pattern of one or more directories. The contents of the given path is returned. Optionally, also the contents of all subdirectories can be returned if the recursive flag is set. The return values is a list of traversed directories and their corresponding contents.

func (*Tree) ItemOp 

func (t *Tree) ItemOp(dir string, opdata map[string]string) (bool, error)

ItemOp executes a file or directory specific operation which can either succeed or fail (e.g. rename or delete). Actions and parameters should be given in the opdata map.

func (*Tree) KnownBranches 

func (t *Tree) KnownBranches() map[string]map[string]string

KnownBranches returns a map of all known branches (active or not reachable). Caution: This map contains also the map of active branches with their fingerprints it should only be used for read operations.

func (*Tree) NotReachableBranches 

func (t *Tree) NotReachableBranches() map[string]map[string]string

NotReachableBranches returns a map of all known branches which couldn't be reached. The map contains the name and the definition of the branch.

func (*Tree) PingBranch 

func (t *Tree) PingBranch(node string, rpc string) (string, error)

PingBranch sends a ping to a remote branch and returns its fingerprint or an error.

func (*Tree) ReadFile 

func (t *Tree) ReadFile(spath string, p []byte, offset int64) (int, error)

ReadFile reads up to len(p) bytes into p from the given offset. It returns the number of bytes read (0 <= n <= len(p)) and any error encountered.

func (*Tree) ReadFileToBuffer 

func (t *Tree) ReadFileToBuffer(spath string, buf io.Writer) error

ReadFileToBuffer reads a complete file into a given buffer which implements io.Writer.

func (*Tree) Refresh 

func (t *Tree) Refresh()

Refresh refreshes all known branches and mappings. Only reachable branches will be mapped into the tree.

func (*Tree) Reset 

func (t *Tree) Reset(branches bool)

Reset either resets only all mounts or if the branches flag is specified also all known branches.

func (*Tree) SetMapping 

func (t *Tree) SetMapping(config string) error

SetMapping adds a given tree mapping configuration in a JSON string.

func (*Tree) Stat 

func (t *Tree) Stat(item string) (os.FileInfo, error)

Stat returns information about a given item. Use this function to find out if a given path is a file or directory.

func (*Tree) String 

func (t *Tree) String() string

String returns a string representation of this tree.

func (*Tree) Sync 

func (t *Tree) Sync(srcDir string, dstDir string, recursive bool,
	updFunc func(op, srcFile, dstFile string, writtenBytes, totalBytes, currentFile, totalFiles int64)) error

Sync a given destination with a given source directory. After this command has finished the dstDir will have the same files and directories as the srcDir.

func (*Tree) WriteFile 

func (t *Tree) WriteFile(spath string, p []byte, offset int64) (int, error)

WriteFile writes p into the given file from the given offset. It returns the number of written bytes and any error encountered.

func (*Tree) WriteFileFromBuffer 

func (t *Tree) WriteFileFromBuffer(spath string, buf io.Reader) error

WriteFileFromBuffer writes a complete file from a given buffer which implements io.Reader.

Source Files

View all Source files

Directories

Path Synopsis
api
Package api contains the REST API for RUFS.
 Package api contains the REST API for RUFS.
v1
Package v1 contains Rufs REST API Version 1.
 Package v1 contains Rufs REST API Version 1.
Rufs main entry point for the standalone server.
 Rufs main entry point for the standalone server.
Package export contains export bindings for Rufs.
 Package export contains export bindings for Rufs.
integration
rumble
Package rumble contains Rumble functions which interface with Rufs.
 Package rumble contains Rumble functions which interface with Rufs.
Package node contains the network communication code for Rufs via RPC calls.
 Package node contains the network communication code for Rufs via RPC calls.
Package term contains a terminal implementation which can control Rufs trees.
 Package term contains a terminal implementation which can control Rufs trees.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.