filefreezer

package module
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 3, 2017 License: GPL-2.0 Imports: 15 Imported by: 0

README

Filefreezer (Alpha 1)

A simple to deploy cloud file storage multi-user system; Licensed under the GPL v3.

Have you ever wanted an easy to deploy server for backing up files and storing them encrypted on a remote machine? Filefreezer does that! It also keeps versions of the files that have been added to the server so that you can go back in the file history and pull up old versions of the files.

Alpha 1 is the first public release of the project. It's passing all of the projects unit tests. Please try it out and report any bugs or any instabilities you find. A GUI front end is under development and a web front end will follow shortly after.

Because it is an alpha release, please don't trust it for reliability yet!

Features

  • Zero-knowledge encryption of file data and file name; the server does not store the user's cryptography password and cannot decrypt any of the data the client sends.

  • File versioning

  • Multi-user capability with quota restrictions

  • Simple data storage backend using Sqlite3

  • Public RESTful API that can be used by other clients

ALPHA RELEASE: API AND DATABASE STABILITY NOT GUARANTEED!

Installation

The quick way to install Filefreezer is to use go get to download the repository and its dependences and then go install to install the freezer CLI executable to $GOROOT/bin.

go get github.com/tbogdala/filefreezer/...
go install github.com/tbogdala/filefreezer/cmd/freezer

Another way to quickly deploy Filefreezer is to use Docker and pull the current image from Docker Hub

sudo docker pull tbogdala/filefreezer:0.9.0
sudo docker run --rm -v $HOME/freezer:/data tbogdala/filefreezer:0.9.0 user add -u admin -p 1234
sudo docker run --read-only --rm -v $HOME/freezer:/data -p 8040:8080 tbogdala/filefreezer:0.9.0

For more information, see the page on Docker Hub.

To build the project manually from source code, you will want to vendor the depenencies used by the project. This process is now managed by Go's dep tool. Simply run the following commands to build the vendor directory for dependencies and then build the freezer CLI executable.

cd $GOPATH/src/github.com/tbogdala/filefreezer
dep ensure
cd cmd/freezer
go build
go install

To serve HTTPS with self-signed TLS keys for development purposes, the necessary files can be generated with openssl using the certgen tool from the Go source code:

cd cmd/freezer/certgen
go run generate_cert.go -ca -ecdsa-curve P384 -host 127.0.0.1
mv cert.pem ../freezer.crt
mv key.pem ../freezer.key

In production you will want to use your own valid certificate public and private keys for serving HTTPS.

Quick Start (work in progress)

Before running the server you must create users for the system or else no one will be able to authenticate and sync files. The act of adding a user will also create the database file that will be used later when running the server.

To setup a user named admin with a password of 1234 run the following command:

freezer user add -u admin -p 1234

If you wanted to remove this user, user the following command:

freezer user rm -u admin

At any point you can modify the user information like name, password and quota using the freezer user mod command. For example you can change the quota of the admin user to 1 KB by running the following command:

freezer user mod -u admin --quota 1024

Once a user has been added to the storage database you can launch the server listening on port 8080 by running the following command:

freezer serve ":8080"

With the server running you can now check the user's stats with this command:

freezer -u admin -p 1234 -h localhost:8080 user stats

Before uploading files the client needs to specify a cryptography password so that all file names and data are encrypted on the client's machine and only the client has knowledge of this crypto password (unlike the login password, which can be setup by the service administrator separately).

To set the cryptography password for a client, run the following which will set the crypto pass to secret:

freezer -u admin -p 1234 -h localhost:8080 user cryptopass secret

Since the file names are encrypted as well as the file data, the crypto password has to be setup before you can see the list of files the user has synchronized with the server.

To get the list of files stored by the user, run the following:

freezer -u admin -p 1234 -h localhost:8080 file ls

A file can be syncrhonized with the server by running the following command, which for test purposes will upload a file called hello.txt from the user's home directory:

freezer -u admin -p 1234 -s secret -h localhost:8080 sync ~/hello.txt hello.txt

The first parameter to the sync command is the local filepath to syncrhonize. A second parameter can be specified to override what the file would be called on the server. If only ~/hello.txt was specified, it will get expanded and named on the server as /home/timothy/hello.txt (depending on the user's home directory). By providing the second parameter of hello.txt it will now be known as only hello.txt on the server.

If at some point you want to remove this file, you can do so with the following command:

freezer -u admin -p 1234 -h localhost:8080 file rm hello.txt

Notice that the command takes the name of the file on the server and not the local file which was originally specified on the command line.

If you wanted to remove a set of files controlled by a regular expression, you can use the --regex flag like so:

freezer -u admin -p 1234 -h localhost:8080 file rm --regex --dryrun "h*"

The regular expression will likely have to be supplied in a quoted string to avoid the shell from evaluating wildcards. The --dryrun flag means freezer will output the filenames matched as if it was going to remove it, but no file deletion will actually happen. Remove the flag to actually remove the matched files.

If you make a change to the ~/hello.txt file and sync again it will upload a new version of that file to the server.

freezer -u admin -p 1234 -s secret -h localhost:8080 sync ~/hello.txt hello.txt

You can get a list of stored versions on the server for a given file by running the following command:

freezer -u admin -p 1234 -s secret -h localhost:8080 versions ls hello.txt

If you wanted to syncronize the local file back to the first version of the file, you can do so with the following command which will overrite the local file with the original version of the file still stored on the server:

freezer -u admin -p 1234 -s secret -h localhost:8080 sync --version=1 ~/hello.txt hello.txt

The local file should now be set back to what it was when it was originally synchronzied.

A shortcut to synchronize an entire directory is this command:

freezer -u admin -p 1234 -s secret -h localhost:8080 syncdir /etc serverbackup/etc

This will upload the entire /etc folder and all of its subfolders to the server under a prefix of serverbackup. By using a prefix like this in the target of a sync or syncdir operation, you can logically organize different groups of files.

If you needed to remove old versions of a file, you can do so by specifying an inclusive range in this command:

freezer -u admin -p 1234 -s secret -h localhost:8080 versions rm 1 2 hello.txt

This deletes the first and second version of the synced hello.txt file but leaves other versions on the server.

If you wished to remove all of the file versions except the current one, you can use this syntax where H~ gets interpreted as (Current Version - 1):

freezer -u admin -p 1234 -s secret -h localhost:8080 versions rm 1 H~ hello.txt

You can also use the regular expression matching to remove all but the current version of all files in storage by running the following command (thereby saving some space):

freezer -u admin -p 1234 -s secret -h localhost:8080 versions rm 1 H~ --regex ".*"

Testing and Benchmarking

This package ships with unit tests and benchmarks included. These are in separate locations to have the tests isolated to the main filefreezer package and then tests specific to the projects located in the cmd directory.

Currently to run all of the tests you would execute the following in a shell:

cd $GOPATH/src/github.com/tbogdala/filefreezer/tests
go test
cd ../cmd/freezer
go test

To run the benchmarks you can execute a similar set of commands which will only run the benchmarks and not the unit tests:

cd $GOPATH/src/github.com/tbogdala/filefreezer/tests
go test -run=xxx -bench=.
cd ../cmd/freezer
go test -run=xxx -bench=.

Known Bugs and Limitations

  • Consider a quota for max fileinfo registered so service cannot be DDOS'd by registering infinite files.

  • Incrementing a user's revision number only happens in some areas like chunk modification. Consider bumping the revision with new files are added or otherwise changed too.

  • userid is taken on some Storage methods, but not all, for checking correct user is accessing data

  • starting a remote sync target name with '/' in win32/msys2 attempts to autocomplete the string as a path and not give the desired results. the fix is to use cmd.exe to perform the command line execution to get the desired results.

TODO / Notes

  • Inspired from a blog post about Dropbox: https://blogs.dropbox.com/tech/2014/07/streaming-file-synchronization/

  • flag: file hashing algo

  • flag: hash on start instead of just checking mod time

  • flag: safetey level for database -- currently it is tuned to be very safe, but a non-zero chance of db corruption on power loss or crash. docs for sqlite say "in practice, you are more likely to suffer a catastrophic disk failure or some other unrecoverable hardware fault" but this should be tunable via command line.

  • work on readability of error messages wrt bubbling up error objects

  • break up unit test functions into more modular test functions

  • multithreading the chunk uploading of files

  • review current code documentation for godoc purposes

  • something like a general db stats command to return total files, chunks, versions per user/system

  • remove output from cmd/freezer/command functions so that they are more reusable

Documentation

Index

Constants

View Source 
const (
	// CurrentDBVersion is set to the current database version and is used
	// by filefreezer to detect when the database tables need to get updated.
	CurrentDBVersion = 1
)

Variables

This section is empty.

Functions

func GenCryptoPasswordHash 

func GenCryptoPasswordHash(password string, makeKeyHash bool, keyHashOpts string) (key []byte, keyHash []byte, keyHashCombo string, err error)

GenCryptoPasswordHash takes the user password then generates a crytpo hash. If makeKeyHash is false, only the key parameter is generated. If keyHashOpts is not an empty string, it attempts to split the string by '$' dividers for scrypt parameters. NOTE: it's intended that keyHashOpts will be the keyHashCombo return value of a previous call.

func GenLoginPasswordHash 

func GenLoginPasswordHash(unsaltedPassword string) (salt string, saltedhash []byte, err error)

GenLoginPasswordHash takes the user password, generates a new random salt, then generates a hash from the salted password combination.

func VerifyCryptoPassword 

func VerifyCryptoPassword(password string, keyHashCombo string) ([]byte, error)

VerifyCryptoPassword takes a plain text password and compares it against a hash of the crypto key to verify that the password is correct and the crypto key is the correct one. On success and successful match a non-nil []byte slice is returned. If the keys do not match nil is returned. Otherwise an non-nil error is returned.

func VerifyLoginPassword 

func VerifyLoginPassword(unsaltedPassowrd string, salt string, saltedHash []byte) bool

VerifyLoginPassword takes the user-supplied unsalted password and the stored salt and hash and verifies that the supplied unsalted password is the correct match. Returns true on match and false on fail.

Types

type FileChunk 

type FileChunk struct {
	FileID      int
	VersionID   int
	ChunkNumber int
	ChunkHash   string
	Chunk       []byte
}

FileChunk contains the information stored about a given file chunk.

type FileInfo 

type FileInfo struct {
	UserID         int
	FileID         int
	FileName       string
	IsDir          bool
	CurrentVersion FileVersionInfo
}

FileInfo contains the information stored about a given file for a particular user.

type FileStats 

type FileStats struct {
	ChunkCount  int
	LastMod     int64
	Permissions uint32
	HashString  string
	IsDir       bool
}

FileStats is a structure used to return information about a given file from the file system.

func CalcFileHashInfo 

func CalcFileHashInfo(maxChunkSize int64, filename string) (stats FileStats, e error)

CalcFileHashInfo takes the file name and calculates the number of chunks, last modified time and hash string for the file. An error is returned on failure.

type FileVersionInfo 

type FileVersionInfo struct {
	VersionID     int
	VersionNumber int
	Permissions   uint32
	LastMod       int64
	ChunkCount    int
	FileHash      string
}

FileVersionInfo contains the version-specific information for a given file.

type Storage 

type Storage struct {
	// ChunkSize is the number of bytes the chunk can maximally be
	ChunkSize int64
	// contains filtered or unexported fields
}

Storage is the backend data model for the file storage logic.

func NewStorage 

func NewStorage(dbPath string) (*Storage, error)

NewStorage creates a new Storage object using the sqlite3 driver at the path given.

func (*Storage) AddFileChunk 

func (s *Storage) AddFileChunk(userID int, fileID int, versionID int, chunkNumber int, chunkHash string, chunk []byte) (*FileChunk, error)

AddFileChunk adds a binary chunk to storage for a given file at a position in the file determined by the chunkNumber passed in and identified by the chunkHash. The userID is used to update the allocation count in the same transaction as well as verify ownership.

func (*Storage) AddFileInfo 

func (s *Storage) AddFileInfo(userID int, filename string, isDir bool, permissions uint32, lastMod int64, chunkCount int, fileHash string) (*FileInfo, error)

AddFileInfo registers a new file for a given user which is identified by the filename string. lastmod (time in seconds since 1/1/1970) and the filehash string are provided as well. The chunkCount parameter should be the number of chunks required for the size of the file. If the file could not be added an error is returned, otherwise nil on success.

func (*Storage) AddUser 

func (s *Storage) AddUser(username string, salt string, saltedHash []byte, quota int) (*User, error)

AddUser should create the user in the USERS table. The username should be unique. saltedHash should be the combined password & salt hash and salt should be the user specific generated salt. This function returns a true bool value if a user was created and false if the user was not created (e.g. username was already taken).

func (*Storage) Close 

func (s *Storage) Close()

Close releases the backend connections to the database.

func (*Storage) CreateTables 

func (s *Storage) CreateTables() error

CreateTables will create the tables needed in the database if they don't already exist. If the tables already exist an error will be returned.

func (*Storage) GetAllUserFileInfos 

func (s *Storage) GetAllUserFileInfos(userID int) ([]FileInfo, error)

GetAllUserFileInfos returns a slice of UserFileInfo objects that describe all known files in storage for a given user ID. If this query was unsuccessful and error is returned.

func (*Storage) GetDBVersion 

func (s *Storage) GetDBVersion() (int, error)

GetDBVersion will return the DB Version number for the opened database.

func (*Storage) GetFileChunk 

func (s *Storage) GetFileChunk(fileID int, chunkNumber int, versionID int) (fc *FileChunk, e error)

GetFileChunk retrieves a file chunk from storage and returns it. An error value is returned on failure.

func (*Storage) GetFileChunkInfos 

func (s *Storage) GetFileChunkInfos(userID int, fileID int, versionID int) ([]FileChunk, error)

GetFileChunkInfos returns a slice of FileChunks containing all of the chunk information except for the chunk bytes themselves.

func (*Storage) GetFileInfo 

func (s *Storage) GetFileInfo(userID int, fileID int) (*FileInfo, error)

GetFileInfo returns a UserFileInfo object that describes the file identified by the fileID parameter. If this query was unsuccessful an error is returned.

func (*Storage) GetFileInfoByName 

func (s *Storage) GetFileInfoByName(userID int, filename string) (*FileInfo, error)

GetFileInfoByName returns a UserFileInfo object that describes the file identified by the userID and filename parameters. If this query was unsuccessful an error is returned.

func (*Storage) GetFileVersions 

func (s *Storage) GetFileVersions(fileID int) ([]FileVersionInfo, error)

GetFileVersions will return a slice of FileVersionInfo that encompases all of the versions registered for a given file ID.

func (*Storage) GetMissingChunkNumbersForFile 

func (s *Storage) GetMissingChunkNumbersForFile(userID int, fileID int) ([]int, error)

GetMissingChunkNumbersForFile will return a slice of chunk numbers that have not been added for a given file.

func (*Storage) GetUser 

func (s *Storage) GetUser(username string) (*User, error)

GetUser queries the Users table for a given username and returns the associated data. If the query fails and error will be returned.

func (*Storage) GetUserStats 

func (s *Storage) GetUserStats(userID int) (*UserStats, error)

GetUserStats returns the user information for a user by user id.

func (*Storage) IsUsernameFree 

func (s *Storage) IsUsernameFree(username string) (bool, error)

IsUsernameFree will return true if there is not already a username with the same text in the Users table.

func (*Storage) RemoveFile 

func (s *Storage) RemoveFile(userID, fileID int) error

RemoveFile removes a file listing and all of the associated chunks in storage. Returns an error on failure

func (*Storage) RemoveFileChunk 

func (s *Storage) RemoveFileChunk(userID int, fileID int, versionID int, chunkNumber int) (bool, error)

RemoveFileChunk removes a chunk from storage identifed by the fileID and chunkNumber. If the chunkNumber specified is out of range of the file's max chunk count, this will simply have no effect. An bool indicating if the chunk was successfully removed is returned as well as an error on failure. userID is required so that the allocation count can updated in the same transaction as well as to verify ownership of the chunk.

func (*Storage) RemoveFileInfo 

func (s *Storage) RemoveFileInfo(fileID int) error

RemoveFileInfo removes a file listing in storage, returning an error on failure.

func (*Storage) RemoveFileVersions 

func (s *Storage) RemoveFileVersions(userID, fileID, minVersion, maxVersion int) error

RemoveFileVersions will remove any file versions of the file specified by fileID that are between the minVersion and maxVersion (inclusive). A non-nil error value is returned on failure.

NOTE: supplying a minVersion and maxVersion that does not include any valid file versions will end up returning an error.

func (*Storage) RemoveUser 

func (s *Storage) RemoveUser(username string) error

RemoveUser removes user and all files and file chunks associated with the user.

func (*Storage) SetUserQuota 

func (s *Storage) SetUserQuota(userID int, quota int) error

SetUserQuota sets the user quota for a user by user id.

func (*Storage) SetUserStats 

func (s *Storage) SetUserStats(userID int, quota int, allocated int, revision int) error

SetUserStats sets the user information for a user by user id and is used to do the first insertion of the user into the stats table.

func (*Storage) TagNewFileVersion 

func (s *Storage) TagNewFileVersion(userID int, fileID int, permissions uint32, lastMod int64, chunkCount int, fileHash string) (*FileInfo, error)

TagNewFileVersion creates a new version of a given file and returns the new version ID as well as the incremented file-local version number.

func (*Storage) UpdateUser 

func (s *Storage) UpdateUser(userID int, name string, salt string, saltedHash []byte, cryptoHash []byte, quota int) error

UpdateUser changes the salt, saltedHash, cryptoHash and quota for a given userID. This will fail if the userID doesn't exist.

func (*Storage) UpdateUserCryptoHash 

func (s *Storage) UpdateUserCryptoHash(userID int, cryptoHash []byte) error

UpdateUserCryptoHash changes the cryptoHash for a given userID. This will fail if the userID doesn't exist.

func (*Storage) UpdateUserStats 

func (s *Storage) UpdateUserStats(userID int, allocDelta int) error

UpdateUserStats increments the user's revision by one and updates the allocated byte counter with the new delta.

type User 

type User struct {
	ID         int
	Name       string
	Salt       string
	SaltedHash []byte
	CryptoHash []byte // a bcrypt hash used to verify the bcrypt hash of the crypto password
}

User contains the basic information stored about a use, but does not include current allocation or revision statistics.

type UserStats 

type UserStats struct {
	Quota     int
	Allocated int
	Revision  int
}

UserStats contains the user specific state information to track data usage.

Source Files

View all Source files

Directories

Path Synopsis
cmd
freezer
freezer/command
freezer/models

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.