desync

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 19, 2018 License: BSD-3-Clause Imports: 39 Imported by: 0

README

desync

This project re-implements many features of upstream casync in Go. It seeks to maintain compatibility with casync's data structures, protocols and types, such as chunk stores (castr), index files (caibx/caidx) and archives (catar) in order to function as a drop-in replacement in many use cases. It also tries to maintain support for platforms other than Linux and simplify build/installation. It consists of a library that implements the features, available for integration into any 3rd-party product as well as a command-line tool.

For support and discussion, see Gitter chat. Feature requests should be discussed there before filing, unless you're interested in doing the work to implement them yourself.

Goals And Non-Goals

Among the distinguishing factors:

  • Supported on MacOS, though there could be incompatibilities when exchanging catar-files between Linux and Mac for example since devices and filemodes differ slightly. *BSD should work as well but hasn't been tested. Windows supports a limited subset of commands.
  • Where the upstream command has chosen to optimize for storage efficiency (f/e, being able to use local files as "seeds", building temporary indexes into them), this command chooses to optimize for runtime performance (maintaining a local explicit chunk store, avoiding the need to reindex) at cost to storage efficiency.
  • Where the upstream command has chosen to take full advantage of Linux platform features, this client chooses to implement a minimum featureset and, while high-value platform-specific features (such as support for btrfs reflinks into a decompressed local chunk cache) might be added in the future, the ability to build without them on other platforms will be maintained.
  • SHA512/256 is currently the only supported hash function.
  • Only chunk store using zstd compression are supported at this point.
  • Supports local stores as well as remote stores (as client) over SSH, SFTP and HTTP
  • Built-in HTTP(S) chunk server that can proxy multiple local or remote stores and also supports caching.
  • Drop-in replacement for casync on SSH servers when serving chunks read-only
  • Support for catar files exists, but ignores XAttr, SELinux, ACLs and FCAPs that may be present in existing catar files and those won't be present when creating a new catar with the tar command
  • Supports chunking with the same algorithm used by casync (see make command) but executed in parallel. Results are identical to what casync produces, same chunks and index files, but with significantly better performance. For example, up to 10x faster than casync if the chunks are already present in the store. If the chunks are new, it heavily depends on I/O, but it's still likely several times faster than casync.
  • While casync supports very small min chunk sizes, optimizations in desync require min chunk sizes larger than the window size of the rolling hash used (currently 48 bytes). The tool's default chunk sizes match the defaults used in casync, min 16k, avg 64k, max 256k.
  • Allows FUSE mounting of blob indexes
  • S3 protocol support to access chunk stores for read operations and some some commands that write chunks
  • Stores and retrieves index files from remote index stores such as HTTP, SFTP and S3

Parallel chunking

One of the significant differences to casync is that desync attempts to make chunking faster by utilizing more CPU resources, chunking data in parallel. Depending on the chosen degree of concurrency, the file is split into N equal parts and each part is chunked independently. While the chunking of each part is ongoing, part1 is trying to align with part2, and part3 is trying to align with part4 and so on. Alignment is achieved once a common split point is found in the overlapping area. If a common split point is found, the process chunking the previous part stops, eg. part1 chunker stops, part2 chunker keeps going until it aligns with part3 and so on until all split points have been found. Once all split points have been determined, the file is opened again (N times) to read, compress and store the chunks. While in most cases this process achieves significantly reduced chunking times at the cost of CPU, there are edge cases where chunking is only about as fast as upstream casync (with more CPU usage). This is the case if no split points can be found in the data between min and max chunk size as is the case if most or all of the file consists of 0-bytes. In this situation, the concurrent chunking processes for each part will not align with each other and a lot of effort is wasted. The table below shows how the type of data that is being chunked can influence runtime of each operation. make refers to the process of chunking, while extract refers to re-assembly of blobs from chunks.

Command Mostly/All 0-bytes Typical data
make Slow (worst-case) - Likely comparable to casync Fast - Parallel chunking
extract Extremely fast - Effectively the speed of a truncate() syscall Fast - Done in parallel, usually limited by I/O

Tool

The tool is provided for convenience. It uses the desync library and makes most features of it available in a consistent fashion. It does not match upsteam casync's syntax exactly, but tries to be similar at least.

Installation

If GOPATH is set correctly, building the tool and installing it into $GOPATH/bin can be done with:

go get -u github.com/folbricht/desync/cmd/desync
Subcommands
  • extract - build a blob from an index file
  • verify - verify the integrity of a local store
  • list-chunks - list all chunk IDs contained in an index file
  • cache - populate a cache from index files without extracting a blob or archive
  • chop - split a blob according to an existing caibx and store the chunks in a local store
  • pull - serve chunks using the casync protocol over stdin/stdout. Set CASYNC_REMOTE_PATH=desync on the client to use it.
  • tar - pack a catar file, optionally chunk the catar and create an index file. Not available on Windows.
  • untar - unpack a catar file or an index referencing a catar. Not available on Windows.
  • prune - remove unreferenced chunks from a local or S3 store. Use with caution, can lead to data loss.
  • verify-index - verify that an index file matches a given blob
  • chunk-server - start a chunk server that serves chunks via HTTP(S)
  • make - split a blob into chunks and create an index file
  • mount-index - FUSE mount a blob index. Will make the blob available as single file inside the mountpoint.
  • info - Show information about an index file, such as number of chunks and optionally chunks from an index that a re present in a store
Options (not all apply to all commands)
  • -s <store> Location of the chunk store, can be local directory or a URL like ssh://hostname/path/to/store. Multiple stores can be specified, they'll be queried for chunks in the same order. The chop, make, tar and prune commands support updating chunk stores in S3, while verify only operates on a local store.
  • -c <store> Location of a chunk store to be used as cache. Needs to be writable.
  • -n <int> Number of concurrent download jobs and ssh sessions to the chunk store.
  • -r Repair a local cache by removing invalid chunks. Only valid for the verify command.
  • -y Answer with yes when asked for confirmation. Only supported by the prune command.
  • -l Listening address for the HTTP chunk server. Can be used multiple times to run on more than one interface or more than one port. Only supported by the chunk-server command.
  • -m Specify the min/avg/max chunk sizes in kb. Only applicable to the make command. Defaults to 16:64:256 and for best results the min should be avg/4 and the max should be 4*avg.
  • -i When packing/unpacking an archive, don't create/read an archive file but instead store/read the chunks and use an index file (caidx) for the archive. Only applicable to tar and untar commands.
  • -t Trust all certificates presented by HTTPS stores. Allows the use of self-signed certs when using a HTTPS chunk server.
  • -key Key file in PEM format used for HTTPS chunk-server command. Also requires a certificate with -cert
  • -cert Certificate file in PEM format used for HTTPS chunk-server command. Also requires -key.
  • -k Keep partially assembled files in place when extract fails or is interrupted. The command can then be restarted and it'll not have to retrieve completed parts again.
Environment variables
  • CASYNC_SSH_PATH overrides the default "ssh" with a command to run when connecting to a remote SSH or SFTP chunk store
  • CASYNC_REMOTE_PATH defines the command to run on the chunk store when using SSH, default "casync"
  • S3_ACCESS_KEY, S3_SECRET_KEY, S3_REGION can be used to define S3 store credentials if only one store is used. Caution, these values take precedence over any S3 credentials set in the config file.
Caching

The -c <store> option can be used to either specify an existing store to act as cache or to populate a new store. Whenever a chunk is requested, it is first looked up in the cache before routing the request to the next (possibly remote) store. Any chunks downloaded from the main stores are added to the cache. In addition, when a chunk is read from the cache and it is a local store, mtime of the chunk is updated to allow for basic garbage collection based on file age. The cache store is expected to be writable. If the cache contains an invalid chunk (checksum does not match the chunk ID), the operation will fail. Invalid chunks are not skipped or removed from the cache automatically. verfiy -r can be used to evict bad chunks from a local store or cache.

Multiple chunk stores

One of the main features of desync is the ability to combine/chain multiple chunk stores of different types and also combine it with a cache store. For example, for a command that reads chunks when assembling a blob, stores can be chained in the command line like so: -s <store1> -s <store2> -s <store3>. A chunk will first be requested from store1, and if not found there, the request will be routed to <store2> and so on. Typically, the fastest chunk store should be listed first to improve performance. It is also possible to combine multiple chunk stores with a cache. In most cases the cache would be a local store, but that is not a requirement. When combining stores and a cache like so: -s <store1> -s <store2> -c <cache>, a chunk request will first be routed to the cache store, then to store1 followed by store2. Any chunks that is not yet in the cache will be stored there upon first request.

Not all types of stores support all operations. The table below lists the supported operations on all store types.

Operation Local store S3 store HTTP store SFTP SSH (casync protocol)
Read chunks yes yes yes yes yes
Write chunks yes yes yes yes no
Use as cache yes yes yes yes no
Prune yes yes no yes no
Verify yes yes no no no
Remote indexes

Indexes can be stored and retrieved from remote locations via SFTP, S3, and HTTP. Storing indexes remotely is optional and deliberately separate from chunk storage. While it's possible to store indexes in the same location as chunks in the case of SFTP and S3, this should only be done in secured environments. The built-in HTTP chunk store (chunk-server command) can not be used as index server. Use the index-server command instead to start an index server that serves indexes and can optionally store them as well (with -w).

Using remote indexes, it is possible to use desync completely file-less. For example when wanting to share a large file with mount-index, one could read the index from an index store like this:

desync mount-index -s http://chunk.store/store http://index.store/myindex.caibx /mnt/image

No file would need to be stored on disk in this case.

S3 chunk stores

desync supports reading from and writing to chunk stores that offer an S3 API, for example hosted in AWS or running on a local server. When using such a store, credentials are passed into the tool either via environment variables S3_ACCESS_KEY and S3_SECRET_KEY or, if multiples are required, in the config file. Care is required when building those URLs. Below a few examples:

AWS

This store is hosted in eu-west-3 in AWS. s3 signals that the S3 protocol is to be used, https should be specified for SSL connections. The first path element of the URL contains the bucket, desync.bucket in this example. Note, when using AWS, no port should be given in the URL!

s3+https://s3-eu-west-3.amazonaws.com/desync.bucket

It's possible to use prefixes (or "directories") to object names like so

s3+https://s3-eu-west-3.amazonaws.com/desync.bucket/prefix
Other service with S3 API

This is a store running on the local machine on port 9000 without SSL.

s3+http://127.0.0.1:9000/store
Previous S3 storage layout

Before April 2018, chunks in S3 stores were kept in a flat layout, with the name being the checksum of the chunk. Since then, the layout was modified to match that of local stores: <4-checksum-chars>/<checksum>.cacnk This change allows the use of other tools to convert or copy stores between local and S3 stores. To convert an existing s3 store from the old format, a command upgrade-s3 is available in the tool.

Configuration

For most use cases, it is sufficient to use the tool's default configuration not requiring a config file. Having a config file $HOME/.config/desync/config.json allows for further customization of timeouts, error retry behaviour or credentials that can't be set via command-line options or environment variables. To view the current configuration, use desync config. If no config file is present, this will show the defaults. To create a config file allowing custom values, use desync config -w which will write the current configuration to the file, then edit the file.

Available configuration values:

  • http-timeout - HTTP request timeout used in HTTP stores (not S3) in nanoseconds
  • http-error-retry - Number of times to retry failed chunk requests from HTTP stores
  • s3-credentials - Defines credentials for use with S3 stores. Especially useful if more than one S3 store is used. The key in the config needs to be the URL scheme and host used for the store, excluding the path, but including the port number if used in the store URL. It is also possible to use a standard aws credentials file in order to store s3 credentials.

Example config

{
  "http-timeout": 60000000000,
  "http-error-retry": 0,
  "s3-credentials": {
       "http://localhost": {
           "access-key": "MYACCESSKEY",
           "secret-key": "MYSECRETKEY"
       },
       "https://127.0.0.1:9000": {
           "aws-credentials-file": "/Users/user/.aws/credentials",
       },
       "https://127.0.0.1:8000": {
           "aws-credentials-file": "/Users/user/.aws/credentials",
           "aws-profile": "profile_static"
       },
       "https://s3.us-west-2.amazonaws.com": {
           "aws-credentials-file": "/Users/user/.aws/credentials",
           "aws-region": "us-west-2",
           "aws-profile": "profile_refreshable"
       }
  }
}

Example aws credentials

[default]
aws_access_key_id = DEFAULT_PROFILE_KEY
aws_secret_access_key = DEFAULT_PROFILE_SECRET

[profile_static]
aws_access_key_id = OTHERACCESSKEY
aws_secret_access_key = OTHERSECRETKEY

[profile_refreshable]
aws_access_key_id = PROFILE_REFRESHABLE_KEY
aws_secret_access_key = PROFILE_REFRESHABLE_SECRET
aws_session_token = PROFILE_REFRESHABLE_TOKEN
Examples:

Re-assemble somefile.tar using a remote chunk store and a blob index file.

desync extract -s ssh://192.168.1.1/path/to/casync.store/ -c /tmp/store somefile.tar.caibx somefile.tar

Use multiple stores, specify the local one first to improve performance.

desync extract -s /some/local/store -s ssh://192.168.1.1/path/to/casync.store/ somefile.tar.caibx somefile.tar

Mix and match remote stores and use a local cache store to improve performance.

desync extract \
       -s ssh://192.168.1.1/path/to/casync.store/ \
       -s http://192.168.1.2/casync.store/ \
       -s https://192.168.1.3/ssl.store/ \
       -c /path/to/cache \
       somefile.tar.caibx somefile.tar

Extract a file in-place (-k option). If this operation fails, the file will remain partially complete and can be restarted without the need to re-download chunks from the remote SFTP store. Use -k when a local cache is not available and the extract may be interrupted.

desync extract -k -s sftp://192.168.1.1/path/to/store file.caibx file.tar

Extract a file using a remote index stored in an HTTP index store

desync extract -k -s sftp://192.168.1.1/path/to/store http://192.168.1.2/file.caibx file.tar

Verify a local cache. Errors will be reported to STDOUT, since -r is not given, nothing invalid will be removed.

desync verify -s /some/local/store

Cache the chunks used in a couple of index files in a local store without actually writing the blob.

desync cache -s ssh://192.168.1.1/path/to/casync.store/ -c /local/cache somefile.tar.caibx other.file.caibx

List the chunks referenced in a caibx.

desync list-chunks somefile.tar.caibx

Chop an existing file according to an existing caibx and store the chunks in a local store. This can be used to populate a local cache from a possibly large blob that already exists on the target system.

desync chop -s /some/local/store somefile.tar.caibx somefile.tar

Pack a directory tree into a catar file.

desync tar archive.catar /some/dir

Pack a directory tree into an archive and chunk the archive, producing an index file.

desync tar -i -s /some/local/store archive.caidx /some/dir

Unpack a catar file.

desync untar archive.catar /some/dir

Unpack a directory tree using an index file referencing a chunked archive.

desync untar -i -s /some/local/store archive.caidx /some/dir

Prune a store to only contain chunks that are referenced in the provided index files. Possible data loss.

desync prune -s /some/local/store index1.caibx index2.caibx

Start a chunk server serving up a local store via port 80.

desync chunk-server -s /some/local/store

Start a chunk server on port 8080 acting as proxy for other remote HTTP and SSH stores and populate a local cache.

desync chunk-server -s http://192.168.1.1/ -s ssh://192.168.1.2/store -c cache -l :8080

Start a writable index server, chunk a file and store the index.

server# desync index-server -s /mnt/indexes -w -l :8080

client# desync make -s /some/store http://192.168.1.1:8080/file.vmdk.caibx file.vmdk

Copy all chunks referenced in an index file from a remote HTTP store to a remote SFTP store.

desync cache -s ssh://192.168.1.2/store -c sftp://192.168.1.3/path/to/store /path/to/index.caibx

Start a TLS chunk server on port 443 acting as proxy for a remote chunk store in AWS with local cache. The credentials for AWS are expected to be in the config file under key https://s3-eu-west-3.amazonaws.com.

desync chunk-server -s s3+https://s3-eu-west-3.amazonaws.com/desync.bucket/prefix -c cache -l 127.0.0.1:https -cert cert.pem -key key.pem

Split a blob, store the chunks and create an index file.

desync make -s /some/local/store index.caibx /some/blob

Split a blob, create an index file and store the chunks in an S3 bucket named store.

S3_ACCESS_KEY=mykey S3_SECRET_KEY=mysecret desync make -s s3+http://127.0.0.1:9000/store index.caibx /some/blob

FUSE mount an index file. This will make the indexed blob available as file underneath the mount point. The filename in the mount matches the name of the index with the extension removed. In this example /some/mnt/ will contain one file index.

desync mount-index -s /some/local/store index.caibx /some/mnt

FUSE mount a chunked and remote index file. First a (small) index file is read from the index-server which is used to re-assemble a larger index file and pipe it into the 2nd command that then mounts it.

desync cat -s http://192.168.1.1/store http://192.168.1.2/small.caibx | desync mount-index -s http://192.168.1.1/store - /mnt/point

Show information about an index file to see how many of its chunks are present in a local store or an S3 store. The local store is queried first, S3 is only queried if the chunk is not present in the local store. The output will be in JSON format (-j) for easier processing in scripts.

desync info -j -s /tmp/store -s s3+http://127.0.0.1:9000/store /path/to/index

TODOs

  • Pre-allocate the output file to avoid fragmentation when using extract command
  • Allow on-disk chunk cache to optionally be stored uncompressed, such that blocks can be directly reflinked (rather than copied) into files, when on a platform and filesystem where reflink support is available.

Documentation

Overview

Package desync implements data structures, protocols and features of https://github.com/systemd/casync in order to allow support for additional platforms and improve performace by way of concurrency and caching.

Supports the following casync data structures: catar archives, caibx/caidx index files, castr stores (local or remote).

See desync/cmd for reference implementations of the available features.

Index

Constants

View Source 
const (
	// Format identifiers used in archive files
	CaFormatEntry             = 0x1396fabcea5bbb51
	CaFormatUser              = 0xf453131aaeeaccb3
	CaFormatGroup             = 0x25eb6ac969396a52
	CaFormatXAttr             = 0xb8157091f80bc486
	CaFormatACLUser           = 0x297dc88b2ef12faf
	CaFormatACLGroup          = 0x36f2acb56cb3dd0b
	CaFormatACLGroupObj       = 0x23047110441f38f3
	CaFormatACLDefault        = 0xfe3eeda6823c8cd0
	CaFormatACLDefaultUser    = 0xbdf03df9bd010a91
	CaFormatACLDefaultGroup   = 0xa0cb1168782d1f51
	CaFormatFCaps             = 0xf7267db0afed0629
	CaFormatSELinux           = 0x46faf0602fd26c59
	CaFormatSymlink           = 0x664a6fb6830e0d6c
	CaFormatDevice            = 0xac3dace369dfe643
	CaFormatPayload           = 0x8b9e1d93d6dcffc9
	CaFormatFilename          = 0x6dbb6ebcb3161f0b
	CaFormatGoodbye           = 0xdfd35c5e8327c403
	CaFormatGoodbyeTailMarker = 0x57446fa533702943
	CaFormatIndex             = 0x96824d9c7b129ff9
	CaFormatTable             = 0xe75b9e112f17417d
	CaFormatTableTailMarker   = 0x4b4f050e5549ecd1

	// SipHash key used in Goodbye elements to hash the filename. It's 16 bytes,
	// split into 2x64bit values, upper and lower part of the key
	CaFormatGoodbyeHashKey0 = 0x8574442b0f1d84b3
	CaFormatGoodbyeHashKey1 = 0x2736ed30d1c22ec1

	// Format feature flags
	CaFormatWith16BitUIDs   = 0x1
	CaFormatWith32BitUIDs   = 0x2
	CaFormatWithUserNames   = 0x4
	CaFormatWithSecTime     = 0x8
	CaFormatWithUSecTime    = 0x10
	CaFormatWithNSecTime    = 0x20
	CaFormatWith2SecTime    = 0x40
	CaFormatWithReadOnly    = 0x80
	CaFormatWithPermissions = 0x100
	CaFormatWithSymlinks    = 0x200
	CaFormatWithDeviceNodes = 0x400
	CaFormatWithFIFOs       = 0x800
	CaFormatWithSockets     = 0x1000

	/* DOS file flags */
	CaFormatWithFlagHidden  = 0x2000
	CaFormatWithFlagSystem  = 0x4000
	CaFormatWithFlagArchive = 0x8000

	/* chattr() flags */
	CaFormatWithFlagAppend         = 0x10000
	CaFormatWithFlagNoAtime        = 0x20000
	CaFormatWithFlagCompr          = 0x40000
	CaFormatWithFlagNoCow          = 0x80000
	CaFormatWithFlagNoDump         = 0x100000
	CaFormatWithFlagDirSync        = 0x200000
	CaFormatWithFlagImmutable      = 0x400000
	CaFormatWithFlagSync           = 0x800000
	CaFormatWithFlagNoComp         = 0x1000000
	CaFormatWithFlagProjectInherit = 0x2000000

	/* btrfs magic */
	CaFormatWithSubvolume   = 0x4000000
	CaFormatWithSubvolumeRO = 0x8000000

	/* Extended Attribute metadata */
	CaFormatWithXattrs  = 0x10000000
	CaFormatWithACL     = 0x20000000
	CaFormatWithSELinux = 0x40000000
	CaFormatWithFcaps   = 0x80000000

	CaFormatSHA512256        = 0x2000000000000000
	CaFormatExcludeSubmounts = 0x4000000000000000
	CaFormatExcludeNoDump    = 0x8000000000000000

	// Protocol message types
	CaProtocolHello      = 0x3c71d0948ca5fbee
	CaProtocolIndex      = 0xb32a91dd2b3e27f8
	CaProtocolIndexEOF   = 0x4f0932f1043718f5
	CaProtocolArchive    = 0x95d6428a69eddcc5
	CaProtocolArchiveEOF = 0x450bef663f24cbad
	CaProtocolRequest    = 0x8ab427e0f89d9210
	CaProtocolChunk      = 0x5213dd180a84bc8c
	CaProtocolMissing    = 0xd010f9fac82b7b6c
	CaProtocolGoodbye    = 0xad205dbf1a3686c3
	CaProtocolAbort      = 0xe7d9136b7efea352

	// Provided services
	CaProtocolReadableStore   = 0x1
	CaProtocolWritableStore   = 0x2
	CaProtocolReadableIndex   = 0x4
	CaProtocolWritableIndex   = 0x8
	CaProtocolReadableArchive = 0x10
	CaProtocolWritableArchive = 0x20

	// Wanted services
	CaProtocolPullChunks      = 0x40
	CaProtocolPullIndex       = 0x80
	CaProtocolPullArchive     = 0x100
	CaProtocolPushChunks      = 0x200
	CaProtocolPushIndex       = 0x400
	CaProtocolPushIndexChunks = 0x800
	CaProtocolPushArchive     = 0x1000

	// Protocol request flags
	CaProtocolRequestHighPriority = 1

	// Chunk properties
	CaProtocolChunkCompressed = 1
)
View Source 
const ChunkerWindowSize = 48

ChunkerWindowSize is the number of bytes in the rolling hash window

Variables

View Source 
var (
	FormatString = map[uint64]string{
		CaFormatEntry:             "CaFormatEntry",
		CaFormatUser:              "CaFormatUser",
		CaFormatGroup:             "CaFormatGroup",
		CaFormatXAttr:             "CaFormatXAttr",
		CaFormatACLUser:           "CaFormatACLUser",
		CaFormatACLGroup:          "CaFormatACLGroup",
		CaFormatACLGroupObj:       "CaFormatACLGroupObj",
		CaFormatACLDefault:        "CaFormatACLDefault",
		CaFormatACLDefaultUser:    "CaFormatACLDefaultUser",
		CaFormatACLDefaultGroup:   "CaFormatACLDefaultGroup",
		CaFormatFCaps:             "CaFormatFCaps",
		CaFormatSELinux:           "CaFormatSELinux",
		CaFormatSymlink:           "CaFormatSymlink",
		CaFormatDevice:            "CaFormatDevice",
		CaFormatPayload:           "CaFormatPayload",
		CaFormatFilename:          "CaFormatFilename",
		CaFormatGoodbye:           "CaFormatGoodbye",
		CaFormatGoodbyeTailMarker: "CaFormatGoodbyeTailMarker",
		CaFormatIndex:             "CaFormatIndex",
		CaFormatTable:             "CaFormatTable",
		CaFormatTableTailMarker:   "CaFormatTableTailMarker",
	}
)
View Source 
var TrustInsecure bool

TrustInsecure determines if invalid certs presented by HTTP stores should be accepted.

Functions

func AssembleFile added in v0.2.0 

func AssembleFile(ctx context.Context, name string, idx Index, s Store, n int, progress func()) error

AssembleFile re-assembles a file based on a list of index chunks. It runs n goroutines, creating one filehandle for the file "name" per goroutine and writes to the file simultaneously. If progress is provided, it'll be called when a chunk has been processed. If the input file exists and is not empty, the algorithm will first confirm if the data matches what is expected and only populate areas that differ from the expected content. This can be used to complete partly written files.

func ChopFile added in v0.2.0 

func ChopFile(ctx context.Context, name string, chunks []IndexChunk, ws WriteStore, n int, pb ProgressBar) error

ChopFile split a file according to a list of chunks obtained from an Index and stores them in the provided store

func Compress added in v0.2.0 

func Compress(b []byte) ([]byte, error)

Compress a block using the only (currently) supported algorithm

func Copy added in v0.2.0 

func Copy(ctx context.Context, ids []ChunkID, src Store, dst WriteStore, n int, progress func()) error

Copy reads a list of chunks from the provided src store, and copies the ones not already present in the dst store. The goal is to load chunks from remote store to populate a cache. If progress is provided, it'll be called when a chunk has been processed. Used to draw a progress bar, can be nil.

func Decompress added in v0.2.0 

func Decompress(out, in []byte) ([]byte, error)

Decompress a block using the only supported algorithm. If you already have a buffer it can be passed into out and will be used. If out=nil, a buffer will be allocated.

func IndexFromFile added in v0.2.0 

func IndexFromFile(ctx context.Context,
	name string,
	n int,
	min, avg, max uint64,
	progress func(uint64),
) (Index, ChunkingStats, error)

IndexFromFile chunks a file in parallel and returns an index. It does not store chunks! Each concurrent chunker starts filesize/n bytes apart and splits independently. Each chunk worker tries to sync with it's next neighbor and if successful stops processing letting the next one continue. The main routine reads and assembles a list of (confirmed) chunks from the workers, starting with the first worker. This algorithm wastes some CPU and I/O if the data doesn't contain chunk boundaries, for example if the whole file contains nil bytes. If progress is not nil, it'll be updated with the confirmed chunk position in the file.

func MountIndex added in v0.2.0 

func MountIndex(ctx context.Context, idx Index, path, name string, s Store, n int) error

func NewHTTPHandler added in v0.2.0 

func NewHTTPHandler(s Store, writable bool) http.Handler

func NewHTTPIndexHandler added in v0.3.0 

func NewHTTPIndexHandler(s IndexStore, writable bool) http.Handler

func SipHash added in v0.2.0 

func SipHash(b []byte) uint64

SipHash is used to calculate the hash in Goodbye element items, hashing the filename.

func Tar added in v0.2.0 

func Tar(ctx context.Context, w io.Writer, src string) error

Tar implements the tar command which recursively parses a directory tree, and produces a stream of encoded casync format elements (catar file).

func UnTar added in v0.2.0 

func UnTar(ctx context.Context, r io.Reader, dst string, opts UntarOptions) error

UnTar implements the untar command, decoding a catar file and writing the contained tree to a target directory.

func UnTarIndex added in v0.2.0 

func UnTarIndex(ctx context.Context, dst string, index Index, s Store, n int, opts UntarOptions) error

UnTarIndex takes an index file (of a chunked catar), re-assembles the catar and decodes it on-the-fly into the target directory 'dst'. Uses n gorountines to retrieve and decompress the chunks.

func VerifyIndex added in v0.2.0 

func VerifyIndex(ctx context.Context, name string, idx Index, n int, progress func()) error

VerifyIndex re-calculates the checksums of a blob comparing it to a given index. Fails if the index does not match the blob.

Types

type ArchiveDecoder added in v0.2.0 

type ArchiveDecoder struct {
	// contains filtered or unexported fields
}

func NewArchiveDecoder added in v0.2.0 

func NewArchiveDecoder(r io.Reader) ArchiveDecoder

func (*ArchiveDecoder) Next added in v0.2.0 

func (a *ArchiveDecoder) Next() (interface{}, error)

Next returns a node from an archive, or nil if the end is reached. If NodeFile is returned, the caller should read the file body before calling Next() again as that invalidates the reader.

type Cache 

type Cache struct {
	// contains filtered or unexported fields
}

Cache is used to connect a (typically remote) store with a local store which functions as disk cache. Any request to the cache for a chunk will first be routed to the local store, and if that fails to the slower remote store. Any chunks retrieved from the remote store will be stored in the local one.

func NewCache 

func NewCache(s Store, l WriteStore) Cache

NewCache returns a cache router that uses a local store as cache before accessing a (supposedly slower) remote one.

func (Cache) Close added in v0.2.0 

func (c Cache) Close() error

func (Cache) GetChunk 

func (c Cache) GetChunk(id ChunkID) ([]byte, error)

GetChunk first asks the local store for the chunk and then the remote one. If we get a chunk from the remote, it's stored locally too.

func (Cache) HasChunk added in v0.2.0 

func (c Cache) HasChunk(id ChunkID) bool

HasChunk first checks the cache for the chunk, then the store.

func (Cache) String added in v0.2.0 

func (c Cache) String() string

type ChunkID 

type ChunkID [32]byte

ChunkID is the SHA512/256 in binary encoding

func ChunkIDFromSlice 

func ChunkIDFromSlice(b []byte) (ChunkID, error)

ChunkIDFromSlice converts a SHA512/256 encoded as byte slice into a ChunkID. It's expected the slice is of the correct length

func ChunkIDFromString 

func ChunkIDFromString(id string) (ChunkID, error)

ChunkIDFromString converts a SHA512/56 encoded as string into a ChunkID

func (ChunkID) String 

func (c ChunkID) String() string

type ChunkInvalid added in v0.2.0 

type ChunkInvalid struct {
	ID  ChunkID
	Sum ChunkID
}

ChunkInvalid means the hash of the chunk content doesn't match its ID

func (ChunkInvalid) Error added in v0.2.0 

func (e ChunkInvalid) Error() string

type ChunkMissing 

type ChunkMissing struct {
	ID ChunkID
}

ChunkMissing is returned by a store that can't find a requested chunk

func (ChunkMissing) Error 

func (e ChunkMissing) Error() string

type ChunkStorage added in v0.2.0 

type ChunkStorage struct {
	sync.Mutex
	// contains filtered or unexported fields
}

func NewChunkStorage added in v0.2.0 

func NewChunkStorage(ws WriteStore) *ChunkStorage

Stores chunks passed in the input channel asynchronously. Wait() will wait for until the input channel is closed or until there's an error, in which case it will return it.

func (*ChunkStorage) StoreChunk added in v0.2.0 

func (s *ChunkStorage) StoreChunk(id ChunkID, b []byte) (err error)

Stores a single chunk in a synchronous manner.

type Chunker added in v0.2.0 

type Chunker struct {
	// contains filtered or unexported fields
}

func NewChunker added in v0.2.0 

func NewChunker(r io.Reader, min, avg, max uint64) (Chunker, error)

func (*Chunker) Avg added in v0.2.0 

func (c *Chunker) Avg() uint64

Avg returns the average chunk size

func (*Chunker) Max added in v0.2.0 

func (c *Chunker) Max() uint64

Max returns the maximum chunk size

func (*Chunker) Min added in v0.2.0 

func (c *Chunker) Min() uint64

Min returns the minimum chunk size

func (*Chunker) Next added in v0.2.0 

func (c *Chunker) Next() (uint64, []byte, error)

Next returns the starting position as well as the chunk data. Returns an empty byte slice when complete

type ChunkingStats added in v0.2.0 

type ChunkingStats struct {
	ChunksAccepted uint64
	ChunksProduced uint64
}

type ConsoleIndexStore added in v0.3.0 

type ConsoleIndexStore struct{}

ConsoleIndexStore is used for writing/reading indexes from STDOUT/STDIN

func NewConsoleIndexStore added in v0.3.0 

func NewConsoleIndexStore() (ConsoleIndexStore, error)

NewConsoleStore creates an instance of an indexStore that reads/writes to and from console

func (ConsoleIndexStore) Close added in v0.3.0 

func (s ConsoleIndexStore) Close() error

func (ConsoleIndexStore) GetIndex added in v0.3.0 

func (s ConsoleIndexStore) GetIndex(string) (i Index, e error)

GetIndex reads an index from STDIN and returns it.

func (ConsoleIndexStore) GetIndexReader added in v0.3.0 

func (s ConsoleIndexStore) GetIndexReader(string) (io.ReadCloser, error)

GetIndexReader returns a reader from STDIN

func (ConsoleIndexStore) StoreIndex added in v0.3.0 

func (s ConsoleIndexStore) StoreIndex(name string, idx Index) error

StoreIndex writes the provided indes to STDOUT. The name is ignored.

func (ConsoleIndexStore) String added in v0.3.0 

func (r ConsoleIndexStore) String() string

type FormatACLDefault added in v0.2.0 

type FormatACLDefault struct {
	FormatHeader
	UserObjPermissions  uint64
	GroupObjPermissions uint64
	OtherPermissions    uint64
	MaskPermissions     uint64
}

type FormatACLGroup added in v0.2.0 

type FormatACLGroup struct {
	FormatHeader
	GID         uint64
	Permissions uint64
	Name        string
}

type FormatACLGroupObj added in v0.2.0 

type FormatACLGroupObj struct {
	FormatHeader
	Permissions uint64
}

type FormatACLUser added in v0.2.0 

type FormatACLUser struct {
	FormatHeader
	UID         uint64
	Permissions uint64
	Name        string
}

type FormatDecoder added in v0.2.0 

type FormatDecoder struct {
	// contains filtered or unexported fields
}

FormatDecoder is used to parse and break up a stream of casync format elements found in archives or index files.

func NewFormatDecoder added in v0.2.0 

func NewFormatDecoder(r io.Reader) FormatDecoder

func (*FormatDecoder) Next added in v0.2.0 

func (d *FormatDecoder) Next() (interface{}, error)

Next returns the next format element from the stream. If an element contains a reader, that reader should be used before any subsequent calls as it'll be invalidated then. Returns nil when the end is reached.

type FormatDevice added in v0.2.0 

type FormatDevice struct {
	FormatHeader
	Major uint64
	Minor uint64
}

type FormatEncoder added in v0.2.0 

type FormatEncoder struct {
	// contains filtered or unexported fields
}

FormatEncoder takes casync format elements and encodes them into a stream.

func NewFormatEncoder added in v0.2.0 

func NewFormatEncoder(w io.Writer) FormatEncoder

func (*FormatEncoder) Encode added in v0.2.0 

func (e *FormatEncoder) Encode(v interface{}) (int64, error)

type FormatEntry added in v0.2.0 

type FormatEntry struct {
	FormatHeader
	FeatureFlags uint64
	Mode         os.FileMode
	Flags        uint64
	UID          int
	GID          int
	MTime        time.Time
}

type FormatFCaps added in v0.2.0 

type FormatFCaps struct {
	FormatHeader
	Data []byte
}

type FormatFilename added in v0.2.0 

type FormatFilename struct {
	FormatHeader
	Name string
}

type FormatGoodbye added in v0.2.0 

type FormatGoodbye struct {
	FormatHeader
	Items []FormatGoodbyeItem
}

type FormatGoodbyeItem added in v0.2.0 

type FormatGoodbyeItem struct {
	Offset uint64
	Size   uint64
	Hash   uint64 // The last item in a list has the CaFormatGoodbyeTailMarker here
}

type FormatGroup added in v0.2.0 

type FormatGroup struct {
	FormatHeader
	Name string
}

type FormatHeader added in v0.2.0 

type FormatHeader struct {
	Size uint64
	Type uint64
}

type FormatIndex added in v0.2.0 

type FormatIndex struct {
	FormatHeader
	FeatureFlags uint64
	ChunkSizeMin uint64
	ChunkSizeAvg uint64
	ChunkSizeMax uint64
}

type FormatPayload added in v0.2.0 

type FormatPayload struct {
	FormatHeader
	Data io.Reader
}

type FormatSELinux added in v0.2.0 

type FormatSELinux struct {
	FormatHeader
	Label string
}
type FormatSymlink struct {
	FormatHeader
	Target string
}

type FormatTable added in v0.2.0 

type FormatTable struct {
	FormatHeader
	Items []FormatTableItem
}

type FormatTableItem added in v0.2.0 

type FormatTableItem struct {
	Offset uint64
	Chunk  ChunkID
}

type FormatUser added in v0.2.0 

type FormatUser struct {
	FormatHeader
	Name string
}

type FormatXAttr added in v0.2.0 

type FormatXAttr struct {
	FormatHeader
	NameAndValue string
}

type HTTPHandler added in v0.2.0 

type HTTPHandler struct {
	HTTPHandlerBase
	// contains filtered or unexported fields
}

func (HTTPHandler) ServeHTTP added in v0.2.0 

func (h HTTPHandler) ServeHTTP(w http.ResponseWriter, r *http.Request)

type HTTPHandlerBase added in v0.3.0 

type HTTPHandlerBase struct {
	// contains filtered or unexported fields
}

type HTTPIndexHandler added in v0.3.0 

type HTTPIndexHandler struct {
	HTTPHandlerBase
	// contains filtered or unexported fields
}

func (HTTPIndexHandler) ServeHTTP added in v0.3.0 

func (h HTTPIndexHandler) ServeHTTP(w http.ResponseWriter, r *http.Request)

type Hash added in v0.2.0 

type Hash struct {
	// contains filtered or unexported fields
}

Hash implements the rolling hash algorithm used to find chunk bounaries in a stream of bytes.

func NewHash added in v0.2.0 

func NewHash(size int, discriminator uint32) Hash

NewHash returns a new instance of a hash. size determines the length of the hash window used and the discriminator is used to find the boundary.

func (*Hash) Initialize added in v0.2.0 

func (h *Hash) Initialize(b []byte)

Initialize the window used for the rolling hash calculation. The size of the slice must match the window size

func (*Hash) IsBoundary added in v0.2.0 

func (h *Hash) IsBoundary() bool

IsBoundary returns true if the discriminator and hash match to signal a chunk boundary has been reached

func (*Hash) Reset added in v0.2.0 

func (h *Hash) Reset()

Reset the hash window and value

func (*Hash) Roll added in v0.2.0 

func (h *Hash) Roll(b byte)

Roll adds a new byte to the hash calculation. No useful value is returned until the hash window has been populated.

type Index 

type Index struct {
	Index  FormatIndex
	Chunks []IndexChunk
}

Index represents the content of an index file

func ChunkStream added in v0.2.0 

func ChunkStream(ctx context.Context, c Chunker, ws WriteStore, n int) (Index, error)

ChunkStream splits up a blob into chunks using the provided chunker (single stream), populates a store with the chunks and returns an index. Hashing and compression is performed in n goroutines while the hashing algorithm is performed serially.

func IndexFromReader added in v0.2.0 

func IndexFromReader(r io.Reader) (c Index, err error)

IndexFromReader parses a caibx structure (from a reader) and returns a populated Caibx object

func (*Index) Length added in v0.2.0 

func (i *Index) Length() int64

Length returns the total (uncompressed) size of the indexed stream

func (*Index) WriteTo added in v0.2.0 

func (i *Index) WriteTo(w io.Writer) (int64, error)

WriteTo writes the index and chunk table into a stream

type IndexChunk added in v0.2.0 

type IndexChunk struct {
	ID    ChunkID
	Start uint64
	Size  uint64
}

IndexChunk is a table entry in an index file containing the chunk ID (SHA256) Similar to an FormatTableItem but with Start and Size instead of just offset to make it easier to use throughout the application.

type IndexMountFS added in v0.2.0 

type IndexMountFS struct {
	FName string // File name in the mountpoint
	Idx   Index  // Index of the blob
	Store Store
	pathfs.FileSystem
}

IndexMountFS is used to FUSE mount an index file (as a blob, not an archive). It present a single file underneath the mountpoint.

func NewIndexMountFS added in v0.2.0 

func NewIndexMountFS(idx Index, name string, s Store) *IndexMountFS

func (*IndexMountFS) GetAttr added in v0.2.0 

func (me *IndexMountFS) GetAttr(name string, context *fuse.Context) (*fuse.Attr, fuse.Status)

func (*IndexMountFS) Open added in v0.2.0 

func (me *IndexMountFS) Open(name string, flags uint32, context *fuse.Context) (file nodefs.File, code fuse.Status)

func (*IndexMountFS) OpenDir added in v0.2.0 

func (me *IndexMountFS) OpenDir(name string, context *fuse.Context) (c []fuse.DirEntry, code fuse.Status)

type IndexMountFile added in v0.2.0 

type IndexMountFile struct {
	nodefs.File
	// contains filtered or unexported fields
}

IndexMountFile represents a (read-only) file handle on a blob in a FUSE mounted filesystem

func NewIndexMountFile added in v0.2.0 

func NewIndexMountFile(idx Index, s Store) *IndexMountFile

func (*IndexMountFile) GetAttr added in v0.2.0 

func (f *IndexMountFile) GetAttr(out *fuse.Attr) fuse.Status

func (*IndexMountFile) Read added in v0.2.0 

func (f *IndexMountFile) Read(dest []byte, off int64) (fuse.ReadResult, fuse.Status)

type IndexPos added in v0.2.0 

type IndexPos struct {
	Store  Store
	Index  Index
	Length int64 // total length of file
	// contains filtered or unexported fields
}

IndexPos represents a position inside an index file, to permit a seeking reader

func NewIndexReadSeeker added in v0.2.0 

func NewIndexReadSeeker(i Index, s Store) *IndexPos

func (*IndexPos) Read added in v0.2.0 

func (ip *IndexPos) Read(p []byte) (n int, err error)

func (*IndexPos) Seek added in v0.2.0 

func (ip *IndexPos) Seek(offset int64, whence int) (int64, error)

type IndexStore added in v0.3.0 

type IndexStore interface {
	GetIndexReader(name string) (io.ReadCloser, error)
	GetIndex(name string) (Index, error)
	io.Closer
	fmt.Stringer
}

type IndexWriteStore added in v0.3.0 

type IndexWriteStore interface {
	IndexStore
	StoreIndex(name string, idx Index) error
}

type Interrupted added in v0.2.0 

type Interrupted struct{}

Interrupted is returned when a user interrupted a long-running operation, for example by pressing Ctrl+C

func (Interrupted) Error added in v0.2.0 

func (e Interrupted) Error() string

type InvalidFormat added in v0.2.0 

type InvalidFormat struct {
	Msg string
}

InvalidFormat is returned when an error occurred when parsing an archive file

func (InvalidFormat) Error added in v0.2.0 

func (e InvalidFormat) Error() string

type LocalIndexStore added in v0.3.0 

type LocalIndexStore struct {
	Path string
}

LocalIndexStore is used to read/write index files on local disk

func NewLocaIndexStore added in v0.3.0 

func NewLocaIndexStore(path string) (LocalIndexStore, error)

NewLocalStore creates an instance of a local castore, it only checks presence of the store

func (LocalIndexStore) Close added in v0.3.0 

func (s LocalIndexStore) Close() error

func (LocalIndexStore) GetIndex added in v0.3.0 

func (s LocalIndexStore) GetIndex(name string) (i Index, e error)

GetIndex returns an Index structure from the store

func (LocalIndexStore) GetIndexReader added in v0.3.0 

func (s LocalIndexStore) GetIndexReader(name string) (rdr io.ReadCloser, e error)

GetIndexReader returns a reader of an index file in the store or an error if the specified index file does not exist.

func (LocalIndexStore) StoreIndex added in v0.3.0 

func (s LocalIndexStore) StoreIndex(name string, idx Index) error

StoreIndex stores an index in the index store with the given name.

func (LocalIndexStore) String added in v0.3.0 

func (r LocalIndexStore) String() string

type LocalStore 

type LocalStore struct {
	Base string

	// When accessing chunks, should mtime be updated? Useful when this is
	// a cache. Old chunks can be identified and removed from the store that way
	UpdateTimes bool
}

LocalStore casync store

func NewLocalStore 

func NewLocalStore(dir string) (LocalStore, error)

NewLocalStore creates an instance of a local castore, it only checks presence of the store

func (LocalStore) Close added in v0.2.0 

func (s LocalStore) Close() error

func (LocalStore) GetChunk 

func (s LocalStore) GetChunk(id ChunkID) ([]byte, error)

GetChunk reads and returns one (compressed!) chunk from the store

func (LocalStore) HasChunk added in v0.2.0 

func (s LocalStore) HasChunk(id ChunkID) bool

HasChunk returns true if the chunk is in the store

func (LocalStore) Prune added in v0.2.0 

func (s LocalStore) Prune(ctx context.Context, ids map[ChunkID]struct{}) error

Prune removes any chunks from the store that are not contained in a list of chunks

func (LocalStore) RemoveChunk added in v0.2.0 

func (s LocalStore) RemoveChunk(id ChunkID) error

RemoveChunk deletes a chunk, typically an invalid one, from the filesystem. Used when verifying and repairing caches.

func (LocalStore) StoreChunk 

func (s LocalStore) StoreChunk(id ChunkID, b []byte) error

StoreChunk adds a new chunk to the store

func (LocalStore) String added in v0.2.0 

func (s LocalStore) String() string

func (LocalStore) Verify added in v0.2.0 

func (s LocalStore) Verify(ctx context.Context, n int, repair bool) error

Verify all chunks in the store. If repair is set true, bad chunks are deleted. n determines the number of concurrent operations.

type Message 

type Message struct {
	Type uint64
	Body []byte
}

Message represents a command sent to, or received from the communication partner.

type NoSuchObject added in v0.3.0 

type NoSuchObject struct {
	// contains filtered or unexported fields
}

NoSuchObject is returned by a store that can't find a requested object

func (NoSuchObject) Error added in v0.3.0 

func (e NoSuchObject) Error() string

type NodeDevice added in v0.2.0 

type NodeDevice struct {
	Name  string
	UID   int
	GID   int
	Mode  os.FileMode
	Major uint64
	Minor uint64
	MTime time.Time
}

type NodeDirectory added in v0.2.0 

type NodeDirectory struct {
	Name  string
	UID   int
	GID   int
	Mode  os.FileMode
	MTime time.Time
}

type NodeFile added in v0.2.0 

type NodeFile struct {
	UID   int
	GID   int
	Mode  os.FileMode
	Name  string
	MTime time.Time
	Data  io.Reader
}
type NodeSymlink struct {
	Name   string
	UID    int
	GID    int
	Mode   os.FileMode
	MTime  time.Time
	Target string
}

type NullChunk added in v0.2.0 

type NullChunk struct {
	Data []byte
	ID   ChunkID
}

NullChunk is used in places where it's common to see requests for chunks containing only 0-bytes. When a chunked file has large areas of 0-bytes, the chunking algorithm does not produce split boundaries, which results in many chunks of 0-bytes of size MAX (max chunk size). The NullChunk can be used to make requesting this kind of chunk more efficient by serving it from memory, rather that request it from disk or network and decompress it repeatedly.

func NewNullChunk added in v0.2.0 

func NewNullChunk(size uint64) *NullChunk

NewNullChunk returns an initialized chunk consisting of 0-bytes of 'size' which must mach the max size used in the index to be effective

type ProgressBar added in v0.2.0 

type ProgressBar interface {
	Add(n int)
	Set(n int)
	Start()
	Stop()
}

type Protocol added in v0.2.0 

type Protocol struct {
	// contains filtered or unexported fields
}

Protocol handles the casync protocol when using remote stores via SSH

func NewProtocol added in v0.2.0 

func NewProtocol(r io.Reader, w io.Writer) *Protocol

NewProtocol creates a new casync protocol handler

func StartProtocol added in v0.2.0 

func StartProtocol(u *url.URL) (*Protocol, error)

StartProtocol initiates a connection to the remote store server using the value in CASYNC_SSH_PATH (default "ssh"), and executes the command in CASYNC_REMOTE_PATH (default "casync"). It then performs the HELLO handshake to initialze the connection

func (*Protocol) Initialize added in v0.2.0 

func (p *Protocol) Initialize(flags uint64) (uint64, error)

Initialize exchanges HELLOs with the other side to start a protocol session. Returns the (capability) flags provided by the other party.

func (*Protocol) ReadMessage added in v0.2.0 

func (p *Protocol) ReadMessage() (Message, error)

ReadMessage reads a generic message from the other end, verifies the length, extracts the type and returns the message body as byte slice

func (*Protocol) RecvHello added in v0.2.0 

func (p *Protocol) RecvHello() (uint64, error)

RecvHello waits for the server to send a HELLO, fails if anything else is received. Returns the flags provided by the server.

func (*Protocol) RequestChunk added in v0.2.0 

func (p *Protocol) RequestChunk(id ChunkID) ([]byte, error)

RequestChunk sends a request for a specific chunk to the server, waits for the response and returns the bytes in the chunk. Returns an error if the server reports the chunk as missing

func (*Protocol) SendGoodbye added in v0.2.0 

func (p *Protocol) SendGoodbye() error

SendGoodbye tells the other side to terminate gracefully

func (*Protocol) SendHello added in v0.2.0 

func (p *Protocol) SendHello(flags uint64) error

SendHello sends a HELLO message to the server, with the flags signaling which service is being requested from it.

func (*Protocol) SendMissing added in v0.2.0 

func (p *Protocol) SendMissing(id ChunkID) error

SendMissing tells the client that the requested chunk is not available

func (*Protocol) SendProtocolChunk added in v0.2.0 

func (p *Protocol) SendProtocolChunk(id ChunkID, flags uint64, chunk []byte) error

SendProtocolChunk responds to a request with the content of a chunk

func (*Protocol) SendProtocolRequest added in v0.2.0 

func (p *Protocol) SendProtocolRequest(id ChunkID, flags uint64) error

SendProtocolRequest requests a chunk from a server

func (*Protocol) WriteMessage added in v0.2.0 

func (p *Protocol) WriteMessage(m Message) error

WriteMessage sends a generic message to the server

type ProtocolServer added in v0.2.0 

type ProtocolServer struct {
	// contains filtered or unexported fields
}

ProtocolServer serves up chunks from a local store using the casync protocol

func NewProtocolServer added in v0.2.0 

func NewProtocolServer(r io.Reader, w io.Writer, s Store) *ProtocolServer

NewProtocolServer returns an initialized server that can serve chunks from a chunk store via the casync protocol

func (*ProtocolServer) Serve added in v0.2.0 

func (s *ProtocolServer) Serve(ctx context.Context) error

Serve starts the protocol server. Blocks unless an error is encountered

type PruneStore added in v0.2.0 

type PruneStore interface {
	WriteStore
	Prune(ctx context.Context, ids map[ChunkID]struct{}) error
}

PruneStore is a store that supports pruning of chunks

type RemoteHTTP added in v0.2.0 

type RemoteHTTP struct {
	*RemoteHTTPBase
}

RemoteHTTP is a remote casync store accessed via HTTP.

func NewRemoteHTTPStore added in v0.2.0 

func NewRemoteHTTPStore(location *url.URL, n int, cert string, key string) (*RemoteHTTP, error)

NewRemoteHTTPStore initializes a new store that pulls chunks via HTTP(S) from a remote web server. n defines the size of idle connections allowed.

func (*RemoteHTTP) GetChunk added in v0.2.0 

func (r *RemoteHTTP) GetChunk(id ChunkID) ([]byte, error)

GetChunk reads and returns one (compressed!) chunk from the store

func (*RemoteHTTP) HasChunk added in v0.2.0 

func (r *RemoteHTTP) HasChunk(id ChunkID) bool

HasChunk returns true if the chunk is in the store

func (*RemoteHTTP) StoreChunk added in v0.2.0 

func (r *RemoteHTTP) StoreChunk(id ChunkID, b []byte) error

StoreChunk adds a new chunk to the store

type RemoteHTTPBase added in v0.3.0 

type RemoteHTTPBase struct {
	// contains filtered or unexported fields
}

RemoteHTTP is a remote casync store accessed via HTTP.

func NewRemoteHTTPStoreBase added in v0.3.0 

func NewRemoteHTTPStoreBase(location *url.URL, n int, cert string, key string) (*RemoteHTTPBase, error)

func (*RemoteHTTPBase) Close added in v0.3.0 

func (r *RemoteHTTPBase) Close() error

func (*RemoteHTTPBase) GetObject added in v0.3.0 

func (r *RemoteHTTPBase) GetObject(name string) ([]byte, error)

GetObject reads and returns an object in the form of []byte from the store

func (*RemoteHTTPBase) SetErrorRetry added in v0.3.0 

func (r *RemoteHTTPBase) SetErrorRetry(n int)

SetErrorRetry defines how many HTTP errors are retried. This can be useful when dealing with unreliable networks that can timeout or where errors are transient.

func (*RemoteHTTPBase) SetTimeout added in v0.3.0 

func (r *RemoteHTTPBase) SetTimeout(timeout time.Duration)

SetTimeout configures the timeout on the HTTP client for all requests

func (*RemoteHTTPBase) StoreObject added in v0.3.0 

func (r *RemoteHTTPBase) StoreObject(name string, rdr io.Reader) error

StoreObject stores an object to the store.

func (*RemoteHTTPBase) String added in v0.3.0 

func (r *RemoteHTTPBase) String() string

type RemoteHTTPIndex added in v0.3.0 

type RemoteHTTPIndex struct {
	*RemoteHTTPBase
}

RemoteHTTP is a remote index store accessed via HTTP.

func NewRemoteHTTPIndexStore added in v0.3.0 

func NewRemoteHTTPIndexStore(location *url.URL, n int, cert string, key string) (*RemoteHTTPIndex, error)

NewRemoteHTTPStore initializes a new store that pulls the specified index file via HTTP(S) from a remote web server.

func (*RemoteHTTPIndex) GetIndex added in v0.3.0 

func (r *RemoteHTTPIndex) GetIndex(name string) (i Index, e error)

GetIndex returns an Index structure from the store

func (RemoteHTTPIndex) GetIndexReader added in v0.3.0 

func (r RemoteHTTPIndex) GetIndexReader(name string) (rdr io.ReadCloser, e error)

Get and Index Reader from an HTTP store, returns an error if the specified index file does not exist.

func (*RemoteHTTPIndex) StoreIndex added in v0.3.0 

func (r *RemoteHTTPIndex) StoreIndex(name string, idx Index) error

StoreChunk adds a new chunk to the store

type RemoteSSH 

type RemoteSSH struct {
	// contains filtered or unexported fields
}

RemoteSSH is a remote casync store accessed via SSH. Supports running multiple sessions to improve throughput.

func NewRemoteSSHStore 

func NewRemoteSSHStore(location *url.URL, n int) (*RemoteSSH, error)

NewRemoteSSHStore establishes up to n connections with a casync chunk server

func (*RemoteSSH) Close added in v0.2.0 

func (r *RemoteSSH) Close() error

Close terminates all client connections

func (*RemoteSSH) GetChunk 

func (r *RemoteSSH) GetChunk(id ChunkID) ([]byte, error)

GetChunk requests a chunk from the server and returns a (compressed) one. It uses any of the n sessions this store maintains in its pool. Blocks until one session becomes available

func (*RemoteSSH) HasChunk added in v0.2.0 

func (r *RemoteSSH) HasChunk(id ChunkID) bool

HasChunk returns true if the chunk is in the store. TODO: Implementing it this way, pulling the whole chunk just to see if it's present, is very inefficient. I'm not aware of a way to implement it with the casync protocol any other way.

func (*RemoteSSH) String added in v0.2.0 

func (r *RemoteSSH) String() string

type S3IndexStore added in v0.3.0 

type S3IndexStore struct {
	S3StoreBase
}

S3Store is a read-write store with S3 backing

func NewS3IndexStore added in v0.3.0 

func NewS3IndexStore(location *url.URL, s3Creds *credentials.Credentials, region string) (s S3IndexStore, e error)

NewS3Store creates an index store with S3 backing. The URL should be provided like this: s3+http://host:port/bucket Credentials are passed in via the environment variables S3_ACCESS_KEY and S3S3_SECRET_KEY, or via the desync config file.

func (S3IndexStore) GetIndex added in v0.3.0 

func (s S3IndexStore) GetIndex(name string) (i Index, e error)

GetIndex returns an Index structure from the store

func (S3IndexStore) GetIndexReader added in v0.3.0 

func (s S3IndexStore) GetIndexReader(name string) (r io.ReadCloser, e error)

Get and Index Reader from an S3 store, returns an error if the specified index file does not exist.

func (S3IndexStore) StoreIndex added in v0.3.0 

func (s S3IndexStore) StoreIndex(name string, idx Index) error

StoreIndex writes the index file to the S3 store

type S3Store added in v0.2.0 

type S3Store struct {
	S3StoreBase
}

S3Store is a read-write store with S3 backing

func NewS3Store added in v0.2.0 

func NewS3Store(location *url.URL, s3Creds *credentials.Credentials, region string) (s S3Store, e error)

NewS3Store creates a chunk store with S3 backing. The URL should be provided like this: s3+http://host:port/bucket Credentials are passed in via the environment variables S3_ACCESS_KEY and S3S3_SECRET_KEY, or via the desync config file.

func (S3Store) GetChunk added in v0.2.0 

func (s S3Store) GetChunk(id ChunkID) ([]byte, error)

GetChunk reads and returns one (compressed!) chunk from the store

func (S3Store) HasChunk added in v0.2.0 

func (s S3Store) HasChunk(id ChunkID) bool

HasChunk returns true if the chunk is in the store

func (S3Store) Prune added in v0.2.0 

func (s S3Store) Prune(ctx context.Context, ids map[ChunkID]struct{}) error

Prune removes any chunks from the store that are not contained in a list (map)

func (S3Store) RemoveChunk added in v0.2.0 

func (s S3Store) RemoveChunk(id ChunkID) error

RemoveChunk deletes a chunk, typically an invalid one, from the filesystem. Used when verifying and repairing caches.

func (S3Store) StoreChunk added in v0.2.0 

func (s S3Store) StoreChunk(id ChunkID, b []byte) error

StoreChunk adds a new chunk to the store

func (S3Store) Upgrade added in v0.2.0 

func (s S3Store) Upgrade(ctx context.Context) error

Upgrade converts the storage layout in S3 from the old format (just a flat layout) to the current layout which prefixes every chunk with the first 4 characters of the checksum as well as a .cacnk extension. This aligns the layout with that of local stores and allows the used of sync tools outside of this tool, local stores could be copied into S3 for example.

type S3StoreBase added in v0.3.0 

type S3StoreBase struct {
	Location string
	// contains filtered or unexported fields
}

S3Store is a read-write store with S3 backing

func NewS3StoreBase added in v0.3.0 

func NewS3StoreBase(u *url.URL, s3Creds *credentials.Credentials, region string) (S3StoreBase, error)

func (S3StoreBase) Close added in v0.3.0 

func (s S3StoreBase) Close() error

func (S3StoreBase) String added in v0.3.0 

func (s S3StoreBase) String() string

type SFTPIndexStore added in v0.3.0 

type SFTPIndexStore struct {
	*SFTPStoreBase
}

func NewSFTPIndexStore added in v0.3.0 

func NewSFTPIndexStore(location *url.URL) (*SFTPIndexStore, error)

NewSFTPIndexStore establishes up to n connections with a casync index server

func (*SFTPIndexStore) GetIndex added in v0.3.0 

func (s *SFTPIndexStore) GetIndex(name string) (i Index, e error)

Get and Index from an SFTP store, returns an error if the specified index file does not exist.

func (*SFTPIndexStore) GetIndexReader added in v0.3.0 

func (s *SFTPIndexStore) GetIndexReader(name string) (r io.ReadCloser, e error)

Get and Index Reader from an SFTP store, returns an error if the specified index file does not exist.

func (*SFTPIndexStore) StoreIndex added in v0.3.0 

func (s *SFTPIndexStore) StoreIndex(name string, idx Index) error

StoreChunk adds a new chunk to the store

type SFTPStore added in v0.2.0 

type SFTPStore struct {
	*SFTPStoreBase
}

func NewSFTPStore added in v0.2.0 

func NewSFTPStore(location *url.URL) (*SFTPStore, error)

NewRemoteSSHStore establishes up to n connections with a casync chunk server

func (*SFTPStore) GetChunk added in v0.2.0 

func (s *SFTPStore) GetChunk(id ChunkID) ([]byte, error)

Get a chunk from an SFTP store, returns ChunkMissing if the file does not exist

func (*SFTPStore) HasChunk added in v0.2.0 

func (s *SFTPStore) HasChunk(id ChunkID) bool

HasChunk returns true if the chunk is in the store

func (*SFTPStore) Prune added in v0.2.0 

func (s *SFTPStore) Prune(ctx context.Context, ids map[ChunkID]struct{}) error

Prune removes any chunks from the store that are not contained in a list of chunks

func (*SFTPStore) RemoveChunk added in v0.2.0 

func (s *SFTPStore) RemoveChunk(id ChunkID) error

RemoveChunk deletes a chunk, typically an invalid one, from the filesystem. Used when verifying and repairing caches.

func (*SFTPStore) StoreChunk added in v0.2.0 

func (s *SFTPStore) StoreChunk(id ChunkID, b []byte) error

StoreChunk adds a new chunk to the store

type SFTPStoreBase added in v0.3.0 

type SFTPStoreBase struct {
	// contains filtered or unexported fields
}

SFTPStore is a remote store that uses SFTP over SSH to access chunks

func (*SFTPStoreBase) Close added in v0.3.0 

func (s *SFTPStoreBase) Close() error

Close terminates all client connections

func (*SFTPStoreBase) StoreObject added in v0.3.0 

func (s *SFTPStoreBase) StoreObject(name string, r io.Reader) error

StoreChunk adds a new chunk to the store

func (*SFTPStoreBase) String added in v0.3.0 

func (s *SFTPStoreBase) String() string

type Store 

type Store interface {
	GetChunk(id ChunkID) ([]byte, error)
	HasChunk(id ChunkID) bool
	io.Closer
	fmt.Stringer
}

Store is a generic interface implemented by read-only stores, like SSH or HTTP remote stores currently.

type StoreRouter added in v0.2.0 

type StoreRouter struct {
	Stores []Store
}

StoreRouter is used to route requests to multiple stores. When a chunk is requested from the router, it'll query the first store and if that returns ChunkMissing, it'll move on to the next.

func NewStoreRouter added in v0.2.0 

func NewStoreRouter(stores ...Store) StoreRouter

NewStoreRouter returns an initialized router

func (StoreRouter) Close added in v0.2.0 

func (r StoreRouter) Close() error

Close calls the Close() method on every store in the router. Returns only the first error encountered.

func (StoreRouter) GetChunk added in v0.2.0 

func (r StoreRouter) GetChunk(id ChunkID) ([]byte, error)

GetChunk queries the available stores in order and moves to the next if it gets a ChunkMissing. Fails if any store returns a different error.

func (StoreRouter) HasChunk added in v0.2.0 

func (r StoreRouter) HasChunk(id ChunkID) bool

HasChunk returns true if one of the containing stores has the chunk. It goes through the stores in order and returns as soon as the chunk is found.

func (StoreRouter) String added in v0.2.0 

func (r StoreRouter) String() string

type UntarOptions added in v0.2.0 

type UntarOptions struct {
	NoSameOwner       bool
	NoSamePermissions bool
}

UntarOptions are used to influence the behaviour of untar

type WriteStore added in v0.2.0 

type WriteStore interface {
	Store
	StoreChunk(id ChunkID, b []byte) error
}

WriteStore is implemented by stores supporting both read and write operations such as a local store or an S3 store.

Source Files

View all Source files

Directories

Path Synopsis
cmd
desync

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.