repository

package
v0.0.0-...-53483ec Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 27, 2023 License: BSD-2-Clause Imports: 30 Imported by: 3

Documentation

Overview

Package repository implements a restic repository on top of a backend. In the following the abstractions used for this package are listed. More information can be found in the restic design document.

File

A file is a named handle for some data saved in the backend. For the local backend, this corresponds to actual files saved to disk. Usually, the SHA256 hash of the content is used for a file's name (hexadecimal, in lower-case ASCII characters). An exception is the file `config`. Most files are encrypted before being saved in a backend. This means that the name is the hash of the ciphertext.

Blob

A blob is a number of bytes that has a type (data or tree). Blobs are identified by an ID, which is the SHA256 hash of the blobs' contents. One or more blobs are bundled together in a Pack and then saved to the backend. Blobs are always encrypted before being bundled in a Pack.

Pack

A Pack is a File in the backend that contains one or more (encrypted) blobs, followed by a header at the end of the Pack. The header is encrypted and contains the ID, type, length and offset for each blob contained in the Pack.

Index

Constants

View Source 
const TestChunkerPol = chunker.Pol(0x3DA3358B4DC173)

Variables

View Source 
var (
	// ErrNoKeyFound is returned when no key for the repository could be decrypted.
	ErrNoKeyFound = errors.New("wrong password or no key found")

	// ErrMaxKeysReached is returned when the maximum number of keys was checked and no key could be found.
	ErrMaxKeysReached = errors.Fatal("maximum number of keys reached")
)
View Source 
var (
	// KDFTimeout specifies the maximum runtime for the KDF.
	KDFTimeout = 500 * time.Millisecond

	// KDFMemory limits the memory the KDF is allowed to use.
	KDFMemory = 60
)
View Source 
var IndexFull = func(idx *Index) bool {
	idx.m.Lock()
	defer idx.m.Unlock()

	debug.Log("checking whether index %p is full", idx)

	var blobs uint
	for typ := range idx.byType {
		blobs += idx.byType[typ].len()
	}
	age := time.Since(idx.created)

	switch {
	case age >= indexMaxAge:
		debug.Log("index %p is old enough", idx, age)
		return true
	case blobs >= indexMaxBlobs:
		debug.Log("index %p has %d blobs", idx, blobs)
		return true
	}

	debug.Log("index %p only has %d blobs and is too young (%v)", idx, blobs, age)
	return false

}

IndexFull returns true iff the index is "full enough" to be saved as a preliminary index.

View Source 
var Params *crypto.Params

Params tracks the parameters used for the KDF. If not set, it will be calibrated on the first run of AddKey().

Functions

func DownloadAndHash 

func DownloadAndHash(ctx context.Context, be Loader, h restic.Handle) (tmpfile *os.File, hash restic.ID, size int64, err error)

DownloadAndHash is all-in-one helper to download content of the file at h to a temporary filesystem location and calculate ID of the contents. Returned (temporary) file is positioned at the beginning of the file; it is the reponsibility of the caller to close and delete the file.

func ForAllIndexes 

func ForAllIndexes(ctx context.Context, repo restic.Repository,
	fn func(id restic.ID, index *Index, oldFormat bool, err error) error) error

ForAllIndexes loads all index files in parallel and calls the given callback. It is guaranteed that the function is not run concurrently. If the callback returns an error, this function is cancelled and also returns that error.

func Repack 

func Repack(ctx context.Context, repo restic.Repository, packs restic.IDSet, keepBlobs restic.BlobSet, p *progress.Counter) (obsoletePacks restic.IDSet, err error)

Repack takes a list of packs together with a list of blobs contained in these packs. Each pack is loaded and the blobs listed in keepBlobs is saved into a new pack. Returned is the list of obsolete packs which can then be removed.

The map keepBlobs is modified by Repack, it is used to keep track of which blobs have been processed.

func RunWorkers 

func RunWorkers(count int, workerFunc func() error) error

RunWorkers runs count instances of workerFunc using an errgroup.Group. If an error occurs in one of the workers, it is returned.

func SaveIndex 

func SaveIndex(ctx context.Context, repo restic.Repository, index *Index) (restic.ID, error)

SaveIndex saves an index in the repository.

func TestBackend 

func TestBackend(t testing.TB) (be restic.Backend, cleanup func())

TestBackend returns a fully configured in-memory backend.

func TestOpenLocal 

func TestOpenLocal(t testing.TB, dir string) (r restic.Repository)

TestOpenLocal opens a local repository.

func TestRepository 

func TestRepository(t testing.TB) (r restic.Repository, cleanup func())

TestRepository returns a repository initialized with a test password on an in-memory backend. When the environment variable RESTIC_TEST_REPO is set to a non-existing directory, a local backend is created there and this is used instead. The directory is not removed, but left there for inspection.

func TestRepositoryWithBackend 

func TestRepositoryWithBackend(t testing.TB, be restic.Backend) (r restic.Repository, cleanup func())

TestRepositoryWithBackend returns a repository initialized with a test password. If be is nil, an in-memory backend is used. A constant polynomial is used for the chunker and low-security test parameters.

func TestUseLowSecurityKDFParameters 

func TestUseLowSecurityKDFParameters(t logger)

TestUseLowSecurityKDFParameters configures low-security KDF parameters for testing.

Types

type EachByPackResult 

type EachByPackResult struct {
	// contains filtered or unexported fields
}

type Index 

type Index struct {
	// contains filtered or unexported fields
}

Index holds lookup tables for id -> pack.

func DecodeIndex 

func DecodeIndex(buf []byte, id restic.ID) (idx *Index, oldFormat bool, err error)

DecodeIndex unserializes an index from buf.

func NewIndex 

func NewIndex() *Index

NewIndex returns a new index.

func (*Index) AddToSupersedes 

func (idx *Index) AddToSupersedes(ids ...restic.ID) error

AddToSupersedes adds the ids to the list of indexes superseded by this index. If the index has already been finalized, an error is returned.

func (*Index) Count 

func (idx *Index) Count(t restic.BlobType) (n uint)

Count returns the number of blobs of type t in the index.

func (*Index) Dump 

func (idx *Index) Dump(w io.Writer) error

Dump writes the pretty-printed JSON representation of the index to w.

func (*Index) Each 

func (idx *Index) Each(ctx context.Context) <-chan restic.PackedBlob

Each returns a channel that yields all blobs known to the index. When the context is cancelled, the background goroutine terminates. This blocks any modification of the index.

func (*Index) EachByPack 

func (idx *Index) EachByPack(ctx context.Context, packBlacklist restic.IDSet) <-chan EachByPackResult

EachByPack returns a channel that yields all blobs known to the index grouped by packID but ignoring blobs with a packID in packPlacklist for finalized indexes. This filtering is used when rebuilding the index where we need to ignore packs from the finalized index which have been re-read into a non-finalized index. When the context is cancelled, the background goroutine terminates. This blocks any modification of the index.

func (*Index) Encode 

func (idx *Index) Encode(w io.Writer) error

Encode writes the JSON serialization of the index to the writer w.

func (*Index) Final 

func (idx *Index) Final() bool

Final returns true iff the index is already written to the repository, it is finalized.

func (*Index) Finalize 

func (idx *Index) Finalize()

Finalize sets the index to final.

func (*Index) Has 

func (idx *Index) Has(bh restic.BlobHandle) bool

Has returns true iff the id is listed in the index.

func (*Index) IDs 

func (idx *Index) IDs() (restic.IDs, error)

IDs returns the IDs of the index, if available. If the index is not yet finalized, an error is returned.

func (*Index) ListPack 

func (idx *Index) ListPack(id restic.ID) (pbs []restic.PackedBlob)

ListPack returns a list of blobs contained in a pack.

func (*Index) Lookup 

func (idx *Index) Lookup(bh restic.BlobHandle, pbs []restic.PackedBlob) []restic.PackedBlob

Lookup queries the index for the blob ID and returns all entries including duplicates. Adds found entries to blobs and returns the result.

func (*Index) LookupSize 

func (idx *Index) LookupSize(bh restic.BlobHandle) (plaintextLength uint, found bool)

LookupSize returns the length of the plaintext content of the blob with the given id.

func (*Index) MixedPacks 

func (idx *Index) MixedPacks() restic.IDSet

MixedPacks returns an IDSet that contain packs which have mixed blobs.

func (*Index) Packs 

func (idx *Index) Packs() restic.IDSet

Packs returns all packs in this index

func (*Index) SetID 

func (idx *Index) SetID(id restic.ID) error

SetID sets the ID the index has been written to. This requires that Finalize() has been called before, otherwise an error is returned.

func (*Index) Store 

func (idx *Index) Store(pb restic.PackedBlob)

Store remembers the id and pack in the index.

func (*Index) StorePack 

func (idx *Index) StorePack(id restic.ID, blobs []restic.Blob)

StorePack remembers the ids of all blobs of a given pack in the index

func (*Index) Supersedes 

func (idx *Index) Supersedes() restic.IDs

Supersedes returns the list of indexes this index supersedes, if any.

type Key 

type Key struct {
	Created  time.Time `json:"created"`
	Username string    `json:"username"`
	Hostname string    `json:"hostname"`

	KDF  string `json:"kdf"`
	N    int    `json:"N"`
	R    int    `json:"r"`
	P    int    `json:"p"`
	Salt []byte `json:"salt"`
	Data []byte `json:"data"`
	// contains filtered or unexported fields
}

Key represents an encrypted master key for a repository.

func AddKey 

func AddKey(ctx context.Context, s *Repository, password, username, hostname string, template *crypto.Key) (*Key, error)

AddKey adds a new key to an already existing repository.

func LoadKey 

func LoadKey(ctx context.Context, s *Repository, name string) (k *Key, err error)

LoadKey loads a key from the backend.

func Open 

func Open(reader io.Reader, password string) (*Key, error)

rubiojr: added Open tries do decrypt the key specified by name with the given password.

func OpenKey 

func OpenKey(ctx context.Context, s *Repository, name string, password string) (*Key, error)

OpenKey tries do decrypt the key specified by name with the given password.

func OpenKeyFromFile 

func OpenKeyFromFile(file, password string) (*Key, error)

rubiojr: added

func SearchKey 

func SearchKey(ctx context.Context, s *Repository, password string, maxKeys int, keyHint string) (k *Key, err error)

SearchKey tries to decrypt at most maxKeys keys in the backend with the given password. If none could be found, ErrNoKeyFound is returned. When maxKeys is reached, ErrMaxKeysReached is returned. When setting maxKeys to zero, all keys in the repo are checked.

func (Key) Name 

func (k Key) Name() string

Name returns an identifier for the key.

func (*Key) String 

func (k *Key) String() string

func (*Key) Valid 

func (k *Key) Valid() bool

Valid tests whether the mac and encryption keys are valid (i.e. not zero)

type Loader 

type Loader interface {
	Load(ctx context.Context, h restic.Handle, length int, offset int64, fn func(rd io.Reader) error) error
}

Loader allows loading data from a backend.

type MasterIndex 

type MasterIndex struct {
	// contains filtered or unexported fields
}

MasterIndex is a collection of indexes and IDs of chunks that are in the process of being saved.

func NewMasterIndex 

func NewMasterIndex() *MasterIndex

NewMasterIndex creates a new master index.

func (*MasterIndex) All 

func (mi *MasterIndex) All() []*Index

All returns all indexes.

func (*MasterIndex) Count 

func (mi *MasterIndex) Count(t restic.BlobType) (n uint)

Count returns the number of blobs of type t in the index.

func (*MasterIndex) Each 

func (mi *MasterIndex) Each(ctx context.Context) <-chan restic.PackedBlob

Each returns a channel that yields all blobs known to the index. When the context is cancelled, the background goroutine terminates. This blocks any modification of the index.

func (*MasterIndex) FinalizeFullIndexes 

func (mi *MasterIndex) FinalizeFullIndexes() []*Index

FinalizeFullIndexes finalizes all indexes that are full and returns that list.

func (*MasterIndex) FinalizeNotFinalIndexes 

func (mi *MasterIndex) FinalizeNotFinalIndexes() []*Index

FinalizeNotFinalIndexes finalizes all indexes that have not yet been saved and returns that list

func (*MasterIndex) Has 

func (mi *MasterIndex) Has(bh restic.BlobHandle) bool

Has queries all known Indexes for the ID and returns the first match. Also returns true if the ID is pending.

func (*MasterIndex) Insert 

func (mi *MasterIndex) Insert(idx *Index)

Insert adds a new index to the MasterIndex.

func (*MasterIndex) IsMixedPack 

func (mi *MasterIndex) IsMixedPack(packID restic.ID) bool

func (*MasterIndex) Lookup 

func (mi *MasterIndex) Lookup(bh restic.BlobHandle) (pbs []restic.PackedBlob)

Lookup queries all known Indexes for the ID and returns all matches.

func (*MasterIndex) LookupSize 

func (mi *MasterIndex) LookupSize(bh restic.BlobHandle) (uint, bool)

LookupSize queries all known Indexes for the ID and returns the first match.

func (*MasterIndex) MergeFinalIndexes 

func (mi *MasterIndex) MergeFinalIndexes() error

MergeFinalIndexes merges all final indexes together. After calling, there will be only one big final index in MasterIndex containing all final index contents. Indexes that are not final are left untouched. This merging can only be called after all index files are loaded - as removing of superseded index contents is only possible for unmerged indexes.

func (*MasterIndex) PackSize 

func (mi *MasterIndex) PackSize(ctx context.Context, onlyHdr bool) map[restic.ID]int64

PackSize returns the size of all packs computed by index information. If onlyHdr is set to true, only the size of the header is returned Note that this function only gives correct sizes, if there are no duplicates in the index.

func (*MasterIndex) Packs 

func (mi *MasterIndex) Packs(packBlacklist restic.IDSet) restic.IDSet

Packs returns all packs that are covered by the index. If packBlacklist is given, those packs are only contained in the resulting IDSet if they are contained in a non-final (newly written) index.

func (*MasterIndex) Save 

func (mi *MasterIndex) Save(ctx context.Context, repo restic.Repository, packBlacklist restic.IDSet, extraObsolete restic.IDs, p *progress.Counter) (obsolete restic.IDSet, err error)

Save saves all known indexes to index files, leaving out any packs whose ID is contained in packBlacklist from finalized indexes. The new index contains the IDs of all known indexes in the "supersedes" field. The IDs are also returned in the IDSet obsolete. After calling this function, you should remove the obsolete index files.

func (*MasterIndex) StorePack 

func (mi *MasterIndex) StorePack(id restic.ID, blobs []restic.Blob)

StorePack remembers the id and pack in the index.

type Packer 

type Packer struct {
	*pack.Packer
	// contains filtered or unexported fields
}

Packer holds a pack.Packer together with a hash writer.

type Repository 

type Repository struct {
	Cache *cache.Cache
	// contains filtered or unexported fields
}

Repository is used to access a repository in a backend.

func New 

func New(be restic.Backend) *Repository

New returns a new repository with backend be.

func (*Repository) Backend 

func (r *Repository) Backend() restic.Backend

Backend returns the backend for the repository.

func (*Repository) Close 

func (r *Repository) Close() error

Close closes the repository by closing the backend.

func (*Repository) Config 

func (r *Repository) Config() restic.Config

Config returns the repository configuration.

func (*Repository) CreateIndexFromPacks 

func (r *Repository) CreateIndexFromPacks(ctx context.Context, packsize map[restic.ID]int64, p *progress.Counter) (invalid restic.IDs, err error)

CreateIndexFromPacks creates a new index by reading all given pack files (with sizes). The index is added to the MasterIndex but not marked as finalized. Returned is the list of pack files which could not be read.

func (*Repository) Delete 

func (r *Repository) Delete(ctx context.Context) error

Delete calls backend.Delete() if implemented, and returns an error otherwise.

func (*Repository) DisableAutoIndexUpdate 

func (r *Repository) DisableAutoIndexUpdate()

DisableAutoIndexUpdate deactives the automatic finalization and upload of new indexes once these are full

func (*Repository) Flush 

func (r *Repository) Flush(ctx context.Context) error

Flush saves all remaining packs and the index

func (*Repository) FlushPacks 

func (r *Repository) FlushPacks(ctx context.Context) error

FlushPacks saves all remaining packs.

func (*Repository) Index 

func (r *Repository) Index() restic.MasterIndex

Index returns the currently used MasterIndex.

func (*Repository) Init 

func (r *Repository) Init(ctx context.Context, password string, chunkerPolynomial *chunker.Pol) error

Init creates a new master key with the supplied password, initializes and saves the repository config.

func (*Repository) Key 

func (r *Repository) Key() *crypto.Key

Key returns the current master key.

func (*Repository) KeyName 

func (r *Repository) KeyName() string

KeyName returns the name of the current key in the backend.

func (*Repository) List 

func (r *Repository) List(ctx context.Context, t restic.FileType, fn func(restic.ID, int64) error) error

List runs fn for all files of type t in the repo.

func (*Repository) ListPack 

func (r *Repository) ListPack(ctx context.Context, id restic.ID, size int64) ([]restic.Blob, uint32, error)

ListPack returns the list of blobs saved in the pack id and the length of the the pack header.

func (*Repository) LoadAndDecrypt 

func (r *Repository) LoadAndDecrypt(ctx context.Context, buf []byte, t restic.FileType, id restic.ID) ([]byte, error)

LoadAndDecrypt loads and decrypts the file with the given type and ID, using the supplied buffer (which must be empty). If the buffer is nil, a new buffer will be allocated and returned.

func (*Repository) LoadBlob 

func (r *Repository) LoadBlob(ctx context.Context, t restic.BlobType, id restic.ID, buf []byte) ([]byte, error)

LoadBlob loads a blob of type t from the repository. It may use all of buf[:cap(buf)] as scratch space.

func (*Repository) LoadIndex 

func (r *Repository) LoadIndex(ctx context.Context) error

LoadIndex loads all index files from the backend in parallel and stores them in the master index. The first error that occurred is returned.

func (*Repository) LoadJSONUnpacked 

func (r *Repository) LoadJSONUnpacked(ctx context.Context, t restic.FileType, id restic.ID, item interface{}) (err error)

LoadJSONUnpacked decrypts the data and afterwards calls json.Unmarshal on the item.

func (*Repository) LoadTree 

func (r *Repository) LoadTree(ctx context.Context, id restic.ID) (*restic.Tree, error)

LoadTree loads a tree from the repository.

func (*Repository) LookupBlobSize 

func (r *Repository) LookupBlobSize(id restic.ID, tpe restic.BlobType) (uint, bool)

LookupBlobSize returns the size of blob id.

func (*Repository) PrefixLength 

func (r *Repository) PrefixLength(ctx context.Context, t restic.FileType) (int, error)

PrefixLength returns the number of bytes required so that all prefixes of all IDs of type t are unique.

func (*Repository) PrepareCache 

func (r *Repository) PrepareCache(indexIDs restic.IDSet) error

PrepareCache initializes the local cache. indexIDs is the list of IDs of index files still present in the repo.

func (*Repository) SaveAndEncrypt 

func (r *Repository) SaveAndEncrypt(ctx context.Context, t restic.BlobType, data []byte, id restic.ID) error

SaveAndEncrypt encrypts data and stores it to the backend as type t. If data is small enough, it will be packed together with other small blobs. The caller must ensure that the id matches the data.

func (*Repository) SaveBlob 

func (r *Repository) SaveBlob(ctx context.Context, t restic.BlobType, buf []byte, id restic.ID, storeDuplicate bool) (newID restic.ID, known bool, err error)

SaveBlob saves a blob of type t into the repository. It takes care that no duplicates are saved; this can be overwritten by setting storeDuplicate to true. If id is the null id, it will be computed and returned. Also returns if the blob was already known before

func (*Repository) SaveFullIndex 

func (r *Repository) SaveFullIndex(ctx context.Context) error

SaveFullIndex saves all full indexes in the backend.

func (*Repository) SaveIndex 

func (r *Repository) SaveIndex(ctx context.Context) error

SaveIndex saves all new indexes in the backend.

func (*Repository) SaveJSONUnpacked 

func (r *Repository) SaveJSONUnpacked(ctx context.Context, t restic.FileType, item interface{}) (restic.ID, error)

SaveJSONUnpacked serialises item as JSON and encrypts and saves it in the backend as type t, without a pack. It returns the storage hash.

func (*Repository) SaveTree 

func (r *Repository) SaveTree(ctx context.Context, t *restic.Tree) (restic.ID, error)

SaveTree stores a tree into the repository and returns the ID. The ID is checked against the index. The tree is only stored when the index does not contain the ID.

func (*Repository) SaveUnpacked 

func (r *Repository) SaveUnpacked(ctx context.Context, t restic.FileType, p []byte) (id restic.ID, err error)

SaveUnpacked encrypts data and stores it in the backend. Returned is the storage hash.

func (*Repository) SearchKey 

func (r *Repository) SearchKey(ctx context.Context, password string, maxKeys int, keyHint string) error

SearchKey finds a key with the supplied password, afterwards the config is read and parsed. It tries at most maxKeys key files in the repo.

func (*Repository) SetDryRun 

func (r *Repository) SetDryRun()

SetDryRun sets the repo backend into dry-run mode.

func (*Repository) SetIndex 

func (r *Repository) SetIndex(i restic.MasterIndex) error

SetIndex instructs the repository to use the given index.

func (*Repository) UseCache 

func (r *Repository) UseCache(c *cache.Cache)

UseCache replaces the backend with the wrapped cache.

type Saver 

type Saver interface {
	Save(context.Context, restic.Handle, restic.RewindReader) error
	Hasher() hash.Hash
}

Saver implements saving data in a backend.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.