set

package
v1.17.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 29, 2026 License: MIT Imports: 27 Imported by: 0

Documentation

Overview

******************************************************************************

  • Copyright (c) 2025 Genome Research Ltd. *
  • Authors: Michael Woolnough <mw31@sanger.ac.uk> *
  • Permission is hereby granted, free of charge, to any person obtaining
  • a copy of this software and associated documentation files (the
  • "Software"), to deal in the Software without restriction, including
  • without limitation the rights to use, copy, modify, merge, publish,
  • distribute, sublicense, and/or sell copies of the Software, and to
  • permit persons to whom the Software is furnished to do so, subject to
  • the following conditions: *
  • The above copyright notice and this permission notice shall be included
  • in all copies or substantial portions of the Software. *
  • THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
  • EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
  • MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
  • IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
  • CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
  • TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
  • SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. *****************************************************************************

Index

Constants

View Source 
const (
	ErrInvalidSetID           = "invalid set ID"
	ErrInvalidRequest         = "request lacks Requester or Set"
	ErrInvalidEntry           = "invalid set entry"
	ErrInvalidTransformerPath = "invalid transformer path concatenation"
	ErrNoAddDuringDiscovery   = "can't add set while set is being discovered"
	ErrPathNotInSet           = "path(s) do not belong to the backup set"
	ErrSetIsNotWritable       = "the set is read-only, you cannot change it"
	ErrTransformerAlreadyUsed = "you cannot edit the transformer on a set with uploaded files"
	ErrTransformerInUse       = "you cannot edit the transformer on a set with unfinished uploads"
	ErrPendingRemovals        = "the set has unfinished removals, you cannot change it"

	AttemptsToBeConsideredFailing = 3

	TrashPrefix = ".trash-"
)
View Source 
const ErrElementNotInSlice = "element not in slice"
View Source 
const (
	ErrInvalidTransformer = "invalid transformer"
)

Variables

This section is empty.

Functions

func FileEntryFilterLastState added in v1.11.0 

func FileEntryFilterLastState(e *Entry) bool

FileEntryFilterLastState is a FileEntryFilter that filters on uploaded files (excluding orphaned files).

func FileEntryFilterUploaded added in v1.11.0 

func FileEntryFilterUploaded(e *Entry) bool

FileEntryFilterUploaded is a FileEntryFilter that filters on uploaded files.

func RemoveElementFromSlice added in v1.8.0 

func RemoveElementFromSlice(slice []string, element string) ([]string, error)

RemoveElementFromSlice returns the given slice without the given element.

func SetFilterTrashed added in v1.11.0 

func SetFilterTrashed(set *Set) bool

SetFilterTrashed is a SetFilter that filters on trashed sets.

Types

type DB 

type DB struct {
	DBRO
	// contains filtered or unexported fields
}

DB is used to create and query a database for storing backup sets (lists of files a user wants to have backed up) and their backup status.

func New 

func New(path, backupPath string, readonly bool) (*DB, error)

New returns a *DB that can be used to create or query a set database. Provide the path to the database file.

Optionally, also provide a path to backup the database to.

Returns an error if path exists but can't be opened, or if it doesn't exist and can't be created.

func (*DB) AddOrUpdate 

func (d *DB) AddOrUpdate(set *Set) error

AddOrUpdate adds or updates the given Set to the database. Errors if the set is read-only, or if the set is being discovered.

func (*DB) Backup added in v0.7.0 

func (d *DB) Backup() error

Backup does an on-line backup of the database to the given file path. There's at least 1 second in between backups, and incoming backup requests are noted during an on-going backup, with 1 further backup occurring after the on-going one completes.

func (*DB) Close 

func (d *DB) Close() error

Close closes the database. Be sure to call this to finalise any writes to disk correctly.

func (*DB) Delete added in v1.11.0 

func (d *DB) Delete(setID string) error

Delete removes the set from the db. Also it deletes its entries and any sub-buckets.

func (*DB) DeleteDiscoveredFoldersBucket added in v1.8.0 

func (d *DB) DeleteDiscoveredFoldersBucket(setID string) error

DeleteSubBucket deletes the provided sub bucket from the set with the provided id.

func (*DB) Discover added in v0.8.0 

func (d *DB) Discover(setID string, cb DiscoverCallback) (*Set, error)

Discover discovers and stores file entry details for the given set. Immediately tries to record in the db that discovery has started and returns any error from doing that. Actual discovery using your callback on directories will then proceed, as will the stat'ing of pure files, and return the updated set.

func (*DB) EnableRemoteBackups added in v0.7.0 

func (d *DB) EnableRemoteBackups(remotePath string, handler transfer.Handler)

EnableRemoteBackups causes the backup file to also be backed up to the remote path.

func (*DB) GetExcludedPaths added in v1.8.0 

func (d *DB) GetExcludedPaths(setID string) ([]string, error)

GetExcludedPaths returns the paths for all files and dirs that should not be included in any discoveries on the given set.

func (*DB) GetExistingDirs added in v1.10.0 

func (d *DB) GetExistingDirs(setID string) (map[string]struct{}, error)

GetExistingDirs returns all existing dir paths.

func (*DB) GetFilesFromInode added in v1.8.0 

func (d *DB) GetFilesFromInode(inode uint64, mountPoint string) ([]string, error)

GetFilesFromInode returns all the paths that share the provided inode on the given mount point.

func (*DB) GetMountPointFromPath added in v0.9.0 

func (d *DB) GetMountPointFromPath(path string) string

GetMountPointFromPath determines the mount point for the given path based on the mount points available on the system when the server started. If nothing matches, returns /.

func (*DB) HardlinkPaths added in v0.8.0 

func (d *DB) HardlinkPaths(e *Entry) ([]string, error)

HardlinkPaths returns all known hardlink paths that share the same mountpoint and inode as the entry provided.

func (*DB) HardlinkRemote added in v0.8.0 

func (d *DB) HardlinkRemote(e *Entry) (string, error)

HardlinkRemote gets the remote path of the first hardlink we uploaded that shares the given entry's inode and mountpoint.

func (*DB) Hide added in v1.10.1 

func (d *DB) Hide(set *Set) error

Hide marks the given Set as hidden in the database. This bypasses the normal AddOrUpdate method, working on read-only sets.

func (*DB) IncrementNumObjectRemoved added in v1.8.0 

func (d *DB) IncrementNumObjectRemoved(setID string) error

IncrementNumObjectRemoved increments the number of objects removed for the given set.

func (*DB) IncrementSetTotalRemoved added in v1.8.0 

func (d *DB) IncrementSetTotalRemoved(setID string) error

IncrementSetTotalRemoved increments the number of objects removed for the set.

func (*DB) IsSetReadyToAddFiles added in v1.10.2 

func (d *DB) IsSetReadyToAddFiles(set *Set) error

IsSetReadyToAddFiles checks if the set is ready to have files/dirs added to it.

func (*DB) MakeSetWritable added in v1.8.8 

func (d *DB) MakeSetWritable(sid string) error

MakeSetWritable sets ReadOnly to false on a given set. This is the only way to change the ReadOnly set.

func (*DB) MergeDirEntries added in v1.10.2 

func (d *DB) MergeDirEntries(setID string, entries []*Dirent) error

MergeDirEntries sets the directory paths for the given backup set. Only supply absolute paths to directories. It keeps the existing entries in the set and handles duplicates by updating the existing entries. If the dir is not already in the set, it will be added with the status Registered.

func (*DB) MergeFileEntries added in v1.10.2 

func (d *DB) MergeFileEntries(setID string, paths []string) error

MergeFileEntries sets the file paths for the given backup set. Only supply absolute paths to files. It keeps the existing entries in the set and handles duplicates by updating the existing entries. If the file is not already in the set, it will be added with the status Registered.

func (*DB) OptimiseRemoveBucket added in v1.8.0 

func (d *DB) OptimiseRemoveBucket(setID string) error

OptimiseRemoveBucket removes all redundant entries from the remove bucket for a given set.

func (*DB) PutEntryInTrash added in v1.11.0 

func (d *DB) PutEntryInTrash(set *Set, entry *Entry) error

PutEntryInTrash puts the given entry into the trash set for the given set.

func (*DB) RemoveDirEntry added in v1.8.0 

func (d *DB) RemoveDirEntry(setID string, path string) error

RemoveDirEntry removes the provided directory from a given set.

func (*DB) RemoveFileEntry added in v1.8.0 

func (d *DB) RemoveFileEntry(setID string, path string) error

RemoveFileEntry removes the provided file from a given set.

func (*DB) RemoveFileFromInode added in v1.8.0 

func (d *DB) RemoveFileFromInode(path string, inode uint64) error

RemoveFileFromInode removes entry for the given path from inode bucket if it is the last file with that inode. Otherwise just removes itself from the list (if the path is the original file 'removal' is setting it to be blank).

func (*DB) RemoveFromRemovedBucket added in v1.10.2 

func (d *DB) RemoveFromRemovedBucket(path, sid string) error

RemoveFromRemovedBucket removes the given path from the removed bucket of the given set.

func (*DB) RemovePathFromFailedBucket added in v1.8.0 

func (d *DB) RemovePathFromFailedBucket(setID, path string) error

RemovePathFromFailedBucket removes the entry with the given setID and path from the failed bucket.

func (*DB) ResetRemoveSize added in v1.8.0 

func (d *DB) ResetRemoveSize(setID string) error

ResetRemoveSize resets the size removed for the set.

func (*DB) SetBackupPath added in v0.7.0 

func (d *DB) SetBackupPath(path string)

SetBackupPath sets the backup path, for if you created the DB without providing one.

func (*DB) SetEntryStatus 

func (d *DB) SetEntryStatus(r *transfer.Request) (*Entry, error)

SetEntryStatus finds the set Entry corresponding to the given Request's Local path, Requester and Set name, and updates its status in the database, and also updates summary status for the Set.

Returns the Entry, which is a reflection of the Request, but contains additional information such as the number of Attempts if the Request is failing and you keep retrying.

Returns an error if a set or entry corresponding to the Request can't be found.

func (*DB) SetError 

func (d *DB) SetError(setID, errMsg string) error

SetError updates a set with the given error message. Returns an error if the setID isn't in the database.

func (*DB) SetMinimumTimeBetweenBackups added in v0.7.0 

func (d *DB) SetMinimumTimeBetweenBackups(dur time.Duration)

SetMinimumTimeBetweenBackups sets the minimum time between successive backups. Defaults to 1 second if this method not called.

func (*DB) SetRemoveRequests added in v1.8.0 

func (d *DB) SetRemoveRequests(sid string, removeReqs []RemoveReq) error

SetRemoveRequests writes a list of remove requests into the database. Directory paths will be put into the database with a trailing slash.

func (*DB) SetWarning added in v0.8.0 

func (d *DB) SetWarning(setID, warnMsg string) error

SetWarning updates a set with the given warning message. Returns an error if the setID isn't in the database.

func (*DB) UpdateBasedOnRemovedEntry added in v1.8.0 

func (d *DB) UpdateBasedOnRemovedEntry(setID string, entry *Entry) error

UpdateBasedOnRemovedEntry updates set counts based on the given entry that's been removed.

func (*DB) UpdateEntry added in v1.8.0 

func (d *DB) UpdateEntry(sid, key string, entry *Entry) error

UpdateEntry puts the updated entry into the database for the given set.

func (*DB) UpdateRemoveRequest added in v1.8.0 

func (d *DB) UpdateRemoveRequest(removeReq RemoveReq) error

UpdateRemoveRequest replaces the given removeReq in the db.

func (*DB) UpdateSetTotalToRemove added in v1.8.0 

func (d *DB) UpdateSetTotalToRemove(setID string, num uint64) error

UpdateSetTotalToRemove sets num of objects to be removed to provided value and resets num of objects removed if the previous removal was successful otherwise just increases number to be removed with provided value.

func (*DB) ValidateFileAndDirPaths added in v1.11.0 

func (d *DB) ValidateFileAndDirPaths(set *Set, paths []string) ([]string, []string, error)

ValidateFileAndDirPaths returns an error if any provided path is not in the given set. Also classifies the valid paths into a slice of filepaths or dirpaths.

type DBRO added in v1.1.0 

type DBRO struct {
	// contains filtered or unexported fields
}

DBRO is the read-only component of the DB struct.

func NewRO added in v1.1.0 

func NewRO(path string) (*DBRO, error)

NewRO returns a *DBRO that can be used to query a set database. Provide the path to the database file.

Returns an error if database can't be opened.

func (*DBRO) GetAll added in v1.1.0 

func (d *DBRO) GetAll() ([]*Set, error)

GetAll returns all the Sets previously added to the database.

func (*DBRO) GetAllDirEntries added in v1.8.0 

func (d *DBRO) GetAllDirEntries(setID string) ([]*Entry, error)

GetDirEntries returns all the dir entries for the given set.

func (*DBRO) GetAllSetsForFile added in v1.11.0 

func (d *DBRO) GetAllSetsForFile(path string) ([]string, error)

GetAllSetsForFile returns a slice of setIDs for sets that contain the given file.

func (*DBRO) GetByID added in v1.1.0 

func (d *DBRO) GetByID(id string) *Set

GetByID returns the Sets with the given ID previously added to the database. Returns nil if such a set does not exist.

func (*DBRO) GetByNameAndRequester added in v1.1.0 

func (d *DBRO) GetByNameAndRequester(name, requester string) (*Set, error)

GetByNameAndRequester returns the set with the given name and requester.

Returns nil error when no set found.

func (*DBRO) GetByRequester added in v1.1.0 

func (d *DBRO) GetByRequester(requester string) ([]*Set, error)

GetByRequester returns all the Sets previously added to the database by the given requester.

func (*DBRO) GetDefinedFileEntry added in v1.1.0 

func (d *DBRO) GetDefinedFileEntry(setID string) (*Entry, error)

GetDefinedFileEntry returns the first defined file entry for the given set.

Will return nil Entry if SetFileEntries hasn't been called.

func (*DBRO) GetDirEntries added in v1.1.0 

func (d *DBRO) GetDirEntries(setID string, filter EntryFilter) ([]*Entry, error)

GetDirEntries returns all the dir entries for the given set.

func (*DBRO) GetDirEntryForSet added in v1.11.0 

func (d *DBRO) GetDirEntryForSet(setID, dirPath string) (*Entry, error)

GetDirEntryForSet returns the dir entry for the given path in the given set.

func (*DBRO) GetDiscoveredFileEntries added in v1.10.0 

func (d *DBRO) GetDiscoveredFileEntries(setID string) ([]*Entry, error)

GetDiscoveredFileEntries returns all the discovered file entries for the given set (only SetDiscoveredEntries, not SetFileEntries).

func (*DBRO) GetFailedEntries added in v1.1.0 

func (d *DBRO) GetFailedEntries(setID string) ([]*Entry, int, error)

GetFailedEntries returns up to 10 of the file entries for the given set (both SetFileEntries and SetDiscoveredEntries) that have a failed status. Also returns the number of failed entries that were not returned.

func (*DBRO) GetFileEntries added in v1.1.0 

func (d *DBRO) GetFileEntries(setID string, filter EntryFilter) ([]*Entry, error)

GetFileEntries returns all the file entries for the given set (both SetFileEntries and SetDiscoveredEntries).

func (*DBRO) GetFileEntryForSet added in v1.1.0 

func (d *DBRO) GetFileEntryForSet(setID, filePath string) (*Entry, error)

GetFileEntryForSet returns the file entry for the given path in the given set.

func (*DBRO) GetFilesInDir added in v1.8.0 

func (d *DBRO) GetFilesInDir(setID string, dirpath string) ([]string, error)

GetFilesInDir returns all file paths from inside the given directory (and all nested inside) for the given set using the db.

func (*DBRO) GetFilteredSets added in v1.11.0 

func (d *DBRO) GetFilteredSets(filter SetFilter) ([]*Set, error)

func (*DBRO) GetFoldersInDir added in v1.8.0 

func (d *DBRO) GetFoldersInDir(setID string, dirpath string) ([]string, error)

GetFoldersInDir returns all folder paths from inside the given directory (and all nested inside) for the given set using the db.

func (*DBRO) GetIncompleteRemoveRequests added in v1.10.2 

func (d *DBRO) GetIncompleteRemoveRequests() ([]RemoveReq, error)

GetIncompleteRemoveRequests returns all incomplete removeReqs from every set.

func (*DBRO) GetPureFileEntries added in v1.1.0 

func (d *DBRO) GetPureFileEntries(setID string) ([]*Entry, error)

GetPureFileEntries returns all the file entries for the given set (only SetFileEntries, not SetDiscoveredEntries).

func (*DBRO) GetRemoveRequests added in v1.8.0 

func (d *DBRO) GetRemoveRequests(sid string) ([]RemoveReq, error)

GetRemoveRequests returns all objects from the remove bucket in the database.

func (*DBRO) GetTrashedSets added in v1.11.0 

func (d *DBRO) GetTrashedSets() ([]*Set, error)

GetTrashedSets returns all the trash Sets previously added to the database.

func (*DBRO) LogSetChangesToSlack added in v1.4.0 

func (d *DBRO) LogSetChangesToSlack(slacker Slacker)

type Dirent added in v1.7.2 

type Dirent struct {
	Path  string
	Mode  fs.FileMode
	Inode uint64
}

Dirent represents and entry produced from a walk of a specified directory.

func DirEntFromWalk added in v1.7.2 

func DirEntFromWalk(de *statter.DirEnt) *Dirent

DirEntFromWalk converts a walk.Dirent to an easier to use local Dirent type.

func (*Dirent) IsCharDevice added in v1.7.2 

func (d *Dirent) IsCharDevice() bool

IsCharDevice returns true if the entry refers to a character device node.

func (*Dirent) IsDevice added in v1.7.2 

func (d *Dirent) IsDevice() bool

IsDevice returns true if the entry refers to a device node.

func (*Dirent) IsDir added in v1.7.2 

func (d *Dirent) IsDir() bool

IsDir returns true if the entry refers to a directory.

func (*Dirent) IsIrregular added in v1.7.2 

func (d *Dirent) IsIrregular() bool

IsIrregular returns true if the entry refers to an irregular file.

func (*Dirent) IsPipe added in v1.7.2 

func (d *Dirent) IsPipe() bool

IsPipe returns true if the entry refers to a named (FIFO) pipe.

func (*Dirent) IsRegular added in v1.7.2 

func (d *Dirent) IsRegular() bool

IsRegular returns true if the entry refers to a regular file.

func (*Dirent) IsSocket added in v1.7.2 

func (d *Dirent) IsSocket() bool

IsSocket returns true if the entry refers to a socket. 

func (d *Dirent) IsSymlink() bool

IsSymlink returns true if the entry refers to a symlink.

type DiscoverCallback added in v0.8.0 

type DiscoverCallback func([]*Entry) ([]*Dirent, []*Dirent, error)

DiscoverCallback will receive the sets directory entries and return a list of the files and a list of the dirs discovered in those directories.

type Entry 

type Entry struct {
	Path        string
	PathForJSON []byte // set by MakeSafeForJSON(); do not set this yourself.
	Size        uint64 // size of local file in bytes.
	Status      EntryStatus
	Type        EntryType
	LastError   string
	LastAttempt time.Time
	Attempts    int
	Inode       uint64
	Dest        string

	TrashDate time.Time // only set for entries in trashed sets.
	// contains filtered or unexported fields
}

Entry holds the status of an entry in a backup set.

func (*Entry) CorrectFromJSON added in v0.6.3 

func (e *Entry) CorrectFromJSON()

CorrectFromJSON copies PathForJSON to Path, so that this value is correct following a decode from JSON.

func (*Entry) InodeStoragePath added in v1.0.3 

func (e *Entry) InodeStoragePath() string

InodeStoragePath returns a relative path that the data for this entry could be stored at if this entry is for a hardlink. The path includes both the inode and the first local path we saw for this inode. This ensures that if an inode gets re-used after previous hardlinks we saw all got deleted locally, the new file will be stored at a new location.

func (*Entry) IsUploaded added in v1.16.0 

func (e *Entry) IsUploaded() bool

IsUploaded returns true if the entry status indicates that it is currently successfully uploaded.

func (*Entry) MakeSafeForJSON added in v0.6.3 

func (e *Entry) MakeSafeForJSON()

MakeSafeForJSON copies Path to PathForJSON, so that if this Entry struct is encoded as JSON, non-UTF8 characters will be preserved. On decode be sure to use CorrectFromJSON().

func (*Entry) ShouldUpload 

func (e *Entry) ShouldUpload(given *Set) bool

ShouldUpload returns true if this Entry is pending or the last attempt was before the given time. Always returns false if the Type is Abnormal.

func (*Entry) WasNotUploaded added in v1.10.2 

func (e *Entry) WasNotUploaded() bool

WasNotUploaded checks whether the entry's status corresponds to a status in which the entry could not be uploaded.

type EntryFilter added in v1.11.0 

type EntryFilter func(*Entry) bool

EntryFilter is a function used to filter the entries for a set retrieved from the database.

A return value of true will keep the file entry, a return of false will discard it.

type EntryStatus 

type EntryStatus int
const (
	// Pending is an Entry status meaning the file has not yet had an upload
	// attempt.
	Pending EntryStatus = iota

	// UploadingEntry is an Entry status meaning the file has started to upload.
	UploadingEntry

	// Uploaded is an Entry status meaning the file has been uploaded for the
	// first time successfully.
	Uploaded

	// Failed is an Entry status meaning the file failed to upload due to some
	// remote issue.
	Failed

	// Missing is an Entry status meaning the local file is missing so can't be
	// uploaded.
	Missing

	// AbnormalEntry is an Entry status meaning the local files is neither
	// regular nor a symlink (ie. it's a fifo or socket etc.), so shouldn't be
	// uploaded.
	AbnormalEntry

	// Replaced is an Entry status meaning the file has been uploaded previously
	// but was uploaded again because the local file was changed.
	Replaced

	// Skipped is an Entry status meaning the file was not uploaded because it
	// was uploaded previously and hasn't changed since.
	Skipped

	// Orphaned is an Entry status meaning the file was uploaded previously but
	// the local file has been deleted, so can't be uploaded again.
	Orphaned

	// Registered is an Entry status meaning the file has been set in the
	// database but with no information.
	Registered
)

func (EntryStatus) String 

func (e EntryStatus) String() string

String lets you convert a EntryStatus to a meaningful string.

type EntryType added in v0.8.0 

type EntryType int
const (
	Regular EntryType = iota
	Hardlink
	Symlink
	Directory
	Abnormal // fifos and sockets etc. that we shouldn't upload

	// Unknown is an Entry type meaning the local file is either missing or
	// there was an error trying to get its type.
	Unknown = -1
)

type Error 

type Error struct {
	Msg string
	// contains filtered or unexported fields
}

func (Error) Error 

func (e Error) Error() string

type RemovalStatus added in v1.8.0 

type RemovalStatus int8
const (
	NotRemoved       RemovalStatus = iota // 0
	AboutToBeRemoved                      // 1
	Removed                               // 2
)

type RemoveAction added in v1.11.0 

type RemoveAction int8
const (
	ToTrash  RemoveAction = iota // 0
	ToRemove                     // 1
)

type RemoveReq added in v1.8.0 

type RemoveReq struct {
	Path                string
	Set                 *Set
	IsDir               bool
	RemoteRemovalStatus RemovalStatus
	IsComplete          bool
	Action              RemoveAction
}

RemoveReq contains information about a remove request for a path.

func NewRemoveRequest added in v1.8.0 

func NewRemoveRequest(path string, set *Set, isDir bool, action RemoveAction) RemoveReq

NewRemoveRequest creates a remove requests using the provided information.

func (RemoveReq) ItemDef added in v1.8.0 

func (rq RemoveReq) ItemDef(ttr time.Duration) *queue.ItemDef

ItemDef returns a queue.ItemDef for the remove request.

func (RemoveReq) Key added in v1.8.0 

func (rq RemoveReq) Key() string

type Set 

type Set struct {
	// An arbitrary (short) name for this backup set.
	Name string

	// The username of the person requesting this backup.
	Requester string

	// The method of transforming local Entries paths in to remote paths, to
	// determine the upload location. "humgen" to use the put.HumgenTransformer,
	// or "prefix=local:remote" to use the put.PrefixTransformer.
	Transformer string

	// Monitor the files and directories and re-upload them whenever they
	// change, checking for changes after the given amount of time. Optional,
	// defaults to unmonitored (a one time upload of Entries).
	MonitorTime time.Duration

	// Tells the monitor if it should remove any files from the set that have
	// been locally deleted.
	MonitorRemovals bool

	// An optional longer free-text description of this backup set.
	Description string

	// Optional additional metadata which will be applied to every file in the
	// set.
	Metadata map[string]string

	// Delete local paths after successful upload. Optional, defaults to no
	// deletions (ie. do a backup, not a move).
	DeleteLocal bool

	// Frozen prevents uploaded files from being overwritten by newer local
	// copies.
	Frozen bool

	// StartedDiscovery provides the last time that discovery started. This is a
	// read-only value.
	StartedDiscovery time.Time

	// LastDiscovery provides the last time that discovery completed. This is a
	// read-only value.
	LastDiscovery time.Time

	// NumFiles provides the total number of set and discovered files in this
	// set, as of the last discovery. This is a read-only value.
	NumFiles uint64

	// SizeTotal provides the total size (bytes) of set and discovered files in
	// this set, as of the last discovery. This is a read-only value.
	SizeTotal uint64

	// Uploaded provides the total number of set and discovered files in this
	// set that have, for the first time, been uploaded or confirmed uploaded
	// since the last discovery. This is a read-only value.
	Uploaded uint64

	// Replaced is like Uploaded, but for files that had previously been
	// uploaded to iRODS, and now uploaded again because the file on local disk
	// was newer.
	Replaced uint64

	// Skipped is like Uploaded, but for files that had previously been
	// uploaded to iRODS, and were not uploaded again because the file on local
	// disk was the same age.
	Skipped uint64

	// Failed provides the total number of set and discovered files in this set
	// that have failed their upload since the last discovery. This is a
	// read-only value.
	Failed uint64

	// Missing provides the total number of set and discovered files in this set
	// that no longer exist locally and remotely since the last discovery. This
	// is a read-only value.
	Missing uint64

	// Abnormal provides the total number of set files in this set that were
	// neither regular files, nor Symlinks (ie. they were fifo or socket files
	// etc).
	Abnormal uint64

	// Symlinks provides the total number of set and discovered files in this
	// set that are symlinks. This is a read-only value.
	Symlinks uint64

	// Hardlinks provides the total number of set and discovered files in this
	// set that are hardlinks and skipped. This is a read-only value.
	Hardlinks uint64

	// Status provides the current status for the set since the last discovery.
	// This is a read-only value.
	Status Status

	// LastCompleted provides the last time that all uploads completed
	// (regardless of success or failure). This is a read-only value.
	LastCompleted time.Time

	// LastCompletedCount provides the count of files on the last upload attempt
	// (those successfully uploaded, those which failed, but not those which
	// were missing locally). This is a read-only value.
	LastCompletedCount uint64

	// LastCompletedSize provides the size of files (bytes) counted in
	// LastCompletedCount. This is a read-only value.
	LastCompletedSize uint64

	// SizeUploaded provides the size of files (bytes) actually uploaded (not
	// skipped) since the last discovery. This is a read-only value.
	SizeUploaded uint64

	// SizeRemoved provides the size of files (bytes) part of the most recent
	// remove. This is a read-only value.
	SizeRemoved uint64

	// NumObjectsToBeRemoved provides the number of objects to be removed in the
	// current remove process.
	NumObjectsToBeRemoved uint64

	// numObjectsRemoved provides the number of objects already removed in the
	// current remove process.
	NumObjectsRemoved uint64

	// Error holds any error that applies to the whole set, such as an issue
	// with the Transformer. This is a read-only value.
	Error string

	// Warning contains errors that do not stop progression. This is a read-only
	// value.
	Warning string

	// ReadOnly prevents user changing the set and blocks monitor and discovery.
	ReadOnly bool

	// Hide allows user to hide the set when viewing the status.
	Hide bool

	// Orphaned provides the total number of set and discovered files in this
	// set that no longer exist locally since the last discovery but do exist
	// remotely. This is a read-only value.
	Orphaned uint64
	// contains filtered or unexported fields
}

Set describes a backup set; a list of files and directories to backup, plus some metadata. All properties are required unless otherwise noted.

func BuildTrashSetFromSet added in v1.11.0 

func BuildTrashSetFromSet(givenSet *Set) Set

BuildTrashSetFromSet builds a brand new set which copies the given set's Requester and Transformer. Use this set to move trashed files to.

func (*Set) Count 

func (s *Set) Count() string

Count provides a string representation of NumFiles, or if 0, returns the LastCompletedCount with a textual note to that effect.

func (*Set) Discovered 

func (s *Set) Discovered() string

Discovered provides a string representation of when discovery last completed, or if it hasn't, when it started, or if it hasn't, says "not started".

func (*Set) DiscoveryCompleted added in v1.4.0 

func (s *Set) DiscoveryCompleted(numFiles uint64)

DiscoveryCompleted should be called when you complete discovering a set. Pass in the number of files you discovered.

func (*Set) HasProblems added in v1.2.0 

func (s *Set) HasProblems() bool

HasProblems returns true if any uploads have failed, if there's an error on the set itself, or if our transformer doesn't work.

Currently does NOT check if all of the user's desired local paths can be transformed, so there might actually be problems.

func (*Set) ID 

func (s *Set) ID() string

ID returns an ID for this set, generated deterministiclly from its Name and Requester. Ie. it is unique between all requesters, and amongst a requester's differently named sets. Sets with the same Name and Requester will have the same ID().

func (*Set) Incomplete added in v1.2.0 

func (s *Set) Incomplete() bool

Incomplete returns true if our Status is not Complete, or if we HasProblems(). (Status can be "Complete" even if there are upload failures, because "Complete" status only means we're not trying to upload any more.)

func (*Set) IsTrash added in v1.11.0 

func (s *Set) IsTrash() bool

IsTrash returns true if the set is trashed.

func (*Set) LogChangesToSlack added in v1.4.0 

func (s *Set) LogChangesToSlack(slacker Slacker)

LogChangesToSlack will cause the set to use the slacker when significant events happen to the set.

func (*Set) MakeTransformer 

func (s *Set) MakeTransformer() (transformer.PathTransformer, error)

MakeTransformer turns our Transformer string in to a transformer.PathTransformer.

func (*Set) Queued added in v1.2.0 

func (s *Set) Queued() bool

Queued returns true if we're either pending discovery or upload. Ie. the set was recently added or updated, but uploads haven't begun yet probably due to uploads for other sets being first in the queue.

func (*Set) RecoveryError added in v1.4.0 

func (s *Set) RecoveryError(err error)

func (*Set) RemovedSize added in v1.8.0 

func (s *Set) RemovedSize() string

RemovedSize provides a string representation of SizeRemoved in a human readable format. This is the size of files part of the most recent remove.

func (*Set) SetError added in v1.4.0 

func (s *Set) SetError(errMsg string)

SetError records the given error against the set, indicating it wont work.

func (*Set) SetWarning added in v1.4.0 

func (s *Set) SetWarning(warnMsg string)

SetWarning records the given warning against the set, indicating it has an issue.

func (*Set) Size 

func (s *Set) Size() string

Size provides a string representation of SizeFiles in a human readable format, or if 0, returns the LastCompletedSize with a textual note to that effect.

func (*Set) SuccessfullyStoredInDB added in v1.4.0 

func (s *Set) SuccessfullyStoredInDB()

SuccessfullyStoredInDB should be called when you successfully store the set in DB.

func (*Set) TransformPath added in v1.1.0 

func (s *Set) TransformPath(path string) (string, error)

func (*Set) UpdateBasedOnEntry added in v1.4.0 

func (s *Set) UpdateBasedOnEntry(entry *Entry, getFileEntries func(string, EntryFilter) ([]*Entry, error)) error

UpdateBasedOnEntry updates set status values based on an updated Entry from updateFileEntry(), assuming that request is for one of set's file entries.

func (*Set) UploadedSize added in v1.5.0 

func (s *Set) UploadedSize() string

UploadedSize provides a string representation of SizeUploaded in a human readable format. This is the size of actual uploads (excluding skipped, unlike Size()) since the last discovery.

func (*Set) UserMetadata added in v1.7.0 

func (s *Set) UserMetadata() string

type SetFilter added in v1.11.0 

type SetFilter func(*Set) bool //nolint:revive

SetFilter is a function used to filter the sets retrieved from the database.

A return value of true will keep the set, a return of false will discard it.

type SizeAndNumber added in v1.1.0 

type SizeAndNumber struct {
	For    string // Aggregation id (set name or requester name or "total").
	Size   uint64 // Size in bytes.
	Number uint64 // Number of files.
}

SizeAndNumber provides the total Size and Number of files in the set(s) this is a summary of, where "For" is the id of aggregation, eg it might be a set requester name if this was for a breakdown by requester.

Files in the set(s) that began an upload (and we thus know their size) are part of these totals.

func (*SizeAndNumber) SizeTiB added in v1.1.0 

func (s *SizeAndNumber) SizeTiB() float64

SizeTiB returns our Size in TiB.

type Slacker added in v1.4.0 

type Slacker interface {
	SendMessage(level slack.Level, msg string)
}

type Status 

type Status int
const (
	// PendingDiscovery is a Set status meaning the set's entries are pending
	// existence, size and directory content discovery.
	PendingDiscovery Status = iota

	// PendingUpload is a Set status meaning discovery has completed, but no
	// entries have been uploaded since then.
	PendingUpload

	// Uploading is a Set status meaning discovery has completed and upload of
	// entries has started.
	Uploading

	// Failing is a Set status meaning at least 1 of the entries has failed to
	// upload after 3 attempts. Other uploads are ongoing.
	Failing

	// Complete is a Set status meaning all entries have had an upload attempt
	// since the last discovery. (Some uploads may have failed, but they had
	// 3 retries.)
	Complete
)

func (Status) String 

func (s Status) String() string

String lets you convert a Status to a meaningful string.

type Usage added in v1.1.0 

type Usage struct {
	// Total size and number of files across all given sets.
	Total *SizeAndNumber

	// ByRequester is a slice where each SizeAndNumber is the total size and
	// number of files for a particular requester amongst the given sets. The
	// slice is sorted from largest size to smallest.
	ByRequester []*SizeAndNumber

	// BySet is a slice where each SizeAndNumber is the total size and number of
	// files in each set. The "For" equals the set requester concatenated with
	// the set name. The slice is sorted from largest size to smallest.
	BySet []*SizeAndNumber

	// ByMonth is a slice where each SizeAndNumber is the total size and number
	// of files in sets that last completed in a certain calendar month. The
	// slice is sorted in date order. Incomplete sets are ignored.
	ByMonth []*SizeAndNumber
}

Usage is the type returned by UsageSummary() and contains various breakdowns of how much iRODS would be used up by the sets.

func UsageSummary added in v1.1.0 

func UsageSummary(sets []*Set) *Usage

UsageSummary returns a Usage that summaries potential iRODS usage of the given sets, broken down in various ways.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.