put

package
v1.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 27, 2024 License: MIT Imports: 24 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	MetaNamespace = "ibackup:"
	MetaKeyMtime  = MetaNamespace + "mtime" // mtime of source file, 1sec truncated UTC RFC 3339
	MetaKeyOwner  = MetaNamespace + "owner" // a username
	MetaKeyGroup  = MetaNamespace + "group" // a unix group name
	MetaKeyDate   = MetaNamespace + "date"  // date upload initiated, 1sec truncated UTC RFC 3339
	// a comma sep list of usernames of the people who reqested the backup.
	MetaKeyRequester      = MetaNamespace + "requesters"
	MetaKeySets           = MetaNamespace + "sets"           // a comma sep list of backup set names this file belongs to
	MetaKeySymlink        = MetaNamespace + "symlink"        // symlink destination if file is a symlink
	MetaKeyHardlink       = MetaNamespace + "hardlink"       // the first path seen with this inode if file is a hardlink
	MetaKeyRemoteHardlink = MetaNamespace + "remotehardlink" // iRODS path that contains the data for this hardlink
	ErrStatFailed         = "stat of local path returned strange results"
)
View Source 
const (
	ErrLocalNotAbs  = "local path could not be made absolute"
	ErrRemoteNotAbs = "remote path not absolute"
	ErrReadTimeout  = "local file read timed out"
	ErrStuckTimeout = "upload killed because it was stuck"
)
View Source 
const ErrMockMetaFail = "meta fail"
View Source 
const ErrMockPutFail = "put fail"
View Source 
const ErrMockStatFail = "stat fail"
View Source 
const (
	ErrOperationTimeout = "iRODS operation timed out"
)

Variables

This section is empty.

Functions

func HumgenTransformer 

func HumgenTransformer(local string) (string, error)

HumgenTransformer is a PathTransformer that will convert a local "lustre" path to a "canonical" path in the humgen iRODS zone.

This transform is specific to the "humgen" group at the Sanger Institute.

func TimeToMeta added in v1.1.0 

func TimeToMeta(t time.Time) (string, error)

TimeToMeta converts a time to a string suitable for storing as metadata, in a way that ObjectInfo.ModTime() will understand and be able to convert back again.

Types

type Baton 

type Baton struct {
	// contains filtered or unexported fields
}

Baton is a Handler that uses Baton (via extendo) to interact with iRODS.

func GetBatonHandler 

func GetBatonHandler() (*Baton, error)

GetBatonHandler returns a Handler that uses Baton to interact with iRODS. If you don't have baton-do in your PATH, you'll get an error.

func (*Baton) AddMeta 

func (b *Baton) AddMeta(path string, meta map[string]string) error

func (*Baton) Cleanup 

func (b *Baton) Cleanup() error

Cleanup stops our clients and closes our client pool.

func (*Baton) CollectionsDone added in v0.6.0 

func (b *Baton) CollectionsDone() error

CollectionsDone closes the connections used for connection creation, and creates new ones for doing puts and metadata operations.

func (*Baton) EnsureCollection 

func (b *Baton) EnsureCollection(collection string) error

EnsureCollection ensures the given collection exists in iRODS, creating it if necessary. You must call Connect() before calling this.

This is safe for calling concurrently, and uses multiple connections. But an artefact is that the error you get might be for a different EnsureCollection() call you made for a different collection. This shouldn't make much difference if you just collect all your errors and don't care about order.

func (*Baton) Put 

func (b *Baton) Put(request *Request) error

Put uploads request Local to the Remote object, overwriting it if it already exists. It calculates and stores the md5 checksum remotely, comparing to the local checksum.

func (*Baton) RemoveMeta 

func (b *Baton) RemoveMeta(path string, meta map[string]string) error

func (*Baton) Stat 

func (b *Baton) Stat(request *Request) (*ObjectInfo, error)

Stat gets mtime and metadata info for the request Remote object.

type Error 

type Error struct {
	// contains filtered or unexported fields
}

func (Error) Error 

func (e Error) Error() string

func (Error) Is added in v0.4.0 

func (e Error) Is(err error) bool

type FileReadTester added in v0.5.0 

type FileReadTester func(ctx context.Context, path string) error

FileReadTester is a function that attempts to open and read the given path, returning any error doing so. Should stop and clean up if the given ctx becomes done before the open and read succeeds.

type FileStatusCallback added in v0.8.0 

type FileStatusCallback func(absPath string, fi os.FileInfo) RequestStatus

FileStatusCallback returns RequestStatusPending if the file is to be uploaded, and returns any other RequestStatus, such as RequestStatusHardlink and RequestStatusSymlink, to not be uploaded.

type Handler 

type Handler interface {
	// EnsureCollection checks if the given collection exists in iRODS, creates
	// it if not, then double-checks it now exists. Must support being called
	// concurrently.
	EnsureCollection(collection string) error

	// CollectionsDone is called after all collections have been created. This
	// method can do things like cleaning up connections created for collection
	// creation. It can also create new connections for subsequent Put() and
	// *Meta calls that are likely to occur.
	CollectionsDone() error

	// Stat checks if the Request's Remote object exists. If it does, records
	// its metadata in the returned ObjectInfo. Returns an error if there was a
	// problem finding out information (but not if the object does not exist).
	Stat(request *Request) (*ObjectInfo, error)

	// Put uploads the Request's Local file to the Remote location, overwriting
	// any existing object, and ensuring that a locally calculated and remotely
	// calculated md5 checksum match.
	Put(request *Request) error

	// RemoveMeta deletes the given metadata from the given object.
	RemoveMeta(path string, meta map[string]string) error

	// AddMeta adds the given metadata to the given object. Given metadata keys
	// should already have been removed with RemoveMeta() from the remote
	// object.
	AddMeta(path string, meta map[string]string) error

	// Cleanup stops any connections created earlier and does any other cleanup
	// needed.
	Cleanup() error
}

Handler is something that knows how to communicate with iRODS and carry out certain operations.

type LocalHandler added in v0.5.0 

type LocalHandler struct {
	// contains filtered or unexported fields
}

LocalHandler satisfies the Handler interface, treating "Remote" as local paths and moving from Local to Remote for the Put().

func GetLocalHandler added in v0.2.0 

func GetLocalHandler() *LocalHandler

GetLocalHandler returns a Handler that doesn't actually interact with iRODS, but instead simply treats "Remote" as local paths and copies from Local to Remote for any Put()s. For use during tests.

func (*LocalHandler) AddMeta added in v0.5.0 

func (l *LocalHandler) AddMeta(path string, meta map[string]string) error

AddMeta adds the given meta to a map stored under path. Returns an error if metafail == path, or if keys were already defined in the map.

func (*LocalHandler) Cleanup added in v0.5.0 

func (l *LocalHandler) Cleanup() error

Cleanup just records this was called.

func (*LocalHandler) CollectionsDone added in v0.6.0 

func (l *LocalHandler) CollectionsDone() error

CollectionsDone says we connected and prepares us for metadata handling.

func (*LocalHandler) EnsureCollection added in v0.5.0 

func (l *LocalHandler) EnsureCollection(dir string) error

EnsureCollection creates the given dir locally and records that we did this.

func (*LocalHandler) GetMeta added in v0.8.0 

func (l *LocalHandler) GetMeta(path string) map[string]string

GetMeta gets the metadata stored for the given path (returns an empty map if path is not known about or has no metadata).

(Currently this is just for testing and not part of the Handler interface.)

func (*LocalHandler) MakeMetaFail added in v0.5.0 

func (l *LocalHandler) MakeMetaFail(remote string)

MakeMetaFail will result in any subsequent *Meta()s for a Request with the given remote path failing.

func (*LocalHandler) MakePutFail added in v0.5.0 

func (l *LocalHandler) MakePutFail(remote string)

MakePutFail will result in any subsequent Put()s for a Request with the given remote path failing.

func (*LocalHandler) MakePutSlow added in v0.5.0 

func (l *LocalHandler) MakePutSlow(local string, dur time.Duration)

MakePutSlow will result in any subsequent Put()s for a Request with the given local path taking the given amount of time.

func (*LocalHandler) MakeStatFail added in v0.5.0 

func (l *LocalHandler) MakeStatFail(remote string)

MakeStatFail will result in any subsequent Stat()s for a Request with the given remote path failing.

func (*LocalHandler) Put added in v0.5.0 

func (l *LocalHandler) Put(request *Request) error

Put just copies from Local to Remote. Returns an error if putFail == Remote.

func (*LocalHandler) RemoveMeta added in v0.5.0 

func (l *LocalHandler) RemoveMeta(path string, meta map[string]string) error

RemoveMeta deletes the given keys from a map stored under path. Returns an error if metaFail == path.

func (*LocalHandler) Stat added in v0.5.0 

func (l *LocalHandler) Stat(request *Request) (*ObjectInfo, error)

Stat returns info about the Remote file, which is a local file on disk. Returns an error if statFail == Remote.

type ObjectInfo 

type ObjectInfo struct {
	Exists bool
	Size   uint64
	Meta   map[string]string
}

func Stat 

func Stat(localPath string) (*ObjectInfo, error)

Stat stats localPath like os.Lstat(), but also returns information about the file in ObjectInfo Meta (mtime, owner and group information).

func (*ObjectInfo) HasSameModTime 

func (o *ObjectInfo) HasSameModTime(as *ObjectInfo) bool

HasSameModTime tells you if this ObjectInfo has the same ModTime() as the given one. Instead of comparing metadata strings, tries to convert them to times in a way that a comparison would be meaningful, in case the string was not generated by us.

func (*ObjectInfo) ModTime 

func (o *ObjectInfo) ModTime() time.Time

ModTime converts our mtime metadata key value string to a time.Time. Returns time zero if the key isn't present or convertable.

type PathTransformer 

type PathTransformer func(local string) (remote string, err error)

PathTransformer is a function that given a local path, returns the corresponding remote path.

func PrefixTransformer 

func PrefixTransformer(localPrefix, remotePrefix string) PathTransformer

PrefixTransformer returns a PathTransformer that will replace localPrefix in any path given to it with remotePrefix, and return the result.

If the given path does not start with localPrefix, returns the path prefixed with remotePrefix (treating the given path as relative to localPrefix).

type Putter 

type Putter struct {
	// contains filtered or unexported fields
}

Putter is used to Put() files in iRODS.

func New 

func New(handler Handler, requests []*Request) (*Putter, error)

New returns a *Putter that will use the given Handler to Put() all the requests in iRODS. You should defer Cleanup() on the return value. All the incoming requests will have their paths validated (they must be absolute).

Requests with Hardlink set will be uploaded to the Hardlink remote location, and an empty file uploaded to the Remote location, with metadata pointing to the Hardlink location.

func (*Putter) Cleanup 

func (p *Putter) Cleanup() error

Cleanup should be deferred after making a New Putter. It handles things like disconnecting.

func (*Putter) CreateCollections 

func (p *Putter) CreateCollections() error

CreateCollections will determine the minimal set of collections that need to be created to support a future Put() request for our requests, checks if those collections exist in iRODS, and creates them if not.

You MUST call this before Put(), even if you think all collections have already been created.

Tries to create all needed collections, potentially returning multiple errors wrapped in to one.

func (*Putter) Put 

func (p *Putter) Put() (chan *Request, chan *Request, chan *Request)

Put will upload all our request Local files to iRODS at the Remote locations. You ought to call CreateCollections() before calling this.

Existing files in iRODS will be overwritten if the local file's mtime is different to the remote's. If the same, the put will be skipped.

MD5 checksums will be calculated locally and remotely and compared to ensure a perfect upload. The request metadata will then replace any existing metadata with the same keys on the Remote object.

Requests that need to be uploaded are first sent on the first returned channel, with Status set to "uploading".

Each of those will then be send on the second returned channel, with Status set like: 

 "uploaded":   a new object was uploaded to iRODS
	"replaced":   an existing object was replaced in iRODS, because Local was
	              newer than Remote
	"failed":     An upload attempt failed; see Error

The third return channel will receive Requests that didn't need to be uploaded, with Status set like: 

"unmodified": Local and Remote had the same modification time, so nothing
              was done
"missing":    Local path could not be accessed, upload skipped; see Error

By calling SetFileStatusCallback() you can decide additional files to not upload.

func (*Putter) SetFileReadTester added in v0.5.0 

func (p *Putter) SetFileReadTester(tester FileReadTester)

SetFileReadTester sets the function used to see if a file can be opened and read. If this attempt hangs, the function should stop and clean up in response to its context becoming done (when the FileReadTimeout elapses).

The default tester shells out to the "head" command so that we don't leak stuck goroutines.

func (*Putter) SetFileReadTimeout added in v0.5.0 

func (p *Putter) SetFileReadTimeout(timeout time.Duration)

SetFileReadTimeout sets how long to wait on a test open and read of each local file before considering it not possible to upload. The default is 10seconds.

type Request 

type Request struct {
	Local         string
	Remote        string
	LocalForJSON  []byte // set by MakeSafeForJSON(); do not set this yourself.
	RemoteForJSON []byte // set by MakeSafeForJSON(); do not set this yourself.
	Requester     string
	Set           string
	Meta          map[string]string
	Status        RequestStatus
	Symlink       string // contains symlink path if request represents a symlink.
	Hardlink      string // contains first seen path if request represents a hard-linked file.
	Size          uint64 // size of Local in bytes, set for you on returned Requests.
	Error         string
	Stuck         *Stuck
	// contains filtered or unexported fields
}

Request represents a local file you would like transferred to a remote iRODS path, and any extra metadata (beyond the defaults which include user, group, mtime, upload date) you'd like to associate with it. Setting Requester and Set will add these to the requesters and sets metadata on upload.

func NewRequestWithTransformedLocal 

func NewRequestWithTransformedLocal(local string, pt PathTransformer) (*Request, error)

NewRequestWithTransformedLocal takes a local path string, uses the given PathTransformer to generate the corresponding remote path, and returns a Request with Local and Remote set.

func (*Request) Clone added in v0.2.0 

func (r *Request) Clone() *Request

Clone returns a copy of this request that can safely be altered without affecting the original.

func (*Request) CorrectFromJSON added in v0.6.3 

func (r *Request) CorrectFromJSON()

CorrectFromJSON copies LocalForJSON and RemoteForJSON to Local and Remote, so that these values are correct following a decode from JSON.

func (*Request) ID added in v0.2.0 

func (r *Request) ID() string

ID returns a deterministic identifier that is unique to this Request's combination of Local, Remote, Requester and Set.

func (*Request) LocalDataPath added in v0.9.0 

func (r *Request) LocalDataPath() string

LocalDataPath should be used instead of Request.Local for upload purposes.

For symlinks, returns a zero-sized file as iRODS doesn't handle links appropriately (there will be metadata allowing its recreation). Otherwise returns Request.Local.

func (*Request) MakeSafeForJSON added in v0.6.3 

func (r *Request) MakeSafeForJSON()

MakeSafeForJSON copies Local and Remote to LocalForJSON and RemoteForJSON, so that if this Request struct is encoded as JSON, non-UTF8 characters will be preserved. On decode be sure to use CorrectFromJSON().

func (*Request) Prepare added in v0.9.0 

func (r *Request) Prepare() error

Prepare calls ValidatePaths and returns its error, if any. Also prepares the request for correct hardlink handling, if Hardlink is set.

func (*Request) Put added in v0.9.0 

func (r *Request) Put(handler Handler) error

Put uses the given handler to upload our Local file to Remote. This has special handling for hardlinks: uploads Local to Hardlink, and an empty file to Remote, with linking metadata.

func (*Request) RemoteDataPath added in v0.9.0 

func (r *Request) RemoteDataPath() string

RemoteDataPath should be used if you want to know where the actual data for this request would get uploaded to. Normally returns Remote, but if a hardlink, returns Hardlink.

func (*Request) Remotes added in v0.9.0 

func (r *Request) Remotes() []string

Remotes normally returns Remote, but if we have a Hardlink, also returns that as a remote.

func (*Request) RemoveAndAddMetadata added in v0.9.0 

func (r *Request) RemoveAndAddMetadata(handler Handler) error

RemoveAndAddMetadata removes and adds metadata on our Remote based on the disk and remote metadata discovered during StatAndAssociateStandardMetadata().

Handles hardlinks (their Hardlink remote and empty file) appropriately.

func (*Request) StatAndAssociateStandardMetadata added in v0.9.0 

func (r *Request) StatAndAssociateStandardMetadata(lInfo *ObjectInfo, handler Handler) (*ObjectInfo, error)

StatAndAssociateStandardMetadata uses the handler to "stat" our Remote, and the given diskMeta to apply standard metadata to ourselves.

For hardlinks, does the same for our Hardlink and empty file to capture that info.

func (*Request) UploadedSize added in v0.8.0 

func (r *Request) UploadedSize() uint64

UploadedSize returns the number of bytes that were uploaded if Request.UploadPath was used.

func (*Request) ValidatePaths 

func (r *Request) ValidatePaths() error

ValidatePaths checks that both Local and Remote paths are absolute. (It does NOT check that either path exists.)

type RequestStatus 

type RequestStatus string
const (
	RequestStatusPending    RequestStatus = "pending"
	RequestStatusReserved   RequestStatus = "reserved"
	RequestStatusUploading  RequestStatus = "uploading"
	RequestStatusUploaded   RequestStatus = "uploaded"
	RequestStatusReplaced   RequestStatus = "replaced"
	RequestStatusUnmodified RequestStatus = "unmodified"
	RequestStatusMissing    RequestStatus = "missing"
	RequestStatusFailed     RequestStatus = "failed"
	ErrNotHumgenLustre                    = "not a valid humgen lustre path"
)

type Stuck added in v0.2.0 

type Stuck struct {
	UploadStarted time.Time
	Host          string
	PID           int
}

Stuck is used to provide details of a potentially "stuck" upload Request.

func NewStuck added in v0.2.0 

func NewStuck(uploadStarted time.Time) *Stuck

NewStuck returns a Stuck with the given UploadStarted and the current Host and PID.

func (*Stuck) String added in v0.2.0 

func (s *Stuck) String() string

String returns a message explaining our stuck details.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.