fs

package
v0.0.0-...-e391311 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 17, 2015 License: MIT Imports: 25 Imported by: 0

Documentation

Overview

Generic file system interface for rclone object storage systems

Index

Constants

View Source 
const (
	// User agent for Fs which can set it
	UserAgent = "rclone/" + Version
	// Very large precision value to show mod time isn't supported
	ModTimeNotSupported = 100 * 365 * 24 * time.Hour
)

Constants

View Source 
const Version = "v1.20"

Variables

View Source 
var (
	// Config file
	ConfigFile *goconfig.ConfigFile
	// Home directory
	HomeDir = configHome()
	// Config file path
	ConfigPath = path.Join(HomeDir, configFileName)
	// Global config
	Config = &ConfigInfo{}
)

Global

View Source 
var (

	// Error returned by NewFs if not found in config file
	NotFoundInConfigFile = fmt.Errorf("Didn't find section in config file")
	ErrorCantCopy        = fmt.Errorf("Can't copy object - incompatible remotes")
	ErrorCantMove        = fmt.Errorf("Can't copy object - incompatible remotes")
	ErrorCantDirMove     = fmt.Errorf("Can't copy directory - incompatible remotes")
	ErrorDirExists       = fmt.Errorf("Can't copy directory - destination already exists")
)

Globals

View Source 
var (
	Stats = NewStats()
)

Globals

Functions

func CalculateModifyWindow 

func CalculateModifyWindow(fs ...Fs)

Work out modify window for fses passed in - sets Config.ModifyWindow

This is the largest modify window of all the fses in use, and the user configured value

func Check 

func Check(fdst, fsrc Fs) error

Checks the files in fsrc and fdst according to Size and MD5SUM

func CheckMd5sums 

func CheckMd5sums(src, dst Object) (equal bool, unset bool, err error)

Check the two files to see if the MD5sums are the same

Returns two bools, the first of which is equality and the second of which is true if either of the MD5SUMs were unset.

May return an error which will already have been logged

If an error is returned it will return equal as false

func Choose 

func Choose(what string, defaults, help []string, newOk bool) string

Choose one of the defaults or type a new string if newOk is set

func ChooseOption 

func ChooseOption(o *Option) string

Choose an option

func ChooseRemote 

func ChooseRemote() string

ChooseRemote chooses a remote name

func Command 

func Command(commands []string) byte

Command - choose one

func Confirm 

func Confirm() bool

Asks the user for Yes or No and returns true or false

func Copy 

func Copy(f Fs, dst, src Object)

Copy src object to dst or f if nil

If dst is nil then the object must not exist already. If you do call Copy() with dst nil on a pre-existing file then some filing systems (eg Drive) may duplicate the file.

func CopyDir 

func CopyDir(fdst, fsrc Fs) error

Copies fsrc into fdst

func Debug 

func Debug(o interface{}, text string, args ...interface{})

Write debuging output for this Object or Fs

func DeleteFiles 

func DeleteFiles(to_be_deleted ObjectsChan)

Delete all the files passed in the channel

func DeleteRemote 

func DeleteRemote(name string)

Delete a remote

func EditConfig 

func EditConfig()

Edit the config file interactively

func EditRemote 

func EditRemote(name string)

Edit a remote

func Equal 

func Equal(src, dst Object) bool

Checks to see if the src and dst objects are equal by looking at size, mtime and MD5SUM

If the src and dst size are different then it is considered to be not equal. If --size-only is in effect then this is the only check that is done.

If the size is the same and the mtime is the same then it is considered to be equal. This check is skipped if using --checksum.

If the size is the same and mtime is different, unreadable or --checksum is set and the MD5SUM is the same then the file is considered to be equal. In this case the mtime on the dst is updated if --checksum is not set.

Otherwise the file is considered to be not equal including if there were errors reading info.

func ErrorLog 

func ErrorLog(o interface{}, text string, args ...interface{})

Write error log output for this Object or Fs Unconditionally logs a message regardless of Config.Quiet or Config.Verbose

func FsSame 

func FsSame(fdst, fsrc Fs) bool

Returns true if fdst and fsrc point to the same underlying Fs

func List 

func List(f Fs, w io.Writer) error

List the Fs to the supplied writer

Shows size and path

Lists in parallel which may get them out of order

func ListDir 

func ListDir(f Fs, w io.Writer) error

List the directories/buckets/containers in the Fs to the supplied writer

func ListFn 

func ListFn(f Fs, fn func(Object)) error

List the Fs to the supplied function

Lists in parallel which may get them out of order

func ListLong 

func ListLong(f Fs, w io.Writer) error

List the Fs to the supplied writer

Shows size, mod time and path

Lists in parallel which may get them out of order

func LoadConfig 

func LoadConfig()

Loads the config file

func Log 

func Log(o interface{}, text string, args ...interface{})

Write log output for this Object or Fs

func Md5sum 

func Md5sum(f Fs, w io.Writer) error

List the Fs to the supplied writer

Produces the same output as the md5sum command

Lists in parallel which may get them out of order

func Md5sumsEqual 

func Md5sumsEqual(src, dst string) bool

Md5sumsEqual checks to see if src == dst, but ignores empty strings

func MimeType 

func MimeType(o Object) string

Returns a guess at the mime type from the extension

func Mkdir 

func Mkdir(f Fs) error

Makes a destination directory or container

func MoveDir 

func MoveDir(fdst, fsrc Fs) error

Moves fsrc into fdst

func NewLoggedTransport 

func NewLoggedTransport(transport http.RoundTripper, logBody bool) *loggedTransport

NewLoggedTransport wraps the transport passed in and logs all roundtrips including the body if logBody is set.

func NewRemote 

func NewRemote(name string)

Make a new remote

func Obscure 

func Obscure(x string) string

Obscure a config value

func OkRemote 

func OkRemote(name string) bool

Print the contents of the remote and ask if it is OK

func OutputLog 

func OutputLog(o interface{}, text string, args ...interface{})

Outputs log for object

func PairChecker 

func PairChecker(in ObjectPairChan, out ObjectPairChan, wg *sync.WaitGroup)

Read Objects~s on in send to out if they need uploading

FIXME potentially doing lots of MD5SUMS at once

func PairCopier 

func PairCopier(in ObjectPairChan, fdst Fs, wg *sync.WaitGroup)

Read Objects on in and copy them

func PairMover 

func PairMover(in ObjectPairChan, fdst Fs, wg *sync.WaitGroup)

Read Objects on in and move them if possible, or copy them if not

func Purge 

func Purge(f Fs) error

Removes a container and all of its contents

FIXME doesn't delete local directories

func ReadLine 

func ReadLine() string

Read some input

func Register 

func Register(info *FsInfo)

Register a filesystem

Fs modules should use this in an init() function

func RemoteConfig 

func RemoteConfig(name string)

Runs the config helper for the remote if needed

func RetryError 

func RetryError(err error) error

RetryError makes an error which indicates it would like to be retried

func RetryErrorf 

func RetryErrorf(format string, a ...interface{}) error

RetryErrorf makes an error which indicates it would like to be retried

func Reveal 

func Reveal(y string) string

Reveal a config value

func Rmdir 

func Rmdir(f Fs) error

Removes a container but not if not empty

func SaveConfig 

func SaveConfig()

Save configuration file.

func ShowRemote 

func ShowRemote(name string)

Show the contents of the remote

func ShowRemotes 

func ShowRemotes()

Show an overview of the config file

func Sync 

func Sync(fdst, fsrc Fs) error

Syncs fsrc into fdst

Types

type Account 

type Account struct {
	// contains filtered or unexported fields
}

Account limits and accounts for one transfer

func NewAccount 

func NewAccount(in io.ReadCloser, obj Object) *Account

NewAccount makes a Account reader for an object

func (*Account) Close 

func (file *Account) Close() error

Close the object

func (*Account) ETA 

func (file *Account) ETA() (eta time.Duration, ok bool)

ETA returns the ETA of the current operation, rounded to full seconds. If the ETA cannot be determined 'ok' returns false.

func (*Account) Progress 

func (file *Account) Progress() (bytes, size int64)

Returns bytes read as well as the size. Size can be <= 0 if the size is unknown.

func (*Account) Read 

func (file *Account) Read(p []byte) (n int, err error)

Read bytes from the object - see io.Reader

func (*Account) Speed 

func (file *Account) Speed() (bps, current float64)

Speed returns the speed of the current file transfer in bytes per second, as well a an exponentially weighted moving average If no read has completed yet, 0 is returned for both values.

func (*Account) String 

func (file *Account) String() string

String produces stats for this file

type ConfigInfo 

type ConfigInfo struct {
	Verbose        bool
	Quiet          bool
	DryRun         bool
	CheckSum       bool
	SizeOnly       bool
	ModifyWindow   time.Duration
	Checkers       int
	Transfers      int
	ConnectTimeout time.Duration // Connect timeout
	Timeout        time.Duration // Data channel timeout
	DumpHeaders    bool
	DumpBodies     bool
}

Filesystem config options

func (*ConfigInfo) Client 

func (ci *ConfigInfo) Client() *http.Client

Transport returns an http.Client with the correct timeouts

func (*ConfigInfo) Transport 

func (ci *ConfigInfo) Transport() http.RoundTripper

Transport returns an http.RoundTripper with the correct timeouts

type Copier 

type Copier interface {
	// Copy src to this remote using server side copy operations.
	//
	// This is stored with the remote path given
	//
	// It returns the destination Object and a possible error
	//
	// Will only be called if src.Fs().Name() == f.Name()
	//
	// If it isn't possible then return fs.ErrorCantCopy
	Copy(src Object, remote string) (Object, error)
}

type Dir 

type Dir struct {
	Name  string    // name of the directory
	When  time.Time // modification or creation time - IsZero for unknown
	Bytes int64     // size of directory and contents -1 for unknown
	Count int64     // number of objects -1 for unknown
}

A structure of directory/container/bucket lists

type DirChan 

type DirChan chan *Dir

A channel of Dir objects

type DirMover 

type DirMover interface {
	// Move src to this remote using server side move operations.
	//
	// Will only be called if src.Fs().Name() == f.Name()
	//
	// If it isn't possible then return fs.ErrorCantDirMove
	//
	// If destination exists then return fs.ErrorDirExists
	DirMove(src Fs) error
}

type Fs 

type Fs interface {
	// The name of the remote (as passed into NewFs)
	Name() string

	// The root of the remote (as passed into NewFs)
	Root() string

	// String returns a description of the FS
	String() string

	// List the Fs into a channel
	List() ObjectsChan

	// List the Fs directories/buckets/containers into a channel
	ListDir() DirChan

	// Find the Object at remote.  Returns nil if can't be found
	NewFsObject(remote string) Object

	// Put in to the remote path with the modTime given of the given size
	//
	// May create the object even if it returns an error - if so
	// will return the object and the error, otherwise will return
	// nil and the error
	Put(in io.Reader, remote string, modTime time.Time, size int64) (Object, error)

	// Make the directory (container, bucket)
	//
	// Shouldn't return an error if it already exists
	Mkdir() error

	// Remove the directory (container, bucket) if empty
	//
	// Return an error if it doesn't exist or isn't empty
	Rmdir() error

	// Precision of the ModTimes in this Fs
	Precision() time.Duration
}

A Filesystem, describes the local filesystem and the remote object store

func NewFs 

func NewFs(path string) (Fs, error)

NewFs makes a new Fs object from the path

The path is of the form remote:path

Remotes are looked up in the config file. If the remote isn't found then NotFoundInConfigFile will be returned.

On Windows avoid single character remote names as they can be mixed up with drive letters.

func NewLimited 

func NewLimited(fs Fs, objects ...Object) Fs

NewLimited maks a limited Fs limited to the objects passed in

type FsInfo 

type FsInfo struct {
	// Name of this fs
	Name string
	// Create a new file system.  If root refers to an existing
	// object, then it should return a Fs which only returns that
	// object.
	NewFs func(name string, root string) (Fs, error)
	// Function to call to help with config
	Config func(string)
	// Options for the Fs configuration
	Options []Option
}

Filesystem info

func Find 

func Find(name string) (*FsInfo, error)

Finds a FsInfo object for the name passed in

Services are looked up in the config file

type Limited 

type Limited struct {
	// contains filtered or unexported fields
}

This defines a Limited Fs which can only return the Objects passed in from the Fs passed in

func (*Limited) Copy 

func (f *Limited) Copy(src Object, remote string) (Object, error)

Copy src to this remote using server side copy operations.

This is stored with the remote path given

It returns the destination Object and a possible error

Will only be called if src.Fs().Name() == f.Name()

If it isn't possible then return fs.ErrorCantCopy

func (*Limited) List 

func (f *Limited) List() ObjectsChan

List the Fs into a channel

func (*Limited) ListDir 

func (f *Limited) ListDir() DirChan

List the Fs directories/buckets/containers into a channel

func (*Limited) Mkdir 

func (f *Limited) Mkdir() error

Make the directory (container, bucket)

func (*Limited) Name 

func (f *Limited) Name() string

The name of the remote (as passed into NewFs)

func (*Limited) NewFsObject 

func (f *Limited) NewFsObject(remote string) Object

Find the Object at remote. Returns nil if can't be found

func (*Limited) Precision 

func (f *Limited) Precision() time.Duration

Precision of the ModTimes in this Fs

func (*Limited) Put 

func (f *Limited) Put(in io.Reader, remote string, modTime time.Time, size int64) (Object, error)

Put in to the remote path with the modTime given of the given size

May create the object even if it returns an error - if so will return the object and the error, otherwise will return nil and the error

func (*Limited) Rmdir 

func (f *Limited) Rmdir() error

Remove the directory (container, bucket) if empty

func (*Limited) Root 

func (f *Limited) Root() string

The root of the remote (as passed into NewFs)

func (*Limited) String 

func (f *Limited) String() string

String returns a description of the FS

type Mover 

type Mover interface {
	// Move src to this remote using server side move operations.
	//
	// This is stored with the remote path given
	//
	// It returns the destination Object and a possible error
	//
	// Will only be called if src.Fs().Name() == f.Name()
	//
	// If it isn't possible then return fs.ErrorCantMove
	Move(src Object, remote string) (Object, error)
}

type Object 

type Object interface {
	// String returns a description of the Object
	String() string

	// Fs returns the Fs that this object is part of
	Fs() Fs

	// Remote returns the remote path
	Remote() string

	// Md5sum returns the md5 checksum of the file
	// If no Md5sum is available it returns ""
	Md5sum() (string, error)

	// ModTime returns the modification date of the file
	// It should return a best guess if one isn't available
	ModTime() time.Time

	// SetModTime sets the metadata on the object to set the modification date
	SetModTime(time.Time)

	// Size returns the size of the file
	Size() int64

	// Open opens the file for read.  Call Close() on the returned io.ReadCloser
	Open() (io.ReadCloser, error)

	// Update in to the object with the modTime given of the given size
	Update(in io.Reader, modTime time.Time, size int64) error

	// Storable says whether this object can be stored
	Storable() bool

	// Removes this object
	Remove() error
}

A filesystem like object which can either be a remote object or a local file/directory

type ObjectPair 

type ObjectPair struct {
	// contains filtered or unexported fields
}

A pair of Objects

type ObjectPairChan 

type ObjectPairChan chan ObjectPair

A channel of ObjectPair

type Objects 

type Objects []Object

A slice of Objects

type ObjectsChan 

type ObjectsChan chan Object

A channel of Objects

type Option 

type Option struct {
	Name     string
	Help     string
	Optional bool
	Examples []OptionExample
}

An options for a Fs

type OptionExample 

type OptionExample struct {
	Value string
	Help  string
}

An example for an option

type Purger 

type Purger interface {
	// Purge all files in the root and the root directory
	//
	// Implement this if you have a way of deleting all the files
	// quicker than just running Remove() on the result of List()
	//
	// Return an error if it doesn't exist
	Purge() error
}

Optional interfaces

type Retry 

type Retry interface {
	error
	Retry() bool
}

An optional interface for error as to whether the operation should be retried

This should be returned from Update or Put methods as required

type SizeSuffix 

type SizeSuffix int64

func (*SizeSuffix) Set 

func (x *SizeSuffix) Set(s string) error

Set a SizeSuffix

func (SizeSuffix) String 

func (x SizeSuffix) String() string

Turn SizeSuffix into a string

func (*SizeSuffix) Type 

func (x *SizeSuffix) Type() string

Type of the value

type StatsInfo 

type StatsInfo struct {
	// contains filtered or unexported fields
}

Stats limits and accounts all transfers

func NewStats 

func NewStats() *StatsInfo

NewStats cretates an initialised StatsInfo

func (*StatsInfo) Bytes 

func (s *StatsInfo) Bytes(bytes int64)

Bytes updates the stats for bytes bytes

func (*StatsInfo) Checking 

func (s *StatsInfo) Checking(o Object)

Checking adds a check into the stats

func (*StatsInfo) DoneChecking 

func (s *StatsInfo) DoneChecking(o Object)

DoneChecking removes a check from the stats

func (*StatsInfo) DoneTransferring 

func (s *StatsInfo) DoneTransferring(o Object)

DoneTransferring removes a transfer from the stats

func (*StatsInfo) Error 

func (s *StatsInfo) Error()

Error adds a single error into the stats

func (*StatsInfo) Errored 

func (s *StatsInfo) Errored() bool

Errored returns whether there have been any errors

func (*StatsInfo) Errors 

func (s *StatsInfo) Errors(errors int64)

Errors updates the stats for errors

func (*StatsInfo) GetErrors 

func (s *StatsInfo) GetErrors() int64

GetErrors reads the number of errors

func (*StatsInfo) GetTransfers 

func (s *StatsInfo) GetTransfers() int64

GetTransfers reads the number of transfers

func (*StatsInfo) Log 

func (s *StatsInfo) Log()

Log outputs the StatsInfo to the log

func (*StatsInfo) ResetCounters 

func (s *StatsInfo) ResetCounters()

ResetCounters sets the counters (bytes, checks, errors, transfers) to 0

func (*StatsInfo) ResetErrors 

func (s *StatsInfo) ResetErrors()

ResetErrors sets the errors count to 0

func (*StatsInfo) String 

func (s *StatsInfo) String() string

String convert the StatsInfo to a string for printing

func (*StatsInfo) Transferring 

func (s *StatsInfo) Transferring(o Object)

Transferring adds a transfer into the stats

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.