fs

package
v1.46.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 9, 2019 License: MIT Imports: 23 Imported by: 0

Documentation

Overview

Package fs is a generic file system interface for rclone object storage systems

Index

Examples

Constants

View Source 
const (
	// ModTimeNotSupported is a very large precision value to show
	// mod time isn't supported on this Fs
	ModTimeNotSupported = 100 * 365 * 24 * time.Hour
	// MaxLevel is a sentinel representing an infinite depth for listings
	MaxLevel = math.MaxInt32
	// EntryDirectory should be used to classify remote paths in directories
	EntryDirectory EntryType = iota // 0
	// EntryObject should be used to classify remote paths in objects
	EntryObject // 1
)

Constants

View Source 
const DurationOff = Duration((1 << 63) - 1)

DurationOff is the default value for flags which can be turned off

Variables

View Source 
var (
	// Config is the global config
	Config = NewConfig()

	// Read a value from the config file
	//
	// This is a function pointer to decouple the config
	// implementation from the fs
	ConfigFileGet = func(section, key string) (string, bool) { return "", false }

	// Set a value into the config file
	//
	// This is a function pointer to decouple the config
	// implementation from the fs
	ConfigFileSet = func(section, key, value string) {
		Errorf(nil, "No config handler to set %q = %q in section %q of the config file", key, value, section)
	}

	// CountError counts an error.  If any errors have been
	// counted then it will exit with a non zero error code.
	//
	// This is a function pointer to decouple the config
	// implementation from the fs
	CountError = func(err error) {}

	// ConfigProvider is the config key used for provider options
	ConfigProvider = "provider"
)

Global

View Source 
var (
	// Filesystem registry
	Registry []*RegInfo
	// ErrorNotFoundInConfigFile is returned by NewFs if not found in config file
	ErrorNotFoundInConfigFile        = errors.New("didn't find section in config file")
	ErrorCantPurge                   = errors.New("can't purge directory")
	ErrorCantCopy                    = errors.New("can't copy object - incompatible remotes")
	ErrorCantMove                    = errors.New("can't move object - incompatible remotes")
	ErrorCantDirMove                 = errors.New("can't move directory - incompatible remotes")
	ErrorDirExists                   = errors.New("can't copy directory - destination already exists")
	ErrorCantSetModTime              = errors.New("can't set modified time")
	ErrorCantSetModTimeWithoutDelete = errors.New("can't set modified time without deleting existing object")
	ErrorDirNotFound                 = errors.New("directory not found")
	ErrorObjectNotFound              = errors.New("object not found")
	ErrorLevelNotSupported           = errors.New("level value not supported")
	ErrorListAborted                 = errors.New("list aborted")
	ErrorListBucketRequired          = errors.New("bucket or container name is needed in remote")
	ErrorIsFile                      = errors.New("is a file not a directory")
	ErrorNotAFile                    = errors.New("is a not a regular file")
	ErrorNotDeleting                 = errors.New("not deleting files as there were IO errors")
	ErrorNotDeletingDirs             = errors.New("not deleting directories as there were IO errors")
	ErrorCantMoveOverlapping         = errors.New("can't move files on overlapping remotes")
	ErrorDirectoryNotEmpty           = errors.New("directory not empty")
	ErrorImmutableModified           = errors.New("immutable file modified")
	ErrorPermissionDenied            = errors.New("permission denied")
)

Globals

View Source 
var DumpFlagsList string

DumpFlagsList is a list of dump flags used in the help

View Source 
var LogPrint = func(level LogLevel, text string) {
	text = fmt.Sprintf("%-6s: %s", level, text)
	_ = log.Output(4, text)
}

LogPrint sends the text to the logger of level

View Source 
var Version = "v1.46"

Version of rclone

Functions

func CheckClose 

func CheckClose(c io.Closer, err *error)

CheckClose is a utility function used to check the return from Close in a defer statement.

func ConfigMap 

func ConfigMap(fsInfo *RegInfo, configName string) (config *configmap.Map)

ConfigMap creates a configmap.Map from the *RegInfo and the configName passed in.

If fsInfo is nil then the returned configmap.Map should only be used for reading non backend specific parameters, such as "type".

func ConfigToEnv 

func ConfigToEnv(section, name string) string

ConfigToEnv converts an config section and name, eg ("myremote", "ignore-size") into an environment name "RCLONE_CONFIG_MYREMOTE_IGNORE_SIZE"

func Debugf 

func Debugf(o interface{}, text string, args ...interface{})

Debugf writes debugging output for this Object or Fs. Use this for debug only. The user must have to specify -vv to see this.

func DirEntryType 

func DirEntryType(d DirEntry) string

DirEntryType returns a string description of the DirEntry, either "object", "directory" or "unknown type XXX"

func Errorf 

func Errorf(o interface{}, text string, args ...interface{})

Errorf writes error log output for this Object or Fs. It should always be seen by the user.

func FileExists 

func FileExists(fs Fs, remote string) (bool, error)

FileExists returns true if a file remote exists. If remote is a directory, FileExists returns false.

func FixRangeOption 

func FixRangeOption(options []OpenOption, size int64)

FixRangeOption looks through the slice of options and adjusts any RangeOption~s found that request a fetch from the end into an absolute fetch using the size passed in and makes sure the range does not exceed filesize. Some remotes (eg Onedrive, Box) don't support range requests which index from the end.

func GetModifyWindow 

func GetModifyWindow(fss ...Info) time.Duration

GetModifyWindow calculates the maximum modify window between the given Fses and the Config.ModifyWindow parameter.

func Infof 

func Infof(o interface{}, text string, args ...interface{})

Infof writes info on transfers for this Object or Fs. Use this level for logging transfers, deletions and things which should appear with the -v flag.

func LogDirName 

func LogDirName(f Fs, dir string) interface{}

LogDirName returns an object for the logger, logging a root directory which would normally be "" as the Fs

func LogLevelPrintf 

func LogLevelPrintf(level LogLevel, o interface{}, text string, args ...interface{})

LogLevelPrintf writes logs at the given level

func LogPrintf 

func LogPrintf(level LogLevel, o interface{}, text string, args ...interface{})

LogPrintf produces a log string from the arguments passed in

func Logf 

func Logf(o interface{}, text string, args ...interface{})

Logf writes log output for this Object or Fs. This should be considered to be Info level logging. It is the default level. By default rclone should not log very much so only use this for important things the user should see. The user can filter these out with the -q flag.

func MimeType 

func MimeType(o ObjectInfo) (mimeType string)

MimeType returns the MimeType from the object, either by calling the MimeTyper interface or using MimeTypeFromName

func MimeTypeDirEntry 

func MimeTypeDirEntry(item DirEntry) string

MimeTypeDirEntry returns the MimeType of a DirEntry

It returns "inode/directory" for directories, or uses MimeType(Object)

func MimeTypeFromName 

func MimeTypeFromName(remote string) (mimeType string)

MimeTypeFromName returns a guess at the mime type from the name

func NewUsageValue 

func NewUsageValue(value int64) *int64

NewUsageValue makes a valid value

func OpenOptionAddHTTPHeaders 

func OpenOptionAddHTTPHeaders(headers http.Header, options []OpenOption)

OpenOptionAddHTTPHeaders Sets each header found in options to the http.Header map provided the key was non empty.

func OpenOptionAddHeaders 

func OpenOptionAddHeaders(options []OpenOption, headers map[string]string)

OpenOptionAddHeaders adds each header found in options to the headers map provided the key was non empty.

func OpenOptionHeaders 

func OpenOptionHeaders(options []OpenOption) (headers map[string]string)

OpenOptionHeaders adds each header found in options to the headers map provided the key was non empty.

It returns a nil map if options was empty

func OptionToEnv 

func OptionToEnv(name string) string

OptionToEnv converts an option name, eg "ignore-size" into an environment name "RCLONE_IGNORE_SIZE"

func ParseDuration 

func ParseDuration(age string) (time.Duration, error)

ParseDuration parses a duration string. Accept ms|s|m|h|d|w|M|y suffixes. Defaults to second if not provided

func Register 

func Register(info *RegInfo)

Register a filesystem

Fs modules should use this in an init() function

Types

type Abouter 

type Abouter interface {
	// About gets quota information from the Fs
	About() (*Usage, error)
}

Abouter is an optional interface for Fs

type BwTimeSlot 

type BwTimeSlot struct {
	DayOfTheWeek int
	HHMM         int
	Bandwidth    SizeSuffix
}

BwTimeSlot represents a bandwidth configuration at a point in time.

type BwTimetable 

type BwTimetable []BwTimeSlot

BwTimetable contains all configured time slots.

func (BwTimetable) LimitAt 

func (x BwTimetable) LimitAt(tt time.Time) BwTimeSlot

LimitAt returns a BwTimeSlot for the time requested.

func (*BwTimetable) Set 

func (x *BwTimetable) Set(s string) error

Set the bandwidth timetable.

func (BwTimetable) String 

func (x BwTimetable) String() string

String returns a printable representation of BwTimetable.

func (BwTimetable) Type 

func (x BwTimetable) Type() string

Type of the value

type ChangeNotifier 

type ChangeNotifier interface {
	// ChangeNotify calls the passed function with a path
	// that has had changes. If the implementation
	// uses polling, it should adhere to the given interval.
	// At least one value will be written to the channel,
	// specifying the initial value and updated values might
	// follow. A 0 Duration should pause the polling.
	// The ChangeNotify implemantion must empty the channel
	// regulary. When the channel gets closed, the implemantion
	// should stop polling and release resources.
	ChangeNotify(func(string, EntryType), <-chan time.Duration)
}

ChangeNotifier is an optional interface for Fs

type CleanUpper 

type CleanUpper interface {
	// CleanUp the trash in the Fs
	//
	// Implement this if you have a way of emptying the trash or
	// otherwise cleaning up old versions of files.
	CleanUp() error
}

CleanUpper is an optional interfaces for Fs

type CommaSepList 

type CommaSepList []string

CommaSepList is a comma separated config value It uses the encoding/csv rules for quoting and escaping

Example 
for _, s := range []string{
	`remotea:test/dir,remoteb:`,
	`"remotea:test/space dir",remoteb:`,
	`"remotea:test/quote""dir",remoteb:`,
} {
	var l CommaSepList
	must(l.Set(s))
	fmt.Printf("%#v\n", l)
}

Output:

fs.CommaSepList{"remotea:test/dir", "remoteb:"}
fs.CommaSepList{"remotea:test/space dir", "remoteb:"}
fs.CommaSepList{"remotea:test/quote\"dir", "remoteb:"}

func (*CommaSepList) Scan 

func (l *CommaSepList) Scan(s fmt.ScanState, ch rune) error

Scan implements the fmt.Scanner interface

func (*CommaSepList) Set 

func (l *CommaSepList) Set(s string) error

Set the List entries

func (CommaSepList) String 

func (l CommaSepList) String() string

func (CommaSepList) Type 

func (CommaSepList) Type() string

Type of the value

type ConfigInfo 

type ConfigInfo struct {
	LogLevel              LogLevel
	StatsLogLevel         LogLevel
	DryRun                bool
	CheckSum              bool
	SizeOnly              bool
	IgnoreTimes           bool
	IgnoreExisting        bool
	IgnoreErrors          bool
	ModifyWindow          time.Duration
	Checkers              int
	Transfers             int
	ConnectTimeout        time.Duration // Connect timeout
	Timeout               time.Duration // Data channel timeout
	Dump                  DumpFlags
	InsecureSkipVerify    bool // Skip server certificate verification
	DeleteMode            DeleteMode
	MaxDelete             int64
	TrackRenames          bool // Track file renames.
	LowLevelRetries       int
	UpdateOlder           bool // Skip files that are newer on the destination
	NoGzip                bool // Disable compression
	MaxDepth              int
	IgnoreSize            bool
	IgnoreChecksum        bool
	NoTraverse            bool
	NoUpdateModTime       bool
	DataRateUnit          string
	BackupDir             string
	Suffix                string
	UseListR              bool
	BufferSize            SizeSuffix
	BwLimit               BwTimetable
	TPSLimit              float64
	TPSLimitBurst         int
	BindAddr              net.IP
	DisableFeatures       []string
	UserAgent             string
	Immutable             bool
	AutoConfirm           bool
	StreamingUploadCutoff SizeSuffix
	StatsFileNameLength   int
	AskPassword           bool
	UseServerModTime      bool
	MaxTransfer           SizeSuffix
	MaxBacklog            int
	StatsOneLine          bool
	Progress              bool
	Cookie                bool
	UseMmap               bool
}

ConfigInfo is filesystem config options

func NewConfig 

func NewConfig() *ConfigInfo

NewConfig creates a new config with everything set to the default value. These are the ultimate defaults and are overriden by the config module.

type Copier 

type Copier interface {
	// Copy src to this remote using server side copy operations.
	//
	// This is stored with the remote path given
	//
	// It returns the destination Object and a possible error
	//
	// Will only be called if src.Fs().Name() == f.Name()
	//
	// If it isn't possible then return fs.ErrorCantCopy
	Copy(src Object, remote string) (Object, error)
}

Copier is an optional interface for Fs

type DeleteMode 

type DeleteMode byte

DeleteMode describes the possible delete modes in the config 

const (
	DeleteModeOff DeleteMode = iota
	DeleteModeBefore
	DeleteModeDuring
	DeleteModeAfter
	DeleteModeOnly
	DeleteModeDefault = DeleteModeAfter
)

DeleteMode constants

type Dir 

type Dir struct {
	// contains filtered or unexported fields
}

Dir describes an unspecialized directory for directory/container/bucket lists

func NewDir 

func NewDir(remote string, modTime time.Time) *Dir

NewDir creates an unspecialized Directory object

func NewDirCopy 

func NewDirCopy(d Directory) *Dir

NewDirCopy creates an unspecialized copy of the Directory object passed in

func (*Dir) ID 

func (d *Dir) ID() string

ID gets the optional ID

func (*Dir) Items 

func (d *Dir) Items() int64

Items returns the count of items in this directory or this directory and subdirectories if known, -1 for unknown

func (*Dir) ModTime 

func (d *Dir) ModTime() time.Time

ModTime returns the modification date of the file It should return a best guess if one isn't available

func (*Dir) Remote 

func (d *Dir) Remote() string

Remote returns the remote path

func (*Dir) SetID 

func (d *Dir) SetID(id string) *Dir

SetID sets the optional ID

func (*Dir) SetItems 

func (d *Dir) SetItems(items int64) *Dir

SetItems sets the number of items in the directory

func (*Dir) SetRemote 

func (d *Dir) SetRemote(remote string) *Dir

SetRemote sets the remote

func (*Dir) SetSize 

func (d *Dir) SetSize(size int64) *Dir

SetSize sets the size of the directory

func (*Dir) Size 

func (d *Dir) Size() int64

Size returns the size of the file

func (*Dir) String 

func (d *Dir) String() string

String returns the name

type DirCacheFlusher 

type DirCacheFlusher interface {
	// DirCacheFlush resets the directory cache - used in testing
	// as an optional interface
	DirCacheFlush()
}

DirCacheFlusher is an optional interface for Fs

type DirEntries 

type DirEntries []DirEntry

DirEntries is a slice of Object or *Dir

func (DirEntries) ForDir 

func (ds DirEntries) ForDir(fn func(dir Directory))

ForDir runs the function supplied on every Directory in the entries

func (DirEntries) ForDirError 

func (ds DirEntries) ForDirError(fn func(dir Directory) error) error

ForDirError runs the function supplied on every Directory in the entries

func (DirEntries) ForObject 

func (ds DirEntries) ForObject(fn func(o Object))

ForObject runs the function supplied on every object in the entries

func (DirEntries) ForObjectError 

func (ds DirEntries) ForObjectError(fn func(o Object) error) error

ForObjectError runs the function supplied on every object in the entries

func (DirEntries) Len 

func (ds DirEntries) Len() int

Len is part of sort.Interface.

func (DirEntries) Less 

func (ds DirEntries) Less(i, j int) bool

Less is part of sort.Interface.

func (DirEntries) Swap 

func (ds DirEntries) Swap(i, j int)

Swap is part of sort.Interface.

type DirEntry 

type DirEntry interface {
	// String returns a description of the Object
	String() string

	// Remote returns the remote path
	Remote() string

	// ModTime returns the modification date of the file
	// It should return a best guess if one isn't available
	ModTime() time.Time

	// Size returns the size of the file
	Size() int64
}

DirEntry provides read only information about the common subset of a Dir or Object. These are returned from directory listings - type assert them into the correct type.

type DirMover 

type DirMover interface {
	// DirMove moves src, srcRemote to this remote at dstRemote
	// using server side move operations.
	//
	// Will only be called if src.Fs().Name() == f.Name()
	//
	// If it isn't possible then return fs.ErrorCantDirMove
	//
	// If destination exists then return fs.ErrorDirExists
	DirMove(src Fs, srcRemote, dstRemote string) error
}

DirMover is an optional interface for Fs

type Directory 

type Directory interface {
	DirEntry

	// Items returns the count of items in this directory or this
	// directory and subdirectories if known, -1 for unknown
	Items() int64

	// ID returns the internal ID of this directory if known, or
	// "" otherwise
	ID() string
}

Directory is a filesystem like directory provided by an Fs

type DumpFlags 

type DumpFlags int

DumpFlags describes the Dump options in force 

const (
	DumpHeaders DumpFlags = 1 << iota
	DumpBodies
	DumpRequests
	DumpResponses
	DumpAuth
	DumpFilters
	DumpGoRoutines
	DumpOpenFiles
)

DumpFlags definitions

func (*DumpFlags) Set 

func (f *DumpFlags) Set(s string) error

Set a DumpFlags as a comma separated list of flags

func (DumpFlags) String 

func (f DumpFlags) String() string

String turns a DumpFlags into a string

func (*DumpFlags) Type 

func (f *DumpFlags) Type() string

Type of the value

type Duration 

type Duration time.Duration

Duration is a time.Duration with some more parsing options

func (Duration) IsSet 

func (d Duration) IsSet() bool

IsSet returns if the duration is != DurationOff

func (*Duration) Scan 

func (d *Duration) Scan(s fmt.ScanState, ch rune) error

Scan implements the fmt.Scanner interface

func (*Duration) Set 

func (d *Duration) Set(s string) error

Set a Duration

func (Duration) String 

func (d Duration) String() string

Turn Duration into a string

func (Duration) Type 

func (d Duration) Type() string

Type of the value

type EntryType 

type EntryType int

EntryType can be associated with remote paths to identify their type

type Features 

type Features struct {
	// Feature flags, whether Fs
	CaseInsensitive         bool // has case insensitive files
	DuplicateFiles          bool // allows duplicate files
	ReadMimeType            bool // can read the mime type of objects
	WriteMimeType           bool // can set the mime type of objects
	CanHaveEmptyDirectories bool // can have empty directories
	BucketBased             bool // is bucket based (like s3, swift etc)
	SetTier                 bool // allows set tier functionality on objects
	GetTier                 bool // allows to retrieve storage tier of objects

	// Purge all files in the root and the root directory
	//
	// Implement this if you have a way of deleting all the files
	// quicker than just running Remove() on the result of List()
	//
	// Return an error if it doesn't exist
	Purge func() error

	// Copy src to this remote using server side copy operations.
	//
	// This is stored with the remote path given
	//
	// It returns the destination Object and a possible error
	//
	// Will only be called if src.Fs().Name() == f.Name()
	//
	// If it isn't possible then return fs.ErrorCantCopy
	Copy func(src Object, remote string) (Object, error)

	// Move src to this remote using server side move operations.
	//
	// This is stored with the remote path given
	//
	// It returns the destination Object and a possible error
	//
	// Will only be called if src.Fs().Name() == f.Name()
	//
	// If it isn't possible then return fs.ErrorCantMove
	Move func(src Object, remote string) (Object, error)

	// DirMove moves src, srcRemote to this remote at dstRemote
	// using server side move operations.
	//
	// Will only be called if src.Fs().Name() == f.Name()
	//
	// If it isn't possible then return fs.ErrorCantDirMove
	//
	// If destination exists then return fs.ErrorDirExists
	DirMove func(src Fs, srcRemote, dstRemote string) error

	// ChangeNotify calls the passed function with a path
	// that has had changes. If the implementation
	// uses polling, it should adhere to the given interval.
	ChangeNotify func(func(string, EntryType), <-chan time.Duration)

	// UnWrap returns the Fs that this Fs is wrapping
	UnWrap func() Fs

	// WrapFs returns the Fs that is wrapping this Fs
	WrapFs func() Fs

	// SetWrapper sets the Fs that is wrapping this Fs
	SetWrapper func(f Fs)

	// DirCacheFlush resets the directory cache - used in testing
	// as an optional interface
	DirCacheFlush func()

	// PublicLink generates a public link to the remote path (usually readable by anyone)
	PublicLink func(remote string) (string, error)

	// Put in to the remote path with the modTime given of the given size
	//
	// May create the object even if it returns an error - if so
	// will return the object and the error, otherwise will return
	// nil and the error
	//
	// May create duplicates or return errors if src already
	// exists.
	PutUnchecked func(in io.Reader, src ObjectInfo, options ...OpenOption) (Object, error)

	// PutStream uploads to the remote path with the modTime given of indeterminate size
	//
	// May create the object even if it returns an error - if so
	// will return the object and the error, otherwise will return
	// nil and the error
	PutStream func(in io.Reader, src ObjectInfo, options ...OpenOption) (Object, error)

	// MergeDirs merges the contents of all the directories passed
	// in into the first one and rmdirs the other directories.
	MergeDirs func([]Directory) error

	// CleanUp the trash in the Fs
	//
	// Implement this if you have a way of emptying the trash or
	// otherwise cleaning up old versions of files.
	CleanUp func() error

	// ListR lists the objects and directories of the Fs starting
	// from dir recursively into out.
	//
	// dir should be "" to start from the root, and should not
	// have trailing slashes.
	//
	// This should return ErrDirNotFound if the directory isn't
	// found.
	//
	// It should call callback for each tranche of entries read.
	// These need not be returned in any particular order.  If
	// callback returns an error then the listing will stop
	// immediately.
	//
	// Don't implement this unless you have a more efficient way
	// of listing recursively that doing a directory traversal.
	ListR ListRFn

	// About gets quota information from the Fs
	About func() (*Usage, error)
}

Features describe the optional features of the Fs

func (*Features) Disable 

func (ft *Features) Disable(name string) *Features

Disable nil's out the named feature. If it isn't found then it will log a message.

func (*Features) DisableList 

func (ft *Features) DisableList(list []string) *Features

DisableList nil's out the comma separated list of named features. If it isn't found then it will log a message.

func (*Features) Fill 

func (ft *Features) Fill(f Fs) *Features

Fill fills in the function pointers in the Features struct from the optional interfaces. It returns the original updated Features struct passed in.

func (*Features) List 

func (ft *Features) List() (out []string)

List returns a slice of all the possible feature names

func (*Features) Mask 

func (ft *Features) Mask(f Fs) *Features

Mask the Features with the Fs passed in

Only optional features which are implemented in both the original Fs AND the one passed in will be advertised. Any features which aren't in both will be set to false/nil, except for UnWrap/Wrap which will be left untouched.

func (*Features) Wrap 

func (ft *Features) Wrap(f Fs) *Features

Wrap makes a Copy of the features passed in, overriding the UnWrap/Wrap method only if available in f.

func (*Features) WrapsFs 

func (ft *Features) WrapsFs(f Fs, w Fs) *Features

WrapsFs adds extra information between `f` which wraps `w`

type Fs 

type Fs interface {
	Info

	// List the objects and directories in dir into entries.  The
	// entries can be returned in any order but should be for a
	// complete directory.
	//
	// dir should be "" to list the root, and should not have
	// trailing slashes.
	//
	// This should return ErrDirNotFound if the directory isn't
	// found.
	List(dir string) (entries DirEntries, err error)

	// NewObject finds the Object at remote.  If it can't be found
	// it returns the error ErrorObjectNotFound.
	NewObject(remote string) (Object, error)

	// Put in to the remote path with the modTime given of the given size
	//
	// When called from outside a Fs by rclone, src.Size() will always be >= 0.
	// But for unknown-sized objects (indicated by src.Size() == -1), Put should either
	// return an error or upload it properly (rather than e.g. calling panic).
	//
	// May create the object even if it returns an error - if so
	// will return the object and the error, otherwise will return
	// nil and the error
	Put(in io.Reader, src ObjectInfo, options ...OpenOption) (Object, error)

	// Mkdir makes the directory (container, bucket)
	//
	// Shouldn't return an error if it already exists
	Mkdir(dir string) error

	// Rmdir removes the directory (container, bucket) if empty
	//
	// Return an error if it doesn't exist or isn't empty
	Rmdir(dir string) error
}

Fs is the interface a cloud storage system must provide

func NewFs 

func NewFs(path string) (Fs, error)

NewFs makes a new Fs object from the path

The path is of the form remote:path

Remotes are looked up in the config file. If the remote isn't found then NotFoundInConfigFile will be returned.

On Windows avoid single character remote names as they can be mixed up with drive letters.

func TemporaryLocalFs 

func TemporaryLocalFs() (Fs, error)

TemporaryLocalFs creates a local FS in the OS's temporary directory.

No cleanup is performed, the caller must call Purge on the Fs themselves.

type GetTierer added in v1.46.0 

type GetTierer interface {
	// GetTier returns storage tier or class of the Object
	GetTier() string
}

GetTierer is an optional interface for Object

type HTTPOption 

type HTTPOption struct {
	Key   string
	Value string
}

HTTPOption defines a general purpose HTTP option

func (*HTTPOption) Header 

func (o *HTTPOption) Header() (key string, value string)

Header formats the option as an http header

func (*HTTPOption) Mandatory 

func (o *HTTPOption) Mandatory() bool

Mandatory returns whether the option must be parsed or can be ignored

func (*HTTPOption) String 

func (o *HTTPOption) String() string

String formats the option into human readable form

type HashesOption 

type HashesOption struct {
	Hashes hash.Set
}

HashesOption defines an option used to tell the local fs to limit the number of hashes it calculates.

func (*HashesOption) Header 

func (o *HashesOption) Header() (key string, value string)

Header formats the option as an http header

func (*HashesOption) Mandatory 

func (o *HashesOption) Mandatory() bool

Mandatory returns whether the option must be parsed or can be ignored

func (*HashesOption) String 

func (o *HashesOption) String() string

String formats the option into human readable form

type IDer 

type IDer interface {
	// ID returns the ID of the Object if known, or "" if not
	ID() string
}

IDer is an optional interface for Object

type Info 

type Info interface {
	// Name of the remote (as passed into NewFs)
	Name() string

	// Root of the remote (as passed into NewFs)
	Root() string

	// String returns a description of the FS
	String() string

	// Precision of the ModTimes in this Fs
	Precision() time.Duration

	// Returns the supported hash types of the filesystem
	Hashes() hash.Set

	// Features returns the optional features of this Fs
	Features() *Features
}

Info provides a read only interface to information about a filesystem.

type ListRCallback 

type ListRCallback func(entries DirEntries) error

ListRCallback defines a callback function for ListR to use

It is called for each tranche of entries read from the listing and if it returns an error, the listing stops.

type ListRFn 

type ListRFn func(dir string, callback ListRCallback) error

ListRFn is defines the call used to recursively list a directory

type ListRer 

type ListRer interface {
	// ListR lists the objects and directories of the Fs starting
	// from dir recursively into out.
	//
	// dir should be "" to start from the root, and should not
	// have trailing slashes.
	//
	// This should return ErrDirNotFound if the directory isn't
	// found.
	//
	// It should call callback for each tranche of entries read.
	// These need not be returned in any particular order.  If
	// callback returns an error then the listing will stop
	// immediately.
	//
	// Don't implement this unless you have a more efficient way
	// of listing recursively that doing a directory traversal.
	ListR(dir string, callback ListRCallback) error
}

ListRer is an optional interfaces for Fs

type LogLevel 

type LogLevel byte

LogLevel describes rclone's logs. These are a subset of the syslog log levels. 

const (
	LogLevelEmergency LogLevel = iota
	LogLevelAlert
	LogLevelCritical
	LogLevelError // Error - can't be suppressed
	LogLevelWarning
	LogLevelNotice // Normal logging, -q suppresses
	LogLevelInfo   // Transfers, needs -v
	LogLevelDebug  // Debug level, needs -vv
)

Log levels. These are the syslog levels of which we only use a subset. 

LOG_EMERG      system is unusable
LOG_ALERT      action must be taken immediately
LOG_CRIT       critical conditions
LOG_ERR        error conditions
LOG_WARNING    warning conditions
LOG_NOTICE     normal, but significant, condition
LOG_INFO       informational message
LOG_DEBUG      debug-level message

func (*LogLevel) Set 

func (l *LogLevel) Set(s string) error

Set a LogLevel

func (LogLevel) String 

func (l LogLevel) String() string

String turns a LogLevel into a string

func (*LogLevel) Type 

func (l *LogLevel) Type() string

Type of the value

type MergeDirser 

type MergeDirser interface {
	// MergeDirs merges the contents of all the directories passed
	// in into the first one and rmdirs the other directories.
	MergeDirs([]Directory) error
}

MergeDirser is an option interface for Fs

type MimeTyper 

type MimeTyper interface {
	// MimeType returns the content type of the Object if
	// known, or "" if not
	MimeType() string
}

MimeTyper is an optional interface for Object

type Mover 

type Mover interface {
	// Move src to this remote using server side move operations.
	//
	// This is stored with the remote path given
	//
	// It returns the destination Object and a possible error
	//
	// Will only be called if src.Fs().Name() == f.Name()
	//
	// If it isn't possible then return fs.ErrorCantMove
	Move(src Object, remote string) (Object, error)
}

Mover is an optional interface for Fs

type Object 

type Object interface {
	ObjectInfo

	// SetModTime sets the metadata on the object to set the modification date
	SetModTime(time.Time) error

	// Open opens the file for read.  Call Close() on the returned io.ReadCloser
	Open(options ...OpenOption) (io.ReadCloser, error)

	// Update in to the object with the modTime given of the given size
	//
	// When called from outside a Fs by rclone, src.Size() will always be >= 0.
	// But for unknown-sized objects (indicated by src.Size() == -1), Upload should either
	// return an error or update the object properly (rather than e.g. calling panic).
	Update(in io.Reader, src ObjectInfo, options ...OpenOption) error

	// Removes this object
	Remove() error
}

Object is a filesystem like object provided by an Fs

type ObjectInfo 

type ObjectInfo interface {
	DirEntry

	// Fs returns read only access to the Fs that this object is part of
	Fs() Info

	// Hash returns the selected checksum of the file
	// If no checksum is available it returns ""
	Hash(hash.Type) (string, error)

	// Storable says whether this object can be stored
	Storable() bool
}

ObjectInfo provides read only information about an object.

type ObjectPair 

type ObjectPair struct {
	Src, Dst Object
}

ObjectPair is a pair of Objects used to describe a potential copy operation.

type ObjectUnWrapper 

type ObjectUnWrapper interface {
	// UnWrap returns the Object that this Object is wrapping or
	// nil if it isn't wrapping anything
	UnWrap() Object
}

ObjectUnWrapper is an optional interface for Object

type Objects 

type Objects []Object

Objects is a slice of Object~s

type ObjectsChan 

type ObjectsChan chan Object

ObjectsChan is a channel of Objects

type OpenOption 

type OpenOption interface {
	fmt.Stringer

	// Header returns the option as an HTTP header
	Header() (key string, value string)

	// Mandatory returns whether this option can be ignored or not
	Mandatory() bool
}

OpenOption is an interface describing options for Open

type Option 

type Option struct {
	Name       string           // name of the option in snake_case
	Help       string           // Help, the first line only is used for the command line help
	Provider   string           // Set to filter on provider
	Default    interface{}      // default value, nil => ""
	Value      interface{}      // value to be set by flags
	Examples   OptionExamples   `json:",omitempty"` // config examples
	ShortOpt   string           // the short option for this if required
	Hide       OptionVisibility // set this to hide the config from the configurator or the command line
	Required   bool             // this option is required
	IsPassword bool             // set if the option is a password
	NoPrefix   bool             // set if the option for this should not use the backend prefix
	Advanced   bool             // set if this is an advanced config option
}

Option is describes an option for the config wizard

This also describes command line options and environment variables

func (*Option) EnvVarName added in v1.46.0 

func (o *Option) EnvVarName(prefix string) string

EnvVarName for the option

func (*Option) FlagName added in v1.46.0 

func (o *Option) FlagName(prefix string) string

FlagName for the option

func (*Option) GetValue added in v1.46.0 

func (o *Option) GetValue() interface{}

GetValue gets the current current value which is the default if not set

func (*Option) Set 

func (o *Option) Set(s string) (err error)

Set a Option from a string

func (*Option) String 

func (o *Option) String() string

String turns Option into a string

func (*Option) Type 

func (o *Option) Type() string

Type of the value

type OptionExample 

type OptionExample struct {
	Value    string
	Help     string
	Provider string
}

OptionExample describes an example for an Option

type OptionExamples 

type OptionExamples []OptionExample

OptionExamples is a slice of examples

func (OptionExamples) Len 

func (os OptionExamples) Len() int

Len is part of sort.Interface.

func (OptionExamples) Less 

func (os OptionExamples) Less(i, j int) bool

Less is part of sort.Interface.

func (OptionExamples) Sort 

func (os OptionExamples) Sort()

Sort sorts an OptionExamples

func (OptionExamples) Swap 

func (os OptionExamples) Swap(i, j int)

Swap is part of sort.Interface.

type OptionVisibility 

type OptionVisibility byte

OptionVisibility controls whether the options are visible in the configurator or the command line. 

const (
	OptionHideCommandLine OptionVisibility = 1 << iota
	OptionHideConfigurator
	OptionHideBoth = OptionHideCommandLine | OptionHideConfigurator
)

Constants Option.Hide

type Options 

type Options []Option

Options is a slice of configuration Option for a backend

type PublicLinker 

type PublicLinker interface {
	// PublicLink generates a public link to the remote path (usually readable by anyone)
	PublicLink(remote string) (string, error)
}

PublicLinker is an optional interface for Fs

type Purger 

type Purger interface {
	// Purge all files in the root and the root directory
	//
	// Implement this if you have a way of deleting all the files
	// quicker than just running Remove() on the result of List()
	//
	// Return an error if it doesn't exist
	Purge() error
}

Purger is an optional interfaces for Fs

type PutStreamer 

type PutStreamer interface {
	// PutStream uploads to the remote path with the modTime given of indeterminate size
	//
	// May create the object even if it returns an error - if so
	// will return the object and the error, otherwise will return
	// nil and the error
	PutStream(in io.Reader, src ObjectInfo, options ...OpenOption) (Object, error)
}

PutStreamer is an optional interface for Fs

type PutUncheckeder 

type PutUncheckeder interface {
	// Put in to the remote path with the modTime given of the given size
	//
	// May create the object even if it returns an error - if so
	// will return the object and the error, otherwise will return
	// nil and the error
	//
	// May create duplicates or return errors if src already
	// exists.
	PutUnchecked(in io.Reader, src ObjectInfo, options ...OpenOption) (Object, error)
}

PutUncheckeder is an optional interface for Fs

type RangeOption 

type RangeOption struct {
	Start int64
	End   int64
}

RangeOption defines an HTTP Range option with start and end. If either start or end are < 0 then they will be omitted.

End may be bigger than the Size of the object in which case it will be capped to the size of the object.

Note that the End is inclusive, so to fetch 100 bytes you would use RangeOption{Start: 0, End: 99}

If Start is specified but End is not then it will fetch from Start to the end of the file.

If End is specified, but Start is not then it will fetch the last End bytes.

Examples: 

RangeOption{Start: 0, End: 99} - fetch the first 100 bytes
RangeOption{Start: 100, End: 199} - fetch the second 100 bytes
RangeOption{Start: 100} - fetch bytes from offset 100 to the end
RangeOption{End: 100} - fetch the last 100 bytes

A RangeOption implements a single byte-range-spec from https://tools.ietf.org/html/rfc7233#section-2.1

func ParseRangeOption 

func ParseRangeOption(s string) (po *RangeOption, err error)

ParseRangeOption parses a RangeOption from a Range: header. It only appects single ranges.

func (*RangeOption) Decode 

func (o *RangeOption) Decode(size int64) (offset, limit int64)

Decode interprets the RangeOption into an offset and a limit

The offset is the start of the stream and the limit is how many bytes should be read from it. If the limit is -1 then the stream should be read to the end.

func (*RangeOption) Header 

func (o *RangeOption) Header() (key string, value string)

Header formats the option as an http header

func (*RangeOption) Mandatory 

func (o *RangeOption) Mandatory() bool

Mandatory returns whether the option must be parsed or can be ignored

func (*RangeOption) String 

func (o *RangeOption) String() string

String formats the option into human readable form

type RangeSeeker 

type RangeSeeker interface {
	// RangeSeek behaves like a call to Seek(offset int64, whence
	// int) with the output wrapped in an io.LimitedReader
	// limiting the total length to limit.
	//
	// RangeSeek with a limit of < 0 is equivalent to a regular Seek.
	RangeSeek(offset int64, whence int, length int64) (int64, error)
}

RangeSeeker is the interface that wraps the RangeSeek method.

Some of the returns from Object.Open() may optionally implement this method for efficiency purposes.

type RegInfo 

type RegInfo struct {
	// Name of this fs
	Name string
	// Description of this fs - defaults to Name
	Description string
	// Prefix for command line flags for this fs - defaults to Name if not set
	Prefix string
	// Create a new file system.  If root refers to an existing
	// object, then it should return a Fs which which points to
	// the parent of that object and ErrorIsFile.
	NewFs func(name string, root string, config configmap.Mapper) (Fs, error) `json:"-"`
	// Function to call to help with config
	Config func(name string, config configmap.Mapper) `json:"-"`
	// Options for the Fs configuration
	Options Options
}

RegInfo provides information about a filesystem

func ConfigFs 

func ConfigFs(path string) (fsInfo *RegInfo, configName, fsPath string, config *configmap.Map, err error)

ConfigFs makes the config for calling NewFs with.

It parses the path which is of the form remote:path

Remotes are looked up in the config file. If the remote isn't found then NotFoundInConfigFile will be returned.

func Find 

func Find(name string) (*RegInfo, error)

Find looks for an RegInfo object for the name passed in. The name can be either the Name or the Prefix.

Services are looked up in the config file

func MustFind 

func MustFind(name string) *RegInfo

MustFind looks for an Info object for the type name passed in

Services are looked up in the config file

Exits with a fatal error if not found

func ParseRemote 

func ParseRemote(path string) (fsInfo *RegInfo, configName, fsPath string, err error)

ParseRemote deconstructs a path into configName, fsPath, looking up the fsName in the config file (returning NotFoundInConfigFile if not found)

func (*RegInfo) FileName added in v1.46.0 

func (ri *RegInfo) FileName() string

FileName returns the on disk file name for this backend

type SeekOption 

type SeekOption struct {
	Offset int64
}

SeekOption defines an HTTP Range option with start only.

func (*SeekOption) Header 

func (o *SeekOption) Header() (key string, value string)

Header formats the option as an http header

func (*SeekOption) Mandatory 

func (o *SeekOption) Mandatory() bool

Mandatory returns whether the option must be parsed or can be ignored

func (*SeekOption) String 

func (o *SeekOption) String() string

String formats the option into human readable form

type SetTierer added in v1.46.0 

type SetTierer interface {
	// SetTier performs changing storage tier of the Object if
	// multiple storage classes supported
	SetTier(tier string) error
}

SetTierer is an optional interface for Object

type SizeSuffix 

type SizeSuffix int64

SizeSuffix is an int64 with a friendly way of printing setting 

const (
	Byte SizeSuffix = 1 << (iota * 10)
	KibiByte
	MebiByte
	GibiByte
	TebiByte
	PebiByte
	ExbiByte
)

Common multipliers for SizeSuffix

func (*SizeSuffix) Scan 

func (x *SizeSuffix) Scan(s fmt.ScanState, ch rune) error

Scan implements the fmt.Scanner interface

func (*SizeSuffix) Set 

func (x *SizeSuffix) Set(s string) error

Set a SizeSuffix

func (SizeSuffix) String 

func (x SizeSuffix) String() string

String turns SizeSuffix into a string

func (*SizeSuffix) Type 

func (x *SizeSuffix) Type() string

Type of the value

func (SizeSuffix) Unit 

func (x SizeSuffix) Unit(unit string) string

Unit turns SizeSuffix into a string with a unit

type SizeSuffixList added in v1.46.0 

type SizeSuffixList []SizeSuffix

SizeSuffixList is a sclice SizeSuffix values

func (SizeSuffixList) Len added in v1.46.0 

func (l SizeSuffixList) Len() int

func (SizeSuffixList) Less added in v1.46.0 

func (l SizeSuffixList) Less(i, j int) bool

func (SizeSuffixList) Sort added in v1.46.0 

func (l SizeSuffixList) Sort()

Sort sorts the list

func (SizeSuffixList) Swap added in v1.46.0 

func (l SizeSuffixList) Swap(i, j int)

type SpaceSepList 

type SpaceSepList []string

SpaceSepList is a space separated config value It uses the encoding/csv rules for quoting and escaping

Example 
for _, s := range []string{
	`remotea:test/dir remoteb:`,
	`"remotea:test/space dir" remoteb:`,
	`"remotea:test/quote""dir" remoteb:`,
} {
	var l SpaceSepList
	must(l.Set(s))
	fmt.Printf("%#v\n", l)
}

Output:

fs.SpaceSepList{"remotea:test/dir", "remoteb:"}
fs.SpaceSepList{"remotea:test/space dir", "remoteb:"}
fs.SpaceSepList{"remotea:test/quote\"dir", "remoteb:"}

func (*SpaceSepList) Scan 

func (l *SpaceSepList) Scan(s fmt.ScanState, ch rune) error

Scan implements the fmt.Scanner interface

func (*SpaceSepList) Set 

func (l *SpaceSepList) Set(s string) error

Set the List entries

func (SpaceSepList) String 

func (l SpaceSepList) String() string

func (SpaceSepList) Type 

func (SpaceSepList) Type() string

Type of the value

type UnWrapper 

type UnWrapper interface {
	// UnWrap returns the Fs that this Fs is wrapping
	UnWrap() Fs
}

UnWrapper is an optional interfaces for Fs

type Usage 

type Usage struct {
	Total   *int64 `json:"total,omitempty"`   // quota of bytes that can be used
	Used    *int64 `json:"used,omitempty"`    // bytes in use
	Trashed *int64 `json:"trashed,omitempty"` // bytes in trash
	Other   *int64 `json:"other,omitempty"`   // other usage eg gmail in drive
	Free    *int64 `json:"free,omitempty"`    // bytes which can be uploaded before reaching the quota
	Objects *int64 `json:"objects,omitempty"` // objects in the storage system
}

Usage is returned by the About call

If a value is nil then it isn't supported by that backend

type Wrapper 

type Wrapper interface {
	// Wrap returns the Fs that is wrapping this Fs
	WrapFs() Fs
	// SetWrapper sets the Fs that is wrapping this Fs
	SetWrapper(f Fs)
}

Wrapper is an optional interfaces for Fs

Source Files

View all Source files

Directories

Path Synopsis
Package accounting providers an accounting and limiting reader
 Package accounting providers an accounting and limiting reader
Package asyncreader provides an asynchronous reader which reads independently of write
 Package asyncreader provides an asynchronous reader which reads independently of write
Package config reads, writes and edits the config file and deals with command line flags
 Package config reads, writes and edits the config file and deals with command line flags
configflags
Package configflags defines the flags used by rclone.
 Package configflags defines the flags used by rclone.
configmap
Package configmap provides an abstraction for reading and writing config
 Package configmap provides an abstraction for reading and writing config
configstruct
Package configstruct parses unstructured maps into structures
 Package configstruct parses unstructured maps into structures
flags
Package flags contains enahnced versions of spf13/pflag flag routines which will read from the environment also.
 Package flags contains enahnced versions of spf13/pflag flag routines which will read from the environment also.
obscure
Package obscure contains the Obscure and Reveal commands
 Package obscure contains the Obscure and Reveal commands
Package filter controls the filtering of files
 Package filter controls the filtering of files
filterflags
Package filterflags implements command line flags to set up a filter
 Package filterflags implements command line flags to set up a filter
Package fserrors provides errors and error handling
 Package fserrors provides errors and error handling
Package fshttp contains the common http parts of the config, Transport and Client
 Package fshttp contains the common http parts of the config, Transport and Client
Package fspath contains routines for fspath manipulation
 Package fspath contains routines for fspath manipulation
Package list contains list functions
 Package list contains list functions
Package log provides logging for rclone
 Package log provides logging for rclone
Package march traverses two directories in lock step
 Package march traverses two directories in lock step
Package object defines some useful Objects
 Package object defines some useful Objects
Package operations does generic operations on filesystems and objects
 Package operations does generic operations on filesystems and objects
rc
Package rc implements a remote control server and registry for rclone To register your internal calls, call rc.Add(path, function).
 Package rc implements a remote control server and registry for rclone To register your internal calls, call rc.Add(path, function).
rcflags
Package rcflags implements command line flags to set up the remote control
 Package rcflags implements command line flags to set up the remote control
rcserver
Package rcserver implements the HTTP endpoint to serve the remote control
 Package rcserver implements the HTTP endpoint to serve the remote control
Package sync is the implementation of sync/copy/move
 Package sync is the implementation of sync/copy/move
Package walk walks directories
 Package walk walks directories

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.