object

package
v1.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 4, 2024 License: Apache-2.0 Imports: 118 Imported by: 2

README

The following object store are supported:

  • file: local files
  • sftp: FTP via SSH
  • s3: Amazon S3
  • hdfs: Hadoop File System (HDFS)
  • gcs: Google Cloud Storage
  • wasb: Windows Azure Blob Storage
  • oss: Aliyun OSS
  • cos: Tencent Cloud COS
  • ks3: KSYun KS3
  • ufile: UCloud UFile
  • qingstor: Qingcloud QingStor
  • bos: Baidu Cloud Object Storage
  • jss: JCloud Object Storage
  • qiniu: Qiniu
  • b2: Backblaze B2
  • space: Digital Ocean Space
  • obs: Huawei Object Storage Service
  • oos: CTYun OOS
  • scw: Scaleway Object Storage
  • minio: MinIO
  • scs: Sina Cloud Storage
  • eos: ECloud (China Mobile Cloud) Object Storage

they should be specified in the following format:

[NAME://][ACCESS_KEY:SECRET_KEY@]BUCKET[.ENDPOINT][/PREFIX]

Some examples:

  • local/path
  • user@host:port:path
  • file:///Users/me/code/
  • hdfs://hdfs@namenode1:9000,namenode2:9000/user/
  • s3://my-bucket/
  • s3://access-key:secret-key-id@my-bucket/prefix
  • wasb://account-name:account-key@my-container/prefix
  • gcs://my-bucket.us-west1.googleapi.com/
  • oss://test
  • cos://test-1234
  • obs://my-bucket
  • bos://my-bucket
  • minio://myip:9000/bucket
  • scs://access-key:secret-key-id@my-bucket.sinacloud.net/prefix

Note:

  • It's recommended to run it in the target region to have better performance.
  • Auto discover endpoint for bucket of S3, OSS, COS, OBS, BOS, SRC and DST can use format NAME://[ACCESS_KEY:SECRET_KEY@]BUCKET[/PREFIX] . ACCESS_KEY and SECRET_KEY can be provided by corresponding environment variables (see below).
  • S3:
    • The access key and secret key for S3 could be provided by AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, or IAM role.
  • Wasb(Windows Azure Storage Blob)
    • The account name and account key can be provided as connection string by AZURE_STORAGE_CONNECTION_STRING.
  • GCS: The machine should be authorized to access Google Cloud Storage.
  • OSS:
    • The credential can be provided by environment variable ALICLOUD_ACCESS_KEY_ID and ALICLOUD_ACCESS_KEY_SECRET , RAM role, EMR MetaService.
  • COS:
    • The AppID should be part of the bucket name.
    • The credential can be provided by environment variable COS_SECRETID and COS_SECRETKEY.
  • OBS:
    • The credential can be provided by environment variable HWCLOUD_ACCESS_KEY and HWCLOUD_SECRET_KEY .
  • BOS:
    • The credential can be provided by environment variable BDCLOUD_ACCESS_KEY and BDCLOUD_SECRET_KEY .
  • Qiniu: The S3 endpoint should be used for Qiniu, for example, abc.cn-north-1-s3.qiniu.com. If there are keys starting with "/", the domain should be provided as QINIU_DOMAIN.
  • sftp: if your target machine uses SSH certificates instead of password, you should pass the path to your private key file to the environment variable SSH_PRIVATE_KEY_PATH, like SSH_PRIVATE_KEY_PATH=/home/someuser/.ssh/id_rsa juicefs sync [src] [dst].
  • Scaleway:
    • The credential can be provided by environment variable SCW_ACCESS_KEY and SCW_SECRET_KEY .
  • MinIO:
    • The credential can be provided by environment variable MINIO_ACCESS_KEY and MINIO_SECRET_KEY .

Documentation

Index

Constants

View Source 
const (
	AES256GCM_RSA = "aes256gcm-rsa"
	CHACHA20_RSA  = "chacha20-rsa"
)

Variables

View Source 
var HEADER_NAMES = []string{"Content-MD5", "Content-Type", "Date"}
View Source 
var OVHCompileRegexp = `^s3\.(\w*)(\.\w*)?\.cloud\.ovh\.net$`
View Source 
var ReqIDCache reqIDCache
View Source 
var TryCFR bool // try copy_file_range
View Source 
var UserAgent = "JuiceFS"

Functions

func ExportRsaPrivateKeyToPem added in v0.11.0 

func ExportRsaPrivateKeyToPem(key *rsa.PrivateKey, passphrase string) string

func GetHttpClient added in v1.0.0 

func GetHttpClient() *http.Client

func IsErrReplicating added in v1.0.0 

func IsErrReplicating(err error) bool

func ListAll added in v0.13.0 

func ListAll(store ObjectStorage, prefix, marker string, followLink bool) (<-chan Object, error)

ListAll on all the keys that starts at marker from object storage.

func ListAllWithDelimiter added in v1.1.0 

func ListAllWithDelimiter(store ObjectStorage, prefix, start, end string, followLink bool) (<-chan Object, error)

func MarshalObject added in v0.12.0 

func MarshalObject(o Object) map[string]interface{}

func ParseRsaPrivateKeyFromPath added in v0.11.0 

func ParseRsaPrivateKeyFromPath(path, passphrase string) (*rsa.PrivateKey, error)

func ParseRsaPrivateKeyFromPem added in v0.11.0 

func ParseRsaPrivateKeyFromPem(enc []byte, passphrase []byte) (*rsa.PrivateKey, error)

func Register added in v0.10.0 

func Register(name string, register Creator)

Types

type COS added in v0.10.0 

type COS struct {
	// contains filtered or unexported fields
}

func (*COS) AbortUpload added in v0.10.0 

func (c *COS) AbortUpload(key string, uploadID string)

func (*COS) CompleteUpload added in v0.10.0 

func (c *COS) CompleteUpload(key string, uploadID string, parts []*Part) error

func (*COS) Copy added in v0.10.0 

func (c *COS) Copy(dst, src string) error

func (*COS) Create added in v0.10.0 

func (c *COS) Create() error

func (*COS) CreateMultipartUpload added in v0.10.0 

func (c *COS) CreateMultipartUpload(key string) (*MultipartUpload, error)

func (*COS) Delete added in v0.10.0 

func (c *COS) Delete(key string) error

func (*COS) Get added in v0.10.0 

func (c *COS) Get(key string, off, limit int64) (io.ReadCloser, error)

func (*COS) Head added in v0.10.0 

func (c *COS) Head(key string) (Object, error)

func (*COS) Limits added in v1.1.0 

func (c *COS) Limits() Limits

func (*COS) List added in v0.10.0 

func (c *COS) List(prefix, marker, delimiter string, limit int64, followLink bool) ([]Object, error)

func (*COS) ListAll added in v0.10.0 

func (c *COS) ListAll(prefix, marker string, followLink bool) (<-chan Object, error)

func (*COS) ListUploads added in v0.10.0 

func (c *COS) ListUploads(marker string) ([]*PendingPart, string, error)

func (*COS) Put added in v0.10.0 

func (c *COS) Put(key string, in io.Reader) error

func (*COS) SetStorageClass added in v1.1.0 

func (c *COS) SetStorageClass(sc string)

func (*COS) String added in v0.10.0 

func (c *COS) String() string

func (*COS) UploadPart added in v0.10.0 

func (c *COS) UploadPart(key string, uploadID string, num int, body []byte) (*Part, error)

func (*COS) UploadPartCopy added in v1.1.0 

func (c *COS) UploadPartCopy(key string, uploadID string, num int, srcKey string, off, size int64) (*Part, error)

type Contents added in v0.10.0 

type Contents struct {
	Key          string
	Size         int64
	LastModified time.Time
}

type Creator 

type Creator func(bucket, accessKey, secretKey, token string) (ObjectStorage, error)

type DataItem added in v0.10.0 

type DataItem struct {
	FileName   string
	Size       int64
	ModifyTime int
}

type DefaultObjectStorage added in v0.10.0 

type DefaultObjectStorage struct{}

func (DefaultObjectStorage) AbortUpload added in v0.10.0 

func (s DefaultObjectStorage) AbortUpload(key string, uploadID string)

func (DefaultObjectStorage) CompleteUpload added in v0.10.0 

func (s DefaultObjectStorage) CompleteUpload(key string, uploadID string, parts []*Part) error

func (DefaultObjectStorage) Copy added in v1.1.0 

func (s DefaultObjectStorage) Copy(dst, src string) error

func (DefaultObjectStorage) Create added in v0.10.0 

func (s DefaultObjectStorage) Create() error

func (DefaultObjectStorage) CreateMultipartUpload added in v0.10.0 

func (s DefaultObjectStorage) CreateMultipartUpload(key string) (*MultipartUpload, error)

func (DefaultObjectStorage) Head added in v0.10.0 

func (s DefaultObjectStorage) Head(key string) (Object, error)

func (DefaultObjectStorage) Limits added in v1.1.0 

func (s DefaultObjectStorage) Limits() Limits

func (DefaultObjectStorage) List added in v0.10.0 

func (s DefaultObjectStorage) List(prefix, marker, delimiter string, limit int64, followLink bool) ([]Object, error)

func (DefaultObjectStorage) ListAll added in v0.10.0 

func (s DefaultObjectStorage) ListAll(prefix, marker string, followLink bool) (<-chan Object, error)

func (DefaultObjectStorage) ListUploads added in v0.10.0 

func (s DefaultObjectStorage) ListUploads(marker string) ([]*PendingPart, string, error)

func (DefaultObjectStorage) UploadPart added in v0.10.0 

func (s DefaultObjectStorage) UploadPart(key string, uploadID string, num int, body []byte) (*Part, error)

func (DefaultObjectStorage) UploadPartCopy added in v1.1.0 

func (s DefaultObjectStorage) UploadPartCopy(key string, uploadID string, num int, srcKey string, off, size int64) (*Part, error)

type Encryptor added in v0.11.0 

type Encryptor interface {
	Encrypt(plaintext []byte) ([]byte, error)
	Decrypt(ciphertext []byte) ([]byte, error)
}

func NewDataEncryptor added in v1.1.0 

func NewDataEncryptor(keyEncryptor Encryptor, algo string) (Encryptor, error)

func NewRSAEncryptor added in v0.11.0 

func NewRSAEncryptor(privKey *rsa.PrivateKey) Encryptor

type File added in v0.10.0 

type File interface {
	Object
	Owner() string
	Group() string
	Mode() os.FileMode
}

type FileSystem added in v0.10.0 

type FileSystem interface {
	MtimeChanger
	Chmod(path string, mode os.FileMode) error
	Chown(path string, owner, group string) error
}

type Limits added in v1.1.0 

type Limits struct {
	IsSupportMultipartUpload bool
	IsSupportUploadPartCopy  bool
	MinPartSize              int
	MaxPartSize              int
	MaxPartCount             int
}

type ListBucketResult added in v0.10.0 

type ListBucketResult struct {
	Contents       []*Contents
	IsTruncated    bool
	Prefix         string
	Marker         string
	MaxKeys        string
	NextMarker     string
	CommonPrefixes []string
}

ListObjectsOutput presents output for ListObjects.

type MtimeChanger added in v0.10.0 

type MtimeChanger interface {
	Chtimes(path string, mtime time.Time) error
}

type MultipartUpload added in v0.10.0 

type MultipartUpload struct {
	MinPartSize int
	MaxCount    int
	UploadID    string
}

type Object added in v0.10.0 

type Object interface {
	Key() string
	Size() int64
	Mtime() time.Time
	IsDir() bool
	IsSymlink() bool
	StorageClass() string
}

func UnmarshalObject added in v0.12.0 

func UnmarshalObject(m map[string]interface{}) Object

type ObjectStorage 

type ObjectStorage interface {
	// Description of the object storage.
	String() string
	// Limits of the object storage.
	Limits() Limits
	// Create the bucket if not existed.
	Create() error
	// Get the data for the given object specified by key.
	Get(key string, off, limit int64) (io.ReadCloser, error)
	// Put data read from a reader to an object specified by key.
	Put(key string, in io.Reader) error
	// Copy an object from src to dst.
	Copy(dst, src string) error
	// Delete a object.
	Delete(key string) error

	// Head returns some information about the object or an error if not found.
	Head(key string) (Object, error)
	// List returns a list of objects.
	List(prefix, marker, delimiter string, limit int64, followLink bool) ([]Object, error)
	// ListAll returns all the objects as an channel.
	ListAll(prefix, marker string, followLink bool) (<-chan Object, error)

	// CreateMultipartUpload starts to upload a large object part by part.
	CreateMultipartUpload(key string) (*MultipartUpload, error)
	// UploadPart upload a part of an object.
	UploadPart(key string, uploadID string, num int, body []byte) (*Part, error)
	// UploadPartCopy Uploads a part by copying data from an existing object as data source.
	UploadPartCopy(key string, uploadID string, num int, srcKey string, off, size int64) (*Part, error)
	// AbortUpload abort a multipart upload.
	AbortUpload(key string, uploadID string)
	// CompleteUpload finish an multipart upload.
	CompleteUpload(key string, uploadID string, parts []*Part) error
	// ListUploads lists existing multipart uploads.
	ListUploads(marker string) ([]*PendingPart, string, error)
}

ObjectStorage is the interface for object storage. all of these API should be idempotent.

func CreateStorage 

func CreateStorage(name, endpoint, accessKey, secretKey, token string) (ObjectStorage, error)

func NewEncrypted added in v0.11.0 

func NewEncrypted(o ObjectStorage, enc Encryptor) ObjectStorage

NewEncrypted returns a encrypted object storage

func NewSharded added in v0.13.0 

func NewSharded(name, endpoint, ak, sk, token string, shards int) (ObjectStorage, error)

func WithPrefix 

func WithPrefix(os ObjectStorage, prefix string) ObjectStorage

WithPrefix return an object storage that add a prefix to keys.

type Part added in v0.10.0 

type Part struct {
	Num  int
	Size int
	ETag string
}

type PendingPart added in v0.10.0 

type PendingPart struct {
	Key      string
	UploadID string
	Created  time.Time
}

type RestfulStorage added in v0.10.0 

type RestfulStorage struct {
	DefaultObjectStorage
	// contains filtered or unexported fields
}

func (*RestfulStorage) Copy added in v0.10.0 

func (s *RestfulStorage) Copy(dst, src string) error

func (*RestfulStorage) Delete added in v0.10.0 

func (s *RestfulStorage) Delete(key string) error

func (*RestfulStorage) Get added in v0.10.0 

func (s *RestfulStorage) Get(key string, off, limit int64) (io.ReadCloser, error)

func (*RestfulStorage) Head added in v0.10.0 

func (s *RestfulStorage) Head(key string) (Object, error)

func (*RestfulStorage) List added in v0.10.0 

func (s *RestfulStorage) List(prefix, marker, delimiter string, limit int64, followLink bool) ([]Object, error)

func (*RestfulStorage) Put added in v0.10.0 

func (u *RestfulStorage) Put(key string, body io.Reader) error

func (*RestfulStorage) String added in v0.10.0 

func (s *RestfulStorage) String() string

type SectionReaderCloser added in v1.0.4 

type SectionReaderCloser struct {
	*io.SectionReader
	io.Closer
}

type SupportStorageClass added in v1.1.0 

type SupportStorageClass interface {
	SetStorageClass(sc string)
}
type SupportSymlink interface {
	// Symlink create a symbolic link
	Symlink(oldName, newName string) error
	// Readlink read a symbolic link
	Readlink(name string) (string, error)
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.