b2

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 20, 2023 License: MIT Imports: 20 Imported by: 0

README

b2

Efficient, idiomatic Go library for Backblaze B2 Cloud Storage.

TODO:

  • Start large file upload: b2_start_large_file.
  • b2_get_upload_part_url
  • Upload to large file part: b2_upload_part.
  • Finish large file: b2_finish_large_file.
  • Download range of file.
  • List files with prefix.

Plan to support large file upload

Large File takes existing sha1 look for existing large file if exists look for existing parts if not exists start large file status and list of parts remaining

Large Status upload each part, if the upload URL doesn't exist or if it is invalid, get a new upload URL. when done, call complete

type LocalLargeFile interface { SHA1() string Size() int64 RangeReader(r Range) (io.Reader, error) }

type CreateLargeFileOptions struct { SHA1 string ModUnixMilli int64 ContentType string }

type LargeFile struct { Local io.ReadSizeSeekCloser FileID string SHA1 string Size int64 }

type LargePart struct { Index int Range Range SHA1 string }

func UploadLargeFile(ctx, remotePath, LocalLargeFile) -> (LargeStatus, error) func NewPartUploader() -> *PartUploader func (LargeStatus) UploadPart(ctx, part, PartUploader) -> error func (LargeStatus) Finish(ctx) -> error

Documentation

Overview

Package b2 provides an idiomatic, efficient interface to Backblaze B2 Cloud Storage.

Uploading

Uploads to B2 require a SHA1 header, so the hash of the file must be known before the upload starts. The (*Bucket).Upload API tries its best not to buffer the entire file in memory, and it will avoid it if passed either a bytes.Buffer or a io.ReadSeeker.

If you know the SHA1 and the length of the file in advance, you can use (*Bucket).UploadWithSHA1 but you are responsible for retrying on transient errors.

Downloading

Downloads from B2 are simple GETs, so if you want more control than the standard functions you can build your own URL according to the API docs. All the information you need is returned by Client.LoginInfo().

Hidden files and versions

There is no first-class support for versions in this library, but most behaviors are transparently exposed. Upload can be used multiple times with the same name, ListFiles will only return the latest version of non-hidden files, and ListFilesVersions will return all files and versions.

Unsupported APIs

Large files (b2_*_large_file, b2_*_part), b2_get_download_authorization, b2_hide_file, b2_update_bucket.

Debug mode

If the B2_DEBUG environment variable is set to 1, all API calls will be logged and also log when new (non-reused) connections are established.

Index

Constants

This section is empty.

Variables

View Source 
var ErrFileNotFound = errors.New("no file with the given name in the bucket")

Functions

This section is empty.

Types

type Bucket 

type Bucket struct {
	ID string
	// contains filtered or unexported fields
}

A Bucket is bound to the Client that created it. It is safe for concurrent use and should be reused to take advantage of connection and URL reuse.

func (*Bucket) Delete 

func (b *Bucket) Delete(ctx context.Context) error

Delete calls b2_delete_bucket. After this call succeeds the Bucket object becomes invalid and any other calls will fail.

func (*Bucket) GetFileInfoByName 

func (b *Bucket) GetFileInfoByName(ctx context.Context, name string) (*FileInfo, error)

GetFileInfoByName obtains a FileInfo for a given name.

If the file doesn't exist, FileNotFoundError is returned. If multiple versions of the file exist, only the latest is returned.

func (*Bucket) ListFileVersions added in v0.3.0 

func (b *Bucket) ListFileVersions(ctx context.Context, o ListOptions) *Listing

ListFilesVersions is like ListFiles, but returns all file versions, alphabetically sorted first, and by reverse of date/time uploaded then.

If fromID is specified, the name-and-id pair is the starting point.

func (*Bucket) ListFiles 

func (b *Bucket) ListFiles(ctx context.Context, o ListOptions) *Listing

ListFiles returns a Listing of files in the Bucket, alphabetically sorted, starting from the file named fromName (included if it exists). To start from the first file in the bucket, set fileName to "".

ListFiles only returns the most recent version of each (non-hidden) file. If you want to fetch all versions, use ListFilesVersions.

func (*Bucket) Upload 

func (b *Bucket) Upload(ctx context.Context, r io.Reader, name, mimeType string, metadata map[string]string) (*FileInfo, error)

Upload uploads a file to a B2 bucket. If mimeType is "", "b2/x-auto" will be used.

Concurrent calls to Upload will use separate upload URLs, but consequent ones will attempt to reuse previously obtained ones to save b2_get_upload_url calls. Upload URL failures are handled transparently.

Since the B2 API requires a SHA1 header, normally the file will first be read entirely into a memory buffer. Two cases avoid the memory copy: if r is a bytes.Buffer, the SHA1 will be computed in place; otherwise, if r implements io.Seeker (like *os.File and *bytes.Reader), the file will be read twice, once to compute the SHA1 and once to upload.

If a file by this name already exist, a new version will be created.

func (*Bucket) UploadWithSHA1 

func (b *Bucket) UploadWithSHA1(ctx context.Context, r io.Reader, name, mimeType, sha1Sum string, length int64, metadata map[string]string) (*FileInfo, error)

UploadWithSHA1 is like Upload, but allows the caller to specify previously known SHA1 and length of the file. It never does any buffering, nor does it retry on failure.

Note that retrying on most upload failures, not just error handling, is mandatory by the B2 API documentation. If the error Status is Unauthorized, a call to (*Client).LoginInfo(true) should be performed first.

sha1Sum should be the hex encoding of the SHA1 sum of what will be read from r.

This is an advanced interface, most clients should use Upload, and consider passing it a bytes.Buffer or io.ReadSeeker to avoid buffering.

type BucketInfo 

type BucketInfo struct {
	Bucket

	Name string
	Type string
}

BucketInfo is an extended Bucket object with metadata.

type Client 

type Client struct {
	// contains filtered or unexported fields
}

A Client is an authenticated API client. It is safe for concurrent use and should be reused to take advantage of connection and URL reuse.

The Client handles refreshing authorization tokens transparently.

func NewClient 

func NewClient(ctx context.Context, accountID, applicationKey string, httpClient *http.Client) (*Client, error)

NewClient calls b2_authorize_account and returns an authenticated Client. httpClient can be nil, in which case http.DefaultClient will be used. ctx is used for initial login and is not stored.

func (*Client) BucketByID 

func (c *Client) BucketByID(id string) *Bucket

BucketByID returns a Bucket bound to the Client. It does NOT check that the bucket actually exists, or perform any network operation.

func (*Client) BucketByName 

func (c *Client) BucketByName(ctx context.Context, name string, createIfNotExists bool) (*BucketInfo, error)

BucketByName returns the Bucket with the given name. If such a bucket is not found and createIfNotExists is true, CreateBucket is called with allPublic set to false. Otherwise, an error is returned.

func (*Client) Buckets 

func (c *Client) Buckets(ctx context.Context, name string) ([]*BucketInfo, error)

Buckets returns a list of buckets sorted by name.

func (*Client) CreateBucket 

func (c *Client) CreateBucket(ctx context.Context, name string, allPublic bool) (*BucketInfo, error)

CreateBucket creates a bucket with b2_create_bucket. If allPublic is true, files in this bucket can be downloaded by anybody.

func (*Client) DeleteFile 

func (c *Client) DeleteFile(ctx context.Context, id, name string) error

DeleteFile deletes a file version.

func (*Client) DownloadFile added in v0.3.0 

func (c *Client) DownloadFile(ctx context.Context, o DownloadOptions) (io.ReadCloser, *FileInfo, error)

DownloadFile gets file contents. The ReadCloser must be closed by the caller once done reading.

Note: the (*FileInfo).CustomMetadata values returned by this function are all represented as strings, because they are delivered by HTTP headers.

func (*Client) DownloadFileByID 

func (c *Client) DownloadFileByID(ctx context.Context, id string) (io.ReadCloser, *FileInfo, error)

DownloadFileByID gets file contents by file ID. The ReadCloser must be closed by the caller once done reading.

Note: the (*FileInfo).CustomMetadata values returned by this function are all represented as strings, because they are delivered by HTTP headers.

func (*Client) DownloadFileByName 

func (c *Client) DownloadFileByName(ctx context.Context, bucket, file string) (io.ReadCloser, *FileInfo, error)

DownloadFileByName gets file contents by file and bucket name. The ReadCloser must be closed by the caller once done reading.

Note: the (*FileInfo).CustomMetadata values returned by this function are all represented as strings, because they are delivered by HTTP headers.

func (*Client) GetFileInfoByID 

func (c *Client) GetFileInfoByID(ctx context.Context, id string) (*FileInfo, error)

GetFileInfoByID obtains a FileInfo for a given ID.

The ID can refer to any file version or "hide" action in any bucket.

func (*Client) LoginInfo 

func (c *Client) LoginInfo(ctx context.Context, refresh bool) (*LoginInfo, error)

LoginInfo returns the LoginInfo object currently in use. If refresh is true, it obtains a new one before returning.

Note that once you obtain this there is no guarantee on its freshness, and it will eventually expire.

type DownloadOptions added in v0.3.0 

type DownloadOptions struct {
	// Download file by ID.
	FileID string

	// Download file by bucket and filename.
	Bucket   string
	FileName string

	// Zero based indicies.
	Range Range
}

type Error 

type Error struct {
	Code    string
	Message string
	Status  int
}

Error is the decoded B2 JSON error return value. It's not the only type of error returned by this package, and it is mostly returned wrapped in a url.Error. Use UnwrapError to access it.

func UnwrapError 

func UnwrapError(err error) (b2Err *Error, ok bool)

UnwrapError attempts to extract the Error that caused err. If there is no Error object to unwrap, ok is false and err is nil. That does not mean that the original error should be ignored.

func (Error) Error 

func (e Error) Error() string

type FileAction added in v0.3.0 

type FileAction string
const (
	FileStart  FileAction = "start"  // A large file has been started, but not finished or canceled.
	FileUpload FileAction = "upload" // A file was uploaded.
	FileHide   FileAction = "hide"   // A file version marking the file as hidden.
	FileFolder FileAction = "folder" // A virtual folder when listing files.
)

type FileInfo 

type FileInfo struct {
	ID   string
	Name string

	ContentSHA1   string // hex encoded
	ContentLength int64
	ContentType   string

	CustomMetadata  map[string]string
	UploadTimestamp time.Time

	// If Action is "hide", this ID does not refer to a file version
	// but to an hiding action. Otherwise "upload".
	Action FileAction
}

A FileInfo is the metadata associated with a specific file version.

type ListOptions added in v0.3.0 

type ListOptions struct {
	FromName  string
	FromID    string // Only used for List File Versions, must set FromName.
	Prefix    string
	Delimiter string
}

type Listing 

type Listing struct {
	// contains filtered or unexported fields
}

A Listing is the result of (*Bucket).ListFiles[Versions]. It works like sql.Rows: use Next to advance and then FileInfo. Check Err once Next returns false. 

l := b.ListFiles("")
for l.Next() {
    fi := l.FileInfo()
    ...
}
if err := l.Err(); err != nil {
    ...
}

A Listing handles pagination transparently, so it iterates until the last file in the bucket. To limit the number of results, do this. 

for i := 0; i < limit && l.Next(); i++ {

func (*Listing) Err 

func (l *Listing) Err() error

Err returns the error, if any, that was encountered while listing.

func (*Listing) FileInfo 

func (l *Listing) FileInfo() *FileInfo

FileInfo returns the FileInfo object made available by Next.

FileInfo must only be called after a call to Next returned true.

func (*Listing) Next 

func (l *Listing) Next() bool

Next calls the list API if needed and prepares the FileInfo results. It returns true on success, or false if there is no next result or an error happened while preparing it. Err should be consulted to distinguish between the two cases.

func (*Listing) SetPageCount 

func (l *Listing) SetPageCount(n int)

SetPageCount controls the number of results to be fetched with each API call. The maximum n is 1000, higher values are automatically limited to 1000.

SetPageCount does not limit the number of results returned by a Listing.

type LoginInfo 

type LoginInfo struct {
	AccountID string
	ApiURL    string

	// DownloadURL is the base URL for file downloads. It is supposed
	// to never change for the same account.
	DownloadURL string
	// AuthorizationToken is the value to pass in the Authorization
	// header of all private calls. This is valid for at most 24 hours.
	AuthorizationToken string
}

LoginInfo holds the information obtained upon login, which are sufficient to interact with the API directly.

type Range added in v0.3.0 

type Range struct {
	Begin int64
	End   int64
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.