uplink

package module
v1.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 16, 2020 License: MIT Imports: 32 Imported by: 82

README

Go library for Storj V3 Network.

Go Report Card Go Doc Beta

Storj is building a decentralized cloud storage network. Check out our white paper for more info!

Storj is an S3-compatible platform and suite of decentralized applications that allows you to store data in a secure and decentralized manner. Your files are encrypted, broken into little pieces and stored in a global decentralized network of computers. Luckily, we also support allowing you (and only you) to retrieve those files!

A Note about Versioning

Our versioning in this repo is intended to primarily support the expectations of the Go modules system, so you can expect that within a major version release, backwards-incompatible changes will be avoided at high cost. This is not the case with our downstream repo, storj/storj.

Documentation

Language bindings

License

This library is distributed under the MIT license (also known as the Expat license).

Support

If you have any questions or suggestions please reach out to us on our community forum or file a support ticket at https://support.tardigrade.io.

Documentation

Overview

Package uplink is the main entrypoint to interacting with Storj Labs' decentralized storage network.

Sign up for an account on a Satellite today! https://tardigrade.io/satellites/

Access Grants

The fundamental unit of access in the Storj Labs storage network is the Access Grant. An access grant is a serialized structure that is internally comprised of an API Key, a set of encryption key information, and information about which Storj Labs or Tardigrade network Satellite is responsible for the metadata. An access grant is always associated with exactly one Project on one Satellite.

If you don't already have an access grant, you will need make an account on a Satellite, generate an API Key, and encapsulate that API Key with encryption information into an access grant.

If you don't already have an account on a Satellite, first make one at https://tardigrade.io/satellites/ and note the Satellite you choose (such as us-central-1.tardigrade.io, europe-west-1.tardigrade.io, etc). Then, make an API Key in the web interface.

The first step to any project is to generate a restricted access grant with the minimal permissions that are needed. Access grants contains all encryption information and they should be restricted as much as possible.

To make an access grant, you can create one using our Uplink CLI tool's 'share' subcommand (after setting up the Uplink CLI tool), or you can make one as follows: 

access, err := uplink.RequestAccessWithPassphrase(ctx, satelliteAddress, apiKey, rootPassphrase)
if err != nil {
    return err
}

// create an access grant for reading bucket "logs"
permissions := uplink.ReadOnlyPermission()
shared := uplink.SharePrefix{Bucket: "logs"}
restrictedAccess, err := access.Share(permissions, shared)
if err != nil {
    return err
}

// serialize the restricted access
serializedAccess, err := restrictedAccess.Serialize()
if err != nil {
    return err
}

In the above example, 'serializedAccess' is a human-readable string that represents read-only access to just the "logs" bucket, and is only able to decrypt that one bucket thanks to hierarchical deterministic key derivation.

Note: RequestAccessWithPassphrase is CPU-intensive, and your application's normal lifecycle should avoid it and use ParseAccess where possible instead.

Projects

Once you have a valid access grant, you can open a Project with the access that access grant allows for. 

project, err := uplink.OpenProject(ctx, access)
if err != nil {
    return err
}
defer project.Close()

Projects allow you to manage buckets and objects within buckets.

Buckets

A bucket represents a collection of objects. You can upload, download, list, and delete objects of any size or shape. Objects within buckets are represented by keys, where keys can optionally be listed using the "/" delimiter.

Note: Objects and object keys within buckets are end-to-end encrypted, but bucket names themselves are not encrypted, so the billing interface on the Satellite can show you bucket line items. 

buckets := project.ListBuckets(ctx, nil)
for buckets.Next() {
    fmt.Println(buckets.Item().Name)
}
if err := buckets.Err(); err != nil {
    return err
}

Download Object

Objects support a couple kilobytes of arbitrary key/value metadata, and arbitrary-size primary data streams with the ability to read at arbitrary offsets. 

object, err := project.DownloadObject(ctx, "logs", "2020-04-18/webserver.log", nil)
if err != nil {
    return err
}
defer object.Close()

_, err = io.Copy(w, object)
return err

If you want to access only a small subrange of the data you uploaded, you can use `uplink.DownloadOptions` to specify the download range. 

object, err := project.DownloadObject(ctx, "logs", "2020-04-18/webserver.log",
    &uplink.DownloadOptions{Offset: 10, Length: 100})
if err != nil {
    return err
}
defer object.Close()

_, err = io.Copy(w, object)
return err

List Objects

Listing objects returns an iterator that allows to walk through all the items: 

objects := project.ListObjects(ctx, "logs", nil)
for objects.Next() {
    item := objects.Item()
    fmt.Println(item.IsPrefix, item.Key)
}
if err := objects.Err(); err != nil {
    return err
}

Index

Constants

This section is empty.

Variables

View Source 
var ErrBandwidthLimitExceeded = errors.New("bandwidth limit exceeded")

ErrBandwidthLimitExceeded is returned when project will exceeded bandwidth limit.

View Source 
var ErrBucketAlreadyExists = errors.New("bucket already exists")

ErrBucketAlreadyExists is returned when the bucket already exists during creation.

View Source 
var ErrBucketNameInvalid = errors.New("bucket name invalid")

ErrBucketNameInvalid is returned when the bucket name is invalid.

View Source 
var ErrBucketNotEmpty = errors.New("bucket not empty")

ErrBucketNotEmpty is returned when the bucket is not empty during deletion.

View Source 
var ErrBucketNotFound = errors.New("bucket not found")

ErrBucketNotFound is returned when the bucket is not found.

View Source 
var ErrObjectKeyInvalid = errors.New("object key invalid")

ErrObjectKeyInvalid is returned when the object key is invalid.

View Source 
var ErrObjectNotFound = errors.New("object not found")

ErrObjectNotFound is returned when the object is not found.

View Source 
var ErrTooManyRequests = errors.New("too many requests")

ErrTooManyRequests is returned when user has sent too many requests in a given amount of time.

View Source 
var ErrUploadDone = errors.New("upload done")

ErrUploadDone is returned when either Abort or Commit has already been called.

Functions

This section is empty.

Types

type Access 

type Access struct {
	// contains filtered or unexported fields
}

An Access Grant contains everything to access a project and specific buckets. It includes a potentially-restricted API Key, a potentially-restricted set of encryption information, and information about the Satellite responsible for the project's metadata.

func ParseAccess 

func ParseAccess(access string) (*Access, error)

ParseAccess parses a serialized access grant string.

This should be the main way to instantiate an access grant for opening a project. See the note on RequestAccessWithPassphrase.

func RequestAccessWithPassphrase 

func RequestAccessWithPassphrase(ctx context.Context, satelliteAddress, apiKey, passphrase string) (*Access, error)

RequestAccessWithPassphrase generates a new access grant using a passhprase. It must talk to the Satellite provided to get a project-based salt for deterministic key derivation.

Note: this is a CPU-heavy function that uses a password-based key derivation function (Argon2). This should be a setup-only step. Most common interactions with the library should be using a serialized access grant through ParseAccess directly.

func (*Access) Serialize 

func (access *Access) Serialize() (string, error)

Serialize serializes an access grant such that it can be used later with ParseAccess or other tools.

func (*Access) Share 

func (access *Access) Share(permission Permission, prefixes ...SharePrefix) (*Access, error)

Share creates a new access grant with specific permissions.

Access grants can only have their existing permissions restricted, and the resulting access grant will only allow for the intersection of all previous Share calls in the access grant construction chain.

Prefixes, if provided, restrict the access grant (and internal encryption information) to only contain enough information to allow access to just those prefixes.

type Bucket 

type Bucket struct {
	Name    string
	Created time.Time
}

Bucket contains information about the bucket.

type BucketIterator 

type BucketIterator struct {
	// contains filtered or unexported fields
}

BucketIterator is an iterator over a collection of buckets.

func (*BucketIterator) Err 

func (buckets *BucketIterator) Err() error

Err returns error, if one happened during iteration.

func (*BucketIterator) Item 

func (buckets *BucketIterator) Item() *Bucket

Item returns the current bucket in the iterator.

func (*BucketIterator) Next 

func (buckets *BucketIterator) Next() bool

Next prepares next Bucket for reading. It returns false if the end of the iteration is reached and there are no more buckets, or if there is an error.

type Config 

type Config struct {
	UserAgent string

	// DialTimeout defines how long client should wait for establishing
	// a connection to peers.
	DialTimeout time.Duration

	// DialContext is how sockets are opened and is called to establish
	// a connection. If unset, net.Dialer is used.
	DialContext func(ctx context.Context, network, address string) (net.Conn, error)
}

Config defines configuration for using uplink library.

func (Config) OpenProject 

func (config Config) OpenProject(ctx context.Context, access *Access) (project *Project, err error)

OpenProject opens a project with the specific access grant.

func (Config) RequestAccessWithPassphrase 

func (config Config) RequestAccessWithPassphrase(ctx context.Context, satelliteAddress, apiKey, passphrase string) (*Access, error)

RequestAccessWithPassphrase generates a new access grant using a passhprase. It must talk to the Satellite provided to get a project-based salt for deterministic key derivation.

Note: this is a CPU-heavy function that uses a password-based key derivation function (Argon2). This should be a setup-only step. Most common interactions with the library should be using a serialized access grant through ParseAccess directly.

type CustomMetadata 

type CustomMetadata map[string]string

CustomMetadata contains custom user metadata about the object.

The keys and values in custom metadata are expected to be valid UTF-8.

When choosing a custom key for your application start it with a prefix "app:key", as an example application named "Image Board" might use a key "image-board:title".

func (CustomMetadata) Clone 

func (meta CustomMetadata) Clone() CustomMetadata

Clone makes a deep clone.

func (CustomMetadata) Verify 

func (meta CustomMetadata) Verify() error

Verify verifies whether CustomMetadata contains only "utf-8".

type Download 

type Download struct {
	// contains filtered or unexported fields
}

Download is a download from Storj Network.

func (*Download) Close 

func (download *Download) Close() error

Close closes the reader of the download.

func (*Download) Info 

func (download *Download) Info() *Object

Info returns the last information about the object.

func (*Download) Read 

func (download *Download) Read(p []byte) (n int, err error)

Read downloads up to len(p) bytes into p from the object's data stream. It returns the number of bytes read (0 <= n <= len(p)) and any error encountered.

type DownloadOptions 

type DownloadOptions struct {
	Offset int64
	// When Length is negative it will read until the end of the blob.
	Length int64
}

DownloadOptions contains additional options for downloading.

type ListBucketsOptions 

type ListBucketsOptions struct {
	// Cursor sets the starting position of the iterator. The first item listed will be the one after the cursor.
	Cursor string
}

ListBucketsOptions defines bucket listing options.

type ListObjectsOptions 

type ListObjectsOptions struct {
	// Prefix allows to filter objects by a key prefix. If not empty, it must end with slash.
	Prefix string
	// Cursor sets the starting position of the iterator. The first item listed will be the one after the cursor.
	Cursor string
	// Recursive iterates the objects without collapsing prefixes.
	Recursive bool

	// System includes SystemMetadata in the results.
	System bool
	// Custom includes CustomMetadata in the results.
	Custom bool
}

ListObjectsOptions defines object listing options.

type Object 

type Object struct {
	Key string
	// IsPrefix indicates whether the Key is a prefix for other objects.
	IsPrefix bool

	System SystemMetadata
	Custom CustomMetadata
}

Object contains information about an object.

type ObjectIterator 

type ObjectIterator struct {
	// contains filtered or unexported fields
}

ObjectIterator is an iterator over a collection of objects or prefixes.

func (*ObjectIterator) Err 

func (objects *ObjectIterator) Err() error

Err returns error, if one happened during iteration.

func (*ObjectIterator) Item 

func (objects *ObjectIterator) Item() *Object

Item returns the current object in the iterator.

func (*ObjectIterator) Next 

func (objects *ObjectIterator) Next() bool

Next prepares next Object for reading. It returns false if the end of the iteration is reached and there are no more objects, or if there is an error.

type Permission 

type Permission struct {
	// AllowDownload gives permission to download the object's content. It
	// allows getting object metadata, but it does not allow listing buckets.
	AllowDownload bool
	// AllowUpload gives permission to create buckets and upload new objects.
	// It does not allow overwriting existing objects unless AllowDelete is
	// granted too.
	AllowUpload bool
	// AllowList gives permission to list buckets. It allows getting object
	// metadata, but it does not allow downloading the object's content.
	AllowList bool
	// AllowDelete gives permission to delete buckets and objects. Unless
	// either AllowDownload or AllowList is granted too, no object metadata and
	// no error info will be returned for deleted objects.
	AllowDelete bool
	// NotBefore restricts when the resulting access grant is valid for.
	// If set, the resulting access grant will not work if the Satellite
	// believes the time is before NotBefore.
	// If set, this value should always be before NotAfter.
	NotBefore time.Time
	// NotAfter restricts when the resulting access grant is valid for.
	// If set, the resulting access grant will not work if the Satellite
	// believes the time is after NotAfter.
	// If set, this value should always be after NotBefore.
	NotAfter time.Time
}

Permission defines what actions can be used to share.

func FullPermission 

func FullPermission() Permission

FullPermission returns a Permission that allows all actions that the parent access grant already allows.

func ReadOnlyPermission 

func ReadOnlyPermission() Permission

ReadOnlyPermission returns a Permission that allows reading and listing (if the parent access grant already allows those things).

func WriteOnlyPermission 

func WriteOnlyPermission() Permission

WriteOnlyPermission returns a Permission that allows writing and deleting (if the parent access grant already allows those things).

type Project 

type Project struct {
	// contains filtered or unexported fields
}

Project provides access to managing buckets and objects.

func OpenProject 

func OpenProject(ctx context.Context, access *Access) (*Project, error)

OpenProject opens a project with the specific access grant.

func (*Project) Close 

func (project *Project) Close() (err error)

Close closes the project and all associated resources.

func (*Project) CreateBucket 

func (project *Project) CreateBucket(ctx context.Context, bucket string) (created *Bucket, err error)

CreateBucket creates a new bucket.

When bucket already exists it returns a valid Bucket and ErrBucketExists.

func (*Project) DeleteBucket 

func (project *Project) DeleteBucket(ctx context.Context, bucket string) (deleted *Bucket, err error)

DeleteBucket deletes a bucket.

When bucket is not empty it returns ErrBucketNotEmpty.

func (*Project) DeleteObject 

func (project *Project) DeleteObject(ctx context.Context, bucket, key string) (deleted *Object, err error)

DeleteObject deletes the object at the specific key.

func (*Project) DownloadObject 

func (project *Project) DownloadObject(ctx context.Context, bucket, key string, options *DownloadOptions) (download *Download, err error)

DownloadObject starts a download from the specific key.

func (*Project) EnsureBucket 

func (project *Project) EnsureBucket(ctx context.Context, bucket string) (ensured *Bucket, err error)

EnsureBucket ensures that a bucket exists or creates a new one.

When bucket already exists it returns a valid Bucket and no error.

func (*Project) ListBuckets 

func (project *Project) ListBuckets(ctx context.Context, options *ListBucketsOptions) *BucketIterator

ListBuckets returns an iterator over the buckets.

func (*Project) ListObjects 

func (project *Project) ListObjects(ctx context.Context, bucket string, options *ListObjectsOptions) *ObjectIterator

ListObjects returns an iterator over the objects.

func (*Project) StatBucket 

func (project *Project) StatBucket(ctx context.Context, bucket string) (info *Bucket, err error)

StatBucket returns information about a bucket.

func (*Project) StatObject 

func (project *Project) StatObject(ctx context.Context, bucket, key string) (info *Object, err error)

StatObject returns information about an object at the specific key.

func (*Project) UploadObject 

func (project *Project) UploadObject(ctx context.Context, bucket, key string, options *UploadOptions) (upload *Upload, err error)

UploadObject starts an upload to the specific key.

type SharePrefix 

type SharePrefix struct {
	Bucket string
	// Prefix is the prefix of the shared object keys.
	//
	// Note: that within a bucket, the hierarchical key derivation scheme is
	// delineated by forward slashes (/), so encryption information will be
	// included in the resulting access grant to decrypt any key that shares
	// the same prefix up until the last slash.
	Prefix string
}

SharePrefix defines a prefix that will be shared.

type SystemMetadata 

type SystemMetadata struct {
	Created       time.Time
	Expires       time.Time
	ContentLength int64
}

SystemMetadata contains information about the object that cannot be changed directly.

type Upload 

type Upload struct {
	// contains filtered or unexported fields
}

Upload is an upload to Storj Network.

func (*Upload) Abort 

func (upload *Upload) Abort() error

Abort aborts the upload.

Returns ErrUploadDone when either Abort or Commit has already been called.

func (*Upload) Commit 

func (upload *Upload) Commit() error

Commit commits data to the store.

Returns ErrUploadDone when either Abort or Commit has already been called.

func (*Upload) Info 

func (upload *Upload) Info() *Object

Info returns the last information about the uploaded object.

func (*Upload) SetCustomMetadata 

func (upload *Upload) SetCustomMetadata(ctx context.Context, custom CustomMetadata) error

SetCustomMetadata updates custom metadata to be included with the object. If it is nil, it won't be modified.

func (*Upload) Write 

func (upload *Upload) Write(p []byte) (n int, err error)

Write uploads len(p) bytes from p to the object's data stream. It returns the number of bytes written from p (0 <= n <= len(p)) and any error encountered that caused the write to stop early.

type UploadOptions 

type UploadOptions struct {
	// When Expires is zero, there is no expiration.
	Expires time.Time
}

UploadOptions contains additional options for uploading.

Source Files

View all Source files

Directories

Path Synopsis
Package backcomp contains utilities for handling backwards incompatible changes.
 Package backcomp contains utilities for handling backwards incompatible changes.
examples
walkthrough
internal
expose
telemetryclient
Package telemetryclient is internal package to support telemetry without introducing a direct dependency to the actual implementation.
 Package telemetryclient is internal package to support telemetry without introducing a direct dependency to the actual implementation.
private
access
ecclient
eestream
metainfo
piecestore
storage/meta
storage/segments
storage/streams
stream
testuplink

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.