Back to godoc.org
storj.io/uplink

Package uplink

v1.1.2
Latest Go to latest
Published: Jun 16, 2020 | License: MIT | Module: storj.io/uplink

Overview

Package uplink is the main entrypoint to interacting with Storj Labs' decentralized storage network.

Sign up for an account on a Satellite today! https://tardigrade.io/satellites/

Access Grants

The fundamental unit of access in the Storj Labs storage network is the Access Grant. An access grant is a serialized structure that is internally comprised of an API Key, a set of encryption key information, and information about which Storj Labs or Tardigrade network Satellite is responsible for the metadata. An access grant is always associated with exactly one Project on one Satellite.

If you don't already have an access grant, you will need make an account on a Satellite, generate an API Key, and encapsulate that API Key with encryption information into an access grant.

If you don't already have an account on a Satellite, first make one at https://tardigrade.io/satellites/ and note the Satellite you choose (such as us-central-1.tardigrade.io, europe-west-1.tardigrade.io, etc). Then, make an API Key in the web interface.

The first step to any project is to generate a restricted access grant with the minimal permissions that are needed. Access grants contains all encryption information and they should be restricted as much as possible.

To make an access grant, you can create one using our Uplink CLI tool's 'share' subcommand (after setting up the Uplink CLI tool), or you can make one as follows:

access, err := uplink.RequestAccessWithPassphrase(ctx, satelliteAddress, apiKey, rootPassphrase)
if err != nil {
    return err
}

// create an access grant for reading bucket "logs"
permissions := uplink.ReadOnlyPermission()
shared := uplink.SharePrefix{Bucket: "logs"}
restrictedAccess, err := access.Share(permissions, shared)
if err != nil {
    return err
}

// serialize the restricted access
serializedAccess, err := restrictedAccess.Serialize()
if err != nil {
    return err
}

In the above example, 'serializedAccess' is a human-readable string that represents read-only access to just the "logs" bucket, and is only able to decrypt that one bucket thanks to hierarchical deterministic key derivation.

Note: RequestAccessWithPassphrase is CPU-intensive, and your application's normal lifecycle should avoid it and use ParseAccess where possible instead.

Projects

Once you have a valid access grant, you can open a Project with the access that access grant allows for.

project, err := uplink.OpenProject(ctx, access)
if err != nil {
    return err
}
defer project.Close()

Projects allow you to manage buckets and objects within buckets.

Buckets

A bucket represents a collection of objects. You can upload, download, list, and delete objects of any size or shape. Objects within buckets are represented by keys, where keys can optionally be listed using the "/" delimiter.

Note: Objects and object keys within buckets are end-to-end encrypted, but bucket names themselves are not encrypted, so the billing interface on the Satellite can show you bucket line items.

buckets := project.ListBuckets(ctx, nil)
for buckets.Next() {
    fmt.Println(buckets.Item().Name)
}
if err := buckets.Err(); err != nil {
    return err
}

Download Object

Objects support a couple kilobytes of arbitrary key/value metadata, and arbitrary-size primary data streams with the ability to read at arbitrary offsets.

object, err := project.DownloadObject(ctx, "logs", "2020-04-18/webserver.log", nil)
if err != nil {
    return err
}
defer object.Close()

_, err = io.Copy(w, object)
return err

If you want to access only a small subrange of the data you uploaded, you can use `uplink.DownloadOptions` to specify the download range.

object, err := project.DownloadObject(ctx, "logs", "2020-04-18/webserver.log",
    &uplink.DownloadOptions{Offset: 10, Length: 100})
if err != nil {
    return err
}
defer object.Close()

_, err = io.Copy(w, object)
return err

List Objects

Listing objects returns an iterator that allows to walk through all the items:

objects := project.ListObjects(ctx, "logs", nil)
for objects.Next() {
    item := objects.Item()
    fmt.Println(item.IsPrefix, item.Key)
}
if err := objects.Err(); err != nil {
    return err
}

Index

Package Files

Variables

var ErrBandwidthLimitExceeded = errors.New("bandwidth limit exceeded")

ErrBandwidthLimitExceeded is returned when project will exceeded bandwidth limit.

var ErrBucketAlreadyExists = errors.New("bucket already exists")

ErrBucketAlreadyExists is returned when the bucket already exists during creation.

var ErrBucketNameInvalid = errors.New("bucket name invalid")

ErrBucketNameInvalid is returned when the bucket name is invalid.

var ErrBucketNotEmpty = errors.New("bucket not empty")

ErrBucketNotEmpty is returned when the bucket is not empty during deletion.

var ErrBucketNotFound = errors.New("bucket not found")

ErrBucketNotFound is returned when the bucket is not found.

var ErrObjectKeyInvalid = errors.New("object key invalid")

ErrObjectKeyInvalid is returned when the object key is invalid.

var ErrObjectNotFound = errors.New("object not found")

ErrObjectNotFound is returned when the object is not found.

var ErrTooManyRequests = errors.New("too many requests")

ErrTooManyRequests is returned when user has sent too many requests in a given amount of time.

var ErrUploadDone = errors.New("upload done")

ErrUploadDone is returned when either Abort or Commit has already been called.

type Access

type Access struct {
	// contains filtered or unexported fields
}

An Access Grant contains everything to access a project and specific buckets. It includes a potentially-restricted API Key, a potentially-restricted set of encryption information, and information about the Satellite responsible for the project's metadata.

func ParseAccess

func ParseAccess(access string) (*Access, error)

ParseAccess parses a serialized access grant string.

This should be the main way to instantiate an access grant for opening a project. See the note on RequestAccessWithPassphrase.

func RequestAccessWithPassphrase

func RequestAccessWithPassphrase(ctx context.Context, satelliteAddress, apiKey, passphrase string) (*Access, error)

RequestAccessWithPassphrase generates a new access grant using a passhprase. It must talk to the Satellite provided to get a project-based salt for deterministic key derivation.

Note: this is a CPU-heavy function that uses a password-based key derivation function (Argon2). This should be a setup-only step. Most common interactions with the library should be using a serialized access grant through ParseAccess directly.

func (*Access) Serialize

func (access *Access) Serialize() (string, error)

Serialize serializes an access grant such that it can be used later with ParseAccess or other tools.

func (*Access) Share

func (access *Access) Share(permission Permission, prefixes ...SharePrefix) (*Access, error)

Share creates a new access grant with specific permissions.

Access grants can only have their existing permissions restricted, and the resulting access grant will only allow for the intersection of all previous Share calls in the access grant construction chain.

Prefixes, if provided, restrict the access grant (and internal encryption information) to only contain enough information to allow access to just those prefixes.

type Bucket

type Bucket struct {
	Name    string
	Created time.Time
}

Bucket contains information about the bucket.

type BucketIterator

type BucketIterator struct {
	// contains filtered or unexported fields
}

BucketIterator is an iterator over a collection of buckets.

func (*BucketIterator) Err

func (buckets *BucketIterator) Err() error

Err returns error, if one happened during iteration.

func (*BucketIterator) Item

func (buckets *BucketIterator) Item() *Bucket

Item returns the current bucket in the iterator.

func (*BucketIterator) Next

func (buckets *BucketIterator) Next() bool

Next prepares next Bucket for reading. It returns false if the end of the iteration is reached and there are no more buckets, or if there is an error.

type Config

type Config struct {
	UserAgent string

	// DialTimeout defines how long client should wait for establishing
	// a connection to peers.
	DialTimeout time.Duration

	// DialContext is how sockets are opened and is called to establish
	// a connection. If unset, net.Dialer is used.
	DialContext func(ctx context.Context, network, address string) (net.Conn, error)
}

Config defines configuration for using uplink library.

func (Config) OpenProject

func (config Config) OpenProject(ctx context.Context, access *Access) (project *Project, err error)

OpenProject opens a project with the specific access grant.

func (Config) RequestAccessWithPassphrase

func (config Config) RequestAccessWithPassphrase(ctx context.Context, satelliteAddress, apiKey, passphrase string) (*Access, error)

RequestAccessWithPassphrase generates a new access grant using a passhprase. It must talk to the Satellite provided to get a project-based salt for deterministic key derivation.

Note: this is a CPU-heavy function that uses a password-based key derivation function (Argon2). This should be a setup-only step. Most common interactions with the library should be using a serialized access grant through ParseAccess directly.

type CustomMetadata

type CustomMetadata map[string]string

CustomMetadata contains custom user metadata about the object.

The keys and values in custom metadata are expected to be valid UTF-8.

When choosing a custom key for your application start it with a prefix "app:key", as an example application named "Image Board" might use a key "image-board:title".

func (CustomMetadata) Clone

func (meta CustomMetadata) Clone() CustomMetadata

Clone makes a deep clone.

func (CustomMetadata) Verify

func (meta CustomMetadata) Verify() error

Verify verifies whether CustomMetadata contains only "utf-8".

type Download

type Download struct {
	// contains filtered or unexported fields
}

Download is a download from Storj Network.

func (*Download) Close

func (download *Download) Close() error

Close closes the reader of the download.

func (*Download) Info

func (download *Download) Info() *Object

Info returns the last information about the object.

func (*Download) Read

func (download *Download) Read(p []byte) (n int, err error)

Read downloads up to len(p) bytes into p from the object's data stream. It returns the number of bytes read (0 <= n <= len(p)) and any error encountered.

type DownloadOptions

type DownloadOptions struct {
	Offset int64
	// When Length is negative it will read until the end of the blob.
	Length int64
}

DownloadOptions contains additional options for downloading.

type ListBucketsOptions

type ListBucketsOptions struct {
	// Cursor sets the starting position of the iterator. The first item listed will be the one after the cursor.
	Cursor string
}

ListBucketsOptions defines bucket listing options.

type ListObjectsOptions

type ListObjectsOptions struct {
	// Prefix allows to filter objects by a key prefix. If not empty, it must end with slash.
	Prefix string
	// Cursor sets the starting position of the iterator. The first item listed will be the one after the cursor.
	Cursor string
	// Recursive iterates the objects without collapsing prefixes.
	Recursive bool

	// System includes SystemMetadata in the results.
	System bool
	// Custom includes CustomMetadata in the results.
	Custom bool
}

ListObjectsOptions defines object listing options.

type Object

type Object struct {
	Key string
	// IsPrefix indicates whether the Key is a prefix for other objects.
	IsPrefix bool

	System SystemMetadata
	Custom CustomMetadata
}

Object contains information about an object.

type ObjectIterator

type ObjectIterator struct {
	// contains filtered or unexported fields
}

ObjectIterator is an iterator over a collection of objects or prefixes.

func (*ObjectIterator) Err

func (objects *ObjectIterator) Err() error

Err returns error, if one happened during iteration.

func (*ObjectIterator) Item

func (objects *ObjectIterator) Item() *Object

Item returns the current object in the iterator.

func (*ObjectIterator) Next

func (objects *ObjectIterator) Next() bool

Next prepares next Object for reading. It returns false if the end of the iteration is reached and there are no more objects, or if there is an error.

type Permission

type Permission struct {
	// AllowDownload gives permission to download the object's content. It
	// allows getting object metadata, but it does not allow listing buckets.
	AllowDownload bool
	// AllowUpload gives permission to create buckets and upload new objects.
	// It does not allow overwriting existing objects unless AllowDelete is
	// granted too.
	AllowUpload bool
	// AllowList gives permission to list buckets. It allows getting object
	// metadata, but it does not allow downloading the object's content.
	AllowList bool
	// AllowDelete gives permission to delete buckets and objects. Unless
	// either AllowDownload or AllowList is granted too, no object metadata and
	// no error info will be returned for deleted objects.
	AllowDelete bool
	// NotBefore restricts when the resulting access grant is valid for.
	// If set, the resulting access grant will not work if the Satellite
	// believes the time is before NotBefore.
	// If set, this value should always be before NotAfter.
	NotBefore time.Time
	// NotAfter restricts when the resulting access grant is valid for.
	// If set, the resulting access grant will not work if the Satellite
	// believes the time is after NotAfter.
	// If set, this value should always be after NotBefore.
	NotAfter time.Time
}

Permission defines what actions can be used to share.

func FullPermission

func FullPermission() Permission

FullPermission returns a Permission that allows all actions that the parent access grant already allows.

func ReadOnlyPermission

func ReadOnlyPermission() Permission

ReadOnlyPermission returns a Permission that allows reading and listing (if the parent access grant already allows those things).

func WriteOnlyPermission

func WriteOnlyPermission() Permission

WriteOnlyPermission returns a Permission that allows writing and deleting (if the parent access grant already allows those things).

type Project

type Project struct {
	// contains filtered or unexported fields
}

Project provides access to managing buckets and objects.

func OpenProject

func OpenProject(ctx context.Context, access *Access) (*Project, error)

OpenProject opens a project with the specific access grant.

func (*Project) Close

func (project *Project) Close() (err error)

Close closes the project and all associated resources.

func (*Project) CreateBucket

func (project *Project) CreateBucket(ctx context.Context, bucket string) (created *Bucket, err error)

CreateBucket creates a new bucket.

When bucket already exists it returns a valid Bucket and ErrBucketExists.

func (*Project) DeleteBucket

func (project *Project) DeleteBucket(ctx context.Context, bucket string) (deleted *Bucket, err error)

DeleteBucket deletes a bucket.

When bucket is not empty it returns ErrBucketNotEmpty.

func (*Project) DeleteObject

func (project *Project) DeleteObject(ctx context.Context, bucket, key string) (deleted *Object, err error)

DeleteObject deletes the object at the specific key.

func (*Project) DownloadObject

func (project *Project) DownloadObject(ctx context.Context, bucket, key string, options *DownloadOptions) (download *Download, err error)

DownloadObject starts a download from the specific key.

func (*Project) EnsureBucket

func (project *Project) EnsureBucket(ctx context.Context, bucket string) (ensured *Bucket, err error)

EnsureBucket ensures that a bucket exists or creates a new one.

When bucket already exists it returns a valid Bucket and no error.

func (*Project) ListBuckets

func (project *Project) ListBuckets(ctx context.Context, options *ListBucketsOptions) *BucketIterator

ListBuckets returns an iterator over the buckets.

func (*Project) ListObjects

func (project *Project) ListObjects(ctx context.Context, bucket string, options *ListObjectsOptions) *ObjectIterator

ListObjects returns an iterator over the objects.

func (*Project) StatBucket

func (project *Project) StatBucket(ctx context.Context, bucket string) (info *Bucket, err error)

StatBucket returns information about a bucket.

func (*Project) StatObject

func (project *Project) StatObject(ctx context.Context, bucket, key string) (info *Object, err error)

StatObject returns information about an object at the specific key.

func (*Project) UploadObject

func (project *Project) UploadObject(ctx context.Context, bucket, key string, options *UploadOptions) (upload *Upload, err error)

UploadObject starts an upload to the specific key.

type SharePrefix

type SharePrefix struct {
	Bucket string
	// Prefix is the prefix of the shared object keys.
	//
	// Note: that within a bucket, the hierarchical key derivation scheme is
	// delineated by forward slashes (/), so encryption information will be
	// included in the resulting access grant to decrypt any key that shares
	// the same prefix up until the last slash.
	Prefix string
}

SharePrefix defines a prefix that will be shared.

type SystemMetadata

type SystemMetadata struct {
	Created       time.Time
	Expires       time.Time
	ContentLength int64
}

SystemMetadata contains information about the object that cannot be changed directly.

type Upload

type Upload struct {
	// contains filtered or unexported fields
}

Upload is an upload to Storj Network.

func (*Upload) Abort

func (upload *Upload) Abort() error

Abort aborts the upload.

Returns ErrUploadDone when either Abort or Commit has already been called.

func (*Upload) Commit

func (upload *Upload) Commit() error

Commit commits data to the store.

Returns ErrUploadDone when either Abort or Commit has already been called.

func (*Upload) Info

func (upload *Upload) Info() *Object

Info returns the last information about the uploaded object.

func (*Upload) SetCustomMetadata

func (upload *Upload) SetCustomMetadata(ctx context.Context, custom CustomMetadata) error

SetCustomMetadata updates custom metadata to be included with the object. If it is nil, it won't be modified.

func (*Upload) Write

func (upload *Upload) Write(p []byte) (n int, err error)

Write uploads len(p) bytes from p to the object's data stream. It returns the number of bytes written from p (0 <= n <= len(p)) and any error encountered that caused the write to stop early.

type UploadOptions

type UploadOptions struct {
	// When Expires is zero, there is no expiration.
	Expires time.Time
}

UploadOptions contains additional options for uploading.

Documentation was rendered with GOOS=linux and GOARCH=amd64.

Jump to identifier

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to identifier