mefs

package module
v0.0.0-...-5f8b615 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 30, 2019 License: Apache-2.0 Imports: 47 Imported by: 0

README

适用于与MEFS的Go SDK

mefs-sdk-go SDK提供了简单的API来访问MEFS的对象存储服务。

本文假设你已经有 Go开发环境

从Github下载

go get -u github.com/memoio/mefs-sdk-go

初始化MinIO Client

MinIO client需要以下4个参数来连接与MEFS兼容的对象存储。

参数 描述
endpoint 对象存储服务的URL
accessKeyID MEFS中的address
secretAccessKey 在该Gateway上的登录密码
secure true代表使用HTTPS(暂未兼容)
package main

import (
	"github.com/memoio/mefs-sdk-go"
	"log"
)

func main() {
	//预先准备好addr
	addr := "0xD60457e090e166305D3CEE0BCF3778C689B7441d"
	endpoint := "127.0.0.1:6711"
	password := "123456"
	mefsClient, err := mefs.New(endpoint, addr, password, true)
	if err != nil {
		log.Fatalln(err)
	}

	fmt.Println("连接成功")
}

示例-文件上传

本示例连接到一个对象存储服务,创建一个存储桶并上传一个文件到存储桶中。

FileUploader.go
package main

import (
	"github.com/memoio/mefs-sdk-go"
	"log"
)

func main() {
	//预先准备好addr
	addr := "0xD60457e090e166305D3CEE0BCF3778C689B7441d"
	endpoint := "127.0.0.1:6711"
	useSSL := true

	// 初使化mefs client对象。
	mefsClient, err := mefs.New(endpoint, accessKeyID, secretAccessKey, useSSL)
	if err != nil {
		log.Fatalln(err)
	}

	// 创建一个叫mymusic的存储桶。
	bucketName := "mymusic"

	err = mefsClient.MakeBucket(bucketName, location)
	if err != nil {
		// 检查存储桶是否已经存在。
		exists, err := mefsClient.BucketExists(bucketName)
		if err == nil && exists {
			log.Printf("We already own %s\n", bucketName)
		} else {
			log.Fatalln(err)
		}
	}
	log.Printf("Successfully created %s\n", bucketName)

	// 上传一个zip文件。
	objectName := "golden-oldies.zip"
	filePath := "/tmp/golden-oldies.zip"
	contentType := "application/zip"

	// 使用FPutObject上传一个zip文件。
	n, err := mefsClient.FPutObject(bucketName, objectName, filePath, minio.PutObjectOptions{ContentType:contentType})
	if err != nil {
		log.Fatalln(err)
	}

	log.Printf("Successfully uploaded %s of size %d\n", objectName, n)
}
运行FileUploader
go run file-uploader.go
2016/08/13 17:03:28 Successfully created mymusic 
2016/08/13 17:03:40 Successfully uploaded golden-oldies.zip of size 16253413

mc ls play/mymusic/
[2016-05-27 16:02:16 PDT]  17MiB golden-oldies.zip
支持API : 操作存储桶
  • MakeBucket
  • ListBuckets
  • BucketExists
  • ListObjects
支持API : 操作文件对象
  • FPutObject
  • FGetObject
  • FPutObjectWithContext
  • FGetObjectWithContext
支持API : 操作对象
  • GetObject
  • PutObject
  • GetObjectWithContext
  • PutObjectWithContext
  • PutObjectStreaming
  • StatObject
具体示例

Documentation

Index

Constants

View Source
const (
	CSVFileHeaderInfoNone   CSVFileHeaderInfo = "NONE"
	CSVFileHeaderInfoIgnore                   = "IGNORE"
	CSVFileHeaderInfoUse                      = "USE"
)

Constants for file header info.

View Source
const (
	SelectCompressionNONE SelectCompressionType = "NONE"
	SelectCompressionGZIP                       = "GZIP"
	SelectCompressionBZIP                       = "BZIP2"
)

Constants for compression types under select API.

View Source
const (
	SelectObjectTypeCSV     SelectObjectType = "CSV"
	SelectObjectTypeJSON                     = "JSON"
	SelectObjectTypeParquet                  = "Parquet"
)

Constants for input data types.

View Source
const (
	DefaultPathName = ".mefs"
	DefaultPathRoot = "~/" + DefaultPathName
	DefaultApiFile  = "api"
	EnvDir          = "MEFS_PATH"
)
View Source
const (
	DirectPin    = "direct"
	RecursivePin = "recursive"
	IndirectPin  = "indirect"
)
View Source
const (
	ObjectCreatedAll                     NotificationEventType = "s3:ObjectCreated:*"
	ObjectCreatedPut                                           = "s3:ObjectCreated:Put"
	ObjectCreatedPost                                          = "s3:ObjectCreated:Post"
	ObjectCreatedCopy                                          = "s3:ObjectCreated:Copy"
	ObjectCreatedCompleteMultipartUpload                       = "s3:ObjectCreated:CompleteMultipartUpload"
	ObjectAccessedGet                                          = "s3:ObjectAccessed:Get"
	ObjectAccessedHead                                         = "s3:ObjectAccessed:Head"
	ObjectAccessedAll                                          = "s3:ObjectAccessed:*"
	ObjectRemovedAll                                           = "s3:ObjectRemoved:*"
	ObjectRemovedDelete                                        = "s3:ObjectRemoved:Delete"
	ObjectRemovedDeleteMarkerCreated                           = "s3:ObjectRemoved:DeleteMarkerCreated"
	ObjectReducedRedundancyLostObject                          = "s3:ReducedRedundancyLostObject"
)

The role of all event types are described in :

http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html#notification-how-to-event-types-and-destinations
View Source
const DefaultRetryCap = time.Second * 30

DefaultRetryCap - Each retry attempt never waits no longer than this maximum time duration.

View Source
const DefaultRetryUnit = time.Second

DefaultRetryUnit - default unit multiplicative per retry. defaults to 1 second.

View Source
const MaxJitter = 1.0

MaxJitter will randomize over the full exponential backoff time

View Source
const NoJitter = 0.0

NoJitter disables the use of jitter for randomizing the exponential backoff time

View Source
const SHOWTIME = "2006-01-02 Mon 15:04:05 MST"

SHOWTIME 用于输出给使用者

Variables

View Source
var DefaultTransport = func(secure bool) (http.RoundTripper, error) {
	tr := &http.Transport{
		Proxy: http.ProxyFromEnvironment,
		DialContext: (&net.Dialer{
			Timeout:   30 * time.Second,
			KeepAlive: 30 * time.Second,
		}).DialContext,
		MaxIdleConns:          1024,
		MaxIdleConnsPerHost:   1024,
		IdleConnTimeout:       90 * time.Second,
		TLSHandshakeTimeout:   10 * time.Second,
		ExpectContinueTimeout: 1 * time.Second,

		DisableCompression: true,
	}

	if secure {
		rootCAs, _ := x509.SystemCertPool()
		if rootCAs == nil {

			rootCAs = x509.NewCertPool()
		}

		tlsConfig := &tls.Config{
			RootCAs: rootCAs,

			MinVersion: tls.VersionTLS12,
		}
		tr.TLSClientConfig = tlsConfig

		if err := http2.ConfigureTransport(tr); err != nil {
			return nil, err
		}
	}
	return tr, nil
}

DefaultTransport - this default transport is similar to http.DefaultTransport but with additional param DisableCompression is set to true to avoid decompressing content with 'gzip' encoding.

View Source
var ErrInvalidObjectPrefix = ErrInvalidObjectName

ErrInvalidObjectPrefix - Invalid object prefix response is similar to object name response.

View Source
var MaxRetry = 10

MaxRetry is the maximum number of retries before stopping.

Functions

func ErrAPINotSupported

func ErrAPINotSupported(message string) error

ErrAPINotSupported - API not supported response The specified API call is not supported

func ErrEntityTooLarge

func ErrEntityTooLarge(totalSize, maxObjectSize int64, bucketName, objectName string) error

ErrEntityTooLarge - Input size is larger than supported maximum.

func ErrEntityTooSmall

func ErrEntityTooSmall(totalSize int64, bucketName, objectName string) error

ErrEntityTooSmall - Input size is smaller than supported minimum.

func ErrInvalidArgument

func ErrInvalidArgument(message string) error

ErrInvalidArgument - Invalid argument response.

func ErrInvalidBucketName

func ErrInvalidBucketName(message string) error

ErrInvalidBucketName - Invalid bucket name response.

func ErrInvalidObjectName

func ErrInvalidObjectName(message string) error

ErrInvalidObjectName - Invalid object name response.

func ErrNoSuchBucketPolicy

func ErrNoSuchBucketPolicy(message string) error

ErrNoSuchBucketPolicy - No Such Bucket Policy response The specified bucket does not have a bucket policy.

func ErrTransferAccelerationBucket

func ErrTransferAccelerationBucket(bucketName string) error

ErrTransferAccelerationBucket - bucket name is invalid to be used with transfer acceleration.

func ErrUnexpectedEOF

func ErrUnexpectedEOF(totalRead, totalSize int64, bucketName, objectName string) error

ErrUnexpectedEOF - Unexpected end of file reached.

Types

type Arn

type Arn struct {
	Partition string
	Service   string
	Region    string
	AccountID string
	Resource  string
}

Arn - holds ARN information that will be sent to the web service, ARN desciption can be found in http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html

func NewArn

func NewArn(partition, service, region, accountID, resource string) Arn

NewArn creates new ARN based on the given partition, service, region, account id and resource

func (Arn) String

func (arn Arn) String() string

Return the string format of the ARN

type BlockStat

type BlockStat struct {
	Key  string
	Size int
}

type BucketInfo

type BucketInfo struct {
	// The name of the bucket.
	Name string `json:"name"`
	// Date the bucket was created.
	CreationDate time.Time `json:"creationDate"`
}

BucketInfo container for bucket metadata.

type BucketLookupType

type BucketLookupType int

BucketLookupType is type of url lookup supported by server.

const (
	BucketLookupAuto BucketLookupType = iota
	BucketLookupDNS
	BucketLookupPath
)

Different types of url lookup supported by the server.Initialized to BucketLookupAuto

type BucketNotification

type BucketNotification struct {
	XMLName       xml.Name       `xml:"NotificationConfiguration"`
	LambdaConfigs []LambdaConfig `xml:"CloudFunctionConfiguration"`
	TopicConfigs  []TopicConfig  `xml:"TopicConfiguration"`
	QueueConfigs  []QueueConfig  `xml:"QueueConfiguration"`
}

BucketNotification - the struct that represents the whole XML to be sent to the web service

func (*BucketNotification) AddLambda

func (b *BucketNotification) AddLambda(lambdaConfig NotificationConfig) bool

AddLambda adds a given lambda config to the general bucket notification config

func (*BucketNotification) AddQueue

func (b *BucketNotification) AddQueue(queueConfig NotificationConfig) bool

AddQueue adds a given queue config to the general bucket notification config

func (*BucketNotification) AddTopic

func (b *BucketNotification) AddTopic(topicConfig NotificationConfig) bool

AddTopic adds a given topic config to the general bucket notification config

func (*BucketNotification) RemoveLambdaByArn

func (b *BucketNotification) RemoveLambdaByArn(arn Arn)

RemoveLambdaByArn removes all lambda configurations that match the exact specified ARN

func (*BucketNotification) RemoveQueueByArn

func (b *BucketNotification) RemoveQueueByArn(arn Arn)

RemoveQueueByArn removes all queue configurations that match the exact specified ARN

func (*BucketNotification) RemoveTopicByArn

func (b *BucketNotification) RemoveTopicByArn(arn Arn)

RemoveTopicByArn removes all topic configurations that match the exact specified ARN

type BucketStat

type BucketStat struct {
	BucketName  string
	BucketID    int32
	Ctime       string
	Policy      int32
	DataCount   int32
	ParityCount int32
}

type Buckets

type Buckets struct {
	Method  string
	Buckets []BucketStat
}

type CSVFileHeaderInfo

type CSVFileHeaderInfo string

CSVFileHeaderInfo - is the parameter for whether to utilize headers.

type CSVInputOptions

type CSVInputOptions struct {
	FileHeaderInfo       CSVFileHeaderInfo
	RecordDelimiter      string
	FieldDelimiter       string `xml:",omitempty"`
	QuoteCharacter       string `xml:",omitempty"`
	QuoteEscapeCharacter string `xml:",omitempty"`
	Comments             string `xml:",omitempty"`
}

CSVInputOptions csv input specific options

type CSVOutputOptions

type CSVOutputOptions struct {
	QuoteFields          CSVQuoteFields `xml:",omitempty"`
	RecordDelimiter      string
	FieldDelimiter       string `xml:",omitempty"`
	QuoteCharacter       string `xml:",omitempty"`
	QuoteEscapeCharacter string `xml:",omitempty"`
}

CSVOutputOptions csv output specific options

type CSVQuoteFields

type CSVQuoteFields string

CSVQuoteFields - is the parameter for how CSV fields are quoted.

const (
	CSVQuoteFieldsAlways   CSVQuoteFields = "Always"
	CSVQuoteFieldsAsNeeded                = "AsNeeded"
)

Constants for csv quote styles.

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client implements Amazon S3 compatible methods.

func New

func New(endpoint, accessKeyID, secretAccessKey string, secure bool) (*Client, error)

New - instantiate minio client, adds automatic verification of signature.

func NewV2

func NewV2(endpoint string, accessKeyID, secretAccessKey string, secure bool) (*Client, error)

NewV2 - instantiate minio client with Amazon S3 signature version '2' compatibility.

func NewV4

func NewV4(endpoint string, accessKeyID, secretAccessKey string, secure bool) (*Client, error)

NewV4 - instantiate minio client with Amazon S3 signature version '4' compatibility.

func NewWithCredentials

func NewWithCredentials(endpoint string, creds *credentials.Credentials, secure bool, region string) (*Client, error)

NewWithCredentials - instantiate minio client with credentials provider for retrieving credentials from various credentials provider such as IAM, File, Env etc.

func NewWithOptions

func NewWithOptions(endpoint string, opts *Options) (*Client, error)

NewWithOptions - instantiate minio client with options

func NewWithRegion

func NewWithRegion(endpoint, accessKeyID, secretAccessKey string, secure bool, region string) (*Client, error)

NewWithRegion - instantiate minio client, with region configured. Unlike New(), NewWithRegion avoids bucket-location lookup operations and it is slightly faster. Use this function when if your application deals with single region.

func (*Client) BlockGet

func (c *Client) BlockGet(path string) ([]byte, error)

func (*Client) BlockPut

func (c *Client) BlockPut(block []byte, format, mhtype string, mhlen int) (string, error)

func (*Client) BlockStat

func (c *Client) BlockStat(path string) (string, int, error)

func (Client) BucketExists

func (c Client) BucketExists(bucketName string) (bool, error)

BucketExists verify if bucket exists and you have permission to access it.

func (Client) ChallengeTest

func (c Client) ChallengeTest(key, to string, options ...LfsOpts) (string, error)

func (Client) ComposeObject

func (c Client) ComposeObject(dst DestinationInfo, srcs []SourceInfo) error

ComposeObject - creates an object using server-side copying of existing objects. It takes a list of source objects (with optional offsets) and concatenates them into a new object using only server-side copying operations.

func (Client) ComposeObjectWithProgress

func (c Client) ComposeObjectWithProgress(dst DestinationInfo, srcs []SourceInfo, progress io.Reader) error

ComposeObjectWithProgress - creates an object using server-side copying of existing objects. It takes a list of source objects (with optional offsets) and concatenates them into a new object using only server-side copying operations. Optionally takes progress reader hook for applications to look at current progress.

func (Client) CopyObject

func (c Client) CopyObject(dst DestinationInfo, src SourceInfo) error

CopyObject - copy a source object into a new object

func (Client) CopyObjectWithProgress

func (c Client) CopyObjectWithProgress(dst DestinationInfo, src SourceInfo, progress io.Reader) error

CopyObjectWithProgress - copy a source object into a new object, optionally takes progress bar input to notify current progress.

func (Client) CreateUser

func (c Client) CreateUser(options ...LfsOpts) (*UserPrivMessage, error)

func (Client) FGetObject

func (c Client) FGetObject(bucketName, objectName, filePath string, opts GetObjectOptions) error

FGetObject - download contents of an object to a local file.

func (Client) FGetObjectWithContext

func (c Client) FGetObjectWithContext(ctx context.Context, bucketName, objectName, filePath string, opts GetObjectOptions) error

FGetObjectWithContext - download contents of an object to a local file. The options can be used to specify the GET request further.

func (Client) FPutObject

func (c Client) FPutObject(bucketName, objectName, filePath string, opts PutObjectOptions) (n int64, err error)

FPutObject - Create an object in a bucket, with contents from file at filePath

func (Client) FPutObjectWithContext

func (c Client) FPutObjectWithContext(ctx context.Context, bucketName, objectName, filePath string, opts PutObjectOptions) (n int64, err error)

FPutObjectWithContext - Create an object in a bucket, with contents from file at filePath. Allows request cancellation.

func (*Client) FindPeer

func (c *Client) FindPeer(peer string) (*PeerInfo, error)

func (Client) Fsync

func (c Client) Fsync(options ...LfsOpts) error

func (Client) GetBlockFrom

func (c Client) GetBlockFrom(key, id string, options ...LfsOpts) (string, error)

func (Client) GetBucketLifecycle

func (c Client) GetBucketLifecycle(bucketName string) (string, error)

GetBucketLifecycle - get bucket lifecycle.

func (Client) GetBucketNotification

func (c Client) GetBucketNotification(bucketName string) (bucketNotification BucketNotification, err error)

GetBucketNotification - get bucket notification at a given path.

func (Client) GetBucketPolicy

func (c Client) GetBucketPolicy(bucketName string) (string, error)

GetBucketPolicy - get bucket policy at a given path.

func (Client) GetFrom

func (c Client) GetFrom(key, id string, options ...LfsOpts) (*QueryEvent, error)

func (Client) GetObject

func (c Client) GetObject(bucketName, objectName string, opts GetObjectOptions) (*Object, error)

GetObject - returns an seekable, readable object.

func (Client) GetObjectACL

func (c Client) GetObjectACL(bucketName, objectName string) (*ObjectInfo, error)

GetObjectACL get object ACLs

func (Client) GetObjectWithContext

func (c Client) GetObjectWithContext(ctx context.Context, bucketName, objectName string, opts GetObjectOptions) (*Object, error)

GetObjectWithContext - returns an seekable, readable object. The options can be used to specify the GET request further.

func (*Client) ID

func (c *Client) ID(peer ...string) (*IdOutput, error)

ID gets information about a given peer. Arguments:

peer: peer.ID of the node to look up. If no peer is specified,

return information about the local peer.

func (*Client) IsUp

func (c *Client) IsUp() bool

func (Client) ListBuckets

func (c Client) ListBuckets() ([]BucketInfo, error)

ListBuckets list all buckets owned by this authenticated user.

This call requires explicit authentication, no anonymous requests are allowed for listing buckets.

api := client.New(....)
for message := range api.ListBuckets() {
    fmt.Println(message)
}

func (Client) ListIncompleteUploads

func (c Client) ListIncompleteUploads(bucketName, objectPrefix string, recursive bool, doneCh <-chan struct{}) <-chan ObjectMultipartInfo

ListIncompleteUploads - List incompletely uploaded multipart objects.

ListIncompleteUploads lists all incompleted objects matching the objectPrefix from the specified bucket. If recursion is enabled it would list all subdirectories and all its contents.

Your input parameters are just bucketName, objectPrefix, recursive and a done channel to pro-actively close the internal go routine. If you enable recursive as 'true' this function will return back all the multipart objects in a given bucket name.

api := client.New(....)
// Create a done channel.
doneCh := make(chan struct{})
defer close(doneCh)
// Recurively list all objects in 'mytestbucket'
recursive := true
for message := range api.ListIncompleteUploads("mytestbucket", "starthere", recursive) {
    fmt.Println(message)
}

func (Client) ListKeepers

func (c Client) ListKeepers(options ...LfsOpts) (*PeerList, error)

func (Client) ListObjects

func (c Client) ListObjects(bucketName, objectPrefix string, recursive bool, doneCh <-chan struct{}) <-chan ObjectInfo

ListObjects - (List Objects) - List some objects or all recursively.

ListObjects lists all objects matching the objectPrefix from the specified bucket. If recursion is enabled it would list all subdirectories and all its contents.

Your input parameters are just bucketName, objectPrefix, recursive and a done channel for pro-actively closing the internal go routine. If you enable recursive as 'true' this function will return back all the objects in a given bucket name and object prefix.

api := client.New(....)
// Create a done channel.
doneCh := make(chan struct{})
defer close(doneCh)
// Recurively list all objects in 'mytestbucket'
recursive := true
for message := range api.ListObjects("mytestbucket", "starthere", recursive, doneCh) {
    fmt.Println(message)
}

func (Client) ListObjectsV2

func (c Client) ListObjectsV2(bucketName, objectPrefix string, recursive bool, doneCh <-chan struct{}) <-chan ObjectInfo

ListObjectsV2 lists all objects matching the objectPrefix from the specified bucket. If recursion is enabled it would list all subdirectories and all its contents.

Your input parameters are just bucketName, objectPrefix, recursive and a done channel for pro-actively closing the internal go routine. If you enable recursive as 'true' this function will return back all the objects in a given bucket name and object prefix.

api := client.New(....)
// Create a done channel.
doneCh := make(chan struct{})
defer close(doneCh)
// Recursively list all objects in 'mytestbucket'
recursive := true
for message := range api.ListObjectsV2("mytestbucket", "starthere", recursive, doneCh) {
    fmt.Println(message)
}

func (Client) ListenBucketNotification

func (c Client) ListenBucketNotification(bucketName, prefix, suffix string, events []string, doneCh <-chan struct{}) <-chan NotificationInfo

ListenBucketNotification - listen on bucket notifications.

func (Client) MakeBucket

func (c Client) MakeBucket(bucketName string, location string) (err error)

MakeBucket creates a new bucket with bucketName.

Location is an optional argument, by default all buckets are created in US Standard Region.

For Amazon S3 for more supported regions - http://docs.aws.amazon.com/general/latest/gr/rande.html For Google Cloud Storage for more supported regions - https://cloud.google.com/storage/docs/bucket-locations

func (Client) PutObject

func (c Client) PutObject(bucketName, objectName string, reader io.Reader, objectSize int64,
	opts PutObjectOptions) (n int64, err error)

PutObject creates an object in a bucket.

You must have WRITE permissions on a bucket to create an object.

  • For size smaller than 128MiB PutObject automatically does a single atomic Put operation.
  • For size larger than 128MiB PutObject automatically does a multipart Put operation.
  • For size input as -1 PutObject does a multipart Put operation until input stream reaches EOF. Maximum object size that can be uploaded through this operation will be 5TiB.

func (Client) PutObjectWithContext

func (c Client) PutObjectWithContext(ctx context.Context, bucketName, objectName string, reader io.Reader, objectSize int64,
	opts PutObjectOptions) (n int64, err error)

PutObjectWithContext - Identical to PutObject call, but accepts context to facilitate request cancellation.

func (Client) RemoveAllBucketNotification

func (c Client) RemoveAllBucketNotification(bucketName string) error

RemoveAllBucketNotification - Remove bucket notification clears all previously specified config

func (Client) RemoveBucket

func (c Client) RemoveBucket(bucketName string) error

RemoveBucket deletes the bucket name.

All objects (including all object versions and delete markers).
in the bucket must be deleted before successfully attempting this request.

func (Client) RemoveIncompleteUpload

func (c Client) RemoveIncompleteUpload(bucketName, objectName string) error

RemoveIncompleteUpload aborts an partially uploaded object.

func (Client) RemoveObject

func (c Client) RemoveObject(bucketName, objectName string) error

RemoveObject remove an object from a bucket.

func (Client) RemoveObjects

func (c Client) RemoveObjects(bucketName string, objectsCh <-chan string) <-chan RemoveObjectError

RemoveObjects removes multiple objects from a bucket. The list of objects to remove are received from objectsCh. Remove failures are sent back via error channel.

func (Client) RemoveObjectsWithContext

func (c Client) RemoveObjectsWithContext(ctx context.Context, bucketName string, objectsCh <-chan string) <-chan RemoveObjectError

RemoveObjectsWithContext - Identical to RemoveObjects call, but accepts context to facilitate request cancellation.

func (*Client) Request

func (c *Client) Request(command string, args ...string) *RequestBuilder

func (*Client) ResolvePath

func (c *Client) ResolvePath(path string) (string, error)

func (Client) SelectObjectContent

func (c Client) SelectObjectContent(ctx context.Context, bucketName, objectName string, opts SelectObjectOptions) (*SelectResults, error)

SelectObjectContent is a implementation of http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectSELECTContent.html AWS S3 API.

func (*Client) SetAppInfo

func (c *Client) SetAppInfo(appName string, appVersion string)

SetAppInfo - add application details to user agent.

func (Client) SetBucketLifecycle

func (c Client) SetBucketLifecycle(bucketName, lifecycle string) error

SetBucketLifecycle set the lifecycle on an existing bucket.

func (Client) SetBucketNotification

func (c Client) SetBucketNotification(bucketName string, bucketNotification BucketNotification) error

SetBucketNotification saves a new bucket notification.

func (Client) SetBucketPolicy

func (c Client) SetBucketPolicy(bucketName, policy string) error

SetBucketPolicy set the access permissions on an existing bucket.

func (*Client) SetCustomTransport

func (c *Client) SetCustomTransport(customHTTPTransport http.RoundTripper)

SetCustomTransport - set new custom transport.

func (*Client) SetS3TransferAccelerate

func (c *Client) SetS3TransferAccelerate(accelerateEndpoint string)

SetS3TransferAccelerate - turns s3 accelerated endpoint on or off for all your requests. This feature is only specific to S3 for all other endpoints this function does nothing. To read further details on s3 transfer acceleration please vist - http://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html

func (*Client) SetTimeout

func (c *Client) SetTimeout(d time.Duration)

func (Client) ShowStorage

func (c Client) ShowStorage(options ...LfsOpts) error

func (Client) StartUser

func (c Client) StartUser(address string, options ...LfsOpts) error

func (Client) StatObject

func (c Client) StatObject(bucketName, objectName string, opts StatObjectOptions) (ObjectInfo, error)

StatObject verifies if object exists and you have permission to access.

func (*Client) SwarmConnect

func (c *Client) SwarmConnect(ctx context.Context, addr ...string) error

SwarmConnect opens a swarm connection to a specific address.

func (*Client) SwarmPeers

func (c *Client) SwarmPeers(ctx context.Context) (*SwarmConnInfos, error)

SwarmPeers gets all the swarm peers

func (*Client) TraceErrorsOnlyOff

func (c *Client) TraceErrorsOnlyOff()

TraceErrorsOnlyOff - Turns off the errors only tracing and everything will be traced after this call. If all tracing needs to be turned off, call TraceOff().

func (*Client) TraceErrorsOnlyOn

func (c *Client) TraceErrorsOnlyOn(outputStream io.Writer)

TraceErrorsOnlyOn - same as TraceOn, but only errors will be traced.

func (*Client) TraceOff

func (c *Client) TraceOff()

TraceOff - disable HTTP tracing.

func (*Client) TraceOn

func (c *Client) TraceOn(outputStream io.Writer)

TraceOn - enable HTTP tracing.

func (*Client) Version

func (c *Client) Version() (string, string, error)

returns ipfs version and commit sha

type CommonPrefix

type CommonPrefix struct {
	Prefix string
}

CommonPrefix container for prefix response.

type CompletePart

type CompletePart struct {
	XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ Part" json:"-"`

	// Part number identifies the part.
	PartNumber int
	ETag       string
}

CompletePart sub container lists individual part numbers and their md5sum, part of completeMultipartUpload.

type Core

type Core struct {
	*Client
}

Core - Inherits Client and adds new methods to expose the low level S3 APIs.

func NewCore

func NewCore(endpoint string, accessKeyID, secretAccessKey string, secure bool) (*Core, error)

NewCore - Returns new initialized a Core client, this CoreClient should be only used under special conditions such as need to access lower primitives and being able to use them to write your own wrappers.

func (Core) AbortMultipartUpload

func (c Core) AbortMultipartUpload(bucket, object, uploadID string) error

AbortMultipartUpload - Abort an incomplete upload.

func (Core) AbortMultipartUploadWithContext

func (c Core) AbortMultipartUploadWithContext(ctx context.Context, bucket, object, uploadID string) error

AbortMultipartUploadWithContext - Abort an incomplete upload.

func (Core) CompleteMultipartUpload

func (c Core) CompleteMultipartUpload(bucket, object, uploadID string, parts []CompletePart) (string, error)

CompleteMultipartUpload - Concatenate uploaded parts and commit to an object.

func (Core) CompleteMultipartUploadWithContext

func (c Core) CompleteMultipartUploadWithContext(ctx context.Context, bucket, object, uploadID string, parts []CompletePart) (string, error)

CompleteMultipartUploadWithContext - Concatenate uploaded parts and commit to an object.

func (Core) CopyObject

func (c Core) CopyObject(sourceBucket, sourceObject, destBucket, destObject string, metadata map[string]string) (ObjectInfo, error)

CopyObject - copies an object from source object to destination object on server side.

func (Core) CopyObjectPart

func (c Core) CopyObjectPart(srcBucket, srcObject, destBucket, destObject string, uploadID string,
	partID int, startOffset, length int64, metadata map[string]string) (p CompletePart, err error)

CopyObjectPart - creates a part in a multipart upload by copying (a part of) an existing object.

func (Core) CopyObjectPartWithContext

func (c Core) CopyObjectPartWithContext(ctx context.Context, srcBucket, srcObject, destBucket, destObject string, uploadID string,
	partID int, startOffset, length int64, metadata map[string]string) (p CompletePart, err error)

CopyObjectPartWithContext - creates a part in a multipart upload by copying (a part of) an existing object.

func (Core) CopyObjectWithContext

func (c Core) CopyObjectWithContext(ctx context.Context, sourceBucket, sourceObject, destBucket, destObject string, metadata map[string]string) (ObjectInfo, error)

CopyObjectWithContext - copies an object from source object to destination object on server side.

func (Core) GetBucketPolicy

func (c Core) GetBucketPolicy(bucket string) (string, error)

GetBucketPolicy - fetches bucket access policy for a given bucket.

func (Core) GetObject

func (c Core) GetObject(bucketName, objectName string, opts GetObjectOptions) (io.ReadCloser, ObjectInfo, http.Header, error)

GetObject is a lower level API implemented to support reading partial objects and also downloading objects with special conditions matching etag, modtime etc.

func (Core) GetObjectWithContext

func (c Core) GetObjectWithContext(ctx context.Context, bucketName, objectName string, opts GetObjectOptions) (io.ReadCloser, ObjectInfo, http.Header, error)

GetObjectWithContext is a lower level API implemented to support reading partial objects and also downloading objects with special conditions matching etag, modtime etc.

func (Core) ListMultipartUploads

func (c Core) ListMultipartUploads(bucket, prefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (result ListMultipartUploadsResult, err error)

ListMultipartUploads - List incomplete uploads.

func (Core) ListObjectParts

func (c Core) ListObjectParts(bucket, object, uploadID string, partNumberMarker int, maxParts int) (result ListObjectPartsResult, err error)

ListObjectParts - List uploaded parts of an incomplete upload.x

func (Core) ListObjects

func (c Core) ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (result ListBucketResult, err error)

ListObjects - List all the objects at a prefix, optionally with marker and delimiter you can further filter the results.

func (Core) ListObjectsV2

func (c Core) ListObjectsV2(bucketName, objectPrefix, continuationToken string, fetchOwner bool, delimiter string, maxkeys int, startAfter string) (ListBucketV2Result, error)

ListObjectsV2 - Lists all the objects at a prefix, similar to ListObjects() but uses continuationToken instead of marker to support iteration over the results.

func (Core) NewMultipartUpload

func (c Core) NewMultipartUpload(bucket, object string, opts PutObjectOptions) (uploadID string, err error)

NewMultipartUpload - Initiates new multipart upload and returns the new uploadID.

func (Core) PutBucketPolicy

func (c Core) PutBucketPolicy(bucket, bucketPolicy string) error

PutBucketPolicy - applies a new bucket access policy for a given bucket.

func (Core) PutObject

func (c Core) PutObject(bucket, object string, data io.Reader, size int64, md5Base64, sha256Hex string, metadata map[string]string, sse encrypt.ServerSide) (ObjectInfo, error)

PutObject - Upload object. Uploads using single PUT call.

func (Core) PutObjectPart

func (c Core) PutObjectPart(bucket, object, uploadID string, partID int, data io.Reader, size int64, md5Base64, sha256Hex string, sse encrypt.ServerSide) (ObjectPart, error)

PutObjectPart - Upload an object part.

func (Core) PutObjectPartWithContext

func (c Core) PutObjectPartWithContext(ctx context.Context, bucket, object, uploadID string, partID int, data io.Reader, size int64, md5Base64, sha256Hex string, sse encrypt.ServerSide) (ObjectPart, error)

PutObjectPartWithContext - Upload an object part.

func (Core) PutObjectWithContext

func (c Core) PutObjectWithContext(ctx context.Context, bucket, object string, data io.Reader, size int64, md5Base64, sha256Hex string, metadata map[string]string, sse encrypt.ServerSide) (ObjectInfo, error)

PutObjectWithContext - Upload object. Uploads using single PUT call.

func (Core) StatObject

func (c Core) StatObject(bucketName, objectName string, opts StatObjectOptions) (ObjectInfo, error)

StatObject is a lower level API implemented to support special conditions matching etag, modtime on a request.

func (Core) StatObjectWithContext

func (c Core) StatObjectWithContext(ctx context.Context, bucketName, objectName string, opts StatObjectOptions) (ObjectInfo, error)

StatObjectWithContext is a lower level API implemented to support special conditions matching etag, modtime on a request.

type DestinationInfo

type DestinationInfo struct {
	// contains filtered or unexported fields
}

DestinationInfo - type with information about the object to be created via server-side copy requests, using the Compose API.

func NewDestinationInfo

func NewDestinationInfo(bucket, object string, sse encrypt.ServerSide, userMeta map[string]string) (d DestinationInfo, err error)

NewDestinationInfo - creates a compose-object/copy-source destination info object.

`encSSEC` is the key info for server-side-encryption with customer provided key. If it is nil, no encryption is performed.

`userMeta` is the user-metadata key-value pairs to be set on the destination. The keys are automatically prefixed with `x-amz-meta-` if needed. If nil is passed, and if only a single source (of any size) is provided in the ComposeObject call, then metadata from the source is copied to the destination.

type Error

type Error struct {
	Command string
	Message string
	Code    int
}

func (*Error) Error

func (e *Error) Error() string

type ErrorResponse

type ErrorResponse struct {
	XMLName    xml.Name `xml:"Error" json:"-"`
	Code       string
	Message    string
	BucketName string
	Key        string
	RequestID  string `xml:"RequestId"`
	HostID     string `xml:"HostId"`

	// Region where the bucket is located. This header is returned
	// only in HEAD bucket and ListObjects response.
	Region string

	// Underlying HTTP status code for the returned error
	StatusCode int `xml:"-" json:"-"`
}

ErrorResponse - Is the typed error returned by all API operations. ErrorResponse struct should be comparable since it is compared inside golang http API (https://github.com/golang/go/issues/29768)

func ToErrorResponse

func ToErrorResponse(err error) ErrorResponse

ToErrorResponse - Returns parsed ErrorResponse struct from body and http headers.

For example:

import s3 "github.com/memoio/mefs-sdk-go"
...
...
reader, stat, err := s3.GetObject(...)
if err != nil {
   resp := s3.ToErrorResponse(err)
}
...

func (ErrorResponse) Error

func (e ErrorResponse) Error() string

Error - Returns S3 error string.

type Filter

type Filter struct {
	S3Key S3Key `xml:"S3Key,omitempty"`
}

Filter - a tag in the notification xml structure which carries suffix/prefix filters

type FilterRule

type FilterRule struct {
	Name  string `xml:"Name"`
	Value string `xml:"Value"`
}

FilterRule - child of S3Key, a tag in the notification xml which carries suffix/prefix filters

type GetBlockResult

type GetBlockResult struct {
	IsExist bool
}

type GetObjectOptions

type GetObjectOptions struct {
	ServerSideEncryption encrypt.ServerSide
	// contains filtered or unexported fields
}

GetObjectOptions are used to specify additional headers or options during GET requests.

func (GetObjectOptions) Header

func (o GetObjectOptions) Header() http.Header

Header returns the http.Header representation of the GET options.

func (*GetObjectOptions) Set

func (o *GetObjectOptions) Set(key, value string)

Set adds a key value pair to the options. The key-value pair will be part of the HTTP GET request headers.

func (*GetObjectOptions) SetMatchETag

func (o *GetObjectOptions) SetMatchETag(etag string) error

SetMatchETag - set match etag.

func (*GetObjectOptions) SetMatchETagExcept

func (o *GetObjectOptions) SetMatchETagExcept(etag string) error

SetMatchETagExcept - set match etag except.

func (*GetObjectOptions) SetModified

func (o *GetObjectOptions) SetModified(modTime time.Time) error

SetModified - set modified time since.

func (*GetObjectOptions) SetRange

func (o *GetObjectOptions) SetRange(start, end int64) error

SetRange - set the start and end offset of the object to be read. See https://tools.ietf.org/html/rfc7233#section-3.1 for reference.

func (*GetObjectOptions) SetUnmodified

func (o *GetObjectOptions) SetUnmodified(modTime time.Time) error

SetUnmodified - set unmodified time since.

type IdOutput

type IdOutput struct {
	ID              string
	PublicKey       string
	Addresses       []string
	AgentVersion    string
	ProtocolVersion string
}

type IntList

type IntList struct {
	ChildLists []int
}

func (IntList) String

func (fl IntList) String() string

type JSONInputOptions

type JSONInputOptions struct {
	Type JSONType
}

JSONInputOptions json input specific options

type JSONOutputOptions

type JSONOutputOptions struct {
	RecordDelimiter string
}

JSONOutputOptions - json output specific options

type JSONType

type JSONType string

JSONType determines json input serialization type.

const (
	JSONDocumentType JSONType = "DOCUMENT"
	JSONLinesType             = "LINES"
)

Constants for JSONTypes.

type LambdaConfig

type LambdaConfig struct {
	NotificationConfig
	Lambda string `xml:"CloudFunction"`
}

LambdaConfig carries one single cloudfunction notification configuration

type LfsOpts

type LfsOpts = func(*RequestBuilder) error

func ForceFlush

func ForceFlush(enabled bool) LfsOpts

func NeedAvailTime

func NeedAvailTime(enabled bool) LfsOpts

func SetAddress

func SetAddress(addr string) LfsOpts

func SetDataCount

func SetDataCount(dataCount int) LfsOpts

func SetObjectName

func SetObjectName(objectName string) LfsOpts

func SetParityCount

func SetParityCount(parityCount int) LfsOpts

func SetPassword

func SetPassword(pwd string) LfsOpts

func SetPolicy

func SetPolicy(policy int) LfsOpts

func SetPrefixFilter

func SetPrefixFilter(prefix string) LfsOpts

func SetSecretKey

func SetSecretKey(sk string) LfsOpts

type ListBucketResult

type ListBucketResult struct {
	// A response can contain CommonPrefixes only if you have
	// specified a delimiter.
	CommonPrefixes []CommonPrefix
	// Metadata about each object returned.
	Contents  []ObjectInfo
	Delimiter string

	// Encoding type used to encode object keys in the response.
	EncodingType string

	// A flag that indicates whether or not ListObjects returned all of the results
	// that satisfied the search criteria.
	IsTruncated bool
	Marker      string
	MaxKeys     int64
	Name        string

	// When response is truncated (the IsTruncated element value in
	// the response is true), you can use the key name in this field
	// as marker in the subsequent request to get next set of objects.
	// Object storage lists objects in alphabetical order Note: This
	// element is returned only if you have delimiter request
	// parameter specified. If response does not include the NextMaker
	// and it is truncated, you can use the value of the last Key in
	// the response as the marker in the subsequent request to get the
	// next set of object keys.
	NextMarker string
	Prefix     string
}

ListBucketResult container for listObjects response.

type ListBucketV2Result

type ListBucketV2Result struct {
	// A response can contain CommonPrefixes only if you have
	// specified a delimiter.
	CommonPrefixes []CommonPrefix
	// Metadata about each object returned.
	Contents  []ObjectInfo
	Delimiter string

	// Encoding type used to encode object keys in the response.
	EncodingType string

	// A flag that indicates whether or not ListObjects returned all of the results
	// that satisfied the search criteria.
	IsTruncated bool
	MaxKeys     int64
	Name        string

	// Hold the token that will be sent in the next request to fetch the next group of keys
	NextContinuationToken string

	ContinuationToken string
	Prefix            string

	// FetchOwner and StartAfter are currently not used
	FetchOwner string
	StartAfter string
}

ListBucketV2Result container for listObjects response version 2.

type ListMultipartUploadsResult

type ListMultipartUploadsResult struct {
	Bucket             string
	KeyMarker          string
	UploadIDMarker     string `xml:"UploadIdMarker"`
	NextKeyMarker      string
	NextUploadIDMarker string `xml:"NextUploadIdMarker"`
	EncodingType       string
	MaxUploads         int64
	IsTruncated        bool
	Uploads            []ObjectMultipartInfo `xml:"Upload"`
	Prefix             string
	Delimiter          string
	// A response can contain CommonPrefixes only if you specify a delimiter.
	CommonPrefixes []CommonPrefix
}

ListMultipartUploadsResult container for ListMultipartUploads response

type ListObjectPartsResult

type ListObjectPartsResult struct {
	Bucket   string
	Key      string
	UploadID string `xml:"UploadId"`

	Initiator initiator
	Owner     owner

	StorageClass         string
	PartNumberMarker     int
	NextPartNumberMarker int
	MaxParts             int

	// Indicates whether the returned list of parts is truncated.
	IsTruncated bool
	ObjectParts []ObjectPart `xml:"Part"`

	EncodingType string
}

ListObjectPartsResult container for ListObjectParts response.

type NotificationConfig

type NotificationConfig struct {
	ID     string                  `xml:"Id,omitempty"`
	Arn    Arn                     `xml:"-"`
	Events []NotificationEventType `xml:"Event"`
	Filter *Filter                 `xml:"Filter,omitempty"`
}

NotificationConfig - represents one single notification configuration such as topic, queue or lambda configuration.

func NewNotificationConfig

func NewNotificationConfig(arn Arn) NotificationConfig

NewNotificationConfig creates one notification config and sets the given ARN

func (*NotificationConfig) AddEvents

func (t *NotificationConfig) AddEvents(events ...NotificationEventType)

AddEvents adds one event to the current notification config

func (*NotificationConfig) AddFilterPrefix

func (t *NotificationConfig) AddFilterPrefix(prefix string)

AddFilterPrefix sets the prefix configuration to the current notification config

func (*NotificationConfig) AddFilterSuffix

func (t *NotificationConfig) AddFilterSuffix(suffix string)

AddFilterSuffix sets the suffix configuration to the current notification config

type NotificationEvent

type NotificationEvent struct {
	EventVersion      string            `json:"eventVersion"`
	EventSource       string            `json:"eventSource"`
	AwsRegion         string            `json:"awsRegion"`
	EventTime         string            `json:"eventTime"`
	EventName         string            `json:"eventName"`
	UserIdentity      identity          `json:"userIdentity"`
	RequestParameters map[string]string `json:"requestParameters"`
	ResponseElements  map[string]string `json:"responseElements"`
	S3                eventMeta         `json:"s3"`
	Source            sourceInfo        `json:"source"`
}

NotificationEvent represents an Amazon an S3 bucket notification event.

type NotificationEventType

type NotificationEventType string

NotificationEventType is a S3 notification event associated to the bucket notification configuration

type NotificationInfo

type NotificationInfo struct {
	Records []NotificationEvent
	Err     error
}

NotificationInfo - represents the collection of notification events, additionally also reports errors if any while listening on bucket notifications.

type Object

type Object struct {
	// contains filtered or unexported fields
}

Object represents an open object. It implements Reader, ReaderAt, Seeker, Closer for a HTTP stream.

func (*Object) Close

func (o *Object) Close() (err error)

Close - The behavior of Close after the first call returns error for subsequent Close() calls.

func (*Object) Read

func (o *Object) Read(b []byte) (n int, err error)

Read reads up to len(b) bytes into b. It returns the number of bytes read (0 <= n <= len(b)) and any error encountered. Returns io.EOF upon end of file.

func (*Object) ReadAt

func (o *Object) ReadAt(b []byte, offset int64) (n int, err error)

ReadAt reads len(b) bytes from the File starting at byte offset off. It returns the number of bytes read and the error, if any. ReadAt always returns a non-nil error when n < len(b). At end of file, that error is io.EOF.

func (*Object) Seek

func (o *Object) Seek(offset int64, whence int) (n int64, err error)

Seek sets the offset for the next Read or Write to offset, interpreted according to whence: 0 means relative to the origin of the file, 1 means relative to the current offset, and 2 means relative to the end. Seek returns the new offset and an error, if any.

Seeking to a negative offset is an error. Seeking to any positive offset is legal, subsequent io operations succeed until the underlying object is not closed.

func (*Object) Stat

func (o *Object) Stat() (ObjectInfo, error)

Stat returns the ObjectInfo structure describing Object.

type ObjectInfo

type ObjectInfo struct {
	// An ETag is optionally set to md5sum of an object.  In case of multipart objects,
	// ETag is of the form MD5SUM-N where MD5SUM is md5sum of all individual md5sums of
	// each parts concatenated into one string.
	ETag string `json:"etag"`

	Key          string    `json:"name"`         // Name of the object
	LastModified time.Time `json:"lastModified"` // Date and time the object was last modified.
	Size         int64     `json:"size"`         // Size in bytes of the object.
	ContentType  string    `json:"contentType"`  // A standard MIME type describing the format of the object data.
	Expires      time.Time `json:"expires"`      // The date and time at which the object is no longer able to be cached.

	// Collection of additional metadata on the object.
	// eg: x-amz-meta-*, content-encoding etc.
	Metadata http.Header `json:"metadata" xml:"-"`

	// Owner name.
	Owner struct {
		DisplayName string `json:"name"`
		ID          string `json:"id"`
	} `json:"owner"`

	// The class of storage used to store the object.
	StorageClass string `json:"storageClass"`

	// Error
	Err error `json:"-"`
}

ObjectInfo container for object metadata.

type ObjectMultipartInfo

type ObjectMultipartInfo struct {
	// Date and time at which the multipart upload was initiated.
	Initiated time.Time `type:"timestamp" timestampFormat:"iso8601"`

	Initiator initiator
	Owner     owner

	// The type of storage to use for the object. Defaults to 'STANDARD'.
	StorageClass string

	// Key of the object for which the multipart upload was initiated.
	Key string

	// Size in bytes of the object.
	Size int64

	// Upload ID that identifies the multipart upload.
	UploadID string `xml:"UploadId"`

	// Error
	Err error
}

ObjectMultipartInfo container for multipart object metadata.

type ObjectPart

type ObjectPart struct {
	// Part number identifies the part.
	PartNumber int

	// Date and time the part was uploaded.
	LastModified time.Time

	// Entity tag returned when the part was uploaded, usually md5sum
	// of the part.
	ETag string

	// Size of the uploaded part data.
	Size int64
}

ObjectPart container for particular part of an object.

type ObjectStat

type ObjectStat struct {
	ObjectName     string
	ObjectSize     int32
	MD5            string
	Ctime          string
	Dir            bool
	LatestChalTime string
}

type Objects

type Objects struct {
	Method  string
	Objects []ObjectStat
}

type Options

type Options struct {
	Creds        *credentials.Credentials
	Secure       bool
	Region       string
	BucketLookup BucketLookupType
}

Options for New method

type ParquetInputOptions

type ParquetInputOptions struct{}

ParquetInputOptions parquet input specific options

type PeerInfo

type PeerInfo struct {
	Addrs []string
	ID    string
}

type PeerList

type PeerList struct {
	Peers []PeerState
}

func (PeerList) String

func (pl PeerList) String() string

type PeerState

type PeerState struct {
	PeerID    string
	Connected bool
}

func (PeerState) String

func (ps PeerState) String() string

type PostPolicy

type PostPolicy struct {
	// contains filtered or unexported fields
}

PostPolicy - Provides strict static type conversion and validation for Amazon S3's POST policy JSON string.

func NewPostPolicy

func NewPostPolicy() *PostPolicy

NewPostPolicy - Instantiate new post policy.

func (*PostPolicy) SetBucket

func (p *PostPolicy) SetBucket(bucketName string) error

SetBucket - Sets bucket at which objects will be uploaded to.

func (*PostPolicy) SetContentLengthRange

func (p *PostPolicy) SetContentLengthRange(min, max int64) error

SetContentLengthRange - Set new min and max content length condition for all incoming uploads.

func (*PostPolicy) SetContentType

func (p *PostPolicy) SetContentType(contentType string) error

SetContentType - Sets content-type of the object for this policy based upload.

func (*PostPolicy) SetExpires

func (p *PostPolicy) SetExpires(t time.Time) error

SetExpires - Sets expiration time for the new policy.

func (*PostPolicy) SetKey

func (p *PostPolicy) SetKey(key string) error

SetKey - Sets an object name for the policy based upload.

func (*PostPolicy) SetKeyStartsWith

func (p *PostPolicy) SetKeyStartsWith(keyStartsWith string) error

SetKeyStartsWith - Sets an object name that an policy based upload can start with.

func (*PostPolicy) SetSuccessStatusAction

func (p *PostPolicy) SetSuccessStatusAction(status string) error

SetSuccessStatusAction - Sets the status success code of the object for this policy based upload.

func (*PostPolicy) SetUserData

func (p *PostPolicy) SetUserData(key string, value string) error

SetUserData - Set user data as a key/value couple. Can be retrieved through a HEAD request or an event.

func (*PostPolicy) SetUserMetadata

func (p *PostPolicy) SetUserMetadata(key string, value string) error

SetUserMetadata - Set user metadata as a key/value couple. Can be retrieved through a HEAD request or an event.

func (PostPolicy) String

func (p PostPolicy) String() string

Stringer interface for printing policy in json formatted string.

type ProgressMessage

type ProgressMessage struct {
	XMLName xml.Name `xml:"Progress" json:"-"`
	StatsMessage
}

ProgressMessage is a struct for progress xml message.

type PutObjectOptions

type PutObjectOptions struct {
	UserMetadata            map[string]string
	Progress                io.Reader
	ContentType             string
	ContentEncoding         string
	ContentDisposition      string
	ContentLanguage         string
	CacheControl            string
	ServerSideEncryption    encrypt.ServerSide
	NumThreads              uint
	StorageClass            string
	WebsiteRedirectLocation string
	PartSize                uint64
}

PutObjectOptions represents options specified by user for PutObject call

func (PutObjectOptions) Header

func (opts PutObjectOptions) Header() (header http.Header)

Header - constructs the headers from metadata entered by user in PutObjectOptions struct

type QueryEvent

type QueryEvent struct {
	ID        peer.ID
	Type      QueryEventType
	Responses []*peer.AddrInfo
	Extra     string
}

type QueryEventType

type QueryEventType int
const (
	SendingQuery QueryEventType = iota
	PeerResponse
	FinalPeer
	QueryError
	Provider
	Value
	AddingPeer
	DialingPeer
)

type QueryExpressionType

type QueryExpressionType string

QueryExpressionType - is of what syntax the expression is, this should only be SQL

const (
	QueryExpressionTypeSQL QueryExpressionType = "SQL"
)

Constants for expression type.

type QueueConfig

type QueueConfig struct {
	NotificationConfig
	Queue string `xml:"Queue"`
}

QueueConfig carries one single queue notification configuration

type RemoveObjectError

type RemoveObjectError struct {
	ObjectName string
	Err        error
}

RemoveObjectError - container of Multi Delete S3 API error

type Request

type Request struct {
	Ctx     context.Context
	ApiBase string
	Command string
	Args    []string
	Opts    map[string]string
	Body    io.Reader
	Headers map[string]string
}

func NewRequest

func NewRequest(ctx context.Context, url, command string, args ...string) *Request

func (*Request) Send

func (r *Request) Send(c *http.Client) (*Response, error)

type RequestBuilder

type RequestBuilder struct {
	// contains filtered or unexported fields
}

RequestBuilder is an IPFS commands request builder.

func (*RequestBuilder) Arguments

func (r *RequestBuilder) Arguments(args ...string) *RequestBuilder

Arguments adds the arguments to the args.

func (*RequestBuilder) Body

func (r *RequestBuilder) Body(body io.Reader) *RequestBuilder

Body sets the request body to the given reader.

func (*RequestBuilder) BodyBytes

func (r *RequestBuilder) BodyBytes(body []byte) *RequestBuilder

BodyBytes sets the request body to the given buffer.

func (*RequestBuilder) BodyString

func (r *RequestBuilder) BodyString(body string) *RequestBuilder

BodyString sets the request body to the given string.

func (*RequestBuilder) Exec

func (r *RequestBuilder) Exec(ctx context.Context, res interface{}) error

Exec sends the request a request and decodes the response.

func (*RequestBuilder) Header

func (r *RequestBuilder) Header(name, value string) *RequestBuilder

Header sets the given header.

func (*RequestBuilder) Option

func (r *RequestBuilder) Option(key string, value interface{}) *RequestBuilder

Option sets the given option.

func (*RequestBuilder) Send

func (r *RequestBuilder) Send(ctx context.Context) (*Response, error)

Send sends the request and return the response.

type Response

type Response struct {
	Output io.ReadCloser
	Error  *Error
}

func (*Response) Close

func (r *Response) Close() error

func (*Response) Decode

func (r *Response) Decode(dec interface{}) error

type S3Key

type S3Key struct {
	FilterRules []FilterRule `xml:"FilterRule,omitempty"`
}

S3Key - child of Filter, a tag in the notification xml which carries suffix/prefix filters

type SelectCompressionType

type SelectCompressionType string

SelectCompressionType - is the parameter for what type of compression is present

type SelectObjectInputSerialization

type SelectObjectInputSerialization struct {
	CompressionType SelectCompressionType
	Parquet         *ParquetInputOptions `xml:"Parquet,omitempty"`
	CSV             *CSVInputOptions     `xml:"CSV,omitempty"`
	JSON            *JSONInputOptions    `xml:"JSON,omitempty"`
}

SelectObjectInputSerialization - input serialization parameters

type SelectObjectOptions

type SelectObjectOptions struct {
	XMLName              xml.Name           `xml:"SelectObjectContentRequest" json:"-"`
	ServerSideEncryption encrypt.ServerSide `xml:"-"`
	Expression           string
	ExpressionType       QueryExpressionType
	InputSerialization   SelectObjectInputSerialization
	OutputSerialization  SelectObjectOutputSerialization
	RequestProgress      struct {
		Enabled bool
	}
}

SelectObjectOptions - represents the input select body

func (SelectObjectOptions) Header

func (o SelectObjectOptions) Header() http.Header

Header returns the http.Header representation of the SelectObject options.

type SelectObjectOutputSerialization

type SelectObjectOutputSerialization struct {
	CSV  *CSVOutputOptions  `xml:"CSV,omitempty"`
	JSON *JSONOutputOptions `xml:"JSON,omitempty"`
}

SelectObjectOutputSerialization - output serialization parameters.

type SelectObjectType

type SelectObjectType string

SelectObjectType - is the parameter which defines what type of object the operation is being performed on.

type SelectResults

type SelectResults struct {
	// contains filtered or unexported fields
}

SelectResults is used for the streaming responses from the server.

func NewSelectResults

func NewSelectResults(resp *http.Response, bucketName string) (*SelectResults, error)

NewSelectResults creates a Select Result parser that parses the response and returns a Reader that will return parsed and assembled select output.

func (*SelectResults) Close

func (s *SelectResults) Close() error

Close - closes the underlying response body and the stream reader.

func (*SelectResults) Progress

func (s *SelectResults) Progress() *ProgressMessage

Progress - information about the progress of a request.

func (*SelectResults) Read

func (s *SelectResults) Read(b []byte) (n int, err error)

Read - is a reader compatible implementation for SelectObjectContent records.

func (*SelectResults) Stats

func (s *SelectResults) Stats() *StatsMessage

Stats - information about a request's stats when processing is complete.

type SourceInfo

type SourceInfo struct {

	// Headers to send with the upload-part-copy request involving
	// this source object.
	Headers http.Header
	// contains filtered or unexported fields
}

SourceInfo - represents a source object to be copied, using server-side copying APIs.

func NewSourceInfo

func NewSourceInfo(bucket, object string, sse encrypt.ServerSide) SourceInfo

NewSourceInfo - create a compose-object/copy-object source info object.

`decryptSSEC` is the decryption key using server-side-encryption with customer provided key. It may be nil if the source is not encrypted.

func (*SourceInfo) SetMatchETagCond

func (s *SourceInfo) SetMatchETagCond(etag string) error

SetMatchETagCond - Set ETag match condition. The object is copied only if the etag of the source matches the value given here.

func (*SourceInfo) SetMatchETagExceptCond

func (s *SourceInfo) SetMatchETagExceptCond(etag string) error

SetMatchETagExceptCond - Set the ETag match exception condition. The object is copied only if the etag of the source is not the value given here.

func (*SourceInfo) SetModifiedSinceCond

func (s *SourceInfo) SetModifiedSinceCond(modTime time.Time) error

SetModifiedSinceCond - Set the modified since condition.

func (*SourceInfo) SetRange

func (s *SourceInfo) SetRange(start, end int64) error

SetRange - Set the start and end offset of the source object to be copied. If this method is not called, the whole source object is copied.

func (*SourceInfo) SetUnmodifiedSinceCond

func (s *SourceInfo) SetUnmodifiedSinceCond(modTime time.Time) error

SetUnmodifiedSinceCond - Set the unmodified since condition.

type StatObjectOptions

type StatObjectOptions struct {
	GetObjectOptions
}

StatObjectOptions are used to specify additional headers or options during GET info/stat requests.

type StatsMessage

type StatsMessage struct {
	XMLName        xml.Name `xml:"Stats" json:"-"`
	BytesScanned   int64
	BytesProcessed int64
	BytesReturned  int64
}

StatsMessage is a struct for stat xml message.

type StringList

type StringList struct {
	ChildLists []string
}

func (StringList) String

func (fl StringList) String() string

type SwarmConnInfo

type SwarmConnInfo struct {
	Addr    string
	Peer    string
	Latency string
	Muxer   string
	Streams []SwarmStreamInfo
}

type SwarmConnInfos

type SwarmConnInfos struct {
	Peers []SwarmConnInfo
}

type SwarmStreamInfo

type SwarmStreamInfo struct {
	Protocol string
}

type TopicConfig

type TopicConfig struct {
	NotificationConfig
	Topic string `xml:"Topic"`
}

TopicConfig carries one single topic notification configuration

type UserPrivMessage

type UserPrivMessage struct {
	Address string
	Sk      string
}

Directories

Path Synopsis
examples
pkg
credentials
Package credentials provides credential retrieval and management for S3 compatible object storage.
Package credentials provides credential retrieval and management for S3 compatible object storage.
set

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL