Documentation
¶
Overview ¶
Package s3blob provides a blob implementation that uses S3. Use OpenBucket to construct a *blob.Bucket.
URLs ¶
For blob.OpenBucket, s3blob registers for the scheme "s3". The default URL opener will use an AWS session with the default credentials and configuration; see https://docs.aws.amazon.com/sdk-for-go/api/aws/session/ for more details. Use "awssdk=v1" or "awssdk=v2" to force a specific AWS SDK version. To customize the URL opener, or for more details on the URL format, see URLOpener. See https://gocloud.dev/concepts/urls/ for background information.
Escaping ¶
Go CDK supports all UTF-8 strings; to make this work with services lacking full UTF-8 support, strings must be escaped (during writes) and unescaped (during reads). The following escapes are performed for s3blob:
- Blob keys: ASCII characters 0-31 are escaped to "__0x<hex>__". Additionally, the "/" in "../" and the trailing "/" in "//" are escaped in the same way.
- Metadata keys: Escaped using URL encoding, then additionally "@:=" are escaped using "__0x<hex>__". These characters were determined by experimentation.
- Metadata values: Escaped using URL encoding.
As ¶
s3blob exposes the following types for As:
- Bucket: (V1) *s3.S3; (V2) *s3v2.Client
- Error: (V1) awserr.Error; (V2) any error type returned by the service, notably smithy.APIError
- ListObject: (V1) s3.Object for objects, s3.CommonPrefix for "directories"; (V2) typesv2.Object for objects, typesv2.CommonPrefix for "directories
- ListOptions.BeforeList: (V1) *s3.ListObjectsV2Input, or *s3.ListObjectsInput when Options.UseLegacyList == true; (V2) *s3v2.ListObjectsV2Input, or *s3v2.ListObjectsInput when Options.UseLegacyList == true
- Reader: (V1) s3.GetObjectOutput; (V2) s3v2.GetObjectInput
- ReaderOptions.BeforeRead: (V1) *s3.GetObjectInput; (V2) *s3v2.GetObjectInput
- Attributes: (V1) s3.HeadObjectOutput; (V2)s3v2.HeadObjectOutput
- CopyOptions.BeforeCopy: *(V1) s3.CopyObjectInput; (V2) s3v2.CopyObjectInput
- WriterOptions.BeforeWrite: (V1) *s3manager.UploadInput, *s3manager.Uploader; (V2) *s3v2.PutObjectInput, *s3v2manager.Uploader
- SignedURLOptions.BeforeSign: (V1) *s3.GetObjectInput; (V2) *s3v2.GetObjectInput, when Options.Method == http.MethodGet, or (V1) *s3.PutObjectInput; (V2) *s3v2.PutObjectInput, when Options.Method == http.MethodPut, or (V1) *s3.DeleteObjectInput; (V2) [not supported] when Options.Method == http.MethodDelete
Example (OpenBucketFromURL) ¶
package main
import (
"context"
"log"
"gocloud.dev/blob"
)
func main() {
// PRAGMA: This example is used on gocloud.dev; PRAGMA comments adjust how it is shown and can be ignored.
// PRAGMA: On gocloud.dev, add a blank import: _ "gocloud.dev/blob/s3blob"
// PRAGMA: On gocloud.dev, hide lines until the next blank line.
ctx := context.Background()
// blob.OpenBucket creates a *blob.Bucket from a URL.
bucket, err := blob.OpenBucket(ctx, "s3://my-bucket?region=us-west-1")
if err != nil {
log.Fatal(err)
}
defer bucket.Close()
// Forcing AWS SDK V2.
bucket, err = blob.OpenBucket(ctx, "s3://my-bucket?region=us-west-1&awssdk=2")
if err != nil {
log.Fatal(err)
}
defer bucket.Close()
}
Output:
Index ¶
Examples ¶
Constants ¶
const Scheme = "s3"
Scheme is the URL scheme s3blob registers its URLOpener under on blob.DefaultMux.
Variables ¶
Set holds Wire providers for this package.
Functions ¶
func OpenBucket ¶
func OpenBucket(ctx context.Context, sess client.ConfigProvider, bucketName string, opts *Options) (*blob.Bucket, error)
OpenBucket returns a *blob.Bucket backed by S3. AWS buckets are bound to a region; sess must have been created using an aws.Config with Region set to the right region for bucketName. See the package documentation for an example.
Example ¶
package main
import (
"context"
"log"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"gocloud.dev/blob/s3blob"
)
func main() {
// PRAGMA: This example is used on gocloud.dev; PRAGMA comments adjust how it is shown and can be ignored.
// PRAGMA: On gocloud.dev, hide lines until the next blank line.
ctx := context.Background()
// Establish an AWS session.
// See https://docs.aws.amazon.com/sdk-for-go/api/aws/session/ for more info.
// The region must match the region for "my-bucket".
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-west-1"),
})
if err != nil {
log.Fatal(err)
}
// Create a *blob.Bucket.
bucket, err := s3blob.OpenBucket(ctx, sess, "my-bucket", nil)
if err != nil {
log.Fatal(err)
}
defer bucket.Close()
}
Output:
func OpenBucketV2 ¶ added in v0.25.0
func OpenBucketV2(ctx context.Context, client *s3v2.Client, bucketName string, opts *Options) (*blob.Bucket, error)
OpenBucketV2 returns a *blob.Bucket backed by S3, using AWS SDK v2.
Example ¶
package main
import (
"context"
"log"
awsv2cfg "github.com/aws/aws-sdk-go-v2/config"
s3v2 "github.com/aws/aws-sdk-go-v2/service/s3"
"gocloud.dev/blob/s3blob"
)
func main() {
// PRAGMA: This example is used on gocloud.dev; PRAGMA comments adjust how it is shown and can be ignored.
// PRAGMA: On gocloud.dev, hide lines until the next blank line.
// Establish a AWS V2 Config.
// See https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/ for more info.
ctx := context.Background()
cfg, err := awsv2cfg.LoadDefaultConfig(ctx)
if err != nil {
log.Fatal(err)
}
// Create a *blob.Bucket.
clientV2 := s3v2.NewFromConfig(cfg)
bucket, err := s3blob.OpenBucketV2(ctx, clientV2, "my-bucket", nil)
if err != nil {
log.Fatal(err)
}
defer bucket.Close()
}
Output:
Types ¶
type Options ¶
type Options struct {
// UseLegacyList forces the use of ListObjects instead of ListObjectsV2.
// Some S3-compatible services (like CEPH) do not currently support
// ListObjectsV2.
UseLegacyList bool
}
Options sets options for constructing a *blob.Bucket backed by fileblob.
type URLOpener ¶ added in v0.10.0
type URLOpener struct {
// UseV2 indicates whether the AWS SDK V2 should be used.
UseV2 bool
// ConfigProvider must be set to a non-nil value if UseV2 is false.
ConfigProvider client.ConfigProvider
// Options specifies the options to pass to OpenBucket.
Options Options
}
URLOpener opens S3 URLs like "s3://mybucket".
The URL host is used as the bucket name.
Use "awssdk=v1" to force using AWS SDK v1, "awssdk=v2" to force using AWS SDK v2, or anything else to accept the default.
For V1, see gocloud.dev/aws/ConfigFromURLParams for supported query parameters for overriding the aws.Session from the URL. For V2, see gocloud.dev/aws/V2ConfigFromURLParams.
Source Files