Documentation
¶
Overview ¶
Package s3blob provides a blob implementation that uses S3. Use OpenBucket to construct a *blob.Bucket.
URLs ¶
For blob.OpenBucket, s3blob registers for the scheme "s3". The default URL opener will use an AWS session with the default credentials and configuration; see https://docs.aws.amazon.com/sdk-for-go/api/aws/session/ for more details. Use "awssdk=v1" or "awssdk=v2" to force a specific AWS SDK version. To customize the URL opener, or for more details on the URL format, To customize the URL opener, or for more details on the URL format, see URLOpener. See https://gocloud.dev/concepts/urls/ for background information.
Escaping ¶
Go CDK supports all UTF-8 strings; to make this work with services lacking full UTF-8 support, strings must be escaped (during writes) and unescaped (during reads). The following escapes are performed for s3blob:
- Blob keys: ASCII characters 0-31 are escaped to "__0x<hex>__". Additionally, the "/" in "../" and the trailing "/" in "//" are escaped in the same way.
- Metadata keys: Escaped using URL encoding, then additionally "@:=" are escaped using "__0x<hex>__". These characters were determined by experimentation.
- Metadata values: Escaped using URL encoding.
As ¶
s3blob exposes the following types for As:
- Bucket: (V1) *s3.S3; (V2) *s3v2.Client
- Error: (V1) awserr.Error; (V2) any error type returned by the service, notably smithy.APIError
- ListObject: (V1) s3.Object for objects, s3.CommonPrefix for "directories"; (V2) typesv2.Object for objects, typesv2.CommonPrefix for "directories
- ListOptions.BeforeList: (V1) *s3.ListObjectsV2Input, or *s3.ListObjectsInput when Options.UseLegacyList == true; (V2) *s3v2.ListObjectsV2Input, or *s3v2.ListObjectsInput when Options.UseLegacyList == true
- Reader: (V1) s3.GetObjectOutput; (V2) s3v2.GetObjectInput
- ReaderOptions.BeforeRead: (V1) *s3.GetObjectInput; (V2) *s3v2.GetObjectInput
- Attributes: (V1) s3.HeadObjectOutput; (V2)s3v2.HeadObjectOutput
- CopyOptions.BeforeCopy: *(V1) s3.CopyObjectInput; (V2) s3v2.CopyObjectInput
- WriterOptions.BeforeWrite: (V1) *s3manager.UploadInput, *s3manager.Uploader; (V2) *s3v2.PutObjectInput, *s3v2manager.Uploader
- SignedURLOptions.BeforeSign: (V1) *s3.GetObjectInput; (V2) *s3v2.GetObjectInput, when Options.Method == http.MethodGet, or (V1) *s3.PutObjectInput; (V2) *s3v2.PutObjectInput, when Options.Method == http.MethodPut, or (V1) *s3.DeleteObjectInput; (V2) [not supported] when Options.Method == http.MethodDelete
Example (OpenBucketFromURL) ¶
package main
import (
"context"
"log"
"gocloud.dev/blob"
)
func main() {
// PRAGMA: This example is used on gocloud.dev; PRAGMA comments adjust how it is shown and can be ignored.
// PRAGMA: On gocloud.dev, add a blank import: _ "gocloud.dev/blob/s3blob"
// PRAGMA: On gocloud.dev, hide lines until the next blank line.
ctx := context.Background()
// blob.OpenBucket creates a *blob.Bucket from a URL.
bucket, err := blob.OpenBucket(ctx, "s3://my-bucket?region=us-west-1")
if err != nil {
log.Fatal(err)
}
defer bucket.Close()
// Forcing AWS SDK V2.
bucket, err = blob.OpenBucket(ctx, "s3://my-bucket?region=us-west-1&awssdk=2")
if err != nil {
log.Fatal(err)
}
defer bucket.Close()
}
Output:
Index ¶
Examples ¶
Constants ¶
const Scheme = "s3"
Scheme is the URL scheme s3blob registers its URLOpener under on blob.DefaultMux.
Variables ¶
Set holds Wire providers for this package.
Functions ¶
func OpenBucket ¶
func OpenBucket(ctx context.Context, sess client.ConfigProvider, bucketName string, opts *Options) (*blob.Bucket, error)
OpenBucket returns a *blob.Bucket backed by S3. AWS buckets are bound to a region; sess must have been created using an aws.Config with Region set to the right region for bucketName. See the package documentation for an example.
Example ¶
package main
import (
"context"
"log"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"gocloud.dev/blob/s3blob"
)
func main() {
// PRAGMA: This example is used on gocloud.dev; PRAGMA comments adjust how it is shown and can be ignored.
// PRAGMA: On gocloud.dev, hide lines until the next blank line.
ctx := context.Background()
// Establish an AWS session.
// See https://docs.aws.amazon.com/sdk-for-go/api/aws/session/ for more info.
// The region must match the region for "my-bucket".
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-west-1"),
})
if err != nil {
log.Fatal(err)
}
// Create a *blob.Bucket.
bucket, err := s3blob.OpenBucket(ctx, sess, "my-bucket", nil)
if err != nil {
log.Fatal(err)
}
defer bucket.Close()
}
Output:
func OpenBucketV2 ¶ added in v0.25.0
func OpenBucketV2(ctx context.Context, client *s3v2.Client, bucketName string, opts *Options) (*blob.Bucket, error)
OpenBucketV2 returns a *blob.Bucket backed by S3, using AWS SDK v2.
Example ¶
package main
import (
"context"
"log"
awsv2cfg "github.com/aws/aws-sdk-go-v2/config"
s3v2 "github.com/aws/aws-sdk-go-v2/service/s3"
"gocloud.dev/blob/s3blob"
)
func main() {
// PRAGMA: This example is used on gocloud.dev; PRAGMA comments adjust how it is shown and can be ignored.
// PRAGMA: On gocloud.dev, hide lines until the next blank line.
// Establish a AWS V2 Config.
// See https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/ for more info.
ctx := context.Background()
cfg, err := awsv2cfg.LoadDefaultConfig(ctx)
if err != nil {
log.Fatal(err)
}
// Create a *blob.Bucket.
clientV2 := s3v2.NewFromConfig(cfg)
bucket, err := s3blob.OpenBucketV2(ctx, clientV2, "my-bucket", nil)
if err != nil {
log.Fatal(err)
}
defer bucket.Close()
}
Output:
Types ¶
type Options ¶
type Options struct {
// UseLegacyList forces the use of ListObjects instead of ListObjectsV2.
// Some S3-compatible services (like CEPH) do not currently support
// ListObjectsV2.
UseLegacyList bool
}
Options sets options for constructing a *blob.Bucket backed by fileblob.
type URLOpener ¶ added in v0.10.0
type URLOpener struct {
// UseV2 indicates whether the AWS SDK V2 should be used.
UseV2 bool
// ConfigProvider must be set to a non-nil value if UseV2 is false.
ConfigProvider client.ConfigProvider
// Options specifies the options to pass to OpenBucket.
Options Options
}
URLOpener opens S3 URLs like "s3://mybucket".
The URL host is used as the bucket name.
Use "awssdk=v1" to force using AWS SDK v1, "awssdk=v2" to force using AWS SDK v2, or anything else to accept the default.
For V1, see gocloud.dev/aws/ConfigFromURLParams for supported query parameters for overriding the aws.Session from the URL. For V2, see gocloud.dev/aws/V2ConfigFromURLParams.
Source Files