sdk

README

IBM Cloud Object Storage - Go SDK

This package allows Go developers to write software that interacts with IBM Cloud Object Storage. It is a fork of the AWS SDK for Go library and can stand as a drop-in replacement if the application needs to connect to object storage using an S3-like API and does not make use of other AWS services.

Notice

IBM has added a Language Support Policy. Language versions will be deprecated on the published schedule without additional notice.

Documentation

• Core documentation for IBM COS
• Go API reference documentation
• REST API reference documentation

For release notes, see the CHANGELOG.

• Getting the SDK
• Example code
• Getting help

Quick start

You'll need:

• An instance of COS.
• An API key from IBM Cloud Identity and Access Management with at least Writer permissions.
• The ID of the instance of COS that you are working with.
• Token acquisition endpoint
• Service endpoint

These values can be found in the IBM Cloud Console by generating a 'service credential'.

Archive Tier Support

You can automatically archive objects after a specified length of time or after a specified date. Once archived, a temporary copy of an object can be restored for access as needed. Restore time may take up to 15 hours.

An archive policy is set at the bucket level by calling the PutBucketLifecycleConfiguration method on a client instance. A newly added or modified archive policy applies to new objects uploaded and does not affect existing objects. For more detail, see the documentation.

Immutable Object Storage

Users can configure buckets with an Immutable Object Storage policy to prevent objects from being modified or deleted for a defined period of time. The retention period can be specified on a per-object basis, or objects can inherit a default retention period set on the bucket. It is also possible to set open-ended and permanent retention periods. Immutable Object Storage meets the rules set forth by the SEC governing record retention, and IBM Cloud administrators are unable to bypass these restrictions. For more detail, see the IBM Cloud documentation.

Note: Immutable Object Storage does not support Aspera transfers via the SDK to upload objects or directories at this stage.

Accelerated Archive

Users can set an archive rule that would allow data restore from an archive in 2 hours or 12 hours.

Getting the SDK

Use go build to add the SDK to your project's Go module dependencies. The SDK requires a minimum version of Go 1.12.

go build ./...

Example code

Create a file main.go, replacing your own values for API key, instance ID, and bucket name:

package main

import (
 "fmt"

 "github.com/IBM/ibm-cos-sdk-go/aws"
 "github.com/IBM/ibm-cos-sdk-go/aws/credentials/ibmiam"
 "github.com/IBM/ibm-cos-sdk-go/aws/session"
 "github.com/IBM/ibm-cos-sdk-go/service/s3"
)

const (
 apiKey            = "<API_KEY>"
 serviceInstanceID = "<RESOURCE_INSTANCE_ID>"
 authEndpoint      = "https://iam.cloud.ibm.com/identity/token"
 serviceEndpoint   = "https://s3-api.us-geo.objectstorage.softlayer.net"
)

func main() {

 newBucket := "new-bucketee"
 newColdBucket := "new-cold-bucketee"

 conf := aws.NewConfig().
  WithEndpoint(serviceEndpoint).
  WithCredentials(ibmiam.NewStaticCredentials(aws.NewConfig(),
   authEndpoint, apiKey, serviceInstanceID)).
  WithS3ForcePathStyle(true)

 sess := session.Must(session.NewSession())
 client := s3.New(sess, conf)

 input := &s3.CreateBucketInput{
  Bucket: aws.String(newBucket),
 }
 client.CreateBucket(input)

 input2 := &s3.CreateBucketInput{
  Bucket: aws.String(newColdBucket),
  CreateBucketConfiguration: &s3.CreateBucketConfiguration{
   LocationConstraint: aws.String("us-cold"),
  },
 }
 client.CreateBucket(input2)

 d, _ := client.ListBuckets(&s3.ListBucketsInput{})
 fmt.Println(d)
}

From the command line, run go run main.go. You should see a list of your buckets.

Getting Help

Feel free to use GitHub issues for tracking bugs and feature requests, but for help please use one of the following resources:

• Read a quick start guide in IBM Cloud Docs.
• Ask a question on Stack Overflow and tag it with ibm and object-storage.
• Open a support ticket with IBM Cloud Support
• If it turns out that you may have found a bug, please open an issue.

Language Support Policy

IBM supports current public releases. IBM will deprecate language versions 90 days after a version reaches end-of-life. All clients will need to upgrade to a supported version before the end of the grace period.

License

This SDK is distributed under the Apache License, Version 2.0, see LICENSE.txt and NOTICE.txt for more information.

Documentation

Overview

Package sdk is the official AWS SDK for the Go programming language.

The AWS SDK for Go provides APIs and utilities that developers can use to build Go applications that use AWS services, such as Amazon Simple Storage Service (Amazon S3).

The SDK removes the complexity of coding directly against a web service interface. It hides a lot of the lower-level plumbing, such as authentication, request retries, and error handling.

The SDK also includes helpful utilities on top of the AWS APIs that add additional capabilities and functionality. For example, the Amazon S3 Download and Upload Manager will automatically split up large objects into multiple parts and transfer them concurrently.

See the s3manager package documentation for more information. https://docs.aws.amazon.com/sdk-for-go/api/service/s3/s3manager/

Getting More Information

Checkout the Getting Started Guide and API Reference Docs detailed the SDK's components and details on each AWS client the SDK supports.

The Getting Started Guide provides examples and detailed description of how to get setup with the SDK. https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/welcome.html

The API Reference Docs include a detailed breakdown of the SDK's components such as utilities and AWS clients. Use this as a reference of the Go types included with the SDK, such as AWS clients, API operations, and API parameters. https://docs.aws.amazon.com/sdk-for-go/api/

Overview of SDK's Packages

The SDK is composed of two main components, SDK core, and service clients. The SDK core packages are all available under the aws package at the root of the SDK. Each client for a supported AWS service is available within its own package under the service folder at the root of the SDK.

• aws - SDK core, provides common shared types such as Config, Logger, and utilities to make working with API parameters easier.

• awserr - Provides the error interface that the SDK will use for all errors that occur in the SDK's processing. This includes service API response errors as well. The Error type is made up of a code and message. Cast the SDK's returned error type to awserr.Error and call the Code method to compare returned error to specific error codes. See the package's documentation for additional values that can be extracted such as RequestId.

• credentials - Provides the types and built in credentials providers the SDK will use to retrieve AWS credentials to make API requests with. Nested under this folder are also additional credentials providers such as stscreds for assuming IAM roles, and ec2rolecreds for EC2 Instance roles.

• endpoints - Provides the AWS Regions and Endpoints metadata for the SDK. Use this to lookup AWS service endpoint information such as which services are in a region, and what regions a service is in. Constants are also provided for all region identifiers, e.g UsWest2RegionID for "us-west-2".

• session - Provides initial default configuration, and load configuration from external sources such as environment and shared credentials file.

• request - Provides the API request sending, and retry logic for the SDK. This package also includes utilities for defining your own request retryer, and configuring how the SDK processes the request.

• service - Clients for AWS services. All services supported by the SDK are available under this folder.

How to Use the SDK's AWS Service Clients

The SDK includes the Go types and utilities you can use to make requests to AWS service APIs. Within the service folder at the root of the SDK you'll find a package for each AWS service the SDK supports. All service clients follows a common pattern of creation and usage.

When creating a client for an AWS service you'll first need to have a Session value constructed. The Session provides shared configuration that can be shared between your service clients. When service clients are created you can pass in additional configuration via the aws.Config type to override configuration provided by in the Session to create service client instances with custom configuration.

Once the service's client is created you can use it to make API requests the AWS service. These clients are safe to use concurrently.

Configuring the SDK

In the AWS SDK for Go, you can configure settings for service clients, such as the log level and maximum number of retries. Most settings are optional; however, for each service client, you must specify a region and your credentials. The SDK uses these values to send requests to the correct AWS region and sign requests with the correct credentials. You can specify these values as part of a session or as environment variables.

See the SDK's configuration guide for more information. https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html

See the session package documentation for more information on how to use Session with the SDK. https://docs.aws.amazon.com/sdk-for-go/api/aws/session/

See the Config type in the aws package for more information on configuration options. https://docs.aws.amazon.com/sdk-for-go/api/aws/#Config

Configuring Credentials

When using the SDK you'll generally need your AWS credentials to authenticate with AWS services. The SDK supports multiple methods of supporting these credentials. By default the SDK will source credentials automatically from its default credential chain. See the session package for more information on this chain, and how to configure it. The common items in the credential chain are the following:

• Environment Credentials - Set of environment variables that are useful when sub processes are created for specific roles.

• Shared Credentials file (~/.aws/credentials) - This file stores your credentials based on a profile name and is useful for local development.

Credentials can be configured in code as well by setting the Config's Credentials value to a custom provider or using one of the providers included with the SDK to bypass the default credential chain and use a custom one. This is helpful when you want to instruct the SDK to only use a specific set of credentials or providers.

This example creates a credential provider for assuming an IAM role, "myRoleARN" and configures the S3 service client to use that role for API requests.

// Initial credentials loaded from SDK's default credential chain. Such as
// the environment, shared credentials (~/.aws/credentials), or EC2 Instance
// Role. These credentials will be used to to make the STS Assume Role API.
sess := session.Must(session.NewSession())

// Create service client value configured for credentials
// from assumed role.
svc := s3.New(sess, &aws.Config{Credentials: creds})/

The SDK has support for the shared configuration file (~/.aws/config). This support can be enabled by setting the environment variable, "AWS_SDK_LOAD_CONFIG=1", or enabling the feature in code when creating a Session via the Option's SharedConfigState parameter.

sess := session.Must(session.NewSessionWithOptions(session.Options{
    SharedConfigState: session.SharedConfigEnable,
}))

Configuring AWS Region

In addition to the credentials you'll need to specify the region the SDK will use to make AWS API requests to. In the SDK you can specify the region either with an environment variable, or directly in code when a Session or service client is created. The last value specified in code wins if the region is specified multiple ways.

To set the region via the environment variable set the "AWS_REGION" to the region you want to the SDK to use. Using this method to set the region will allow you to run your application in multiple regions without needing additional code in the application to select the region.

AWS_REGION=us-west-2

The endpoints package includes constants for all regions the SDK knows. The values are all suffixed with RegionID. These values are helpful, because they reduce the need to type the region string manually.

To set the region on a Session use the aws package's Config struct parameter Region to the AWS region you want the service clients created from the session to use. This is helpful when you want to create multiple service clients, and all of the clients make API requests to the same region.

sess := session.Must(session.NewSession(&aws.Config{
    Region: aws.String(endpoints.UsWest2RegionID),
}))

In addition to setting the region when creating a Session you can also set the region on a per service client bases. This overrides the region of a Session. This is helpful when you want to create service clients in specific regions different from the Session's region.

svc := s3.New(sess, &aws.Config{
    Region: aws.String(endpoints.UsWest2RegionID),
})

See the Config type in the aws package for more information and additional options such as setting the Endpoint, and other service client configuration options. https://docs.aws.amazon.com/sdk-for-go/api/aws/#Config

Making API Requests

Once the client is created you can make an API request to the service. Each API method takes a input parameter, and returns the service response and an error. The SDK provides methods for making the API call in multiple ways.

In this list we'll use the S3 ListObjects API as an example for the different ways of making API requests.

• ListObjects - Base API operation that will make the API request to the service.

• ListObjectsRequest - API methods suffixed with Request will construct the API request, but not send it. This is also helpful when you want to get a presigned URL for a request, and share the presigned URL instead of your application making the request directly.

• ListObjectsPages - Same as the base API operation, but uses a callback to automatically handle pagination of the API's response.

• ListObjectsWithContext - Same as base API operation, but adds support for the Context pattern. This is helpful for controlling the canceling of in flight requests. See the Go standard library context package for more information. This method also takes request package's Option functional options as the variadic argument for modifying how the request will be made, or extracting information from the raw HTTP response.

• ListObjectsPagesWithContext - same as ListObjectsPages, but adds support for the Context pattern. Similar to ListObjectsWithContext this method also takes the request package's Option function option types as the variadic argument.

In addition to the API operations the SDK also includes several higher level methods that abstract checking for and waiting for an AWS resource to be in a desired state. In this list we'll use WaitUntilBucketExists to demonstrate the different forms of waiters.

• WaitUntilBucketExists. - Method to make API request to query an AWS service for a resource's state. Will return successfully when that state is accomplished.

• WaitUntilBucketExistsWithContext - Same as WaitUntilBucketExists, but adds support for the Context pattern. In addition these methods take request package's WaiterOptions to configure the waiter, and how underlying request will be made by the SDK.

The API method will document which error codes the service might return for the operation. These errors will also be available as const strings prefixed with "ErrCode" in the service client's package. If there are no errors listed in the API's SDK documentation you'll need to consult the AWS service's API documentation for the errors that could be returned.

ctx := context.Background()

result, err := svc.GetObjectWithContext(ctx, &s3.GetObjectInput{
    Bucket: aws.String("my-bucket"),
    Key: aws.String("my-key"),
})
if err != nil {
    // Cast err to awserr.Error to handle specific error codes.
    aerr, ok := err.(awserr.Error)
    if ok && aerr.Code() == s3.ErrCodeNoSuchKey {
        // Specific error code handling
    }
    return err
}

// Make sure to close the body when done with it for S3 GetObject APIs or
// will leak connections.
defer result.Body.Close()

fmt.Println("Object Size:", aws.StringValue(result.ContentLength))

API Request Pagination and Resource Waiters

Pagination helper methods are suffixed with "Pages", and provide the functionality needed to round trip API page requests. Pagination methods take a callback function that will be called for each page of the API's response.

objects := []string{}
err := svc.ListObjectsPagesWithContext(ctx, &s3.ListObjectsInput{
    Bucket: aws.String(myBucket),
}, func(p *s3.ListObjectsOutput, lastPage bool) bool {
    for _, o := range p.Contents {
        objects = append(objects, aws.StringValue(o.Key))
    }
    return true // continue paging
})
if err != nil {
    panic(fmt.Sprintf("failed to list objects for bucket, %s, %v", myBucket, err))
}

fmt.Println("Objects in bucket:", objects)

Waiter helper methods provide the functionality to wait for an AWS resource state. These methods abstract the logic needed to to check the state of an AWS resource, and wait until that resource is in a desired state. The waiter will block until the resource is in the state that is desired, an error occurs, or the waiter times out. If a resource times out the error code returned will be request.WaiterResourceNotReadyErrorCode.

err := svc.WaitUntilBucketExistsWithContext(ctx, &s3.HeadBucketInput{
    Bucket: aws.String(myBucket),
})
if err != nil {
    aerr, ok := err.(awserr.Error)
    if ok && aerr.Code() == request.WaiterResourceNotReadyErrorCode {
        fmt.Fprintf(os.Stderr, "timed out while waiting for bucket to exist")
    }
    panic(fmt.Errorf("failed to wait for bucket to exist, %v", err))
}
fmt.Println("Bucket", myBucket, "exists")

Complete SDK Example

This example shows a complete working Go file which will upload a file to S3 and use the Context pattern to implement timeout logic that will cancel the request if it takes too long. This example highlights how to use sessions, create a service client, make a request, handle the error, and process the response.

 package main

 import (
 	"context"
 	"flag"
 	"fmt"
 	"os"
 	"time"

 	"github.com/IBM/ibm-cos-sdk-go/aws"
 	"github.com/IBM/ibm-cos-sdk-go/aws/awserr"
 	"github.com/IBM/ibm-cos-sdk-go/aws/request"
 	"github.com/IBM/ibm-cos-sdk-go/aws/session"
 	"github.com/IBM/ibm-cos-sdk-go/service/s3"
 )

 // Uploads a file to S3 given a bucket and object key. Also takes a duration
 // value to terminate the update if it doesn't complete within that time.
 //
 // The AWS Region needs to be provided in the AWS shared config or on the
 // environment variable as `AWS_REGION`. Credentials also must be provided
 // Will default to shared config file, but can load from environment if provided.
 //
 // Usage:
 //   # Upload myfile.txt to myBucket/myKey. Must complete within 10 minutes or will fail
 //   go run withContext.go -b mybucket -k myKey -d 10m < myfile.txt
 func main() {
 	var bucket, key string
 	var timeout time.Duration

 	flag.StringVar(&bucket, "b", "", "Bucket name.")
 	flag.StringVar(&key, "k", "", "Object key name.")
 	flag.DurationVar(&timeout, "d", 0, "Upload timeout.")
 	flag.Parse()

 	// All clients require a Session. The Session provides the client with
	// shared configuration such as region, endpoint, and credentials. A
	// Session should be shared where possible to take advantage of
	// configuration and credential caching. See the session package for
	// more information.
 	sess := session.Must(session.NewSession())

	// Create a new instance of the service's client with a Session.
	// Optional aws.Config values can also be provided as variadic arguments
	// to the New function. This option allows you to provide service
	// specific configuration.
 	svc := s3.New(sess)

 	// Create a context with a timeout that will abort the upload if it takes
 	// more than the passed in timeout.
 	ctx := context.Background()
 	var cancelFn func()
 	if timeout > 0 {
 		ctx, cancelFn = context.WithTimeout(ctx, timeout)
 	}
 	// Ensure the context is canceled to prevent leaking.
 	// See context package for more information, https://golang.org/pkg/context/
 	defer cancelFn()

 	// Uploads the object to S3. The Context will interrupt the request if the
 	// timeout expires.
 	_, err := svc.PutObjectWithContext(ctx, &s3.PutObjectInput{
 		Bucket: aws.String(bucket),
 		Key:    aws.String(key),
 		Body:   os.Stdin,
 	})
 	if err != nil {
 		if aerr, ok := err.(awserr.Error); ok && aerr.Code() == request.CanceledErrorCode {
 			// If the SDK can determine the request or retry delay was canceled
 			// by a context the CanceledErrorCode error code will be returned.
 			fmt.Fprintf(os.Stderr, "upload canceled due to timeout, %v\n", err)
 		} else {
 			fmt.Fprintf(os.Stderr, "failed to upload object, %v\n", err)
 		}
 		os.Exit(1)
 	}

 	fmt.Printf("successfully uploaded file to %s/%s\n", bucket, key)
 }