cloud

package module
v0.13.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 24, 2019 License: Apache-2.0 Imports: 0 Imported by: 1

README

The Go Cloud Development Kit (Go CDK)

Write once, run on any cloud ☁️

Build Status godoc Coverage

The Go Cloud Development Kit (Go CDK) allows Go application developers to seamlessly deploy cloud applications on any combination of cloud providers. It does this by providing stable, idiomatic interfaces for common uses like storage and databases. Think database/sql for cloud products.

Imagine writing this to read from blob storage (like Google Cloud Storage or S3):

ctx := context.Background()
bucket, err := blob.OpenBucket(ctx, "s3://my-bucket")
if err != nil {
    return err
}
defer bucket.Close()
blobReader, err := bucket.NewReader(ctx, "my-blob", nil)
if err != nil {
    return err
}

and being able to run that code on any cloud you want, avoiding all the ceremony of cloud-specific authorization, tracing, SDKs and all the other code required to make an application portable across cloud platforms.

The project works well with a code generator called Wire. It creates human-readable code that only imports the cloud SDKs for providers you use. This allows the Go CDK to grow to support any number of cloud providers, without increasing compile times or binary sizes, and avoiding any side effects from init() functions.

You can learn more about the project from our announcement blog post, or our talk at Next 2018:

Installation

# First "cd" into your project directory if you have one to ensure "go get" uses
# Go modules (or not) appropriately. See "go help modules" for more info.
go get gocloud.dev

The Go CDK builds at the latest stable release of Go. Previous Go versions may compile but are not supported.

Documentation

Documentation for the project lives primarily on https://gocloud.dev/, including tutorials.

You can also browse Go package reference on godoc.org.

Project status

The APIs are still in alpha, but we think they are production-ready and are actively looking for feedback from early adopters. If you have comments or questions, you can post to the go-cloud mailing list or email us at go-cdk-feedback@google.com.

Current features

The Go CDK provides generic APIs for:

  • Unstructured binary (blob) storage
  • Publish/Subscribe (pubsub)
  • Variables that change at runtime (runtimevar)
  • Connecting to MySQL and PostgreSQL databases (mysql, postgres)
  • Server startup and diagnostics: request logging, tracing, and health checking (server)

Contributing

Thank you for your interest in contributing to the Go Cloud Development Kit! ❤

Everyone is welcome to contribute, whether it's in the form of code, documentation, bug reports, feature requests, or anything else. We encourage you to experiment with the Go CDK and make contributions to help evolve it to meet your needs!

The GitHub repository at google/go-cloud contains some provider implementations for each portable API. We intend to include Google Cloud Platform, Amazon Web Services, and Azure implementations, as well as prominent open source providers and at least one implementation suitable for use in local testing. Unfortunately, we cannot support every cloud provider directly from the project; however, we encourage contributions in separate repositories.

If you create a repository that implements the Go CDK interfaces for other providers, let us know! We would be happy to link to it here and give you a heads-up before making any breaking changes.

See the contributing guide for more details.

Community

You can contact us on the go-cloud mailing list.

This project is covered by the Go Code of Conduct.

Documentation

Overview

Package cloud contains a library and tools for open cloud development in Go.

The Go Cloud Development Kit (Go CDK) allows application developers to seamlessly deploy cloud applications on any combination of cloud providers. It does this by providing stable, idiomatic interfaces for common uses like storage and databases. Think `database/sql` for cloud products.

At the core of the Go CDK are common "portable types" implemented by cloud providers. For example, objects of the blob.Bucket portable type can be created using gcsblob.OpenBucket, s3blob.OpenBucket, or any other provider. Then, the blob.Bucket can be used throughout your application without worrying about the underlying implementation.

The Go CDK works well with a code generator called Wire (https://github.com/google/wire/blob/master/README.md). It creates human-readable code that only imports the cloud SDKs for providers you use. This allows the Go CDK to grow to support any number of cloud providers, without increasing compile times or binary sizes, and avoiding any side effects from `init()` functions.

For non-reference documentation, see https://gocloud.dev/

URLs

In addition to creating portable types via provider-specific constructors (e.g., creating a blob.Bucket using s3blob.OpenBucket), many portable types can also be created using a URL. The scheme of the URL specifies the provider, and each provider implementation has code to convert the URL into the data needed to call its constructor. For example, calling blob.OpenBucket with s3blob://my-bucket will return a blob.Bucket created using s3blob.OpenBucket.

Each portable API package will document the types that it supports opening by URL; for example, the blob package supports Buckets, while the pubsub package supports Topics and Subscriptions. Each provider implementation will document what scheme(s) it registers for, and what format of URL it expects.

Each portable type URL opener will accept URL schemes with an <api>+ prefix (e.g., blob+file:///dir" instead of "file:///dir", as well as schemes with an <api>+<type>+ prefix (e.g., blob+bucket+file:///dir).

Each portable API package should include an example using a URL, and many providers will include provider-specific examples as well.

URL Muxes

Each portable type that's openable via URL will have a top-level function you can call, like blob.OpenBucket. This top-level function uses a default instance of an URLMux multiplexer to map schemes to a provider-specific opener for the type. For example, blob has a BucketURLOpener interface that providers implement and then register using RegisterBucket.

Many applications will work just fine using the default mux through the top-level Open functions. However, if you want more control, you can create your own URLMux and register the provider URLOpeners you need. Most providers will export URLOpeners that give you more fine grained control over the arguments needed by the constructor. In particular, portable types opened via URL will often use default credentials from the environment. For example, the AWS URL openers use the credentials saved by "aws login" (we don't want to include credentials in the URL itself, since they are likely to be sensitive).

  • Instantiate the provider's URLOpener with the specific fields you need. For example, s3blob.URLOpener{ConfigProvider: myAWSProvider} using a ConfigProvider that holds explicit AWS credentials.
  • Create your own instance of the URLMux, e.g., mymux := new(blob.URLMux).
  • Register your custom URLOpener on your mux, e.g., mymux.RegisterBucket(s3blob.Scheme, myS3URLOpener).
  • Now use your mux to open URLs, e.g. mymux.OpenBucket('s3://my-bucket').

Escaping the abstraction

It is not feasible or desirable for APIs like blob.Bucket to encompass the full functionality of every provider. Rather, we intend to provide a subset of the most commonly used functionality. There will be cases where a developer wants to access provider-specific functionality, such as unexposed APIs or data fields, errors or options. This can be accomplished using As functions.

As

As functions in the APIs provide the user a way to escape the Go CDK abstraction to access provider-specific types. They might be used as an interim solution until a feature request to the Go CDK is implemented. Or, the Go CDK may choose not to support specific features, and the use of As will be permanent.

Using As implies that the resulting code is no longer portable; the provider-specific code will need to be ported in order to switch providers. Therefore, it should be avoided if possible.

Each API will include examples demonstrating how to use its various As functions, and each provider implementation will document what types it supports for each.

Usage:

1. Declare a variable of the provider-specific type you want to access.

2. Pass a pointer to it to As.

3. If the type is supported, As will return true and copy the provider-specific type into your variable. Otherwise, it will return false.

Provider-specific types that are intended to be mutable will be exposed as a pointer to the underlying type.

Source Files

View all Source files

Directories

Path Synopsis
aws
Package aws provides fundamental Wire providers for Amazon Web Services (AWS).
 Package aws provides fundamental Wire providers for Amazon Web Services (AWS).
awscloud
Package awscloud contains Wire providers for AWS services.
 Package awscloud contains Wire providers for AWS services.
rds
Package rds contains Wire providers that are common across RDS.
 Package rds contains Wire providers that are common across RDS.
azure
azurecloud
Package azurecloud contains Wire providers for Azure services.
 Package azurecloud contains Wire providers for Azure services.
azuredb
Package azuredb contains Wire providers that are common across Azure Database.
 Package azuredb contains Wire providers that are common across Azure Database.
Package blob provides an easy and portable way to interact with blobs within a storage location, hereafter called a "bucket".
 Package blob provides an easy and portable way to interact with blobs within a storage location, hereafter called a "bucket".
azureblob
Package azureblob provides a blob implementation that uses Azure Storage’s BlockBlob.
 Package azureblob provides a blob implementation that uses Azure Storage’s BlockBlob.
driver
Package driver defines a set of interfaces that the blob package uses to interact with the underlying blob services.
 Package driver defines a set of interfaces that the blob package uses to interact with the underlying blob services.
drivertest
Package drivertest provides a conformance test for implementations of driver.
 Package drivertest provides a conformance test for implementations of driver.
fileblob
Package fileblob provides a blob implementation that uses the filesystem.
 Package fileblob provides a blob implementation that uses the filesystem.
gcsblob
Package gcsblob provides a blob implementation that uses GCS.
 Package gcsblob provides a blob implementation that uses GCS.
memblob
Package memblob provides an in-memory blob implementation.
 Package memblob provides an in-memory blob implementation.
s3blob
Package s3blob provides a blob implementation that uses S3.
 Package s3blob provides a blob implementation that uses S3.
docstore
mongodocstore Module
Package gcerrors provides support for getting error codes from errors returned by Go CDK APIs.
 Package gcerrors provides support for getting error codes from errors returned by Go CDK APIs.
gcp
Package gcp provides fundamental Wire providers and types for Google Cloud Platform (GCP).
 Package gcp provides fundamental Wire providers and types for Google Cloud Platform (GCP).
cloudsql
Package cloudsql contains Wire providers that are common across Google Cloud SQL.
 Package cloudsql contains Wire providers that are common across Google Cloud SQL.
gcpcloud
Package gcpcloud contains Wire providers for GCP services.
 Package gcpcloud contains Wire providers for GCP services.
Package health provides health check handlers.
 Package health provides health check handlers.
sqlhealth
Package sqlhealth provides a health check for a SQL database connection.
 Package sqlhealth provides a health check for a SQL database connection.
internal
batcher
Package batcher supports batching of items.
 Package batcher supports batching of items.
docstore
Package docstore provides a portable implementation of a document store.
 Package docstore provides a portable implementation of a document store.
docstore/driver
Package driver defines a set of interfaces that the docstore package uses to interact with the underlying services.
 Package driver defines a set of interfaces that the docstore package uses to interact with the underlying services.
docstore/drivertest
Package drivertest provides a conformance test for implementations of driver.
 Package drivertest provides a conformance test for implementations of driver.
docstore/dynamodocstore
Package dynamodocstore provides a docstore implementation backed by AWS DynamoDB.
 Package dynamodocstore provides a docstore implementation backed by AWS DynamoDB.
docstore/firedocstore
Package firedocstore provides an implementation of the docstore API for Google Cloud Firestore.
 Package firedocstore provides an implementation of the docstore API for Google Cloud Firestore.
docstore/internal/fields
Package fields provides a view of the fields of a struct that follows the Go rules, amended to consider tags and case insensitivity.
 Package fields provides a view of the fields of a struct that follows the Go rules, amended to consider tags and case insensitivity.
docstore/memdocstore
Package memdocstore provides an in-memory implementation of the docstore API.
 Package memdocstore provides an in-memory implementation of the docstore API.
docstore/mongodocstore
Package mongodocstore provides an implementation of the docstore API for MongoDB.
 Package mongodocstore provides an implementation of the docstore API for MongoDB.
escape
Package escape includes helpers for escaping and unescaping strings.
 Package escape includes helpers for escaping and unescaping strings.
gcerr
Package gcerr provides an error type for Go CDK APIs.
 Package gcerr provides an error type for Go CDK APIs.
oc
Package oc supports OpenCensus tracing and metrics for the Go Cloud Development Kit.
 Package oc supports OpenCensus tracing and metrics for the Go Cloud Development Kit.
openurl
Package openurl provides helpers for URLMux and URLOpeners in portable APIs.
 Package openurl provides helpers for URLMux and URLOpeners in portable APIs.
retry
Package retry provides retry logic.
 Package retry provides retry logic.
testing/octest
Package octest supports testing of OpenCensus integrations.
 Package octest supports testing of OpenCensus integrations.
testing/replay
Package replay provides the ability to record and replay HTTP requests.
 Package replay provides the ability to record and replay HTTP requests.
testing/setup
testing/terraform
Package terraform provides a function to read Terraform output.
 Package terraform provides a function to read Terraform output.
trace
Package trace provides support for OpenCensus tracing.
 Package trace provides support for OpenCensus tracing.
useragent
Package useragent includes constants and utilitiesfor setting the User-Agent for Go CDK connections to GCP.
 Package useragent includes constants and utilitiesfor setting the User-Agent for Go CDK connections to GCP.
cmd/gocdk Module
contributebot Module
Package mysql provides functions to open MySQL databases with OpenCensus instrumentation.
 Package mysql provides functions to open MySQL databases with OpenCensus instrumentation.
azuremysql
Package azuremysql provides connections to Azure Database for MySQL.
 Package azuremysql provides connections to Azure Database for MySQL.
cloudmysql
Package cloudmysql provides connections to managed MySQL Cloud SQL instances.
 Package cloudmysql provides connections to managed MySQL Cloud SQL instances.
rdsmysql
Package rdsmysql provides connections to AWS RDS MySQL instances.
 Package rdsmysql provides connections to AWS RDS MySQL instances.
Package postgres provides functions to open PostgreSQL databases with OpenCensus instrumentation.
 Package postgres provides functions to open PostgreSQL databases with OpenCensus instrumentation.
cloudpostgres
Package cloudpostgres provides connections to managed PostgreSQL Cloud SQL instances.
 Package cloudpostgres provides connections to managed PostgreSQL Cloud SQL instances.
rdspostgres
Package rdspostgres provides connections to AWS RDS PostgreSQL instances.
 Package rdspostgres provides connections to AWS RDS PostgreSQL instances.
Package pubsub provides an easy and portable way to interact with publish/ subscribe systems.
 Package pubsub provides an easy and portable way to interact with publish/ subscribe systems.
awssnssqs
Package awssnssqs provides an implementation of pubsub that uses AWS SNS (Simple Notification Service) and SQS (Simple Queueing Service).
 Package awssnssqs provides an implementation of pubsub that uses AWS SNS (Simple Notification Service) and SQS (Simple Queueing Service).
azuresb
Package azuresb provides an implementation of pubsub using Azure Service Bus Topic and Subscription.
 Package azuresb provides an implementation of pubsub using Azure Service Bus Topic and Subscription.
driver
Package driver defines a set of interfaces that the pubsub package uses to interact with the underlying pubsub services.
 Package driver defines a set of interfaces that the pubsub package uses to interact with the underlying pubsub services.
drivertest
Package drivertest provides a conformance test for implementations of driver.
 Package drivertest provides a conformance test for implementations of driver.
gcppubsub
Package gcppubsub provides a pubsub implementation that uses GCP PubSub.
 Package gcppubsub provides a pubsub implementation that uses GCP PubSub.
mempubsub
Package mempubsub provides an in-memory pubsub implementation.
 Package mempubsub provides an in-memory pubsub implementation.
natspubsub
Package natspubsub provides a pubsub implementation for NATS.io.
 Package natspubsub provides a pubsub implementation for NATS.io.
rabbitpubsub
Package rabbitpubsub provides an pubsub implementation for RabbitMQ.
 Package rabbitpubsub provides an pubsub implementation for RabbitMQ.
kafkapubsub Module
Package requestlog provides an http.Handler that logs information about requests.
 Package requestlog provides an http.Handler that logs information about requests.
Package runtimevar provides an easy and portable way to watch runtime configuration variables.
 Package runtimevar provides an easy and portable way to watch runtime configuration variables.
awsparamstore
Package awsparamstore provides a runtimevar implementation with variables read from AWS Systems Manager Parameter Store (https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-paramstore.html) Use OpenVariable to construct a *runtimevar.Variable.
 Package awsparamstore provides a runtimevar implementation with variables read from AWS Systems Manager Parameter Store (https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-paramstore.html) Use OpenVariable to construct a *runtimevar.Variable.
blobvar
Package blobvar provides a runtimevar implementation with variables read from a blob.Bucket.
 Package blobvar provides a runtimevar implementation with variables read from a blob.Bucket.
constantvar
Package constantvar provides a runtimevar implementation with Variables that never change.
 Package constantvar provides a runtimevar implementation with Variables that never change.
driver
Package driver provides the interface for providers of runtimevar.
 Package driver provides the interface for providers of runtimevar.
drivertest
Package drivertest provides a conformance test for implementations of runtimevar.
 Package drivertest provides a conformance test for implementations of runtimevar.
etcdvar
Package etcdvar provides a runtimevar implementation with variables backed by etcd.
 Package etcdvar provides a runtimevar implementation with variables backed by etcd.
filevar
Package filevar provides a runtimevar implementation with variables backed by the filesystem.
 Package filevar provides a runtimevar implementation with variables backed by the filesystem.
gcpruntimeconfig
Package gcpruntimeconfig provides a runtimevar implementation with variables read from GCP Cloud Runtime Configurator (https://cloud.google.com/deployment-manager/runtime-configurator).
 Package gcpruntimeconfig provides a runtimevar implementation with variables read from GCP Cloud Runtime Configurator (https://cloud.google.com/deployment-manager/runtime-configurator).
httpvar
Package httpvar provides a runtimevar implementation with variables backed by http endpoint.
 Package httpvar provides a runtimevar implementation with variables backed by http endpoint.
samples
gocdk-blob
gocdk-blob demonstrates the use of the Go CDK blob package in a simple command-line application.
 gocdk-blob demonstrates the use of the Go CDK blob package in a simple command-line application.
gocdk-pubsub
gocdk-pubsub demonstrates the use of the Go CDK pubsub package in a simple command-line application.
 gocdk-pubsub demonstrates the use of the Go CDK pubsub package in a simple command-line application.
gocdk-runtimevar
gocdk-runtimevar demonstrates the use of the Go CDK runtimevar package in a simple command-line application.
 gocdk-runtimevar demonstrates the use of the Go CDK runtimevar package in a simple command-line application.
gocdk-secrets
gocdk-secrets demonstrates the use of the Go CDK secrets package in a simple command-line application.
 gocdk-secrets demonstrates the use of the Go CDK secrets package in a simple command-line application.
guestbook
guestbook is a sample application that records visitors' messages, displays a cloud banner, and an administrative message.
 guestbook is a sample application that records visitors' messages, displays a cloud banner, and an administrative message.
guestbook/aws/provision_db
The provision_db program connects to an RDS database and initializes it with SQL from stdin.
 The provision_db program connects to an RDS database and initializes it with SQL from stdin.
guestbook/gcp/deploy
The deploy program builds the Guestbook server locally and deploys it to GKE.
 The deploy program builds the Guestbook server locally and deploys it to GKE.
guestbook/gcp/provision_db
The provision_db program connects to a Cloud SQL database and initializes it with SQL from a file.
 The provision_db program connects to a Cloud SQL database and initializes it with SQL from a file.
guestbook/localdb
server
Command server runs a simple HTTP server with integrated Stackdriver tracing and health checks.
 Command server runs a simple HTTP server with integrated Stackdriver tracing and health checks.
tutorial
Command upload saves files to blob storage on GCP, AWS, and Azure.
 Command upload saves files to blob storage on GCP, AWS, and Azure.
Package secrets provides an easy and portable way to encrypt and decrypt messages.
 Package secrets provides an easy and portable way to encrypt and decrypt messages.
awskms
Package awskms provides a secrets implementation backed by AWS KMS.
 Package awskms provides a secrets implementation backed by AWS KMS.
azurekeyvault
Package azurekeyvault provides a secrets implementation backed by Azure KeyVault.
 Package azurekeyvault provides a secrets implementation backed by Azure KeyVault.
driver
Package driver defines interfaces to be implemented for providers of the secrets package.
 Package driver defines interfaces to be implemented for providers of the secrets package.
drivertest
Package drivertest provides a conformance test for implementations of the secrets driver.
 Package drivertest provides a conformance test for implementations of the secrets driver.
gcpkms
Package gcpkms provides a secrets implementation backed by Google Cloud KMS.
 Package gcpkms provides a secrets implementation backed by Google Cloud KMS.
localsecrets
Package localsecrets provides a secrets implementation using a locally locally provided symmetric key.
 Package localsecrets provides a secrets implementation using a locally locally provided symmetric key.
vault
Package vault provides a secrets implementation using the Transit Secrets Engine of Vault by Hashicorp.
 Package vault provides a secrets implementation using the Transit Secrets Engine of Vault by Hashicorp.
hashivault Module
Package server provides a preconfigured HTTP server with diagnostic hooks.
 Package server provides a preconfigured HTTP server with diagnostic hooks.
driver
Package driver defines an interface for custom HTTP listeners.
 Package driver defines an interface for custom HTTP listeners.
sdserver
Package sdserver provides the diagnostic hooks for a server using Stackdriver.
 Package sdserver provides the diagnostic hooks for a server using Stackdriver.
xrayserver
Package xrayserver provides the diagnostic hooks for a server using AWS X-Ray.
 Package xrayserver provides the diagnostic hooks for a server using AWS X-Ray.
tests
aws/app
gcp/app
The app command is a test app that is initialized with the GCP SDK.
 The app command is a test app that is initialized with the GCP SDK.
internal/testutil
Package testutil contains utility functions used by server tests against different platforms.
 Package testutil contains utility functions used by server tests against different platforms.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.