dynamodbcopy

README

dynamodbcopy

Dynamodbcopy is a cli tool wrapper around the aws-sdk that allows you to copy information from one dynamodb table to another.

Main Features

• Provides a CLI to easily copy dynamodb records from one place to another
• Allows you to set read and write capacity units for the source and target table
• Integrates with aws-sdk, sharing it's credentials
• Allows you to parameterize the source and target table with specific roles, enabling you to perform cross-account copies
• Stores current provisioning values before performing a copy, restoring the inital values at the end of the copy or if any error occurs during the copy.

Usage

Use "dynamodbcopy [command] --help" for more information about a command.

Installing

Use go get to retrieve dynamodbcopy to add it to your GOPATH workspace, or project's Go module dependencies.

go get github.com/uniplaces/dynamodbcopy/cmd/dynamodbcopy

To update run with -u

go get -u github.com/uniplaces/dynamodbcopy/cmd/dynamodbcopy

Go Modules

If you are using Go modules, your go get will default to the latest tagged version. To get a specific release version of the dynamodbcopy use @<tag> in your go get command.

go get github.com/uniplaces/dynamodbcopy/cmd/dynamodbcopy@v1.0.0

To get the latest repository change use @latest tag.

go get github.com/uniplaces/dynamodbcopy/cmd/dynamodbcopy@latest

Opening Issues

If you encounter a bug, please start by searching the existing issues and see if others are also experiencing the issue before opening a new one. Please include the version for dynamodbcopy and Go that you are using. Please also include reproduction case when appropriate.

Contributing

Please feel free to make suggestions, create issues, fork the repository and send pull requests!

Licence

Copyright 2018 UNIPLACES

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Documentation

Overview

Package dynamodbcopy provides a simple interface for you to copy info across dynamodb tables

Index

• func RandomSleeper(ms int) int
• type Capacity
• type Config
  • func NewConfig(readUnits, writeUnits, readWorkers, writeWorkers int) Config
  • func (c Config) Provisioning(current Provisioning) Provisioning
  • func (c Config) Workers() (int, int)
• type Copier
  • func NewCopier(srcTableService, trgTableService DynamoDBService, copierChan CopierChan, ...) Copier
• type CopierChan
  • func NewCopierChan(itemsChanSize int) CopierChan
• type DynamoDBClient
  • func NewDynamoClient(roleArn string) DynamoDBClient
• type DynamoDBItem
• type DynamoDBService
  • func NewDynamoDBService(tableName string, client DynamoDBClient, sleepFn Sleeper, logger Logger) DynamoDBService
• type Logger
  • func NewDebugLogger(logger Logger, debug bool) Logger
• type Provisioner
  • func NewProvisioner(srcTableService, trgTableService DynamoDBService, logger Logger) Provisioner
• type Provisioning
  • func NewProvisioning(srcDescription, trgDescription *dynamodb.TableDescription) Provisioning
• type Sleeper

Functions

func RandomSleeper

func RandomSleeper(ms int) int

RandomSleeper will sleep for the provided time (in ms) plus an additional random factor (ranging between [0, 100[). Returns the total sleep time

Types

type Capacity

type Capacity struct {
	Read  int64
	Write int64
}

Capacity abstracts the read and write units capacities values

type Config

type Config struct {
	// contains filtered or unexported fields
}

Config encapsulates the values nedeed for the command

func NewConfig

func NewConfig(readUnits, writeUnits, readWorkers, writeWorkers int) Config

NewConfig creates a new Config to store the parameters user defined parameters

func (Config) Provisioning

func (c Config) Provisioning(current Provisioning) Provisioning

Provisioning calculates a new Provisioning value based on the passed argument and the current Config (receiver). The returned Provisioning value will have the higher values for read and write capacity units of the 2

func (Config) Workers

func (c Config) Workers() (int, int)

Workers returns the Config read and write worker count

type Copier

type Copier interface {
	Copy(readers, writers int) error
}

Copier is the interface that allows you to copy records from the source to target table

func NewCopier

func NewCopier(srcTableService, trgTableService DynamoDBService, copierChan CopierChan, logger Logger) Copier

NewCopier returns a new Copier to copy records

type CopierChan

type CopierChan struct {
	Items  chan []DynamoDBItem
	Errors chan error
}

CopierChan encapsulates the value and error channel used by the copier

func NewCopierChan

func NewCopierChan(itemsChanSize int) CopierChan

NewCopierChan creates a new CopierChan with a buffered chan []DynamoDBItem of itemsChanSize

type DynamoDBClient

type DynamoDBClient interface {
	dynamodbiface.DynamoDBAPI
}

DynamoDBClient is a wrapper interface over aws-sdk dynamodbiface.DynamoDBClient for mocking purposes

func NewDynamoClient

func NewDynamoClient(roleArn string) DynamoDBClient

NewDynamoClient creates a DynamoDB client wrapper around the AWS-SDK with a predefined Session. By default, it creates a new Session with SharedConfigEnable, so you can use AWS SDK's environment variables and AWS credentials to connect to DynamoDB.

The provided ARN role allows you to configure the Session to assume a specific IAM Role.

If an empty string is provided, it will create a new session with SharedConfigEnable Please refer to https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html for more information on how you can set up the SDK

type DynamoDBItem

type DynamoDBItem map[string]*dynamodb.AttributeValue

DynamoDBItem type to abstract a DynamoDB item

type DynamoDBService

type DynamoDBService interface {
	DescribeTable() (*dynamodb.TableDescription, error)
	UpdateCapacity(capacity Capacity) error
	WaitForReadyTable() error
	BatchWrite(items []DynamoDBItem) error
	Scan(totalSegments, segment int, itemsChan chan<- []DynamoDBItem) error
}

DynamoDBService interface provides methods to call the aws sdk

func NewDynamoDBService

func NewDynamoDBService(tableName string, client DynamoDBClient, sleepFn Sleeper, logger Logger) DynamoDBService

NewDynamoDBService creates new service for a given DynamoDB table with a previously configured DynamoDB client

type Logger

type Logger interface {
	Printf(format string, msg ...interface{})
}

Logger defines the logging interface used by the command

func NewDebugLogger

func NewDebugLogger(logger Logger, debug bool) Logger

NewDebugLogger creates a wrapper around the argument logger to only log when debug flag is true

type Provisioner

type Provisioner interface {
	Fetch() (Provisioning, error)
	Update(provisioning Provisioning) (Provisioning, error)
}

Provisioner is the interface that provides the methods to manipulate DynamoDB's provisioning values

func NewProvisioner

func NewProvisioner(srcTableService, trgTableService DynamoDBService, logger Logger) Provisioner

NewProvisioner returns a new Provisioner to manipulate the source and target table provisioning values

type Provisioning

type Provisioning struct {
	Source *Capacity
	Target *Capacity
}

Provisioning stores the provisioning capacities for the source and target tables The Capacity for each table will be nil when the table's billing mode isn't BillingModeProvisioned

func NewProvisioning

func NewProvisioning(srcDescription, trgDescription *dynamodb.TableDescription) Provisioning

NewProvisioning creates a new Provisioning based on the source and target tables dynamodb.TableDescription It will only set capacity for each table if the is BillingModeProvisioned

type Sleeper

type Sleeper func(ms int) int

Sleeper abstracts out sleep side effects to allow for better testing, receiving the intended sleep time (in ms) and returning the total time spent sleeping