dynamoathenamigration

README

Dynamomigration tool

Dynamomigration tool allows to export Teleport audit events logs from DynamoDB table into Athena Audit log. It's using DynamoDB export to S3 to export data.

Requirements:

• Point-in-time recovery (PITR) on DynamoDB table
• Writable filesystem on machine where script will be executed
• IAM permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowDynamoExportAndList",
            "Effect": "Allow",
            "Action": [
                "dynamodb:ExportTableToPointInTime"
            ],
            "Resource": "arn:aws:dynamodb:region:account:table/tablename"
        },
        {
            "Sid": "AllowDynamoExportDescribe",
            "Effect": "Allow",
            "Action": [
                "dynamodb:DescribeExport"
            ],
            "Resource": "arn:aws:dynamodb:region:account:table/tablename/*"
        },
        {
            "Sid": "AllowWriteReadDestinationBucket",
            "Effect": "Allow",
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::export-bucket/*"
        },
        {
            "Sid": "AllowWriteLargePayloadsBucket",
            "Effect": "Allow",
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::large-payloads-bucket/*"
        },
        {
            "Sid": "AllowPublishToAthenaTopic",
            "Effect": "Allow",
            "Action": [
                "sns:Publish"
            ],
            "Resource": "arn:aws:sns:region:account:topicname"
        }
    ]
}

Example usage

Build: cd examples/dynamoathenamigration/cmd && go build -o dynamoathenamigration.

It is recommended to test export first using -dryRun flag. DryRun does not emit any events, it makes sure that export is in valid format and events can be parsed.

Dry run example:

./dynamoathenamigration -dynamoARN='arn:aws:dynamodb:region:account:table/tablename' \
  -exportPath='s3://bucket/prefix' \
  -dryRun

Full migration:

./dynamoathenamigration -dynamoARN='arn:aws:dynamodb:region:account:table/tablename' \
  -exportPath='s3://bucket/prefix' \
  -snsTopicARN=arn:aws:sns:region:account:topicname \
  -largePayloadsPath=s3://bucket/prefix

To reuse existing export without triggering new one, use -exportARN=xxx.

Documentation

Index

• Constants
• func Migrate(ctx context.Context, cfg Config) error
• func MigrateWithAWS(ctx context.Context, cfg Config, awsCfg aws.Config) error
• type Config
  • func (cfg *Config) CheckAndSetDefaults() error

Constants

const (
	DefaultMaxMemoryUsedForSortingExportInMB = 500
)

Functions

func Migrate

func Migrate(ctx context.Context, cfg Config) error

Migrate executed dynamodb -> athena migration.

func MigrateWithAWS

func MigrateWithAWS(ctx context.Context, cfg Config, awsCfg aws.Config) error

MigrateWithAWS executed dynamodb -> athena migration. Provide your own awsCfg

Types

type Config

type Config struct {
	// ExportTime is time in the past from which to export table data.
	ExportTime time.Time

	// ExportARN allows to use already finished export without triggering new.
	ExportARN string

	// DynamoTableARN that will be exported.
	DynamoTableARN string

	// ExportLocalDir specifies where export files will be downloaded (it must exists).
	// If empty os.TempDir() will be used.
	ExportLocalDir string

	// MaxMemoryUsedForSortingExportInMB (MB) is used to define how large amount of events
	// will be loaded into memory when doing sorting of events before publishing it.
	MaxMemoryUsedForSortingExportInMB int

	// Bucket used to store export.
	Bucket string
	// Prefix is s3 prefix where to store export inside bucket.
	Prefix string

	// DryRun allows to generate export and convert it to AuditEvents.
	// Nothing is published to athena publisher.
	// Can be used to test if export is valid.
	DryRun bool

	// NoOfEmitWorkers defines how many workers are used to emit audit events.
	NoOfEmitWorkers int

	// CheckpointPath is full path of file where checkpoint data should be stored.
	// Defaults to file in current directory (athenadynamomigration.json)
	// Checkpoint allow to resume export which failed during emitting.
	CheckpointPath string

	// TopicARN is topic of athena logger.
	TopicARN string
	// LargePayloadBucket is s3 bucket configured for large payloads in athena logger.
	LargePayloadBucket string
	// LargePayloadPrefix is s3 prefix configured for large payloads in athena logger.
	LargePayloadPrefix string

	Logger log.FieldLogger
	// contains filtered or unexported fields
}

func (*Config) CheckAndSetDefaults

func (cfg *Config) CheckAndSetDefaults() error