appencryption

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 17, 2021 License: MIT Imports: 11 Imported by: 5

README

Asherah - Go

Application level envelope encryption SDK for Golang with support for cloud-agnostic data storage and key management.

GoDoc

Quick Start

package main

import (
    "context"

    "github.com/godaddy/asherah/go/appencryption"
    "github.com/godaddy/asherah/go/appencryption/pkg/crypto/aead"
    "github.com/godaddy/asherah/go/appencryption/pkg/kms"
    "github.com/godaddy/asherah/go/appencryption/pkg/persistence"
)

func main() {
    crypto := aead.NewAES256GCM()
    config := &appencryption.Config{
        Service: "reference_app",
        Product: "productId",
        Policy:  appencryption.NewCryptoPolicy(),
    }
    metastore := persistence.NewMemoryMetastore()
    key, err := kms.NewStatic("thisIsAStaticMasterKeyForTesting", crypto)
    if err != nil {
        panic(err)
    }

    // Create a session factory. The builder steps used below are for testing only.
    factory := appencryption.NewSessionFactory(config, metastore, key, crypto)
    defer factory.Close()

    // Now create a cryptographic session for a partition.
    sess, err := factory.GetSession("shopper123")
    if err != nil {
        panic(err)
    }
    // Close frees the memory held by the intermediate keys used in this session
    defer sess.Close()

    // Now encrypt some data
    dataRow, err := sess.Encrypt(context.Background(), []byte("mysupersecretpayload"))
    if err != nil {
        panic(err)
    }

    //Decrypt the data
    data, err := sess.Decrypt(context.Background(), *dataRow)
    if err != nil {
        panic(err)
    }
}

A more extensive example is the Reference Application, which will evolve along with the SDK.

How to Use Asherah

Before you can start encrypting data, you need to define Asherah's required pluggable components. Below we show how to build the various options for each component.

Define the Metastore

Detailed information about the Metastore, including any provisioning steps, can be found here.

RDBMS Metastore

Asherah can connect to a relational database by accepting a connection string and opening a connection to the database using a database driver name. See https://golang.org/s/sqldrivers for a list of third-party drivers.

// Open a DB connection using a database driver name and connection string
connectionString := ...;

// Parse the DSN string to a Config
dsn, err := mysql.ParseDSN(connectionString)
if err != nil {
    return err
}

// Open a connection to the database using the driver name and the connection string
db, err := sql.Open("driver-name", dsn.FormatDSN())
if err != nil {
    return err
}

// Build the Metastore
metastore := persistence.NewSQLMetastore(db)
DynamoDB Metastore
awsConfig := &aws.Config{
    Region: aws.String("us-west-2"), // specify preferred region here
}

sess, err = session.NewSession(awsConfig)
if err != nil {
    panic(err)
}

// To configure an endpoint
awsConfig.Endpoint = aws.String("http://localhost:8000"),

You can also either use the WithXXX functional options to configure the metastore properties.

  • WithDynamoDBRegionSuffix: Specifies whether regional suffixes should be enabled for DynamoDB. Enabling this suffixes the keys with the DynamoDb preferred region. This is required to enable Global Tables.
  • WithTableName: Specifies the name of the DynamoDb table.
// Build the Metastore
metastore := persistence.NewDynamoDBMetastore(
    sess,
    persistence.WithDynamoDBRegionSuffix(true),
    persistence.WithTableName("CustomTableName") ,
)
In-memory Metastore (FOR TESTING ONLY)
metastore := persistence.NewMemoryMetastore()
Define the Key Management Service

Detailed information about the Key Management Service can be found here.

AWS KMS
// Create a map of region and ARN pairs that will all be used when creating a System Key
regionArnMap := map[string]string {
    "us-west-2": "ARN FOR US-WEST-2",
    "us-east-2": "ARN FOR US-EAST-2",
    "eu-west-2": "ARN FOR EU-WEST-2",
    ...,
}
crypto := aead.NewAES256GCM()

// Build the Key Management Service using the region dictionary and your preferred (usually current) region
keyManagementService :=  kms.NewAWS(crypto, "us-west-2", regionArnMap)
Static KMS (FOR TESTING ONLY)
crypto := aead.NewAES256GCM()
keyManagementService := kms.NewStatic("thisIsAStaticMasterKeyForTesting", crypto)
Define the Crypto Policy

Detailed information on Crypto Policy can be found here. The Crypto Policy's effect on key caching is explained here.

Basic Expiring Crypto Policy
cryptoPolicy := appencryption.NewCryptoPolicy()

The default key expiration limit is 90 days and revoke check interval is 60 minutes. These can be changed using functional options.

cryptoPolicy := appencryption.NewCryptoPolicy(
    appencryption.WithExpireAfterDuration(24 * time.Hour),
    appencryption.WithRevokeCheckInterval(30 * time.Minute))
(Optional) Enable Session Caching

Session caching is disabled by default. Enabling it is primarily useful if you are working with stateless workloads and the shared session can't be used by the calling app.

To enable session caching, simply use the WithSessionCache option when creating the crypto policy.

cryptoPolicy := appencryption.NewCryptoPolicy(
    appencryption.WithExpireAfterDuration(24 * time.Hour),
    appencryption.WithRevokeCheckInterval(30 * time.Minute),
    appencryption.WithSessionCache(),
    appencryption.SessionCacheMaxSize(200),
    appencryption.WithSessionCacheDuration(5 * time.Minute),
)
(Optional) Enable Metrics

Asherah's Go implementation uses go-metrics for metrics, which are enabled by default. If metrics are to be disabled, we simply use the WithMetrics functional option while creating the SessionFactory.

factory := NewSessionFactory(config, metastore, kms, crypto, WithMetrics(false))

The following metrics are available:

  • ael.drr.decrypt: Total time spent on all operations that were needed to decrypt.
  • ael.drr.encrypt: Total time spent on all operations that were needed to encrypt.
  • ael.kms.aws.decrypt.<region>: Time spent on decrypting the region-specific keys.
  • ael.kms.aws.decryptkey: Total time spend in decrypting the key which would include the region-specific decrypt calls in case of transient failures.
  • ael.kms.aws.encrypt.<region>: Time spent on data key plain text encryption for each region.
  • ael.kms.aws.encryptkey: Total time spent in encrypting the key which would include the region-specific generatedDataKey and parallel encrypt calls.
  • ael.kms.aws.generatedatakey.<region>: Time spent to generate the first data key which is then encrypted in remaining regions.
  • ael.metastore.sql.load: Time spent to load a record from sql metastore.
  • ael.metastore.sql.loadlatest: Time spent to get the latest record from sql metastore.
  • ael.metastore.sql.store: Time spent to store a record into sql metastore.
  • ael.metastore.dynamodb.load: Time spent to load a record from DynamoDB metastore.
  • ael.metastore.dynamodb.loadlatest: Time spent to get the latest record from DynamoDB metastore.
  • ael.metastore.dynamodb.store: Time spent to store a record into DynamoDB metastore.
Build a Session Factory

A session factory can now be built using the components we defined above.

sessionFactory := appencryption.NewSessionFactory(
    &appencryption.Config{
        Service: "reference_app",
        Product: "productId",
        Policy:  appencryption.NewCryptoPolicy(),
    },
    metastore,
    kms,
    crypto,
)

NOTE: We recommend that every service have its own session factory, preferably as a singleton instance within the service. This will allow you to leverage caching and minimize resource usage. Always remember to close the session factory before exiting the service to ensure that all resources held by the factory, including the cache, are disposed of properly.

Performing Cryptographic Operations

Create a session to be used for cryptographic operations.

sess, err := factory.GetSession("shopper123")
if err != nil {
    panic(err)
}
// Close frees the memory held by the intermediate keys used in this session
defer sess.Close()

NOTE: Remember to close the session after all cryptographic operations to dispose of associated resources.

Encrypt/Decrypt

This usage style is similar to common encryption utilities where payloads are simply encrypted and decrypted, and it is completely up to the calling application for storage responsibility.

originalPayloadString := "mysupersecretpayload";

// encrypt the payload
dataRowRecord, err := sess.Encrypt(context.Background(), []byte(originalPayloadString))
if err != nil {
    panic(err)
}

// decrypt the payload
decryptedPayload, err := sess.Decrypt(context.Background(), *dataRowRecord)
if err != nil {
    panic(err)
}
Custom Persistence via Store/Load methods

Asherah supports a key-value/document storage model. Use custom Storer and Loader implementations to hook into the session's Store and Load methods.

An example map-backed implementation:

type storage map[string][]byte

func (s storage) Store(_ context.Context, d appencryption.DataRowRecord) (interface{}, error) {
    b, err := json.Marshal(d)
    if err != nil {
      return nil, err
    }

    key := uuid.NewString()
    s[key] = b

    return key, nil
}

func (s storage) Load(_ context.Context, key string) (*appencryption.DataRowRecord, error) {
    var d appencryption.DataRowRecord
    err := json.Unmarshal(s[key], &d)

    return &d, err
}

An example end-to-end use of the session Store and Load calls:

mystore := make(storage)

originalPayloadData := []byte("mysupersecretpayload")

// Encrypts the payload and stores it in the map
key, err := sess.Store(context.Background(), originalPayloadData, mystore)
if err != nil {
    panic(err)
}

// Uses the persistenceKey to look-up the payload in the map, then decrypts the payload and returns it
payload, err := sess.Load(context.Background(), key, mystore)

Documentation

appencryption package: See the godocs for api documentation.

Development Notes

Unit Tests

Some unit tests will use the AWS SDK, If you don’t already have a local AWS credentials file, create a dummy file called ~/.aws/credentials with the below contents:

[default]
aws_access_key_id = foobar
aws_secret_access_key = barfoo

Alternately, you can set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

Documentation

Overview

Package appencryption contains the implementation to securely persist and encrypt data in the public cloud. Your main interaction with the library will most likely be the SessionFactory which should be created on application start up and stored for the lifetime of the app.

A session should closed as close as possible to the creation of the session. It should also be short lived to avoid running into the limits on the amount of memory that can be locked. See mlock documentation on how to set/check the current limits. It can also be checked using ulimit.

Index

Constants

View Source 
const (
	DefaultExpireAfter          = time.Hour * 24 * 90 // 90 days
	DefaultRevokedCheckInterval = time.Minute * 60
	DefaultCreateDatePrecision  = time.Minute
	DefaultSessionCacheMaxSize  = 1000
	DefaultSessionCacheDuration = time.Hour * 2
	DefaultSessionCacheEngine   = "default"
)

Default values for CryptoPolicy if not overridden.

View Source 
const AES256KeySize int = 32

AES256KeySize is the size of the AES key used by the AEAD implementation

View Source 
const MetricsPrefix = "ael"

MetricsPrefix prefixes all metrics names

Variables

This section is empty.

Functions

This section is empty.

Types

type AEAD 

type AEAD interface {
	// Encrypt encrypts data using the provided key bytes.
	Encrypt(data, key []byte) ([]byte, error)
	// Decrypt decrypts data using the provided key bytes.
	Decrypt(data, key []byte) ([]byte, error)
}

AEAD contains the functions required to encrypt and decrypt data using a specific cipher.

type Config 

type Config struct {
	// Service is the identifier for this service.
	Service string
	// Product is the identifier for the team or group that owns the calling service.
	Product string
	// Policy contains the information on when to expire keys.
	// If no policy is provided, 90 days rotations will be set
	// as defaults.
	Policy *CryptoPolicy
}

Config contains the required information to setup and use this library.

type CryptoPolicy 

type CryptoPolicy struct {
	// ExpireKeyAfter is used to determine when a key is considered expired based on its creation time
	// (regularly-scheduled rotation).
	ExpireKeyAfter time.Duration
	// RevokeCheckInterval controls the cache TTL (if caching is enabled) to check if a cached key has been marked as
	// revoked (irregularly-scheduled rotation).
	RevokeCheckInterval time.Duration
	// CreateDatePrecision is used to truncate a new key's creation timestamp to avoid concurrent callers from
	// excessively creating keys in race condition scenarios.
	CreateDatePrecision time.Duration
	// CacheIntermediateKeys determines whether Intermediate Keys will be cached.
	CacheIntermediateKeys bool
	// CacheSystemKeys determines whether System Keys will be cached.
	CacheSystemKeys bool
	// CacheSessions determines whether sessions will be cached.
	CacheSessions bool
	// SessionCacheMaxSize controls the maximum size of the cache if session caching is enabled.
	SessionCacheMaxSize int
	// SessionCacheDuration controls the amount of time a session will remain cached without being accessed
	// if session caching is enabled.
	SessionCacheDuration time.Duration
	// WithSessionCacheEngine determines the underlying cache implemenataion in use by the session cache
	// if session caching is enabled.
	//
	// Deprecated: multiple cache implementations are no longer supported and this option will be removed
	// in a future release.
	SessionCacheEngine string
}

CryptoPolicy contains options to customize various behaviors in the SDK.

func NewCryptoPolicy 

func NewCryptoPolicy(opts ...PolicyOption) *CryptoPolicy

NewCryptoPolicy returns a new CryptoPolicy with default values.

type DataRowRecord 

type DataRowRecord struct {
	Key  *EnvelopeKeyRecord
	Data []byte
}

DataRowRecord contains the encrypted key and provided data, as well as the information required to decrypt the key encryption key. This struct should be stored in your data persistence as it's required to decrypt data.

type Encryption 

type Encryption interface {
	// EncryptPayload encrypts a provided slice of bytes and returns a DataRowRecord which contains
	// required information to decrypt the data in the future.
	EncryptPayload(ctx context.Context, data []byte) (*DataRowRecord, error)
	// DecryptDataRowRecord decrypts a DataRowRecord key and returns the original byte slice
	// provided to the encrypt function.
	DecryptDataRowRecord(ctx context.Context, d DataRowRecord) ([]byte, error)
	// Close frees up any resources. It should be called as soon as an instance is
	// no longer in use.
	Close() error
}

Encryption implements the required methods to perform encryption/decryption on a payload

type EnvelopeKeyRecord 

type EnvelopeKeyRecord struct {
	Revoked       bool     `json:"Revoked,omitempty"`
	ID            string   `json:"-"`
	Created       int64    `json:"Created"`
	EncryptedKey  []byte   `json:"Key"`
	ParentKeyMeta *KeyMeta `json:"ParentKeyMeta,omitempty"`
}

EnvelopeKeyRecord represents an encrypted key and is the data structure used to persist the key in our key table. It also contains the meta data of the key used to encrypt it.

type FactoryOption 

type FactoryOption func(*SessionFactory)

FactoryOption is used to configure additional options in a SessionFactory.

func WithMetrics 

func WithMetrics(enabled bool) FactoryOption

WithMetrics enables or disables metrics.

func WithSecretFactory 

func WithSecretFactory(f securememory.SecretFactory) FactoryOption

WithSecretFactory sets the factory to use for creating Secrets

type KeyManagementService 

type KeyManagementService interface {
	// EncryptKey takes in an unencrypted byte slice and encrypts it with the master key.
	// The returned value should then be inserted into the Metastore before being
	// used.
	EncryptKey(context.Context, []byte) ([]byte, error)
	// DecryptKey decrypts the encrypted byte slice using the master key.
	DecryptKey(context.Context, []byte) ([]byte, error)
}

KeyManagementService contains the logic required to encrypt a system key with a master key.

type KeyMeta 

type KeyMeta struct {
	ID      string `json:"KeyId"`
	Created int64  `json:"Created"`
}

KeyMeta contains the ID and Created timestamp for an encryption key.

func (KeyMeta) String 

func (m KeyMeta) String() string

String returns a string with the KeyMeta values.

type Loader added in v0.1.6 

type Loader interface {
	// Load returns a DataRowRecord corresponding to the specified key, if found, along with any errors encountered.
	Load(ctx context.Context, key interface{}) (*DataRowRecord, error)
}

Loader declares the behavior for loading data from a persistence store.

type Metastore 

type Metastore interface {
	// Load retrieves a specific key by id and created timestamp.
	// The return value will be nil if not already present.
	Load(ctx context.Context, id string, created int64) (*EnvelopeKeyRecord, error)
	// LoadLatest returns the latest key matching the provided ID.
	// The return value will be nil if not already present.
	LoadLatest(ctx context.Context, id string) (*EnvelopeKeyRecord, error)
	// Store attempts to insert the key into the metastore if one is not
	// already present. If a key exists, the method will return false. If
	// one is not present, the value will be inserted and we return true.
	Store(ctx context.Context, id string, created int64, envelope *EnvelopeKeyRecord) (bool, error)
}

Metastore implements the required methods to retrieve an encryption key from it's storage.

type PolicyOption 

type PolicyOption func(*CryptoPolicy)

PolicyOption is used to configure a CryptoPolicy.

func WithExpireAfterDuration 

func WithExpireAfterDuration(d time.Duration) PolicyOption

WithExpireAfterDuration sets amount of time a key is considered valid.

func WithNoCache 

func WithNoCache() PolicyOption

WithNoCache disables caching of both System and Intermediate Keys.

func WithRevokeCheckInterval 

func WithRevokeCheckInterval(d time.Duration) PolicyOption

WithRevokeCheckInterval sets the interval to check for revoked keys in the cache.

func WithSessionCache added in v0.1.3 

func WithSessionCache() PolicyOption

WithSessionCache enables session caching. When used all sessions for a given partition will share underlying System and Intermediate Key caches.

func WithSessionCacheDuration added in v0.1.3 

func WithSessionCacheDuration(d time.Duration) PolicyOption

WithSessionCacheDuration specifies the amount of time a session will remain cached without being accessed if session caching is enabled.

func WithSessionCacheEngine deprecated added in v0.1.3 

func WithSessionCacheEngine(engine string) PolicyOption

WithSessionCacheEngine determines the underlying cache implemenataion in use by the session cache if session caching is enabled.

Deprecated: multiple cache implementations are no longer supported and this option will be removed in a future release.

func WithSessionCacheMaxSize added in v0.1.3 

func WithSessionCacheMaxSize(size int) PolicyOption

WithSessionCacheMaxSize specifies the session cache max size to use if session caching is enabled.

type Session 

type Session struct {
	// contains filtered or unexported fields
}

Session is used to encrypt and decrypt data related to a specific partition ID.

func (*Session) Close 

func (s *Session) Close() error

Close will close any open resources owned by this session (e.g. cache of keys). It should be called as soon as it's no longer in use.

func (*Session) Decrypt 

func (s *Session) Decrypt(ctx context.Context, d DataRowRecord) ([]byte, error)

Decrypt decrypts a DataRowRecord and returns the original byte slice provided to the encrypt function.

func (*Session) Encrypt 

func (s *Session) Encrypt(ctx context.Context, data []byte) (*DataRowRecord, error)

Encrypt encrypts a provided slice of bytes and returns a DataRowRecord, which contains required information to decrypt the data in the future.

func (*Session) Load added in v0.1.6 

func (s *Session) Load(ctx context.Context, key interface{}, store Loader) ([]byte, error)

Load uses a persistence key to load a DataRowRecord from the provided data persistence store, if any, and returns the decrypted payload.

func (*Session) Store added in v0.1.6 

func (s *Session) Store(ctx context.Context, payload []byte, store Storer) (interface{}, error)

Store encrypts a payload, stores the resulting DataRowRecord into the provided data persistence store. It returns the key that serves as a unique identifier for the stored payload as well as any error encountered along the way.

type SessionFactory 

type SessionFactory struct {
	Config        *Config
	Metastore     Metastore
	Crypto        AEAD
	KMS           KeyManagementService
	SecretFactory securememory.SecretFactory
	// contains filtered or unexported fields
}

SessionFactory is used to create new encryption sessions and manage the lifetime of the intermediate keys.

func NewSessionFactory 

func NewSessionFactory(config *Config, store Metastore, kms KeyManagementService, crypto AEAD, opts ...FactoryOption) *SessionFactory

NewSessionFactory creates a new session factory with default implementations.

func (*SessionFactory) Close 

func (f *SessionFactory) Close() error

Close will close any open resources owned by this factory (e.g. cache of system keys). It should be called when the factory is no longer required

func (*SessionFactory) GetSession 

func (f *SessionFactory) GetSession(id string) (*Session, error)

GetSession returns a new session for the provided partition ID.

type Storer added in v0.1.6 

type Storer interface {
	// Store persists the DataRowRecord and returns its associated key for future lookup (e.g. UUID, etc.).
	Store(ctx context.Context, d DataRowRecord) (interface{}, error)
}

Storer declares the behavior for storing data in a persistence store.

Source Files

View all Source files

Directories

Path Synopsis
cmd
example Module
pkg
crypto/aead
kms
log
Package log implements simple logging functionality with a focus on debug level logging.
 Package log implements simple logging functionality with a focus on debug level logging.
persistence
persistence/persistencetest

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.