README
¶
go-ds-dynamodb
A DynamoDB Datastore Implementation
This is an implementation of go-datastore that is backed by DynamoDB.
ddbds includes support for optimized prefix queries. When you setup your table's key schema correctly and register it with ddbds, then incoming queries that match the schema will be converted into DynamoDB queries instead of table scans, enabling high performance, ordered, high-cardinality prefix queries.
Note that ddbds currently only stores values up to 400 kb (the DynamoDB maximum item size). This makes ddbds inappropriate for block storage. It could be extended to fall back to S3, but that is not yet implemented. Within the InterPlanetary ecosystem, it's designed for storing DHT records, IPNS records, peerstore records, etc.
Setup
Simple Setup with Unoptimized Queries
ddbds can be used as a simple key-value store, without optimized queries.
In this case, all datastore queries will result in full table scans using the ParallelScan API, and filtering/ordering/etc. will be performed client-side.
This is a good option if your table is small or your data and access patterns would not significantly benefit from optimized queries.
var ddbClient *dynamodb.DynamoDB = ...
tableName := "datastore-table"
ddbDS := ddbds.New(ddbClient, tableName)
By default the expected partition key is DSKey of type string. The name can be customized with the WithPartitionKey() option.
Optimized Queries
To use optimized prefix queries, you must specify a sort key.
Also, elements written into the datastore should have at least 2 parts, such as /a/b and not /a.
ddbds splits the key into partition and sort keys. Examples:
/a-> error (not enough parts)/a/b-> [a,b]/a/b/c-> [a,b/c]- etc.
To use optimized queries, simply specify the sort key name using the WithSortKey() option:
var ddbClient *dynamodb.DynamoDB = ...
tableName := "datastore-table"
ddbDS := ddbds.New(
ddbClient,
tableName,
ddbds.WithPartitionKey("PartitionKey"),
ddbds.WithSortKey("SortKey"),
)
Composing Datastores
This datastore can be composed using mount datastores for optimized prefix queries under different namespaces and DynamoDB tables.
Example:
ddbDS := mount.New([]mount.Mount{
{
Prefix: ds.NewKey("/peers/addrs"),
Datastore: ddbds.New(
ddbClient,
"datastore-peers-addrs",
ddbds.WithPartitionkey("PeerID"),
),
},
{
Prefix: ds.NewKey("/providers"),
Datastore: ddbds.New(
ddbClient,
"datastore-providers",
ddbds.WithPartitionkey("ContentHash"),
ddbds.WithSortKey("PeerID"),
),
},
{
Prefix: ds.NewKey("/"),
Datastore: ddbds.New(
ddbClient,
"datastore-all",
),
},
})
IAM Permissions
The following describes the IAM actions and the datastore methods that use them:
- dynamodb:GetItem
Get()GetExpiration()GetSize()Has()
- dynamodb:PutItem
Put()PutWithTTL()
- dynamodb:DeleteItem
Delete()
- dynamodb:Scan
Scan()(if there is no sort key defined)
- dynamodb:Query
Query()(if there is a sort key defined)
- dynamodb:DescribeTable
DiskUsage()
- dynamodb:UpdateItem
SetTTL()
- dynamodb:BatchWriteItem
Batch.Commit()
Datastore Features
- Batching
- TTL
- Disk Usage
- Transactions
- Checked (not applicable)
- Scrubbed (not applicable)
- GC (not applicable)
Contributing
Contributions are welcome! This repository is part of the IPFS project and therefore governed by our contributing guidelines.
License
Documentation
¶
Index ¶
- Variables
- func WithPartitionkey(partitionKey string) func(o *Options)
- func WithScanParallelism(n int) func(o *Options)
- func WithSortKey(sortKey string) func(o *Options)
- func WithStronglyConsistentReads() func(o *Options)
- type DDBDatastore
- func (d *DDBDatastore) Batch(_ context.Context) (datastore.Batch, error)
- func (d *DDBDatastore) Close() error
- func (d *DDBDatastore) Delete(ctx context.Context, key ds.Key) error
- func (d *DDBDatastore) DiskUsage(ctx context.Context) (uint64, error)
- func (d *DDBDatastore) EntryCount(ctx context.Context) (uint64, error)
- func (d *DDBDatastore) Get(ctx context.Context, key ds.Key) ([]byte, error)
- func (d *DDBDatastore) GetExpiration(ctx context.Context, key ds.Key) (time.Time, error)
- func (d *DDBDatastore) GetSize(ctx context.Context, key ds.Key) (size int, err error)
- func (d *DDBDatastore) Has(ctx context.Context, key ds.Key) (bool, error)
- func (d *DDBDatastore) Put(ctx context.Context, key ds.Key, value []byte) error
- func (d *DDBDatastore) PutWithTTL(ctx context.Context, key ds.Key, value []byte, ttl time.Duration) error
- func (d *DDBDatastore) Query(ctx context.Context, q query.Query) (query.Results, error)
- func (d *DDBDatastore) SetTTL(ctx context.Context, key ds.Key, ttl time.Duration) error
- func (d *DDBDatastore) Sync(ctx context.Context, prefix ds.Key) error
- type Options
Constants ¶
This section is empty.
Variables ¶
var (
ErrInvalidKey = errors.New("invalid key for DynamoDB datastore")
)
Functions ¶
func WithPartitionkey ¶
func WithScanParallelism ¶
func WithSortKey ¶
func WithStronglyConsistentReads ¶
func WithStronglyConsistentReads() func(o *Options)
Types ¶
type DDBDatastore ¶
type DDBDatastore struct {
// contains filtered or unexported fields
}
func New ¶
func New(ddbClient *dynamodb.DynamoDB, table string, optFns ...func(o *Options)) *DDBDatastore
func (*DDBDatastore) Batch ¶
func (d *DDBDatastore) Batch(_ context.Context) (datastore.Batch, error)
func (*DDBDatastore) Close ¶
func (d *DDBDatastore) Close() error
func (*DDBDatastore) DiskUsage ¶
func (d *DDBDatastore) DiskUsage(ctx context.Context) (uint64, error)
DiskUsage returns the size of the DynamoDB table. Note that DynamoDB only updates this size once every few hours. The underlying call is heavily throttled so this should only be called occasionally.
func (*DDBDatastore) EntryCount ¶
func (d *DDBDatastore) EntryCount(ctx context.Context) (uint64, error)
Source Files