s3ds

README

S3 Datastore Implementation

This is an implementation of the datastore interface backed by amazon s3.

NOTE: Plugins only work on Linux and MacOS at the moment. You can track the progress of this issue here: https://github.com/golang/go/issues/19282

Bundling

As go plugins can be finicky to correctly compile and install, you may want to consider bundling this plugin and re-building go-ipfs. If you do it this way, you won't need to install the .so file in your local repo, i.e following the above Building and Installing section, and you won't need to worry about getting all the versions to match up.

# We use go modules for everything.
> export GO111MODULE=on

# Clone go-ipfs.
> git clone https://github.com/ipfs/go-ipfs
> cd go-ipfs

# Pull in the datastore plugin (you can specify a version other than latest if you'd like).
> go get github.com/opensaucerer/go-ds-s3/plugin@latest

# Add the plugin to the preload list.
> echo -en "\ns3ds github.com/opensaucerer/go-ds-s3/plugin 0" >> plugin/loader/preload_list

# ( this first pass will fail ) Try to build go-ipfs with the plugin
> make build

# Update the deptree
> go mod tidy

# Now rebuild go-ipfs with the plugin
> make build

# (Optionally) install go-ipfs
> make install

Detailed Installation

For a brand new ipfs instance (no data stored yet):

1. Edit $IPFS_DIR/config to include s3 details (see Configuration below).

Configuration

The config file should include the following:

{
  "Datastore": {
  ...

    "Spec": {
      "mounts": [
        {
          "child": {
            "type": "s3ds",
            "region": "us-east-1",
            "bucket": "$bucketname",
            "rootDirectory": "$bucketsubdirectory",
            "accessKey": "",
            "secretKey": ""
          },
          "mountpoint": "/blocks",
          "prefix": "s3.datastore",
          "type": "measure"
        },

If the access and secret key are blank they will be loaded from the usual ~/.aws/. If you are on another S3 compatible provider, e.g. Linode, then your config should be:

{
  "Datastore": {
  ...

    "Spec": {
      "mounts": [
        {
          "child": {
            "type": "s3ds",
            "region": "us-east-1",
            "bucket": "$bucketname",
            "rootDirectory": "$bucketsubdirectory",
            "regionEndpoint": "us-east-1.linodeobjects.com",
            "accessKey": "",
            "secretKey": ""
          },
          "mountpoint": "/blocks",
          "prefix": "s3.datastore",
          "type": "measure"
        },

1. Overwrite $IPFS_DIR/datastore_spec as specified below (Don't do this on an instance with existing data - it will be lost).

If you are configuring a brand new ipfs instance without any data, you can overwrite the datastore_spec file with:

{"mounts":[{"bucket":"$bucketname","mountpoint":"/blocks","region":"us-east-1","rootDirectory":"$bucketsubdirectory"},{"mountpoint":"/","path":"datastore","type":"levelds"}],"type":"mount"}

Otherwise, you need to do a datastore migration.

Contribute

Feel free to join in. All welcome. Open an issue! Or create a Pull Request

Documentation

Index

• type Config
• type S3Bucket
  • func NewS3Datastore(conf Config) (*S3Bucket, error)
  • func (s *S3Bucket) Batch(_ context.Context) (ds.Batch, error)
  • func (s *S3Bucket) Close() error
  • func (s *S3Bucket) Delete(ctx context.Context, k ds.Key) error
  • func (s *S3Bucket) Get(ctx context.Context, k ds.Key) ([]byte, error)
  • func (s *S3Bucket) GetSize(ctx context.Context, k ds.Key) (size int, err error)
  • func (s *S3Bucket) Has(ctx context.Context, k ds.Key) (exists bool, err error)
  • func (s *S3Bucket) Put(ctx context.Context, k ds.Key, value []byte) error
  • func (s *S3Bucket) Query(ctx context.Context, q dsq.Query) (dsq.Results, error)
  • func (s *S3Bucket) Sync(ctx context.Context, prefix ds.Key) error

Types

type Config

type Config struct {
	AccessKey           string
	SecretKey           string
	SessionToken        string
	Bucket              string
	Region              string
	RegionEndpoint      string
	RootDirectory       string
	Workers             int
	CredentialsEndpoint string
}

type S3Bucket

type S3Bucket struct {
	Config
	S3 *s3.S3
}

func NewS3Datastore

func NewS3Datastore(conf Config) (*S3Bucket, error)

func (*S3Bucket) Batch

func (s *S3Bucket) Batch(_ context.Context) (ds.Batch, error)

func (*S3Bucket) Close

func (s *S3Bucket) Close() error

func (*S3Bucket) Delete

func (s *S3Bucket) Delete(ctx context.Context, k ds.Key) error

func (*S3Bucket) Get

func (s *S3Bucket) Get(ctx context.Context, k ds.Key) ([]byte, error)

func (*S3Bucket) GetSize

func (s *S3Bucket) GetSize(ctx context.Context, k ds.Key) (size int, err error)

func (*S3Bucket) Has

func (s *S3Bucket) Has(ctx context.Context, k ds.Key) (exists bool, err error)

func (*S3Bucket) Put

func (s *S3Bucket) Put(ctx context.Context, k ds.Key, value []byte) error

func (*S3Bucket) Query

func (s *S3Bucket) Query(ctx context.Context, q dsq.Query) (dsq.Results, error)

func (*S3Bucket) Sync

func (s *S3Bucket) Sync(ctx context.Context, prefix ds.Key) error