s3ds

README

S3 Datastore Implementation

This is an implementation of the datastore interface backed by amazon s3.

NOTE: Plugins only work on Linux and MacOS at the moment. You can track the progress of this issue here: https://github.com/golang/go/issues/19282

Building and Installing

You must build the plugin with the exact version of go used to build the go-ipfs binary you will use it with. You can find the go version for go-ipfs builds from dist.ipfs.io in the build-info file, e.g. https://dist.ipfs.io/go-ipfs/v0.4.22/build-info or by running ipfs version --all

You can this plugin by running make build. You can then install it into your local IPFS repo by running make install.

Plugins need to be built against the correct version of go-ipfs. This package generally tracks the latest go-ipfs release but if you need to build against a different version, please set the IPFS_VERSION environment variable.

You can set IPFS_VERSION to:

• vX.Y.Z to build against that version of IPFS.
• $commit or $branch to build against a specific go-ipfs commit or branch.
• /absolute/path/to/source to build against a specific go-ipfs checkout.

To update the go-ipfs, run:

> make go.mod IPFS_VERSION=version

Detailed Installation

For a brand new ipfs instance (no data stored yet)

1. copy s3plugin.so $IPFS_DIR/plugins/go-ds-s3.so (or run make install if you are installing locally)
2. ipfs init
3. edit $IPFS_DIR/config to include s3 details (see Configuration below)
4. overwrite $IPFS_DIR/datastore_spec as specified below (Don't do this on an instance with existing data - it will be lost)

Configuration

config file should include the following:

{
  "Datastore": {
  ...

    "Spec": {
      "mounts": [
        {
          "child": {
            "type": "s3ds",
            "region": "us-east-1",
            "bucket": "$bucketname",
            "accessKey": "",
            "secretKey": ""
          },
          "mountpoint": "/blocks",
          "prefix": "s3.datastore",
          "type": "measure"
        },

If the access and secret key are blank they will be loaded from the usual ~/.aws/ If you are on another S3 compatible provider, e.g. Linode, then your config should be:

{
  "Datastore": {
  ...

    "Spec": {
      "mounts": [
        {
          "child": {
            "type": "s3ds",
            "region": "us-east-1",
            "bucket": "$bucketname",
            "regionEndpoint": "us-east-1.linodeobjects.com",
            "accessKey": "",
            "secretKey": ""
          },
          "mountpoint": "/blocks",
          "prefix": "s3.datastore",
          "type": "measure"
        },

If you are configuring a brand new ipfs instance without any data, you can overwrite the datastore_spec file with:

{"mounts":[{"bucket":"$bucketname","mountpoint":"/blocks","region":"us-east-1","rootDirectory":""},{"mountpoint":"/","path":"datastore","type":"levelds"}],"type":"mount"} otherwise you need to do a datastore migration.

Contribute

Feel free to join in. All welcome. Open an issue!

This repository falls under the IPFS Code of Conduct.

Want to hack on IPFS?

License

MIT

Documentation

Index

• type Config
• type S3Bucket
  • func NewS3Datastore(conf Config) (*S3Bucket, error)
  • func (s *S3Bucket) Batch() (ds.Batch, error)
  • func (s *S3Bucket) Close() error
  • func (s *S3Bucket) Delete(k ds.Key) error
  • func (s *S3Bucket) Get(k ds.Key) ([]byte, error)
  • func (s *S3Bucket) GetSize(k ds.Key) (size int, err error)
  • func (s *S3Bucket) Has(k ds.Key) (exists bool, err error)
  • func (s *S3Bucket) Put(k ds.Key, value []byte) error
  • func (s *S3Bucket) Query(q dsq.Query) (dsq.Results, error)

Types

type Config

type Config struct {
	AccessKey      string
	SecretKey      string
	SessionToken   string
	Bucket         string
	Region         string
	RegionEndpoint string
	RootDirectory  string
	Workers        int
}

type S3Bucket

type S3Bucket struct {
	Config
	S3 *s3.S3
}

func NewS3Datastore

func NewS3Datastore(conf Config) (*S3Bucket, error)

func (*S3Bucket) Batch

func (s *S3Bucket) Batch() (ds.Batch, error)

func (*S3Bucket) Close

func (s *S3Bucket) Close() error

func (*S3Bucket) Delete

func (s *S3Bucket) Delete(k ds.Key) error

func (*S3Bucket) Get

func (s *S3Bucket) Get(k ds.Key) ([]byte, error)

func (*S3Bucket) GetSize

func (s *S3Bucket) GetSize(k ds.Key) (size int, err error)

func (*S3Bucket) Has

func (s *S3Bucket) Has(k ds.Key) (exists bool, err error)

func (*S3Bucket) Put

func (s *S3Bucket) Put(k ds.Key, value []byte) error

func (*S3Bucket) Query

func (s *S3Bucket) Query(q dsq.Query) (dsq.Results, error)