blobcache

README

Blobcache

Blobcache is a content addressed data store, designed to be a replicated data layer for applications.

For more information checkout a brief Introduction. And the gRPC API docs.

Goals

• Define a simple API for content-addressed storage suitable for use by multiple applications.
• Efficiently multiplex multiple logical content-addressed stores onto a single real store backed by the file system.
• Store data with other Blobcache peers.
• Store data with cloud object storage providers.

Web Of Trust Storage Network

Blobcache is an experiment in creating a Web of Trust storage network, which is also economically aware.

The peer-to-peer components of Blobcache are still in the early development phases, but many of the components have been prototyped and are available in this repository.

Blobcache depends on INET256 for connections to peers.

Non-Goals

• Create a blockchain or cryptocurrency.
• Store data with untrusted peers.
• Altruistic data storage and retrieval like BitTorrent or IPFS.
• Merkle data structures. Blobcache only deals with blobs.

Getting Started

You should be able to install with

$ go install ./cmd/blobcache

And then if ${go env GOPATH}/bin is on your path, you should be able to run the blobcache command with

$ blobcache 

or to put the executable somewhere specific

$ go build -o ./put/the/binary/here/please ./cmd/blobcache 

Running a Node in Memory

This is a good option if you just want to play around with the API, and don't want to persist any data, or connect to peers.

$ blobcache island

Creating a Private Key

Blobcache requires a private key for each instance. This is used to uniquely identify the instance and for communicating securely with peers using INET256

$ blobcache keygen > blobcache_pk.pem

Running the daemon

The following commands create a directory for blobcache data and runs a daemon on localhost on the default port. Change the name of the private-key to whereever your key is.

$ mkdir my-data
$ blobcache run --private-key ./blobcache_pk.pem --data-dir my-data

Once the daemon is running, you should be all set to connect to it and start building your application on top of content-addressed storage.

Help, I need to store data larger than a single Blobcache blob

Blobcache is not a filesystem or object store. It only allows storing blobs up to a fixed maximum size (currently 2MB). It is more of a building block than a storage solution on it's own.

Take a look at glfs, a Git-Like FileSystem which breaks apart large files into chunks which will fit into Blobcache.

Also take a look at GotFS which is a more complicated, but in many ways more efficient, alternative to glfs.

Help, I need to store many small things efficiently in Blobcache

Take a look at GotKV, an immutable key-value store on top of content-addressable storage. It's Blobcache compatible.

Community

You can get in touch via either:

• Our dedicated Matrix room #blobcache:matrix.org
• The INET256 Discord, in the #applications channel.

License

The Blobcache implementation is licensed under GPLv3.

All of the clients are licensed under Apache 2.0

What this means is: if you improve Blobcache you have to make the improvements available, but if you are just using a client you can do whatever you want with it.