simpleblobstore

README

go-common/simpleblobstore

This repository contains the go-common/simpleblobstore library.

To install:

go get github.com/ugorji/go-common/simpleblobstore

Package Documentation

Package simpleblobstore provides simple blob storage.

Exported Package API

type BlobDriver struct{ ... }

Documentation

Overview

Package simpleblobstore provides simple blob storage.

Index

• type BlobDriver
  • func (l BlobDriver) BlobInfo(ctx app.Context, key string) (bi *app.BlobInfo, err error)
  • func (l BlobDriver) BlobReader(ctx app.Context, key string) (br app.BlobReader, err error)
  • func (l BlobDriver) BlobServe(c app.Context, key string, response http.ResponseWriter) (err error)
  • func (l BlobDriver) BlobWriter(ctx app.Context, contentType string) (b app.BlobWriter, err error)

Types

type BlobDriver

type BlobDriver struct {
	Dir string
}

BlobDriver manages all blob handling on the backend.

For now, blobdir and datadir must be on a shared drive (e.g. NFS mounted). All backends can write to them simultaneously. There might be some issues, but it's best to keep it simple for now.

Blob handling is decentralized. Multiple backends do not have to use a single master.

Blobs can be referred to by a long key string, or by a uint64.

• With a uint64, it just contains a reference to enough information to get to that file.
• With a long key string, that key contains the uint64 id along with all information about the blob (ie timestamp, content type, etc).

We typically do not use the uint64 at this time. We don't have a clean way of checking if the file is unique.

The key is a base64 encoding of:

shardL1(1) shardL2(1) shardL3(1) random(5)
size(8) creationTimeSec(8) creationTimeNs(4) contentType(n)

Note that the top gives a unique representation of the blob, and bottom are just metadata.

On the file system, the files will be in the directory structure: (shardL1, shardL2 and shardL3 are just the 3 level directories below).

BLOBDIR/
  n(00-ff)/      (configurable number of sub-directories at this level)
    n(00-ff)/    (configurable number of sub-directories at this level)
      n(00-ff)/  (configurable number of sub-directories at this level)
        (00-ff)5-ZZZZZZZZZZZZZZZZZZ (files here)

Note that we do not store blobs onto a flat directory. This has issues with reaching the maximum sub-directory depth and number of files in a directory. Even though most filesystems can now handle immense number of files, tools like ls, du, etc will blow up.

With the model we have, we have 3 levels of subdirectories each having up to 256 directories below them. This means we can have up to 256^3 directories containing files (i.e. 16 million sub directories).

However, for the same reasons as above, we'd stop at about 1000 children per directory, giving us up to 16 billion entities to store (with room to grow).

func (BlobDriver) BlobInfo

func (l BlobDriver) BlobInfo(ctx app.Context, key string) (bi *app.BlobInfo, err error)

func (BlobDriver) BlobReader

func (l BlobDriver) BlobReader(ctx app.Context, key string) (br app.BlobReader, err error)

func (BlobDriver) BlobServe

func (l BlobDriver) BlobServe(c app.Context, key string,
	response http.ResponseWriter) (err error)

func (BlobDriver) BlobWriter

func (l BlobDriver) BlobWriter(ctx app.Context, contentType string,
) (b app.BlobWriter, err error)