bobstore

README

bobstore

A blobstore for not-so-large blobs.

It is meant to store a couple of TB of 10k-50k json blobs, to take pressure of a non-distributed database.

It uses a directory with max. 64k data files each 1GB long. There is one lock file which is locked exclusively by the single writing process.

It is safe to use the same opened db handle from multiple goroutines. If this is the writing process, writing from multiple goroutines is supported.

Blobs are only appended. They are never modified, and individual blobs can not be deleted (whole files can be deleted, just don't reference their data after deletion).

Copyright & License

Copyright (c) YC118, The Elders of Pator Tech School.

See LICENSE.md

Documentation

Overview

Package bobstore - a blobstore for small/medium blobs.

It is meant to store a couple of TB of 10k-50k json blobs, to take pressure of a non-distributed database.

It uses a directory with max. 64k data files each 1GB long. There is one lock file which is locked exclusively by the single writing process.

It is safe to use the same opened db handle from multiple goroutines. If this is the writing process, writing from multiple goroutines is supported.

Blobs are only appended. They are never modified, and individual blobs can not be deleted (whole files can be deleted, just don't reference their data after deletion).

Index

• Constants
• type Codec
  • func CodecFor(name string) *Codec
  • func GZIPCodec() *Codec
  • func SnappyCodec() *Codec
• type Cursor
  • func (c *Cursor) Compressed() uint32
  • func (c *Cursor) Error() error
  • func (c *Cursor) Length() uint32
  • func (c *Cursor) Next() bool
  • func (c *Cursor) Ref() Ref
  • func (c *Cursor) Typ() string
• type DB
  • func Open(name string) (*DB, error)
  • func OpenRW(name string) (*DB, error)
  • func (db *DB) Close() (xerr error)
  • func (db *DB) Cursor(next Ref) *Cursor
  • func (db *DB) Read(ref Ref) ([]byte, error)
  • func (db *DB) Write(b []byte) (Ref, error)
  • func (db *DB) WritePosition() (ref Ref, err error)
  • func (db *DB) WriteWithCodec(b []byte, codec *Codec) (Ref, error)
• type Ref
  • func ParseRef(s string) (Ref, error)
  • func (ref Ref) String() string

Constants

const MaxFileLength = 1024 * 1024 * 1024

MaxFileLength hard max for length of single file: 1GB this means a single blob may not be larger than 1GB - headerSize

const MaxNumberFiles = 0xFFFF

MaxNumberFiles is 64k

Types

type Codec

type Codec struct {
	// contains filtered or unexported fields
}

Codec for compression/decompression

func CodecFor

func CodecFor(name string) *Codec

CodecFor returns the codec for name

func GZIPCodec

func GZIPCodec() *Codec

GZIPCodec - gzip codec

func SnappyCodec

func SnappyCodec() *Codec

SnappyCodec - snapy codec

type Cursor

type Cursor struct {
	// contains filtered or unexported fields
}

Cursor keeps track of the current position and record for iteration.

func (*Cursor) Compressed

func (c *Cursor) Compressed() uint32

Compressed returns the compressed length of the current blob.

func (*Cursor) Error

func (c *Cursor) Error() error

Err gives the error that caused Next() to return false, if any.

func (*Cursor) Length

func (c *Cursor) Length() uint32

Length returns the length of the current blob.

func (*Cursor) Next

func (c *Cursor) Next() bool

Next advances to the next blob.

It returns true if there is a current blob in which case the methods returning information on the current blob can be called. It returns false after the last blob has been visited, or after an error occurred.

Before Next() is called the first time, all other method results are undefined. After Next() returned false, only Error() has a defined result.

func (*Cursor) Ref

func (c *Cursor) Ref() Ref

Ref returns the current ref.

func (*Cursor) Typ

func (c *Cursor) Typ() string

Typ returns the typ of the current blob. One of SNAP, GZIP, NONE.

type DB

type DB struct {
	MaxFileLength uint32
	// contains filtered or unexported fields
}

DB is an opaque handle to an opened blob storage

func Open

func Open(name string) (*DB, error)

Open a DB for reading

func OpenRW

func OpenRW(name string) (*DB, error)

OpenRW opens a DB for RW access

func (*DB) Close

func (db *DB) Close() (xerr error)

Close an open DB

func (*DB) Cursor

func (db *DB) Cursor(next Ref) *Cursor

Cursor iterates over the db. next is the initial ref, null value means beginning of DB.

func (*DB) Read

func (db *DB) Read(ref Ref) ([]byte, error)

Read the blob at ref

func (*DB) Write

func (db *DB) Write(b []byte) (Ref, error)

Write to the database. Will use the default SnappyCodec()

func (*DB) WritePosition

func (db *DB) WritePosition() (ref Ref, err error)

WritePosition gives the current write position (where the next write would be) only implemented if opened read-write

func (*DB) WriteWithCodec

func (db *DB) WriteWithCodec(b []byte, codec *Codec) (Ref, error)

WriteWithCodec - write the blob with explicit codec.

type Ref

type Ref struct {
	// Fno
	Fno uint16

	// Position within the file
	Pos uint32
	// contains filtered or unexported fields
}

Ref is a reference to a storage file and position the string representation is fixed: 5 digits : 8 hex digits ==> 14 characters in all

func ParseRef

func ParseRef(s string) (Ref, error)

ParseRef parses a refs string representation

func (Ref) String

func (ref Ref) String() string