mcdb

README

Multi-CDB

Write Once Read Many Constant Database. It wraps github.com/colinmarc/cdb (See the original cdb specification and C implementation by D. J. Bernstein at http://cr.yp.to/cdb.html) to hide the 4GiB size limit.

The database(s) can be written only once, and you must set the number of tables to use beforehand.

Rationale

CDB is fast and very simple. For read-only access, it's ideal. The size limit is too low nowadays.

CLI

./cmd/cdb is a command-line program that can dump in cdbmake-format, and load from that format.

Documentation

Overview

Package mcdb contains multiple Constant Databases (DJB's cdb), as one cdb can only contain 4GiB maximum.

This is a thin wrapper over github.com/colinmarc/cdb.

Index

• Constants
• func Hash(p []byte) uint32
• type Iterator
  • func (m *Iterator) Dump(w io.Writer) error
  • func (m *Iterator) Err() error
  • func (m *Iterator) Key() []byte
  • func (m *Iterator) Next() bool
  • func (m *Iterator) Value() []byte
• type Reader
  • func NewReader(dir string) (*Reader, error)
  • func (m *Reader) Close() error
  • func (m *Reader) Dump(w io.Writer) error
  • func (m *Reader) Get(key []byte) ([]byte, error)
  • func (m *Reader) Iter() *Iterator
• type Writer
  • func NewWriter(dir string, n int) (*Writer, error)
  • func (m *Writer) Close() error
  • func (m *Writer) Load(r io.Reader) error
  • func (m *Writer) Put(key, val []byte) error

Constants

const FileName = "mcdb-%d,%b.cdb"

Functions

func Hash

func Hash(p []byte) uint32

Hash is the used hash. FNV-1 at the moment.

Types

type Iterator

type Iterator struct {
	// contains filtered or unexported fields
}

Iterator iterates through all keys of all CDB files.

func (*Iterator) Dump

func (m *Iterator) Dump(w io.Writer) error

Dump the current Iterator position data in cdbmake format ("+%d,%d:%s->%s\n", len(key), len(value), key, value).

func (*Iterator) Err

func (m *Iterator) Err() error

Err returns the last error.

func (*Iterator) Key

func (m *Iterator) Key() []byte

Key returns the current key (after Next).

func (*Iterator) Next

func (m *Iterator) Next() bool

Next advances the iterator, if possible.

func (*Iterator) Value

func (m *Iterator) Value() []byte

Value returns the current value (after Next).

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader is a reader for multiple CDB files.

func NewReader

func NewReader(dir string) (*Reader, error)

NewReader opens the multiple CDB files for reading.

func (*Reader) Close

func (m *Reader) Close() error

Close the underlying readers.

func (*Reader) Dump

func (m *Reader) Dump(w io.Writer) error

Dump all the underlying data in cdbmake format ("+%d,%d:%s->%s\n", len(key), len(value), key, value)

func (*Reader) Get

func (m *Reader) Get(key []byte) ([]byte, error)

Get the key from the reader.

func (*Reader) Iter

func (m *Reader) Iter() *Iterator

Iter returns an iterator.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer is the writer. It needs the number of tables beforehand.

func NewWriter

func NewWriter(dir string, n int) (*Writer, error)

NewWriter returns a new Writer.

The next power-of-two number of tables are created, so for example with n=3, 4 tables are created, containing maximum 16GiB of data.

func (*Writer) Close

func (m *Writer) Close() error

Close the underlying writers.

func (*Writer) Load

func (m *Writer) Load(r io.Reader) error

Load the Writer from cdbmake format ("+%d,%d:%s->%s\n", len(key), len(value), key, value).

func (*Writer) Put

func (m *Writer) Put(key, val []byte) error

Put the key into one of the underlying writers.