diskmap

Documentation

Overview

Package diskmap provides disk storage of key/value pairs. The data is immutable once written. In addition the diskmap utilizes mmap on reads to make the random access faster.

Usage is simplistic:

// Create a new diskmap.
p := path.Join(os.TempDir(), nextSuffix())
w, err := diskmap.New(p)
if err != nil {
  panic(err)
}

// Write a key/value to the diskmap.
if err := w.Write([]byte("hello"), []byte("world")); err != nil {
  panic(err)
}

// Close the file to writing.
w.Close()

// Open the file for reading.
m, err := diskmap.Open(p)
if err != nil {
  panic(err)
}

// Read the value at key "hello".
v, err := m.Read([]byte("hello"))
if err != nil {
  panic(err)
}

// Print the value at key "hello".
fmt.Println(string(v))

// Loop through all entries in the map.
ctx, cancel := context.WithCancel(context.Background())
defer cancel() // Make sure if we end the "range" early we don't leave any leaky goroutines.

for kv := range m.Range(ctx) {
  fmt.Printf("key: %s, value: %s", string(kv.Key), string(kv.Value))
}

Storage details:

The disk storage is fairly straight forward. Once all values have been written to disk, an index of the keys and offsets is written to disk. 64 bytes are reserved at the start of the file to hold a int64 that gives the offset where the index starts and the number of key/value pairs stored. The additional space is reserved for future use. All numbers are int64 values.

The file structure looks as follows:

<file>
  <reserve header space>
    [index offset]
    [number of key/value pairs]
  </reserve header space>
  <data section>
    [byte value]
    [byte value]
    ...
  </data section>
  <index section>
    [data offset]
    [data length]
    [key length]
    [key]
    ...
  </index section>
</file>

Reading the file is simply:

• read the initial 8 bytes into a int64 to get the offset to the index
• seek to the index offset
• read the data storage offset
• read the key length
• read the key
• build map of key to disk offset using the data above.

Index

• type KeyValue
• type Reader
  • func Open(p string) (Reader, error)
• type Writer
  • func New(p string) (Writer, error)

Types

type KeyValue

type KeyValue struct {
	// Key is the key the value was stored at.
	Key []byte

	// Value is the value stored at Key.
	Value []byte
}

KeyValue holds a key/value pair.

type Reader

type Reader interface {
	// Get fetches key "k" and returns the value. Errors when key not found. Thread-safe.
	Read(k []byte) ([]byte, error)

	// Range allows iteration over all the key/value pairs stored in the diskmap. If not interating
	// over all values, Cancel() or a timeout should be used on the Context to prevent a goroutine leak.
	Range(ctx context.Context) chan KeyValue

	// Close closes the diskmap file.
	Close() error
}

Reader provides read access to the the diskmap file.

func Open

func Open(p string) (Reader, error)

Open returns a Reader for a file written by a Writer.

type Writer

type Writer interface {
	// Write writes a key/value pair to disk.  Thread-safe.
	Write(k, v []byte) error

	// Close syncronizes the file to disk and closes it.
	Close() error
}

Writer provides write access to the diskmap file. An error on write makes the Writer unusable.

func New

func New(p string) (Writer, error)

New returns a new Writer that writes to file "p".