README
¶
voidDB
goos: linux
goarch: arm64
pkg: github.com/voidDB/voidDB/test
BenchmarkVoidPut-2 131072 14933 ns/op
BenchmarkVoidGet-2 131072 1060 ns/op
BenchmarkVoidGetNext-2 131072 245.8 ns/op
BenchmarkLMDBPut-2 131072 22414 ns/op
BenchmarkLMDBGet-2 131072 1826 ns/op
BenchmarkLMDBGetNext-2 131072 602.2 ns/op
BenchmarkBoltPut-2 131072 66984 ns/op
BenchmarkBoltGet-2 131072 2552 ns/op
BenchmarkBoltGetNext-2 131072 254.6 ns/op
BenchmarkLevelPut-2 131072 44182 ns/op
BenchmarkLevelGet-2 131072 30949 ns/op
BenchmarkLevelGetNext-2 131072 3441 ns/op
BenchmarkBadgerPut-2 131072 15182 ns/op
BenchmarkBadgerGet-2 131072 33114 ns/op
BenchmarkBadgerGetNext-2 131072 12895 ns/op
BenchmarkNothing-2 131072 0.3239 ns/op
Documentation
¶
Overview ¶
voidDB is a memory-mapped key-value store: simultaneously in-memory and persistent on disk. An embedded database manager, it is meant to be integrated into application software to eliminate protocol overheads and achieve zero-copy performance. This library supplies interfaces for storage and retrieval of arbitrary bytes on 64-bit computers running the Linux operating system.
voidDB features Put, Get, and Del operations as well as forward and backward iteration over self-sorting data in ACID (atomic, consistent, isolated, and durable) transactions. Readers retain a consistent view of the data throughout their lifetime, even as newer transactions are being committed: only pages freed by transactions older than the oldest surviving reader are actively recycled.
voidDB employs a copy-on-write strategy to maintain data in a multi-version concurrency-controlled (MVCC) B+ tree structure. It allows virtually any number of concurrent readers, but only one active writer at any given moment. Readers (and the sole writer) neither compete nor block one another, even though they may originate from and operate within different threads and processes.
voidDB is resilient against torn writes. It automatically restores a database to its last stable state in the event of a mid-write crash. Once a transaction is committed and flushed to disk it is safe, but even if not it could do no harm to existing data in storage. Applications need not be concerned about broken lockfiles or lingering effects of unfinished transactions should an uncontrolled shutdown occur; its design guarantees automatic and immediate release of resources upon process termination.
Getting Started ¶
To begin developing with voidDB, Install Go on your 64-bit Linux machine.
$ go version go version go1.22.3 linux/arm64
Then, import voidDB in your Go application. The following would result in the creation of a database file and its reader table in the working directory. Set the database capacity to any reasonably large value to make sufficient room for the data you intend to store, even if it exceeds the total amount of physical memory; neither memory nor disk is immediately consumed to capacity.
package main
import (
"errors"
"os"
"github.com/voidDB/voidDB"
)
func main() {
const (
capacity = 1 << 40 // 1 TiB
path = "void"
)
void, err := voidDB.NewVoid(path, capacity)
if errors.Is(err, os.ErrExist) {
void, err = voidDB.OpenVoid(path, capacity)
}
if err != nil {
panic(err)
}
defer void.Close()
}
Use *Void.View (or *Void.Update only when modifying data) for convenience and peace of mind. Ensure all changes are safely synced to disk with mustSync set to true if even the slightest risk of losing those changes is a concern.
mustSync := true
err = void.Update(mustSync,
func(txn *voidDB.Txn) (err error) {
err = txn.Put(
[]byte("greeting"),
[]byte("Hello, World!"),
)
return
},
)
if err != nil {
panic(err)
}
Open a cursor if more than one keyspace is required. An application can map different values to the same key so long as they reside in separate keyspaces. The transaction handle doubles as a cursor in the default keyspace.
cur0, err := txn.OpenCursor(
[]byte("hello"),
)
err = cur0.Put(
[]byte("greeting"),
[]byte("Hello, World!"),
)
cur1, err := txn.OpenCursor(
[]byte("goodbye"),
)
err = cur1.Put(
[]byte("greeting"),
[]byte("さようなら、世界。"),
)
val, err := cur0.Get(
[]byte("greeting"),
)
log.Printf("%s", val) // Hello, World!
val, err = cur1.Get(
[]byte("greeting"),
)
log.Printf("%s", val) // さようなら、世界。
To iterate over a keyspace, use *cursor.Cursor.GetNext/GetPrev. Position the cursor with *cursor.Cursor.Get/GetFirst/GetLast.
for {
key, val, err := cur.GetNext()
if errors.Is(err, common.ErrorNotFound) {
break
}
log.Printf("%s -> %s", key, val)
}
Author ¶
voidDB builds upon ideas in the celebrated Lightning Memory-Mapped Database Manager on several key points of its high-level design, but otherwise it is elegantly implemented from scratch to break free of limitations in function, clarity, and performance.
voidDB is a cherished toy, a journey into the Unknown, a heroic struggle, and a work of love. It is the “Twee!” of a bird; a tree falling in the forest; yet another programmer pouring their drop into the proverbial [bit] bucket. Above all, it is a shrine unto simple, readable, and functional code; an assertion that the dichotomy between such aesthetics and practical performance is mere illusion.
Copyright 2024 Joel Ling.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Txn ¶
A Txn is a transaction handle necessary for interacting with a database. A transaction is the sum of the state of the database as at the beginning of that transaction and any changes made within it. See *Void.BeginTxn for more information.
func (*Txn) Abort ¶
Abort discards all changes made in a read-write transaction, and releases the exclusive write lock. In the case of a read-only transaction, Abort ends the moratorium on recycling of pages constituting its view of the dataset. For this reason, applications should not be slow to abort transactions that have outlived their usefulness lest they prevent effective resource utilisation. Following an invocation of Abort, the transaction handle must no longer be used.
func (*Txn) Commit ¶
Commit persists all changes to data made in a transaction. The state of the database is not really updated until Commit has been invoked. If it returns a nil error, effects of the transaction would be perceived in subsequent transactions, whereas pre-existing transactions will remain oblivious as intended. Whether Commit waits on *os.File.Sync depends on the mustSync argument passed to *Void.BeginTxn. The transaction handle is not safe to reuse after the first invocation of Commit, regardless of the result.
func (*Txn) OpenCursor ¶
OpenCursor returns a handle on a cursor associated with the transaction and a particular keyspace. Keyspaces allow multiple datasets with potentially intersecting (overlapping) sets of keys to reside within the same database without conflict, provided that all keys are unique within their respective keyspaces. Argument keyspace must not be simultaneously non-nil and of zero length, or otherwise longer than node.MaxKeySize. Passing nil as the argument causes OpenCursor to return a cursor in the default keyspace.
CAUTION: An application utilising keyspaces should avoid modifying records within the default keyspace, as it is used to store pointers to all the other keyspaces. There is virtually no limit on the number of keyspaces in a database.
Unless multiple keyspaces are required, there is usually no need to invoke OpenCursor because the transaction handle embeds a *cursor.Cursor associated with the default keyspace.
func (*Txn) SerialNumber ¶ added in v0.1.1
SerialNumber returns a serial number identifying a particular state of the database as at the beginning of the transaction. All transactions beginning from the same state share the same serial number.
type Void ¶
type Void struct {
// contains filtered or unexported fields
}
A Void is a handle on a database. To interact with the database, enter a transaction through *Void.BeginTxn.
func NewVoid ¶
NewVoid creates and initialises a database file and its reader table at path and path.readers respectively, and returns a handle on the database, or os.ErrExist if a file already exists at path. See also OpenVoid for an explanation of the capacity parameter.
func OpenVoid ¶
OpenVoid returns a handle on the database persisted to the file at path.
The capacity argument sets a hard limit on the size of the database file in number of bytes, but it applies only to transactions entered into via the database handle returned. The database file never shrinks, but it will not be allowed to grow if its size already exceeds capacity as at the time of invocation. A transaction running against the limit would incur common.ErrorFull on commit.
func (*Void) BeginTxn ¶
BeginTxn begins a new transaction. The resulting transaction cannot modify data if readonly is true: any changes made are isolated to the transaction and non-durable; otherwise it is a write transaction. Since there cannot be more than one ongoing write transaction per database at any point in time, the function may return syscall.EAGAIN or syscall.EWOULDBLOCK (same error, “resource temporarily unavailable”) if an uncommitted/unaborted incumbent is present in any thread/process in the system.
Setting mustSync to true ensures that all changes to data are flushed to disk when the transaction is committed, at a cost to write performance; setting it to false empowers the filesystem to optimise writes at a risk of data loss in the event of a crash at the level of the operating system or lower, e.g. hardware or power failure. Database corruption is also conceivable, albeit only if the filesystem does not preserve write order. TL;DR: set mustSync to true if safety matters more than speed; false if vice versa.
BeginTxn returns common.ErrorResized if the database file has grown beyond the capacity initially passed to OpenVoid. This can happen if another database handle with a higher capacity has been obtained via a separate invocation of OpenVoid in the meantime. To adapt to the new size and proceed, close the database handle and replace it with a new invocation of OpenVoid.
func (*Void) Close ¶
Close closes the database file and releases the corresponding memory map, rendering both unusable to any remaining transactions already entered into using the database handle. These stranded transactions could give rise to undefined behaviour if their use is attempted, which could disrupt the application, but in any case they pose no danger whatsoever to the data safely jettisoned.
func (*Void) Update ¶ added in v0.1.1
Update is a convenient wrapper around *Void.BeginTxn, *Txn.Commit, and *Txn.Abort, to help applications ensure timely termination of writers. If operation is successful (in that it returns a nil error), the transaction is automatically committed and the result of *Txn.Commit is returned. Otherwise, the transaction is aborted and the output of errors.Join wrapping the return values of operation and *Txn.Abort is returned. IMPORTANT: See also *Void.BeginTxn for an explanation of the mustSync parameter.
func (*Void) View ¶ added in v0.1.1
View is similar to *Void.Update, except that it begins and passes to operation a read-only transaction. If operation results in a non-nil error, that error is errors.Join-ed with the result of *Txn.Abort; otherwise only the latter is returned.
Source Files
Directories