dqlite

README

go-dqlite

This repository provides the go-dqlite Go package, containing bindings for the dqlite C library and a pure-Go client for the dqlite wire protocol.

Usage

The current major version of the package is v3:

$ go get github.com/canonical/go-dqlite/v3

The v2 major version is used for an LTS series that gets fixes but no new features. Major version 1 is no longer receiving updates of any kind and should not be used.

The best way to understand how to use the package is probably by looking at the source code of the demo program and using it as an example.

In general your application will use code such as:

dir := "/path/to/data/directory"
address := "1.2.3.4:666" // Unique node address
cluster := []string{...} // Optional list of existing nodes, when starting a new node
app, err := app.New(dir, app.WithAddress(address), app.WithCluster(cluster))
if err != nil {
        // ...
}

db, err := app.Open(context.Background(), "my-database")
if err != nil {
        // ...
}

// db is a *sql.DB object
if _, err := db.Exec("CREATE TABLE my_table (n INT)"); err != nil
        // ...
}

Build

In order to use the go-dqlite package in your application, you'll need to have the dqlite C library installed on your system, along with its dependencies.

go-dqlite can use the go-sqlite3 package to store some information about the cluster locally on each node in a SQLite database file. Pass -tags libsqlite3 when building go-dqlite to request that go-sqlite3 link to your system's libsqlite3, instead of building its own. Alternatively, pass -tags nosqlite3 to disable the go-sqlite3 dependency; go-dqlite can store the same information in YAML files instead.

Documentation

The documentation for this package can be found on pkg.go.dev.

Demo

To see dqlite in action, either install the Debian package from the PPA:

$ sudo add-apt-repository -y ppa:dqlite/dev
$ sudo apt install dqlite-tools-v3 libdqlite-dev

or build the dqlite C library and its dependencies from source, as described here, and then run:

$ go install -tags libsqlite3 ./cmd/dqlite-demo

from the top-level directory of this repository.

This builds a demo dqlite application, which exposes a simple key/value store over an HTTP API.

Once the dqlite-demo binary is installed (normally under ~/go/bin or /usr/bin/), start three nodes of the demo application:

$ dqlite-demo --api 127.0.0.1:8001 --db 127.0.0.1:9001 &
$ dqlite-demo --api 127.0.0.1:8002 --db 127.0.0.1:9002 --join 127.0.0.1:9001 &
$ dqlite-demo --api 127.0.0.1:8003 --db 127.0.0.1:9003 --join 127.0.0.1:9001 &

The --api flag tells the demo program where to expose its HTTP API.

The --db flag tells the demo program to use the given address for internal database replication.

The --join flag is optional and should be used only for additional nodes after the first one. It informs them about the existing cluster, so they can automatically join it.

Now we can start using the cluster. Let's insert a key pair:

$ curl -X PUT -d my-value http://127.0.0.1:8001/my-key

and then retrieve it from the database:

$ curl http://127.0.0.1:8001/my-key

Currently the first node is the leader. If we stop it and then try to query the key again curl will fail, but we can simply change the endpoint to another node and things will work since an automatic failover has taken place:

$ kill -TERM %1; curl http://127.0.0.1:8002/my-key

Shell

A basic SQLite-like dqlite shell is available in the dqlite-tools-v3 package or can be built with:

$ go install -tags libsqlite3 ./cmd/dqlite

The general usage is:

dqlite -s <servers> <database> [command] [flags]

Example usage in the case of the dqlite-demo example listed above:

$ dqlite -s 127.0.0.1:9001 demo
dqlite> SELECT * FROM model;
my-key|my-value

The shell supports normal SQL queries plus the special .cluster and .leader commands to inspect the cluster members and the current leader.

Documentation

Index

• Constants
• func ConfigMultiThread() error
• func GenerateID(address string) uint64
• func ReconfigureMembership(dir string, cluster []NodeInfo) error
• func ReconfigureMembershipExt(dir string, cluster []NodeInfo) error
• type LastEntryInfo
  • func ReadLastEntryInfo(dir string) (LastEntryInfo, error)
  • func (lhs LastEntryInfo) Before(rhs LastEntryInfo) bool
• type Node
  • func New(id uint64, address string, dir string, options ...Option) (*Node, error)
  • func (s *Node) BindAddress() string
  • func (s *Node) Close() error
  • func (s *Node) Recover(cluster []NodeInfo) error
  • func (s *Node) Start() error
• type NodeInfo
• type Option
  • func WithAutoRecovery(recovery bool) Option
  • func WithBindAddress(address string) Option
  • func WithBlockSize(size uint) Option
  • func WithBusyTimeout(msecs int64) Option
  • func WithDialFunc(dial client.DialFunc) Option
  • func WithDiskMode(disk bool) Option
  • func WithFailureDomain(code uint64) Option
  • func WithNetworkLatency(latency time.Duration) Option
  • func WithSnapshotParams(params SnapshotParams) Option
• type SnapshotParams
• type TrailingStrategy

Examples

• LastEntryInfo

Constants

const (
	TrailingStrategyStatic  = bindings.TrailingStrategyStatic
	TrailingStrategyDynamic = bindings.TrailingStrategyDynamic
)

const BootstrapID = 0x2dc171858c3155be

BootstrapID is a magic ID that should be used for the fist node in a cluster. Alternatively ID 1 can be used as well.

Functions

func ConfigMultiThread

func ConfigMultiThread() error

ConfigMultiThread sets the threading mode of SQLite to Multi-thread.

By default go-dqlite configures SQLite to Single-thread mode, because the dqlite engine itself is single-threaded, and enabling Multi-thread or Serialized modes would incur in a performance penality.

If your Go process also uses SQLite directly (e.g. using the github.com/mattn/go-sqlite3 bindings) you might need to switch to Multi-thread mode in order to be thread-safe.

IMPORTANT: It's possible to successfully change SQLite's threading mode only if no SQLite APIs have been invoked yet (e.g. no database has been opened yet). Therefore you'll typically want to call ConfigMultiThread() very early in your process setup. Alternatively you can set the GO_DQLITE_MULTITHREAD environment variable to 1 at process startup, in order to prevent go-dqlite from setting Single-thread mode at all.

func GenerateID

func GenerateID(address string) uint64

GenerateID generates a unique ID for a new node, based on a hash of its address and the current time.

func ReconfigureMembership

func ReconfigureMembership(dir string, cluster []NodeInfo) error

ReconfigureMembership forces a new cluster configuration.

Deprecated: this function ignores the provided node roles and makes every node in the new configuration a voter. Use ReconfigureMembershipExt, which respects the provided roles.

func ReconfigureMembershipExt

func ReconfigureMembershipExt(dir string, cluster []NodeInfo) error

ReconfigureMembershipExt forces a new cluster configuration.

This function is useful to revive a cluster that can't achieve quorum in its old configuration because some nodes can't be brought online. Forcing a new configuration is unsafe, and you should follow these steps to avoid data loss and inconsistency:

1. Make sure no dqlite node in the cluster is running.
2. Identify all dqlite nodes that have survived and that you want to be part of the recovered cluster. Call this the "new member list".
3. Call ReadLastEntryInfo on each node in the member list, and find which node has the most recent entry according to LastEntryInfo.Before. Call this the "template node".
4. Invoke ReconfigureMembershipExt exactly one time, on the template node. The arguments are the data directory of the template node and the new member list.
5. Copy the data directory of the template node to all other nodes in the new member list, replacing their previous data directories.
6. Restart all nodes in the new member list.

Types

type LastEntryInfo

type LastEntryInfo struct {
	Term, Index uint64
}

LastEntryInfo holds information about the last entry in the persistent raft log of a node.

The zero value is not a valid entry description, and can be used as a sentinel.

Example

package main

import (
	"fmt"
	"sort"

	dqlite "github.com/canonical/go-dqlite/v3"
)

type infoSorter []dqlite.LastEntryInfo

func (is infoSorter) Len() int {
	return len(is)
}

func (is infoSorter) Less(i, j int) bool {
	return is[i].Before(is[j])
}

func (is infoSorter) Swap(i, j int) {
	is[i], is[j] = is[j], is[i]
}

func main() {
	infos := []dqlite.LastEntryInfo{
		{Term: 1, Index: 2},
		{Term: 2, Index: 2},
		{Term: 1, Index: 1},
		{Term: 2, Index: 1},
	}
	sort.Sort(infoSorter(infos))
	fmt.Println(infos)
}

Output:
[{1 1} {1 2} {2 1} {2 2}]

func ReadLastEntryInfo

func ReadLastEntryInfo(dir string) (LastEntryInfo, error)

ReadLastEntryInfo reads information about the last entry in the raft persistent log from a node's data directory.

This is intended to be used during the cluster recovery process, see ReconfigureMembershipExt. The node must not be running.

This is a non-destructive operation, but is not read-only, since it has the side effect of renaming raft open segment files to closed segment files.

func (LastEntryInfo) Before

func (lhs LastEntryInfo) Before(rhs LastEntryInfo) bool

Before tells whether the entry described by the receiver is strictly less recent than another entry.

Entry A is less recent than entry B when A has a lower term number, or when A and B have the same term number and A has a lower index.

type Node

type Node struct {
	// contains filtered or unexported fields
}

Node runs a dqlite node.

func New

func New(id uint64, address string, dir string, options ...Option) (*Node, error)

New creates a new Node instance.

func (*Node) BindAddress

func (s *Node) BindAddress() string

BindAddress returns the network address the node is listening to.

func (*Node) Close

func (s *Node) Close() error

Close the server, releasing all resources it created.

func (*Node) Recover

func (s *Node) Recover(cluster []NodeInfo) error

Recover a node by forcing a new cluster configuration.

Deprecated: use ReconfigureMembershipExt instead, which does not require instantiating a new Node object.

func (*Node) Start

func (s *Node) Start() error

Start serving requests.

type NodeInfo

type NodeInfo = client.NodeInfo

NodeInfo is a convenience alias for client.NodeInfo.

type Option

type Option func(*options)

Option can be used to tweak node parameters.

func WithAutoRecovery

func WithAutoRecovery(recovery bool) Option

WithAutoRecovery enables or disables auto-recovery of persisted data at startup for this node.

When auto-recovery is enabled, raft snapshots and segment files may be deleted at startup if they are determined to be corrupt. This helps the startup process to succeed in more cases, but can lead to data loss.

Auto-recovery is enabled by default.

func WithBindAddress

func WithBindAddress(address string) Option

WithBindAddress sets a custom bind address for the server.

func WithBlockSize

func WithBlockSize(size uint) Option

WithBlockSize sets the block size of the node.

func WithBusyTimeout

func WithBusyTimeout(msecs int64) Option

WithBusyTimeout sets the timeout for how long a database operation will wait for a lock to be released before returning an error (SQLITE_BUSY), that is the amount of time a writer will wait for others to finish writing on the same database.

Readers never wait writers nor readers.

The default behavior is to fail immediately.

func WithDialFunc

func WithDialFunc(dial client.DialFunc) Option

WithDialFunc sets a custom dial function for the server.

func WithDiskMode

func WithDiskMode(disk bool) Option

WithDiskMode enables dqlite disk-mode on the node. DEPRECATED: this API will always fail on dqlite 1.18.3+ as support for disk mode has been dropped.

func WithFailureDomain

func WithFailureDomain(code uint64) Option

WithFailureDomain sets the code of the failure domain the node belongs to.

func WithNetworkLatency

func WithNetworkLatency(latency time.Duration) Option

WithNetworkLatency sets the average one-way network latency.

func WithSnapshotParams

func WithSnapshotParams(params SnapshotParams) Option

WithSnapshotParams sets the snapshot parameters of the node.

type SnapshotParams

type SnapshotParams struct {
	Threshold uint64
	Trailing  uint64
	Strategy  TrailingStrategy
}

SnapshotParams exposes bindings.SnapshotParams. Used for setting dqlite's snapshot parameters. SnapshotParams.Threshold controls after how many raft log entries a snapshot is taken. The higher this number, the lower the frequency of the snapshots. SnapshotParams.Trailing controls how many raft log entries are retained after taking a snapshot.

type TrailingStrategy

type TrailingStrategy = bindings.TrailingStrategy