Documentation
¶
Overview ¶
Package crdt implements a conflict-free replicated data type for plain text: a replicated character sequence that any number of replicas may edit concurrently, offline, and in any delivery order, and that is guaranteed to converge to the same text on every replica.
The sequence is an RGA (Replicated Growable Array). Every character carries a unique ID and the ID of the character it was inserted after, so insertions are placed relative to content rather than to an index that concurrent edits would invalidate. Deletions are tombstones, so an insertion may still refer to a character another replica has already removed.
Determinism ¶
The package never reads the wall clock and never draws random numbers, so the same Doc compiled to js/wasm behaves exactly as it does on a server. Replica identity is injected by the caller as a SiteID; see DeriveSiteID for a deterministic way to obtain one from bytes the caller already has.
Two counters ¶
Each operation carries two numbers, and they are not the same thing:
- ID.Seq is a per-site counter that increases by exactly one per operation the site issues. It gives the operation its identity and lets a VersionVector describe, exactly, which operations a replica holds.
- Op.Clock is a Lamport timestamp, bumped past every clock a replica has seen. It orders concurrent insertions at the same position, and it is what makes RGA integration convergent.
Folding the two into one counter would create gaps in a site's own sequence, and a version vector cannot describe a sequence with gaps.
Delivery ¶
Doc.Apply tolerates duplicates, and it tolerates operations that arrive before the operations they depend on: an operation that is not yet ready is buffered and integrated as soon as its dependencies land. Callers therefore do not need an ordered transport, only an eventually-complete one.
Example ¶
Two people edit the same document at the same time, neither having seen the other's change. Once the two operations have crossed, both replicas hold the same text — with no server deciding anything.
package main
import (
"fmt"
"github.com/go-crdt/crdt"
)
func main() {
ada, grace := crdt.New(1), crdt.New(2)
opening, err := ada.Insert(0, "the quick fox")
if err != nil {
panic(err)
}
if err := grace.Apply(opening...); err != nil {
panic(err)
}
// Both edit at once, neither seeing the other.
fromAda, err := ada.Insert(10, "brown ")
if err != nil {
panic(err)
}
fromGrace, err := grace.Insert(13, " jumps")
if err != nil {
panic(err)
}
if err := ada.Apply(fromGrace...); err != nil {
panic(err)
}
if err := grace.Apply(fromAda...); err != nil {
panic(err)
}
fmt.Println(ada)
fmt.Println(grace)
}
Output: the quick brown fox jumps the quick brown fox jumps
Index ¶
- Variables
- func AppendOps(dst []byte, ops []Op) ([]byte, error)
- type AuthorRun
- type Change
- type Doc
- func (d *Doc) Anchor(pos int) (ID, error)
- func (d *Doc) Apply(ops ...Op) error
- func (d *Doc) ApplyChanges(ops ...Op) ([]Change, error)
- func (d *Doc) Author(pos int) (SiteID, error)
- func (d *Doc) AuthorRuns() []AuthorRun
- func (d *Doc) Delete(pos, length int) ([]Op, error)
- func (d *Doc) DeleteUTF16(pos, length int) ([]Op, error)
- func (d *Doc) Insert(pos int, text string) ([]Op, error)
- func (d *Doc) InsertUTF16(pos int, text string) ([]Op, error)
- func (d *Doc) Len() int
- func (d *Doc) LenUTF16() int
- func (d *Doc) OpsSince(vv VersionVector) []Op
- func (d *Doc) Pending() int
- func (d *Doc) Position(anchor ID) (pos int, ok bool)
- func (d *Doc) RuneOffset(pos int) (int, error)
- func (d *Doc) Site() SiteID
- func (d *Doc) Snapshot() []byte
- func (d *Doc) String() string
- func (d *Doc) Tombstones() int
- func (d *Doc) UTF16Offset(pos int) (int, error)
- func (d *Doc) Version() VersionVector
- func (d *Doc) Visible(anchor ID) bool
- type ID
- type Op
- type OpKind
- type SiteID
- type VersionVector
- func (v VersionVector) Clone() VersionVector
- func (v VersionVector) Equal(other VersionVector) bool
- func (v VersionVector) Get(site SiteID) uint64
- func (v VersionVector) Includes(id ID) bool
- func (v VersionVector) MarshalBinary() ([]byte, error)
- func (v *VersionVector) UnmarshalBinary(data []byte) error
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ErrInvalidOp = errors.New("crdt: invalid operation")
ErrInvalidOp reports an operation that cannot be applied because it is not well formed — an unknown kind, a missing identity, an unusable character, or a field set that does not belong to its kind.
var ErrInvalidText = errors.New("crdt: invalid UTF-8")
ErrInvalidText reports text that is not valid UTF-8. The package refuses it rather than substituting replacement characters, which would silently corrupt a document that no later edit could repair.
var ErrMalformed = errors.New("crdt: malformed encoding")
ErrMalformed reports bytes that are not a valid encoding.
var ErrOutOfRange = errors.New("crdt: position out of range")
ErrOutOfRange reports a position or length outside the document.
var ErrSurrogateBoundary = errors.New("crdt: UTF-16 offset splits a surrogate pair")
ErrSurrogateBoundary reports a UTF-16 offset that falls between the two code units of one character.
Such an offset names a position that does not exist: half of an emoji is not a place a cursor can be, and no editor's user ever put it there. It is refused rather than rounded because rounding would move an edit somewhere the caller did not ask for and leave nothing behind to say so — the same reasoning that has Doc.Insert refuse invalid UTF-8 rather than substitute replacement characters.
It is not a hypothetical. JavaScript will happily do the operation, and `"a😀b".slice(0, 2) + "x"` is a string containing a lone high surrogate: not text, not valid UTF-8, and not anything this package can hold. An offset that splits a character has already lost the information needed to honour it.
A caller who must tolerate such an offset can round it down in one step, without a second API: an offset that splits a character is always exactly one past that character's first unit, so Doc.RuneOffset of pos-1 is the position of the character it landed inside.
Functions ¶
Types ¶
type AuthorRun ¶ added in v0.7.0
type AuthorRun struct {
// Pos is the visible offset the stretch starts at.
Pos int
// Len is how many characters it covers.
Len int
// Site is the replica that wrote them.
Site SiteID
}
An AuthorRun is a stretch of the visible text one replica wrote.
type Change ¶ added in v0.7.0
A Change is one contiguous edit to the visible text: remove Removed characters at Pos, then put Text there. Either part may be empty.
Offsets are in runes, and each change is expressed against the text as it stands after the changes before it. Applying them in order to a copy of the text is what brings the copy up to date.
func ChangesFrom ¶ added in v0.7.0
ChangesFrom returns the edits that turn text into the document's current text, for a caller holding a copy it cannot otherwise reconcile — a view that has just been reconnected, say. It is a convenience over Doc.String, not a cheaper path: it compares the two.
type Doc ¶
type Doc struct {
// contains filtered or unexported fields
}
A Doc is one replica of a text document. It is not safe for concurrent use; serialize access from the outside, as an editor naturally does.
The zero Doc is unusable — construct one with New or Load.
func Load ¶
Load rebuilds a document from a snapshot, to be edited as site. The site need not be one that appears in the snapshot — a client joining an existing document brings its own.
Example ¶
A replica joining an existing session is given a snapshot rather than the whole history, and can take part immediately.
package main
import (
"fmt"
"github.com/go-crdt/crdt"
)
func main() {
server := crdt.New(1)
if _, err := server.Insert(0, "shared draft"); err != nil {
panic(err)
}
client, err := crdt.Load(2, server.Snapshot())
if err != nil {
panic(err)
}
edit, err := client.Insert(client.Len(), " — revised")
if err != nil {
panic(err)
}
if err := server.Apply(edit...); err != nil {
panic(err)
}
fmt.Println(server)
}
Output: shared draft — revised
func New ¶
New returns an empty document that issues operations as site. Every replica editing a document concurrently must pass a distinct site; see SiteID.
func (*Doc) Anchor ¶ added in v0.7.0
Anchor returns the identity of the character at visible offset pos.
The identity does not move. Insertions and deletions elsewhere change which offset the character sits at, and never change what it is, so an anchor is what a comment, a mark or a selection should be stored as — an offset stored instead would point somewhere else the moment anyone edits above it.
pos may equal Doc.Len, which anchors to the end of the document and returns the zero ID: the position after every character there is, and the one thing insertions at the end do not move.
func (*Doc) Apply ¶
Apply integrates operations from peers. Duplicates are ignored, and an operation that arrives before the operations it depends on is buffered until they do, so the caller needs no ordered delivery.
A malformed operation is rejected and nothing in the batch is applied.
func (*Doc) ApplyChanges ¶ added in v0.7.0
ApplyChanges is Doc.Apply, and also reports what the document did: the edits a view of the text has to make to catch up, in the order it has to make them.
Only what actually happened is reported. An operation already applied, or one still waiting for the operations it depends on, changes nothing and says nothing; when it does land, the change is reported then.
Finding where each edit landed costs a walk up the index per operation, which Doc.Apply does not pay. Use that one when nothing is watching.
func (*Doc) Author ¶ added in v0.7.0
Author returns the replica that wrote the character at visible offset pos.
func (*Doc) AuthorRuns ¶ added in v0.7.0
AuthorRuns splits the visible text into stretches by who wrote them, in order. It is what colouring a document by author needs, and it costs one pass rather than one lookup per character.
Adjacent stretches by the same replica are joined, so the result depends on the text rather than on how the document happens to be stored: two replicas holding the same document return the same runs.
func (*Doc) Delete ¶
Delete removes length runes starting at rune offset pos and returns the operations that describe it. Deleting nothing is a no-op.
func (*Doc) DeleteUTF16 ¶ added in v0.8.0
DeleteUTF16 is Doc.Delete with pos and length counted in UTF-16 code units rather than in runes.
Both ends of the range are converted, so length is a number of code units and the number of characters removed may be fewer — deleting the four units of two emoji removes two characters. A range whose either end splits a character is refused; see ErrSurrogateBoundary.
func (*Doc) Insert ¶
Insert adds text at rune offset pos and returns the operations that describe it. The operations are already applied here; send them to every peer.
pos may equal Doc.Len, which appends. Inserting the empty string is a no-op that returns no operations.
func (*Doc) InsertUTF16 ¶ added in v0.8.0
InsertUTF16 is Doc.Insert with pos counted in UTF-16 code units rather than in runes. The text itself is a Go string, and so is measured in neither.
Example ¶
A browser's cursor offset counts UTF-16 code units, and an emoji is two of them. Handing that offset to Insert would put the text one place to the left of where the user asked for it, without an error and without a trace.
package main
import (
"fmt"
"github.com/go-crdt/crdt"
)
func main() {
doc := crdt.New(1)
if _, err := doc.Insert(0, "ship \U0001F680 it"); err != nil {
panic(err)
}
// Nine characters, ten code units: the rocket is one and two.
fmt.Println(doc.Len(), doc.LenUTF16())
// The editor reports its caret just after the rocket, which is code unit
// seven and character six.
if _, err := doc.InsertUTF16(7, " now"); err != nil {
panic(err)
}
fmt.Println(doc)
// The offset between the rocket's two units names no position at all.
_, err := doc.InsertUTF16(6, "x")
fmt.Println(err)
}
Output: 9 10 ship 🚀 now it crdt: UTF-16 offset splits a surrogate pair
func (*Doc) LenUTF16 ¶ added in v0.8.0
LenUTF16 returns the length of the document in UTF-16 code units — the number JavaScript's String.prototype.length reports for Doc.String.
It is a counter, not a walk: the count of visible supplementary characters is maintained beside the count of visible characters, so this reads the document no more than Doc.Len does.
func (*Doc) OpsSince ¶
func (d *Doc) OpsSince(vv VersionVector) []Op
OpsSince returns the operations this replica holds that vv does not, ready to be sent to the peer that produced vv. Pass a nil vector for the whole history.
The result is in document order, which for insertions is a causal order: a character always follows its origin. Deletions may arrive before the insertions they refer to, which the receiving Doc.Apply buffers.
A deletion's Lamport timestamp is not retained — it never affects ordering — so replayed deletions carry their sequence number as their clock.
Example ¶
A replica that has been away asks for what it missed by handing over its version vector; it is sent those operations and nothing else.
package main
import (
"fmt"
"github.com/go-crdt/crdt"
)
func main() {
online, offline := crdt.New(1), crdt.New(2)
start, err := online.Insert(0, "notes")
if err != nil {
panic(err)
}
if err := offline.Apply(start...); err != nil {
panic(err)
}
// The offline replica misses everything that follows.
if _, err := online.Insert(5, ": chapter one"); err != nil {
panic(err)
}
if _, err := online.Delete(0, 1); err != nil {
panic(err)
}
missed := online.OpsSince(offline.Version())
fmt.Println(len(missed), "operations missed")
if err := offline.Apply(missed...); err != nil {
panic(err)
}
fmt.Println(offline)
}
Output: 14 operations missed otes: chapter one
func (*Doc) Pending ¶
Pending returns the number of operations buffered awaiting their dependencies. A healthy replica returns to zero once delivery catches up; a number that only grows means a peer is withholding operations.
func (*Doc) Position ¶ added in v0.7.0
Position returns where the character an anchor names sits now.
A deleted character still has a place — the offset it would occupy, which is where the text around it closed up — and that is returned too, because a comment on a deleted sentence belongs where the sentence was rather than nowhere. Use Doc.Visible to tell the two apart.
The zero ID anchors to the end of the document. ok is false only for an identity this document has never seen, which means the anchor came from somewhere else or the operations that would explain it have not arrived.
func (*Doc) RuneOffset ¶ added in v0.8.0
RuneOffset converts a UTF-16 offset into the rune offset naming the same position. pos may equal Doc.LenUTF16, which converts to Doc.Len.
An offset falling between the two code units of one character is refused with ErrSurrogateBoundary; see there for why, and for the one-line way to round it down instead.
func (*Doc) Snapshot ¶
Snapshot encodes the whole document — every character, alive or tombstoned, in document order, plus the version vector. It is what a server sends a client joining an existing session, and what it persists.
Characters are written in runs: one header for a stretch one site typed consecutively, then its text, then the stretches of it that have been deleted. Writing one record per character instead, as version 1 did, cost twenty-five bytes for every character of a real document — measured against other implementations, between eight and twenty-four times what they need.
The runs written are maximal, whatever boundaries the document happens to hold in memory. That is what keeps the encoding canonical: two replicas that have applied the same operations produce identical bytes even if the operations arrived in different orders, so a snapshot doubles as a convergence check. It also keeps the format independent of the layout a replica stores, which is what let that layout change twice without a flag day.
The full history is recoverable from a snapshot: Doc.OpsSince on a loaded document returns the same operations it would have on the original.
func (*Doc) Tombstones ¶
Tombstones returns the number of deleted characters still held in memory. They cannot be dropped, because a concurrent insertion may name one as its origin; see docs/performance.md.
func (*Doc) UTF16Offset ¶ added in v0.8.0
UTF16Offset converts a rune offset into the UTF-16 offset naming the same position. pos may equal Doc.Len, which converts to Doc.LenUTF16.
This is the direction an editor needs to place someone else's cursor, or to report where an edit of its own landed.
Example ¶
Awareness offsets are rune positions, because both peers have to agree what an offset means and an update has nowhere to say. A peer whose editor counts UTF-16 converts at its own edge — where it has to clamp in any case, since a cursor may describe a document longer than the one it now has.
package main
import (
"fmt"
"github.com/go-crdt/crdt"
"github.com/go-crdt/crdt/awareness"
)
func main() {
doc := crdt.New(1)
if _, err := doc.Insert(0, "\U0001F600 hello"); err != nil {
panic(err)
}
peers := awareness.New()
peers.Apply(awareness.Update{Site: 2, Clock: 1, Cursor: awareness.Cursor{Anchor: 2, Head: 99}})
for _, peer := range peers.Peers() {
head := min(max(peer.Cursor.Head, 0), doc.Len())
at, err := doc.UTF16Offset(head)
if err != nil {
panic(err)
}
fmt.Println("caret at code unit", at)
}
}
Output: caret at code unit 8
func (*Doc) Version ¶
func (d *Doc) Version() VersionVector
Version returns a copy of the version vector describing which operations this replica holds. Pass it to a peer's Doc.OpsSince to be sent exactly what is missing.
type ID ¶
ID names a single operation, and — for an insertion — the character that operation created. It is unique across replicas because Site is unique and Seq counts that site's own operations.
The zero ID is the document root: the virtual character that precedes all content. It is a valid insertion origin and is never the ID of a real operation, because Seq starts at one.
func (ID) IsRoot ¶
IsRoot reports whether id names the virtual character at the start of every document rather than a real operation.
The test is on Seq alone, because Seq counts from one: no operation ever carries zero, whatever its site. A decoder that only compared against the zero ID would let a sequence number of zero paired with a non-zero site through as if it named something real.
type Op ¶
type Op struct {
// Kind selects insertion or deletion.
Kind OpKind
// ID names this operation. Its Seq is the issuing site's own counter and
// increases by exactly one per operation.
ID ID
// Clock is the Lamport timestamp that orders this operation against
// concurrent ones. See the package documentation.
Clock uint64
// Origin is the character the new one is inserted after; the zero ID means
// the start of the document. OpInsert only.
Origin ID
// Char is the inserted character. OpInsert only.
Char rune
// Target is the character to tombstone. OpDelete only.
Target ID
}
An Op is one indivisible change to a document. It is the only thing replicas exchange, it is self-describing, and applying it is idempotent, so a transport may duplicate or reorder operations freely.
Which fields carry meaning depends on Kind: Origin and Char belong to OpInsert, Target to OpDelete. The unused fields must be zero, and are checked, so a garbled operation is rejected rather than silently reinterpreted.
func (Op) MarshalBinary ¶
MarshalBinary encodes the operation. It reports ErrInvalidOp rather than producing bytes that would be rejected on arrival.
func (*Op) UnmarshalBinary ¶
UnmarshalBinary decodes an operation written by MarshalBinary. Trailing bytes are an error: an operation is decoded from exactly its own encoding.
type SiteID ¶
type SiteID uint64
SiteID identifies a replica. It is chosen by the caller and must be distinct for every replica that concurrently edits a document; two replicas sharing a SiteID can mint the same ID for different characters, which breaks convergence.
The package never generates one itself: a random or clock-derived identifier is unavailable, or not reproducible, under js/wasm. See DeriveSiteID.
func DeriveSiteID ¶
DeriveSiteID hashes b into a SiteID with FNV-1a. It gives callers a deterministic way to turn an identifier they already hold — a session token, a user ID, a tab identifier — into a replica identity, on any platform, including js/wasm where the usual sources of randomness are absent.
Distinctness is the caller's responsibility: distinct b almost always yields distinct SiteIDs, but a hash cannot promise it.
Example ¶
Site identities have to be distinct and cannot be drawn at random under js/wasm, so they are derived from something the caller already holds.
package main
import (
"fmt"
"github.com/go-crdt/crdt"
)
func main() {
first := crdt.DeriveSiteID([]byte("session-8f2c"))
second := crdt.DeriveSiteID([]byte("session-8f2c"))
fmt.Println(first == second, first == crdt.DeriveSiteID([]byte("session-91ab")))
}
Output: true false
type VersionVector ¶
A VersionVector records, per site, the highest sequence number a replica has applied. Because a site's sequence numbers have no gaps and Doc refuses to apply an operation until its predecessor has landed, the vector describes a replica's state exactly: it holds operation Seq from Site if and only if Seq <= v[Site].
The nil vector is valid and reads as "nothing applied".
func (VersionVector) Clone ¶
func (v VersionVector) Clone() VersionVector
Clone returns an independent copy. The clone of a nil vector is empty but non-nil, so it can be written to.
func (VersionVector) Equal ¶
func (v VersionVector) Equal(other VersionVector) bool
Equal reports whether v and other describe the same set of operations. Sites recorded with a zero sequence number count as absent, so a nil vector equals an empty one.
func (VersionVector) Get ¶
func (v VersionVector) Get(site SiteID) uint64
Get returns the highest sequence number applied for site, or zero.
func (VersionVector) Includes ¶
func (v VersionVector) Includes(id ID) bool
Includes reports whether the operation named by id has been applied.
func (VersionVector) MarshalBinary ¶ added in v0.1.1
func (v VersionVector) MarshalBinary() ([]byte, error)
MarshalBinary encodes the vector. Entries are written in ascending site order, so the same state always produces the same bytes and a caller may compare or cache them.
A replica sends this to be told what it has missed; see Doc.OpsSince.
func (*VersionVector) UnmarshalBinary ¶ added in v0.1.1
func (v *VersionVector) UnmarshalBinary(data []byte) error
UnmarshalBinary decodes a vector written by MarshalBinary. A site listed twice, a sequence number of zero, and trailing bytes are all rejected: each would leave what the vector means dependent on decoding order.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package awareness tracks who else has a document open and where their cursor is — the coloured carets and name labels a collaborative editor shows.
|
Package awareness tracks who else has a document open and where their cursor is — the coloured carets and name labels a collaborative editor shows. |