README
¶
collab — the wire for collaborative editing
github.com/go-crdt/collab carries a go-crdt/crdt
document between the people editing it: a gRPC service, a server that hosts
documents, and a client that joins one. Pure Go, CGO=0.
The service is thin on purpose. The document is a CRDT, so the server never transforms an operation and never decides an outcome — it applies what it is sent to its own replica and hands it to everyone else. Two things follow that a server-authoritative design cannot offer: a participant may edit while disconnected and reconcile on return, and the server may be restarted or replaced without anyone losing work.
The client builds for js/wasm, so a browser tab and the server run the same
code down to the merge — over either of two carriers, from one server:
| carrier | for | browser client, gzipped |
|---|---|---|
collab.WebSocket — the session's own framing |
anywhere, browsers included | 919 KB |
collab.GRPC — over gRPC |
native peers | 4 461 KB |
Everything a session carries is already bytes crdt encoded and will check on
arrival, so protobuf describes fields nobody reads through it — and compiled to
wasm its machinery cannot be linked away. For scale, the CRDT alone is 633 KB.
Outside a browser none of that matters, which is why gRPC is still there.
From a page
The editor this was built for is TypeScript and cannot call Go, so ./wasm
compiles to a binding a page uses directly. Offsets are UTF-16 code units
throughout — the units a JavaScript string counts in — and an offset splitting a
character is refused rather than rounded:
const session = await collab.join({ url, document: "project:default", site });
const body = await session.text("file:main.tex");
const chat = await session.list("chat");
const cells = await session.map("cells");
await body.insert(0, "bonjour");
await cells.set("B7", new TextEncoder().encode("42"));
await session.onChange(parts => {
for (const part of parts) {
if (part.kind === "text") applyEdits(part.text); // {pos, removed, insert}
if (part.kind === "map") reread(part.keys); // the keys that changed
if (part.kind === "list") rereadWhole(part.name); // a list says only that it moved
}
});
Types are in wasm/collab.d.ts. A value is Uint8Array in
both directions, never a string and never JSON: the CRDT does not interpret it
and neither does the binding.
Using it
Server — any grpc.Server, any carrier:
srv := collab.NewServer(collab.Config{Store: myStore})
collabpb.RegisterCollabServer(grpcServer, srv)
Participant. A document holds named parts, so a caller reaches for the one it means and gets a handle that edits and publishes:
c, err := collab.Join(ctx, conn, collab.ClientConfig{Document: "notes", Site: 1})
body, _ := c.Text("file:main.tex") // the buffer an editor binds to
chat, _ := c.List("chat") // the messages beside it
cells, _ := c.Map("cells") // a sheet
body.Insert(0, "hello")
chat.Append([]byte("on commence"))
cells.Set("B7", []byte("42"))
c.SetCursor(awareness.Cursor{Anchor: 5, Head: 5}, map[string]string{"name": "ada"})
for range c.Changes() {
for _, part := range c.TakeChanges() { // which part moved, and how
render(part)
}
render(c.Peers())
}
A handle is what a caller touches rather than the replicated structure, and that is forced: editing the structure directly would produce operations nobody ever heard, so the participant would drift away from everyone else while its own screen looked right.
Coming back after a disconnection, keeping the work done offline:
c, err := collab.Join(ctx, conn, collab.ClientConfig{
Document: "notes",
Site: 1,
Resume: savedSnapshot, // from Client.Snapshot()
})
Shape of a session
One bidirectional stream per participant per document. The client opens with a
Join; the server answers with a Welcome holding either the whole document or,
for a participant that says what it already has, only what it missed — plus who
else is present and where the server stands, so the participant can push whatever
it wrote while away. After that, operations and presence flow both ways.
What it guarantees
- Convergence, proven end to end. The acceptance test runs three replicas across two runtimes — one native, two compiled to WebAssembly and executed by Node through a real WebSocket — editing concurrently and converging on the same text.
- Offline work is never stranded. A resuming participant pushes what the server lacks and is sent what it missed. Both directions are tested.
- Nobody stalls the document. A participant that stops reading is
disconnected with
ResourceExhaustedand caught up when it rejoins, rather than holding everyone up or being served state that is quietly out of date. - Nothing is trusted. Malformed operations, presence, version vectors and
snapshots are each refused with
InvalidArgument, on both sides of the wire. - One replica identity per participant. Two participants sharing a site is
silent data loss rather than a conflict — both mint the same operation
identities for different characters — so the arriving session takes the
identity and the one already holding it is disconnected with
Aborted. Site zero, the server's own replica, is refused outright. - Documents outlive sessions. The last participant out writes the document;
Server.Flushwrites it without waiting. A write that fails is retried rather than forgotten.
What a binding needs
An editor cannot be handed the whole text on every keystroke somebody else
makes: that throws away the selection, the scroll position and every decoration.
Changes says something happened, TakeChanges says what — the edits, in the
order they have to be made.
c.TakeChanges() // []crdt.Change: remove this many here, put this there
c.Anchor(pos) // a handle on a character, for a comment or a stored selection
c.Position(anchor) // where it is now, or where it was if it has gone
c.AuthorRuns() // the text split by who wrote each stretch
c.InsertUTF16(pos, text) // offsets in the units a browser counts
A browser counts UTF-16 code units, and an emoji is one character and two units. A session that took the browser's offsets for runes would edit in the wrong place, silently, from the first emoji onwards — so the same operations are addressed both ways, and an offset landing inside a character is refused rather than moved.
Who may open what
Config.Authorize is asked once per session, after the join arrives and before
the document is touched, so a refused session neither reads the store nor reveals
whether the document exists:
collab.NewServer(collab.Config{
Authorize: func(ctx context.Context, document string, site crdt.SiteID) error {
return myACL.Check(userFrom(ctx), document)
},
})
It lives here rather than in a gRPC interceptor, which is where one would first look for it: an interceptor sees the method and the request metadata, and the document being joined is in neither — it arrives in the stream's first message. Authentication, being per connection rather than per document, still belongs in an interceptor; the context carries whatever it put there.
Persistence
Store is a two-method seam — Load and Save on snapshots, which are
self-contained, so a document restored from one can still serve a participant
that has been away. MemoryStore is the default.
collab/pgstore keeps documents in PostgreSQL, over a plain *sql.DB
and with no driver of its own, so the caller picks one:
db, _ := sql.Open("pgx", os.Getenv("DATABASE_URL"))
store, _ := pgstore.New(db)
store.Migrate(ctx)
srv := collab.NewServer(collab.Config{Store: store})
It is a module of its own, so importing collab does not drag a database driver
into anyone's build. Its tests run against a real PostgreSQL — CI fails the job
if one is missing rather than skipping it.
Status
Version 0.1. 100% statement coverage, race-clean, six-arch CI, and the WebAssembly end-to-end test running on every pull request — where a missing toolchain is a failure, not a skipped test.
License
BSD-3-Clause — see LICENSE. Copyright the go-crdt authors.
Documentation
¶
Overview ¶
Package collab carries a github.com/go-crdt/crdt document between the people editing it: a gRPC service, a server that hosts documents, and a client that joins one.
The service is thin on purpose. The document is a CRDT, so the server never transforms an operation and never decides an outcome — it applies what it is sent to its own replica and hands it to everyone else. Two consequences follow that a server-authoritative design cannot offer: a participant may edit while disconnected and reconcile later, and the server may be restarted or replaced without any client losing work.
Over what ¶
Two carriers, and which one to use is decided by where the code runs rather than by taste. WebSocket carries a session's own framing over a plain WebSocket; GRPC carries it over gRPC. One server serves both at once — Server.ServeWebSocket beside the registered service — and a participant on each edits the same document.
The reason there are two is measured. Everything a session carries is bytes some encoder in github.com/go-crdt/crdt produced and will check on arrival, so protobuf is describing fields nobody reads through it — and compiled to wasm its reflection and registry machinery cannot be linked away. The browser test client, gzipped, is 919 KB over the framing and 4 461 KB over gRPC, against 633 KB for the CRDT alone. Outside a browser none of that matters, and gRPC brings deadlines, interceptors and the tooling built around them.
The client builds for js/wasm either way, so a browser tab and a server run the same code down to the merge.
A document holds named parts ¶
What an editor holds is not one structure: the text of a file, the comments anchored into it, the record of who changed what, the messages beside it, the cells of a sheet. A document here is a github.com/go-crdt/crdt.Composite, so they travel together — one snapshot, one version, one decision about who may open it, and no instant at which the set of them disagrees.
A caller reaches for a part by name and gets a handle: Client.Text, Client.List, Client.Map. A handle edits and publishes in one step, which is why it exists rather than the replicated structure itself — a caller editing that directly would produce operations nobody ever heard, and drift away from everyone else while its own screen looked right.
Shape of a session ¶
One bidirectional stream per participant per document. The client opens with a collabpb.Join; the server answers with a collabpb.Welcome holding either the whole document or, for a participant that says what it already has, only what it missed. After that, operations and presence flow both ways until either side hangs up.
Index ¶
- Constants
- Variables
- type Client
- func (c *Client) Changes() <-chan struct{}
- func (c *Client) Close() error
- func (c *Client) Document() string
- func (c *Client) Done() <-chan struct{}
- func (c *Client) Err() error
- func (c *Client) List(name string) (*List, error)
- func (c *Client) Map(name string) (*Map, error)
- func (c *Client) Parts() []crdt.Part
- func (c *Client) Peers() []awareness.Peer
- func (c *Client) SetCursor(cursor awareness.Cursor, meta map[string]string) error
- func (c *Client) Site() crdt.SiteID
- func (c *Client) Snapshot() []byte
- func (c *Client) TakeChanges() []crdt.PartChange
- func (c *Client) Text(name string) (*Text, error)
- func (c *Client) Version() crdt.CompositeVersion
- type ClientConfig
- type Config
- type List
- func (l *List) Append(values ...[]byte) error
- func (l *List) Delete(pos, count int) error
- func (l *List) Get(pos int) ([]byte, error)
- func (l *List) Insert(pos int, values ...[]byte) error
- func (l *List) Len() int
- func (l *List) Name() string
- func (l *List) Part() crdt.Part
- func (l *List) Values() [][]byte
- type Map
- type MemoryStore
- type Server
- type Store
- type Text
- func (t *Text) Anchor(pos int) (crdt.ID, error)
- func (t *Text) AnchorUTF16(pos int) (crdt.ID, error)
- func (t *Text) AuthorRuns() []crdt.AuthorRun
- func (t *Text) AuthorRunsUTF16() []crdt.AuthorRun
- func (t *Text) Delete(pos, length int) error
- func (t *Text) DeleteUTF16(pos, length int) error
- func (t *Text) Insert(pos int, text string) error
- func (t *Text) InsertUTF16(pos int, text string) error
- func (t *Text) Len() int
- func (t *Text) LenUTF16() int
- func (t *Text) Name() string
- func (t *Text) Part() crdt.Part
- func (t *Text) Position(anchor crdt.ID) (int, bool)
- func (t *Text) PositionUTF16(anchor crdt.ID) (pos int, ok bool)
- func (t *Text) String() string
- func (t *Text) Visible(anchor crdt.ID) bool
- type Transport
- type WebSocketOption
Constants ¶
const DefaultBacklog = 256
DefaultBacklog is how many messages may be queued for one participant before the server gives up on it. See Config.
Variables ¶
var ErrClosed = errors.New("collab: session closed")
ErrClosed is why a session ended when this participant closed it, and what an edit made afterwards returns.
var ErrProtocol = errors.New("collab: unexpected message")
ErrProtocol reports a message that is not part of a session: a kind that cannot arrive at that moment — a second welcome, or a join halfway through — or bytes that are not a message at all.
var ErrTransport = errors.New("collab: transport")
ErrTransport reports a carrier that could not be opened or that failed.
Functions ¶
This section is empty.
Types ¶
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
A Client is one participant's view of a document: a replica that edits locally and is kept in step with everyone else.
It is safe for concurrent use. It builds for js/wasm, so a browser tab runs this code and the server's merge logic unchanged.
func Join ¶
Join opens a session over transport and returns once the document has arrived, so the client is usable the moment it is returned.
Use WebSocket unless there is a reason not to; it is what the same code compiled for a browser can afford. GRPC is there for a native peer that wants what gRPC brings with it.
The session lives until ctx is cancelled or Client.Close is called.
func (*Client) Changes ¶
func (c *Client) Changes() <-chan struct{}
Changes receives a value whenever the document or the participants changed. It coalesces: a reader that is slow sees one wake-up, not a queue of them.
func (*Client) Close ¶
Close ends the session. The local document is left intact, so its Client.Snapshot can resume later.
func (*Client) Done ¶
func (c *Client) Done() <-chan struct{}
Done is closed when the session has ended, whatever the reason.
func (*Client) Err ¶
Err returns why the session ended, or nil while it is still running. Once Client.Done is closed it is never nil: a session that was closed deliberately reports ErrClosed rather than the transport's cancellation.
func (*Client) Parts ¶ added in v0.10.0
Parts returns the parts this replica holds, in the canonical order. A part that has never been written to is not among them.
func (*Client) Peers ¶
Peers returns the other participants and where their cursors are, ordered by site.
func (*Client) SetCursor ¶
SetCursor publishes where this participant is. meta carries whatever the editor wants shown — a display name, a colour — and is not interpreted here.
Cursor positions are ephemeral and are never persisted.
func (*Client) Snapshot ¶
Snapshot returns the document in a form ClientConfig.Resume accepts, which is how a participant keeps its place across a disconnection.
func (*Client) TakeChanges ¶ added in v0.7.0
func (c *Client) TakeChanges() []crdt.PartChange
TakeChanges returns the edits made by everyone else since it was last called, in the order a view of the text has to make them, and forgets them.
It pairs with Client.Changes: that says something happened, this says what. A view that only ever applies these holds what the document holds — see crdt.Change.
Local edits are not reported. A caller that made them already knows.
func (*Client) Text ¶
Text returns a handle on the text part with this name, which is created the first time anybody writes to it. The name is arbitrary UTF-8 and is expected to carry structure — "file:src/main.tex". An empty or invalid name is refused; see crdt.Part.
func (*Client) Version ¶
func (c *Client) Version() crdt.CompositeVersion
Version returns what this participant holds, for ClientConfig.Resume or for diagnostics.
type ClientConfig ¶
type ClientConfig struct {
// Document names the document to join. It is created if it does not exist.
Document string
// Site is this participant's replica identity, and must differ from every
// other participant's in the document. See [crdt.DeriveSiteID].
Site crdt.SiteID
// Resume is a snapshot from an earlier session, obtained from
// [Client.Snapshot]. When set, the participant keeps the work it did while
// disconnected and is sent only what it missed, rather than the whole
// document.
Resume []byte
}
ClientConfig describes a participant joining a document.
type Config ¶
type Config struct {
// Store keeps documents between sessions. Defaults to a [MemoryStore].
Store Store
// Backlog is how many messages may be queued for one participant.
// A participant that falls further behind than this is disconnected with
// ResourceExhausted rather than served stale state or allowed to stall
// everyone else; it rejoins and is caught up from its version vector.
// Defaults to [DefaultBacklog].
Backlog int
// Authorize, when set, decides whether a participant may open a document.
// It is asked once per session, after the join arrives and before the
// document is touched, so a refused session neither reads the store nor
// reveals whether the document exists.
//
// This belongs here rather than in a gRPC interceptor, which is where one
// would first look for it: an interceptor sees the method and the request
// metadata, and the document being joined is in neither — it arrives in the
// stream's first message. Anything deciding per document has to run after
// that message, which means here. Authentication, which is per connection
// rather than per document, still belongs in an interceptor; ctx carries
// whatever it put there.
//
// Returning a gRPC status error passes that status to the participant
// unchanged; any other error is reported as PermissionDenied.
Authorize func(ctx context.Context, document string, site crdt.SiteID) error
}
Config configures a Server.
type List ¶ added in v0.10.0
type List struct {
// contains filtered or unexported fields
}
A List is a handle on one list part — comments, a change log, the messages beside a document.
func (*List) Append ¶ added in v0.10.0
Append adds values after the last one, which is what a chat or a log does.
func (*List) Insert ¶ added in v0.10.0
Insert adds values at index pos, locally and then everywhere.
func (*List) Values ¶ added in v0.10.0
Values returns copies of every value present, in order. It is what a view of a list reads when it is told the list changed; see crdt.PartChange.
type Map ¶ added in v0.10.0
type Map struct {
// contains filtered or unexported fields
}
A Map is a handle on one map part, such as the cells of a sheet.
func (*Map) Delete ¶ added in v0.10.0
Delete removes key, locally and then everywhere. It writes a tombstone whether or not this replica holds the key; see crdt.Map.
func (*Map) Get ¶ added in v0.10.0
Get returns a copy of the value at key, and whether the key is present. It is what a view reads for each key a crdt.PartChange names.
type MemoryStore ¶
type MemoryStore struct {
// contains filtered or unexported fields
}
MemoryStore keeps documents in memory. It is the default, it is what the tests use, and it is enough for a single process that does not need to survive a restart. Anything else — Postgres, object storage — implements Store.
func (*MemoryStore) Documents ¶
func (s *MemoryStore) Documents() []string
Documents returns the names of the documents held, which is what a caller needs to inspect or migrate a store.
type Server ¶
type Server struct {
collabpb.UnimplementedCollabServer
// contains filtered or unexported fields
}
A Server hosts documents. Register it with collabpb.RegisterCollabServer on any grpc.Server — over github.com/grpc-transports/websocket for browsers, over plain TCP for anything else.
Documents stay in memory once opened, so a long-lived server holds every document it has served. Call Server.Flush to persist them.
func (*Server) Flush ¶
Flush persists every document that has changed since it was last written. A server that wants durability without waiting for participants to leave calls this on a timer, or before shutting down.
func (*Server) ServeWebSocket ¶ added in v0.11.0
ServeWebSocket returns an http.Handler that runs sessions over WebSockets — the carrier a browser can afford, and the one WebSocket dials.
Mount it where the browser will reach it. Everything else is the same server: the same documents, the same store, the same Config.Authorize, and a participant here edits the same document as one arriving over gRPC.
origins, when not empty, are the Origin header values allowed to open a session, which is the check that stops another site's page from opening one with the visitor's cookies. An empty list allows only same-origin requests.
type Store ¶
type Store interface {
// Load returns the snapshot for a document, or nil if there is none yet.
// Returning nil is how a store says "new document", and is not an error.
Load(ctx context.Context, document string) ([]byte, error)
// Save records the current snapshot, replacing any previous one.
Save(ctx context.Context, document string, snapshot []byte) error
}
A Store keeps documents between sessions. It holds snapshots, which are self-contained: a document restored from one can still serve a participant that has been away, because the snapshot carries the whole history.
Implementations must be safe for concurrent use.
type Text ¶ added in v0.10.0
type Text struct {
// contains filtered or unexported fields
}
A Text is a handle on one text part: the buffer of a file, and what an editor binds to.
func (*Text) Anchor ¶ added in v0.10.0
Anchor returns the identity of the character at rune offset pos, which keeps naming that character however the text moves around it. It is what a comment or a stored selection should hold; see crdt.Doc.Anchor.
func (*Text) AnchorUTF16 ¶ added in v0.12.0
AnchorUTF16 is Text.Anchor with pos counted in UTF-16 code units, the units a page counts in. An offset falling between the two units of one character is refused rather than rounded; see crdt.ErrSurrogateBoundary.
func (*Text) AuthorRuns ¶ added in v0.10.0
AuthorRuns splits the visible text into stretches by who wrote them, which is what colouring a document by author needs.
func (*Text) AuthorRunsUTF16 ¶ added in v0.12.0
AuthorRunsUTF16 is Text.AuthorRuns with every offset and length counted in UTF-16 code units, so that a page can colour the string it holds without converting anything by hand.
func (*Text) Delete ¶ added in v0.10.0
Delete removes length runes at rune offset pos, locally and then everywhere.
func (*Text) DeleteUTF16 ¶ added in v0.10.0
DeleteUTF16 removes length code units at an offset counted in the same units.
func (*Text) Insert ¶ added in v0.10.0
Insert adds text at rune offset pos, locally and then everywhere.
func (*Text) InsertUTF16 ¶ added in v0.10.0
InsertUTF16 adds text at an offset counted in UTF-16 code units.
func (*Text) LenUTF16 ¶ added in v0.10.0
LenUTF16 returns the length a browser would report, counting UTF-16 code units. Its companions InsertUTF16 and DeleteUTF16 take offsets in the same units, so a caller in the browser never converts by hand; see crdt.Doc.
func (*Text) Part ¶ added in v0.10.0
Part names this handle's part, which is what a crdt.PartChange from Client.TakeChanges carries.
func (*Text) Position ¶ added in v0.10.0
Position returns where the character an anchor names sits now — or where it was, if it has been deleted. See crdt.Doc.Position.
func (*Text) PositionUTF16 ¶ added in v0.12.0
PositionUTF16 is Text.Position with the offset reported in UTF-16 code units. ok is false for an anchor this replica has never seen, exactly as it is there — which is not the same question as whether the character is still in the text; that one is Text.Visible.
type Transport ¶ added in v0.11.0
type Transport interface {
// contains filtered or unexported methods
}
A Transport is how a participant reaches a server. WebSocket works anywhere, a browser included; GRPC works outside one.
There are two because of what they cost where they run. Outside a browser a carrier costs nothing anybody notices, and gRPC brings deadlines, interceptors and everything already built around them. Inside one it is paid for on every load: protobuf alone is six times the size of the whole CRDT compiled to wasm — see wire.go for the measurements — so the browser gets a framing of four message kinds over a plain WebSocket instead.
Both carry the same session, byte for byte in the fields that matter, because every field in these messages is something github.com/go-crdt/crdt encoded and will check on arrival. A participant on one and a participant on the other can edit the same document.
func GRPC ¶ added in v0.11.0
func GRPC(conn grpc.ClientConnInterface) Transport
GRPC returns a transport that opens sessions on a gRPC connection.
It is deliberately not the default. Everything it carries is bytes some encoder in github.com/go-crdt/crdt produced, so protobuf is describing fields nobody reads through it — and compiled for a browser it costs six times the CRDT itself. Outside a browser that does not matter, and gRPC brings deadlines, interceptors and the tooling built around them, which is reason enough to keep it. See Transport and WebSocket.
func WebSocket ¶ added in v0.11.0
func WebSocket(url string, opts ...WebSocketOption) Transport
WebSocket returns a transport that opens sessions at url, which is "ws://" or "wss://" and the path the server's handler is mounted at.
This is the transport a browser uses, and the one to reach for by default: it is what the same code compiled to wasm can afford. See Transport.
type WebSocketOption ¶ added in v0.11.0
type WebSocketOption func(*wsTransport)
A WebSocketOption configures WebSocket.
func WithHTTPHeader ¶ added in v0.11.0
func WithHTTPHeader(h http.Header) WebSocketOption
WithHTTPHeader sends these headers with the opening handshake, which is where a cookie or a bearer token goes when the participant is not a browser.
It does not exist for a browser, because a page cannot put a header on a WebSocket handshake — and does not need to, since the browser sends the cookies for that origin itself. Code meant to run in both places should let the cookie do the work; see Config.Authorize.
Source Files
Directories