Documentation
¶
Overview ¶
Package gitstore keeps collaborative documents in a git repository: the state, so it comes back exactly, and the text, so the history is one a person can read.
Why both ¶
A collab.Store is handed a snapshot, which is the whole document — characters with their identities, who wrote each of them, the comments anchored to them, what has been deleted and by whom. Committing that alone would version the document perfectly and give a repository nobody can read: a diff between two snapshots says nothing, and a clone is a directory of opaque files.
Writing only the text is the other half of the same mistake. It reads beautifully and it is lossy: the comments, the authorship and the identities every anchor depends on are gone, so a document restored from it is a different document that happens to say the same words. Somebody's comment on a sentence would come back attached to nothing.
So this writes both, from one snapshot, in one commit. The text is what the history is about and what a diff shows; the state is what a restore uses. They cannot drift, because neither is edited: both are written from the same bytes at the same moment.
What a release is ¶
A version vector is not one. It says which operations a replica holds, which is a causal fact and not a decision anybody made — two replicas can hold the same document and describe it differently, and neither is "version 3".
A release is a decision, so it is a tag: Store.Release names a commit that already exists. What is tagged is a state somebody chose, and it restores exactly, because the snapshot is in the commit beside the text.
Two instances ¶
A repository both of them can reach is a channel between them and not only a record of what they did. Store.Push sends what this instance committed, Store.Pull brings back what another one did and merges it, and what the two of them disagree about is the state file — the one conflict in this design that never needs a person, because a snapshot is a set of operations and merging two sets of operations is what this package does for a living.
The latency is a pull interval rather than a link, and what it costs is a repository both instances may reach rather than two servers up and reachable at once. A store with no remote does none of it and is exactly what it was before there was anything to configure.
Example (Federation) ¶
Two institutions federating: what an operator writes.
Everything here is deliberate. The identity comes from a scoped federated identifier, because a bare one is the same replica in every datacentre in the world. The store is two stores, so the same save answers both "what is it now" and "what did it say last Tuesday". The link is FollowWithRetry rather than Follow, because a link that drops stays dropped and the loop that brings it back is the one thing everybody writes and most people write without jitter. And AuthorizeOperations, rather than Authorize alone, because a participant speaks for itself while a link speaks for an institution.
The carrier here is collab.Pipe, which keeps the example to one process. A real deployment dials the other institution — collab.GRPC over a connection, or a WebSocket — and changes nothing else: the dialler is the only part that knows there is a network.
package main
import (
"context"
"errors"
"fmt"
"os"
"os/exec"
"strings"
"time"
"github.com/go-crdt/collab"
"github.com/go-crdt/collab/gitstore"
"github.com/go-crdt/crdt"
)
// siteFor derives a replica identity from a federated identifier.
//
// The identifier must be scoped, and this is the whole of what GÉANT settles
// for this design: eduGAIN hands over an identifier that is globally unique
// because only the home organisation issues inside its own scope, so two
// institutions that have never spoken cannot mint the same one.
//
// A bare identifier would be the one failure this design cannot merge its way
// out of: DeriveSiteID is a hash, so "42" is the same site on every instance in
// the world, and two institutions each with a user "42" would silently share a
// replica.
func siteFor(eppn string) crdt.SiteID { return crdt.DeriveSiteID([]byte(eppn)) }
// federates says whether an institution accepts work carried on behalf of a
// site. Real deployments would ask their federation metadata; this one holds
// the scopes it has agreed with, which is the same question asked of a smaller
// register.
func federates(name string, with ...string) func(context.Context, string, crdt.SiteID, []crdt.PartOps) error {
allowed := map[crdt.SiteID]bool{}
for _, scope := range with {
for _, who := range []string{"ada", "grace", "link"} {
allowed[siteFor(who+"@"+scope)] = true
}
}
return func(_ context.Context, _ string, _ crdt.SiteID, batches []crdt.PartOps) error {
for _, batch := range batches {
for _, site := range sitesIn(batch) {
if !allowed[site] {
return fmt.Errorf("site %d is not in a scope %s federates with", site, name)
}
}
}
return nil
}
}
// sitesIn reports which replicas wrote the operations in a batch.
func sitesIn(batch crdt.PartOps) []crdt.SiteID {
seen := map[crdt.SiteID]bool{}
var out []crdt.SiteID
for _, op := range batch.Text {
if !seen[op.ID.Site] {
seen[op.ID.Site] = true
out = append(out, op.ID.Site)
}
}
return out
}
func gitIn(dir string, args ...string) (string, error) {
cmd := exec.Command("git", args...)
cmd.Dir = dir
out, err := cmd.CombinedOutput()
if err != nil {
return string(out), errors.New(strings.TrimSpace(string(out)))
}
return string(out), nil
}
func main() {
must := func(err error) {
if err != nil {
panic(err)
}
}
// Deferred first so it runs last: the directories go once the servers have
// stopped writing into them, not while.
parisDir, lyonDir := tempDir(), tempDir()
defer func() { _ = os.RemoveAll(parisDir); _ = os.RemoveAll(lyonDir) }()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
// One institution. Its store is a database-shaped one beside a git
// repository; MultiStore writes both and merges what both hold, so adding
// the repository to a server that was already running backfills it.
open := func(dir, name, scope string, rule func(context.Context, string, crdt.SiteID, []crdt.PartOps) error) (*collab.Server, *gitstore.Store) {
repo, err := gitstore.New(dir, gitstore.WithAuthor(name, name+"@"+scope))
must(err)
return collab.NewServer(collab.Config{
Store: collab.NewMultiStore(collab.NewMemoryStore(), repo),
PersistEvery: time.Millisecond,
AuthorizeOperations: rule,
}), repo
}
paris, parisGit := open(parisDir, "paris", "paris.example.ac",
federates("paris", "paris.example.ac", "lyon.example.ac"))
lyon, _ := open(lyonDir, "lyon", "lyon.example.ac", nil)
// A server is closed, not merely cancelled: it has documents to write out,
// and a directory removed while it is still writing is a race an operator
// would inherit from this example.
defer func() {
stop, done := context.WithTimeout(context.Background(), 5*time.Second)
defer done()
_ = paris.Close(stop)
_ = lyon.Close(stop)
}()
// Lyon follows Paris, and keeps following: the policy for coming back
// belongs to the operator, and this is one written down.
dial := func(context.Context) (collab.Transport, error) {
transport, conn := collab.Pipe()
go func() { _ = paris.ServePipe(ctx, conn) }()
return transport, nil
}
go func() {
_ = lyon.FollowWithRetry(ctx, dial, "project:paper",
crdt.DeriveSiteID([]byte("link@lyon.example.ac")),
collab.RetryPolicy{Wait: 50 * time.Millisecond, Ceiling: 2 * time.Second})
}()
// A person at each institution, editing the document their own server holds.
join := func(s *collab.Server, eppn string) *collab.Client {
transport, conn := collab.Pipe()
go func() { _ = s.ServePipe(ctx, conn) }()
c, err := collab.Join(ctx, transport, collab.ClientConfig{
Document: "project:paper",
Site: crdt.DeriveSiteID([]byte(eppn)),
})
must(err)
return c
}
ada, grace := join(paris, "ada@paris.example.ac"), join(lyon, "grace@lyon.example.ac")
defer func() { _ = ada.Close(); _ = grace.Close() }()
adaBody, err := ada.Text("file:paper.tex")
must(err)
must(adaBody.Insert(0, "On rivers."))
graceBody, err := grace.Text("file:paper.tex")
must(err)
waitUntil(func() bool { return graceBody.String() == "On rivers." })
must(graceBody.Insert(graceBody.Len(), " They run downhill."))
waitUntil(func() bool { return adaBody.String() == "On rivers. They run downhill." })
fmt.Println("paris:", adaBody.String())
fmt.Println("lyon: ", graceBody.String())
// The repository holds the text a person reads and the state a document is
// restored from, in the same commit, so the two cannot drift.
waitUntil(func() bool {
out, err := gitIn(parisDir, "show", "HEAD:project%3Apaper/paper.tex")
return err == nil && strings.TrimSpace(out) == "On rivers. They run downhill."
})
out, err := gitIn(parisDir, "show", "--stat", "--format=", "HEAD")
must(err)
fmt.Println("committed:", strings.Join(filesIn(out), " and "))
// A release names a commit that already exists.
must(parisGit.Release(ctx, "v1.0", "the first version we showed anybody"))
kept, err := parisGit.At("project:paper", "v1.0")
must(err)
restored, err := crdt.LoadComposite(1, kept)
must(err)
released, err := restored.Text("file:paper.tex")
must(err)
fmt.Println("v1.0: ", released.String())
}
// tempDir is what an example has instead of t.TempDir. Its caller removes it:
// an example somebody copies should not teach them to leave directories behind.
func tempDir() string {
dir, err := os.MkdirTemp("", "gitstore-example")
if err != nil {
panic(err)
}
return dir
}
// waitUntil is what an example has instead of a test's deadline helper. A
// program would wait on Client.Changes rather than poll.
func waitUntil(want func() bool) {
deadline := time.Now().Add(15 * time.Second)
for !want() {
if time.Now().After(deadline) {
panic("the two replicas did not converge")
}
time.Sleep(2 * time.Millisecond)
}
}
// filesIn reads the paths out of a git --stat, so the example can say what a
// commit held without printing a hash that changes every run.
func filesIn(stat string) []string {
var out []string
for _, line := range strings.Split(stat, "\n") {
name, _, found := strings.Cut(strings.TrimSpace(line), "|")
if !found {
continue
}
out = append(out, strings.TrimSpace(name[strings.LastIndex(name, "/")+1:]))
}
return out
}
Output: paris: On rivers. They run downhill. lyon: On rivers. They run downhill. committed: paper.tex and state.crdt v1.0: On rivers. They run downhill.
Index ¶
- Variables
- func Merge(ours, theirs []byte) ([]byte, error)
- type Option
- type Remote
- type Revision
- type Store
- func (s *Store) At(document, revision string) ([]byte, error)
- func (s *Store) Documents() ([]string, error)
- func (s *Store) History(document string) ([]Revision, error)
- func (s *Store) Load(_ context.Context, document string) ([]byte, error)
- func (s *Store) Pull(ctx context.Context) error
- func (s *Store) Push(ctx context.Context) error
- func (s *Store) Reconcile(ctx context.Context, document string, theirs []byte) error
- func (s *Store) Release(ctx context.Context, name, message string) error
- func (s *Store) Save(ctx context.Context, document string, snapshot []byte) error
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ErrNoDocument = errors.New("gitstore: a document must have a name")
ErrNoDocument reports a document with no name, which cannot have a directory.
var ErrNoRemote = errors.New("gitstore: no remote is configured")
ErrNoRemote reports an operation that needs a remote on a store that has none. It is not federation failing; it is being asked to federate by something that was never told where.
Functions ¶
func Merge ¶
Merge combines two snapshots of one document into one that holds everything both hold.
It is what makes a git repository a channel between instances rather than only a record. Two servers sharing a repository will diverge — that is what working separately means — and git will then report a conflict on the state file, which is the one conflict in this design that never needs a person: a snapshot is a set of operations, and the merge of two sets of operations is what this package does for a living.
The rendered text is not merged and must not be. It is derived, so whichever side of a conflict is taken is wrong: it is written again from the merged state, and a conflict marker never reaches a document.
Merging is symmetric to the byte — Merge(a, b) and Merge(b, a) encode identically — because the snapshot encoding is canonical and both hold the same operations. That is what a merge driver needs: two instances resolving the same conflict independently must reach the same commit, or they have merely disagreed somewhere new.
It is collab.MergeSnapshots, which is where this now lives: the operation is about snapshots and not about git, it needs nothing this module has, and keeping a second copy here would mean two functions that have to agree forever about what merging means. This one stays because it is the name this package's own vocabulary uses, and because Reconcile and a pull both call it.
One thing changed in the move, and for the better: either side may now be empty, which is how a store says it has never held the document, and merging with nothing gives back the other side. It used to be an error, which is why Reconcile checks for it before calling. That check stays, because it also saves reading a document back in order to hand it straight to Save, and because saying what an absent document means is worth a line.
A pull's check is a different one and must not be confused with it: it tests the error, not the emptiness, because a state that will not open is not a state this instance does not have.
Types ¶
type Option ¶
type Option func(*Store)
An Option changes how a store writes.
func WithAuthor ¶
WithAuthor names who the commits are by. The default is the store itself, which is honest: a commit here is made by a server writing down what a document holds, not by whoever last typed into it. Authorship of the text is in the snapshot, per character, and is not what a commit's author line means.
func WithFiles ¶
WithFiles decides which parts are written out as files and where.
The default writes a text part named "file:<path>" to <path>, which is the convention a document of files already uses, and writes nothing for anything else — a list of chat messages is not a file, and rendering it as one would invent a format nobody asked for. Its content is in the snapshot either way.
func WithRemote ¶ added in v0.2.0
WithRemote gives the store a repository to push to and pull from.
type Remote ¶ added in v0.2.0
type Remote struct {
// URL is anything go-git can reach: an https or ssh address, or the path
// of a repository on this machine. Empty means there is no remote.
URL string
// Auth is asked what to authenticate as, and may be nil, which means
// nothing — see [Store.Push] for why this package does not go looking.
Auth func(context.Context) (transport.AuthMethod, error)
// PushFailed is told when a push made by a save does not reach the remote,
// and may be nil, which drops it.
//
// A save does not fail for that — see [Store.Save] — so this is the only
// place the failure is heard. Whether it is logged, counted or paged on is
// the operator's, but a store that has not federated since yesterday
// should be saying so somewhere.
PushFailed func(error)
}
A Remote is the repository this one federates through: where Store.Push sends what was committed here and where Store.Pull finds what was committed elsewhere.
A zero Remote is no remote, and a store without one writes locally and does exactly what it did before any of this existed. Federating is something an operator turns on, not something a store does because it can.
type Revision ¶
type Revision struct {
// Hash names the commit, and is what [Store.At] takes.
Hash string
// When it was written down.
When time.Time
// Message is the commit's subject.
Message string
// Release is the tag on it, if it has one.
Release string
}
A Revision is one commit of a document's history.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is a collab.Store backed by a git repository.
It is safe for concurrent use. go-git is not — two goroutines committing to one worktree race on the index — so every operation here takes one lock. A server saves a document every few seconds, so what that costs is nothing worth measuring against what a corrupt index costs.
A push and a fetch are held under that same lock, and they are the one thing here that waits on somebody else's machine. A remote that hangs therefore hangs the store, which is why every remote operation takes a context and nothing in this package invents one: how long a document may go unsaved because a git host is slow is a decision, and it belongs to whoever passes it.
func (*Store) At ¶
At returns a document's snapshot as it stood at a revision, which may be a commit hash or a release name.
What comes back is the whole document, not the text: the comments, the authorship and the identities every anchor depends on are in it, so a document restored from a release is the document that was released rather than one that says the same words.
func (*Store) Pull ¶ added in v0.2.0
Pull brings back what other instances committed and merges it into this one.
It fetches the remote's branch and, if that holds anything this instance does not, merges every document there into the copy here and records the result as one commit with two parents. The second parent is the whole point: it is what puts their history into this branch, and a branch that does not hold theirs cannot be pushed. So the commit is made even when the documents come out unchanged — what it records is not that anything moved but that this instance now holds what the other one wrote.
Nothing is checked out. Git's own merge would write over the worktree, and the worktree here may be holding a save whose commit was lost — see Store.Save — so taking a tree from anywhere else would discard exactly what that failure direction exists to protect. The merged state is written from this side instead, and the text is written again with it, so a conflict marker never reaches a document.
It does not push the merge afterwards. Whether the other instances see it now or at the next save is a policy, and the operator is the one with the loop.
func (*Store) Push ¶ added in v0.2.0
Push sends everything committed here to the remote.
It pushes a branch and the tags on it rather than a commit, so whatever has accumulated locally goes in one go and a push that failed yesterday costs nothing to recover from beyond the next one arriving.
A push that is not a fast-forward fails rather than forcing. That means another instance has committed something this one has not seen, and the answer to that is Store.Pull — which merges, which is the operation this whole package exists for — and not overwriting a history somebody else is also writing.
func (*Store) Reconcile ¶
Reconcile merges another instance's snapshot of a document into this one's and commits the result, text and all.
It is the operation a pull performs after git has said the state file conflicts: not "take one side", which would throw away whatever the other instance did, but "hold both", which is the only answer that loses nothing.
func (*Store) Release ¶
Release tags the state as it stands now, which is what a version somebody decided on is.
It tags the commit the last Store.Save made rather than making one of its own: a release is a name for a state that already exists, and inventing a commit for it would put a state in the history that nobody was ever editing.
It pushes on the same terms a save does, because a release nobody else can name is not one, and because one rule for the whole store is easier to rely on than two: the tag is here whatever the network did, every push carries every tag, so the next push that arrives carries this one.
func (*Store) Save ¶
Save writes the snapshot and the text it renders to, commits both, and — if there is a remote — sends the commit on.
A save that changes nothing makes no commit. A server persists on a timer, so most saves of a document nobody is editing are identical to the last, and a history of thousands of empty commits is a history nobody can read.
What a failure leaves ¶
The files are written before they are staged, so a save that fails at staging, at reading the worktree or at committing leaves the worktree holding the new state and the history holding the old. **The commit is lost and the document is not**: the next Load reads the worktree and returns the newer of the two, and the next successful save commits it.
That is the direction to fail in. The other one — putting the old bytes back so the two agree — would mean a full disk losing the work of everyone editing rather than losing a line of history, and a store whose whole point is to keep what was written should not be the thing that discards it.
What a failed push leaves ¶
A push that does not arrive does not fail the save, and the commit is not undone either. The two failures are not the same kind: a save that cannot write has not stored the document, and a push that cannot reach the remote has stored it and not yet shared it. Reporting the second as the first would make an unreachable peer look like a full disk, and what a server does about those two is not the same thing — one of them is a reason to stop.
Argued from what the caller could do instead: nothing that helps. The commit is here, it is on this instance's branch, and git pushes a branch rather than a commit, so the next push that gets through carries this one and everything after it — with nothing queued, nothing replayed and nothing to reconcile in the meantime. Which is why a save that changed nothing and made no commit still pushes when a commit here has not reached the remote: that save is the retry, and it is the only one there needs to be. A save with nothing owed says nothing to anybody, because a server persists on a timer and a connection every few seconds to report that a document is unchanged is how a remote becomes something an operator switches off.
What must not happen is that it goes unheard, so Remote.PushFailed is told. A store whose remote has been unreachable since yesterday is still storing documents perfectly and has stopped federating, and only somebody's log can say so.
Source Files