Documentation
¶
Overview ¶
Package qi provides transport-agnostic helpers for replicating JSON object documents on top of github.com/iamseth/yin.
yin owns the CRDT data structures, version vectors, JSON object encoding, and delta formats. qi layers visible-file, checkpoint, and transport-agnostic session behavior around those primitives. Store is the owning, synchronized generation-validation boundary; JSONFile and Session are lower-level and require caller coordination.
NewStore constructs an in-memory Store. OpenDurableStore instead makes an application-owned CheckpointCommitter the acceptance boundary for every mutation: ordinary errors leave memory and authority unchanged, while an error wrapping ErrDurabilityUncertain means publication occurred and memory was updated to match. The application loads the initial checkpoint bytes and must give the Store exclusive ownership of their matching authority. OpenStore is the filesystem-backed CLI-oriented convenience constructor when dataDir is non-empty. Its visible path is startup input only; Store retains no paths and exposes later projection I/O through byte-oriented methods.
Visible JSON, qi checkpoint envelopes, yin snapshots, and yin deltas are distinct artifacts. A committer receives complete restart state; visible JSON and separately requested synchronization artifacts are not a substitute for the bytes accepted by that authority. The p2p subpackage provides pull-based libp2p synchronization. Runtime-local causal replica identity remains separate from document identity and network peer identity.
Index ¶
- Constants
- Variables
- func SnapshotPath(dataDir string) string
- type CatchUp
- type CheckpointCommitter
- type Cursor
- type DeltaArtifact
- type DocumentIdentity
- type JSONFile
- type Session
- func (s *Session) ApplyDeltaFrom(peer yin.ReplicaID, artifact DeltaArtifact) (bool, error)
- func (s *Session) Cursor(peer yin.ReplicaID) Cursor
- func (s *Session) ExtractDeltaFor(peer yin.ReplicaID) (DeltaArtifact, bool, error)
- func (s *Session) ExtractDeltaSince(cursor Cursor) (DeltaArtifact, bool, error)
- func (s *Session) Mark(peer yin.ReplicaID, cursor Cursor) error
- func (s *Session) SetCursor(peer yin.ReplicaID, cursor Cursor) error
- type SnapshotArtifact
- type Store
- func NewStore(replica yin.ReplicaID) (*Store, error)
- func NewStoreForDocument(replica yin.ReplicaID, identity DocumentIdentity) (*Store, error)
- func NewStoreFromJSONFile(replica yin.ReplicaID, file *JSONFile) (*Store, error)
- func OpenDurableStore(replica yin.ReplicaID, expectedIdentity DocumentIdentity, ...) (*Store, error)
- func OpenStore(replica yin.ReplicaID, dataDir, visiblePath string) (*Store, error)
- func OpenStoreForDocument(replica yin.ReplicaID, identity DocumentIdentity, dataDir, visiblePath string) (*Store, error)
- func (s *Store) ApplyCatchUp(catchUp CatchUp) (bool, error)
- func (s *Store) ApplyDeltaArtifact(artifact DeltaArtifact) (bool, error)
- func (s *Store) Bootstrap(artifact SnapshotArtifact) (bool, error)
- func (s *Store) CatchUp(cursor Cursor) (CatchUp, error)
- func (s *Store) Cursor() Cursor
- func (s *Store) Delete(key string) error
- func (s *Store) ExportVisibleJSON() ([]byte, error)
- func (s *Store) Identity() DocumentIdentity
- func (s *Store) ImportVisibleJSON(data []byte) (bool, error)
- func (s *Store) ReplicaID() yin.ReplicaID
- func (s *Store) RotateCheckpoint() (DocumentIdentity, error)
- func (s *Store) Set(key string, value json.RawMessage) error
- func (s *Store) Snapshot() (SnapshotArtifact, error)
- func (s *Store) Version() yin.VersionVector
Examples ¶
Constants ¶
const ( // DefaultDocumentID identifies the single document used by the demo CLI. DefaultDocumentID = "default" // InitialGeneration is the first checkpoint generation. InitialGeneration uint64 = 1 )
const DocumentSnapshotFilename = "document.snapshot.json"
DocumentSnapshotFilename is the durable qi generation-checkpoint filename stored under a data directory. A checkpoint envelopes document identity and a yin snapshot; it is not the visible JSON projection, a bare yin snapshot, or a yin delta.
Variables ¶
var ( // ErrDocumentMismatch means an artifact was routed to a different document. ErrDocumentMismatch = errors.New("qi: document mismatch") // ErrGenerationMismatch means an artifact belongs to another checkpoint generation. ErrGenerationMismatch = errors.New("qi: generation mismatch") // ErrMalformedSyncArtifact distinguishes invalid artifact or yin bytes from identity mismatches. ErrMalformedSyncArtifact = errors.New("qi: malformed sync artifact") )
var ErrDurabilityUncertain = errors.New("qi: checkpoint durability uncertain")
ErrDurabilityUncertain means a checkpoint was published, but the committer could not confirm its durability. The Store adopts the published checkpoint before returning an error wrapping this sentinel.
var ErrEmptyReplicaID = errors.New("qi: replica id is empty")
ErrEmptyReplicaID means a qi document was requested without its runtime-local yin author identity. qi validates only non-emptiness; replica IDs are opaque.
var ErrInvalidVisibleJSON = errors.New("qi: invalid visible JSON")
ErrInvalidVisibleJSON identifies malformed input to ImportVisibleJSON, including valid JSON values that are not top-level objects.
Functions ¶
func SnapshotPath ¶
SnapshotPath returns the durable qi generation-checkpoint path for dataDir.
Types ¶
type CatchUp ¶
type CatchUp struct {
Snapshot *SnapshotArtifact `json:"snapshot,omitempty"`
Delta *DeltaArtifact `json:"delta,omitempty"`
}
CatchUp contains at most one generation-sensitive artifact. A snapshot means the requester must bootstrap; a delta is normal same-generation catch-up.
type CheckpointCommitter ¶
type CheckpointCommitter interface {
CommitCheckpoint(checkpoint []byte) (published bool, err error)
}
CheckpointCommitter publishes complete qi checkpoint bytes to an application-owned durable authority.
CommitCheckpoint must return (true, nil) after successful publication. An error before publication must return published false. If publication occurred but its durability cannot be confirmed, it must return published true with a non-nil error. Returning false with a nil error is invalid.
The checkpoint slice is owned by the caller: CommitCheckpoint must not mutate it or retain it after returning. The application must give a Store exclusive authority over the corresponding checkpoint while that Store uses the committer. Calls are serialized by Store; implementations need not support concurrent or reentrant calls.
Checkpoint bytes are the complete qi checkpoint envelope, including document identity and a yin snapshot. They are not visible JSON, a bare yin snapshot, or a delta.
type Cursor ¶
type Cursor struct {
Identity DocumentIdentity `json:"identity"`
Version yin.VersionVector `json:"version"`
}
Cursor is generation-scoped causal coverage. A version vector without this identity must never be used for extraction in another generation.
type DeltaArtifact ¶
type DeltaArtifact struct {
Identity DocumentIdentity `json:"identity"`
Payload []byte `json:"payload"`
}
DeltaArtifact associates encoded yin delta bytes with their document generation.
type DocumentIdentity ¶
type DocumentIdentity struct {
DocumentID string `json:"document_id"`
Generation uint64 `json:"generation"`
}
DocumentIdentity is the compatibility boundary for one synchronized document. DocumentID is stable for the document's lifetime; Generation increases on checkpoint rotation and is never inferred from yin causal versions.
type JSONFile ¶
type JSONFile struct {
// contains filtered or unexported fields
}
JSONFile owns a yin LWW map exposed as a top-level JSON object. It is a lower-level, caller-synchronized API: methods must not be called concurrently. Its constructor replica is the runtime-local author of future writes.
func NewJSONFile ¶
NewJSONFile creates an empty top-level JSON object document. replica must be non-empty and becomes the runtime-local yin author for the document's lifetime.
func ParseJSONFile ¶
ParseJSONFile parses a visible top-level JSON object for a non-empty local replica. Parsing the projection creates local writes; it does not restore yin causal metadata or tombstones.
func ParseJSONFileSnapshot ¶
ParseJSONFileSnapshot parses a durable yin LWW map snapshot for a non-empty receiving replica. Snapshots keep causal metadata and tombstones, unlike the visible JSON projection parsed by ParseJSONFile. Decode retains replica as the local author for future writes and never adopts an identity from snapshot data.
func (*JSONFile) MarshalSnapshot ¶
MarshalSnapshot encodes a durable CRDT snapshot, including metadata not present in the visible top-level JSON object projection.
func (*JSONFile) Set ¶
func (f *JSONFile) Set(key string, value json.RawMessage) error
Set stores value at key after validating that value is a complete JSON value. Pass []byte("null") explicitly to store JSON null.
func (*JSONFile) Version ¶
func (f *JSONFile) Version() yin.VersionVector
Version returns the current document version vector.
type Session ¶
type Session struct {
// contains filtered or unexported fields
}
Session tracks generation-scoped per-peer cursors around a caller-supplied JSONFile. Its mutex serializes Session operations and validates identity before yin ingest, but callers retain the file and must not mutate it concurrently. Store is the preferred owning, concurrency-safe end-to-end boundary.
func NewSession ¶
NewSession wraps file for the default document generation. It does not copy or take exclusive ownership of file; the caller must coordinate all direct use. NewSession panics if file is nil.
func NewSessionForDocument ¶
func NewSessionForDocument(file *JSONFile, identity DocumentIdentity) *Session
NewSessionForDocument wraps file at an explicit document boundary. It does not copy or take exclusive ownership of file; the caller must coordinate all direct use. It panics if file is nil or identity is invalid.
func (*Session) ApplyDeltaFrom ¶
ApplyDeltaFrom rejects a mismatched artifact before decoding or applying yin and marks the sender cursor only after successful decoding.
func (*Session) ExtractDeltaFor ¶
ExtractDeltaFor returns changes beyond a peer's generation-scoped cursor without advancing it. Call Mark or SetCursor only after the caller's protocol has established the corresponding peer coverage.
Example ¶
ExampleSession_ExtractDeltaFor shows that extraction does not advance a peer cursor; the caller marks coverage only after its protocol establishes that the peer applied the delta.
package main
import (
"fmt"
qi "github.com/iamseth/qi"
)
func main() {
aliceFile, _ := qi.NewJSONFile("alice")
bobFile, _ := qi.NewJSONFile("bob")
_ = aliceFile.Set("message", []byte(`"hello"`))
alice := qi.NewSession(aliceFile)
bob := qi.NewSession(bobFile)
artifact, pendingBeforeMark, _ := alice.ExtractDeltaFor("bob")
_, _ = bob.ApplyDeltaFrom("alice", artifact)
_ = alice.Mark("bob", qi.Cursor{
Identity: qi.DocumentIdentity{DocumentID: qi.DefaultDocumentID, Generation: qi.InitialGeneration},
Version: bobFile.Version(),
})
_, pendingAfterMark, _ := alice.ExtractDeltaFor("bob")
fmt.Println(pendingBeforeMark, pendingAfterMark)
}
Output: true false
func (*Session) ExtractDeltaSince ¶
func (s *Session) ExtractDeltaSince(cursor Cursor) (DeltaArtifact, bool, error)
ExtractDeltaSince returns changes beyond cursor without advancing it.
type SnapshotArtifact ¶
type SnapshotArtifact struct {
FormatVersion int `json:"format_version"`
Identity DocumentIdentity `json:"identity"`
Payload []byte `json:"payload"`
}
SnapshotArtifact associates a durable yin snapshot with its document generation.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is the concurrency-safe embedding boundary for one generation-scoped document. It owns its JSONFile, serializes mutations and synchronization, and validates document generations before ingesting artifacts. Its non-empty local yin replica ID is runtime configuration, remains fixed for the Store's lifetime, and is not loaded from checkpoints or remote snapshots.
Stores created by in-memory constructors accept mutations in memory. Stores created by OpenStore with a non-empty data directory, or by OpenDurableStore, accept every state change only through their CheckpointCommitter. An unpublished commit error leaves Store state unchanged. If an error wraps ErrDurabilityUncertain, the checkpoint was published and Store state has changed to match it.
Example ¶
ExampleStore demonstrates the canonical, concurrency-safe embedding API and transport-independent delta exchange. Lower-level integrations that provide their own synchronization can compose JSONFile and Session directly.
package main
import (
"fmt"
qi "github.com/iamseth/qi"
)
func main() {
alice, err := qi.NewStore("alice")
if err != nil {
panic(err)
}
bob, err := qi.NewStore("bob")
if err != nil {
panic(err)
}
_, _ = alice.ImportVisibleJSON([]byte(`{"message":"hello","settings":{"theme":"dark"}}`))
catchUp, _ := alice.CatchUp(bob.Cursor())
_, _ = bob.ApplyCatchUp(catchUp)
visible, _ := bob.ExportVisibleJSON()
fmt.Println(string(visible))
}
Output: {"message":"hello","settings":{"theme":"dark"}}
func NewStore ¶
NewStore creates an empty, in-memory concurrency-safe document store. replica is the runtime-local yin author of future writes, must be non-empty, and remains fixed for the returned Store's lifetime. Mutations are accepted without checkpoint publication.
func NewStoreForDocument ¶
func NewStoreForDocument(replica yin.ReplicaID, identity DocumentIdentity) (*Store, error)
NewStoreForDocument creates an empty, in-memory Store at an explicit document boundary. It applies the same local-replica contract and mutation acceptance behavior as NewStore.
func NewStoreFromJSONFile ¶
NewStoreFromJSONFile copies file's complete yin snapshot, including causal metadata and tombstones, into a new in-memory Store. It does not take ownership of file: subsequent use of either value cannot mutate the other. replica must be non-empty and becomes the copy's fixed runtime-local author for future writes. Mutations are accepted without checkpoint publication.
func OpenDurableStore ¶
func OpenDurableStore(replica yin.ReplicaID, expectedIdentity DocumentIdentity, initialCheckpoint []byte, committer CheckpointCommitter) (*Store, error)
OpenDurableStore opens a Store whose mutations are accepted only by publishing a complete checkpoint through committer. committer must be non-nil. Nil or empty initialCheckpoint creates fresh state at expectedIdentity; supplied bytes restore their generation and yin state, and must carry expectedIdentity.DocumentID. The expected generation is ignored when restoring because the checkpoint is authoritative for generation.
The application owns loading and exclusive authority over the checkpoint passed here. On an unpublished commit error Store state is unchanged. On an error wrapping ErrDurabilityUncertain, publication is known to have occurred and Store state has changed to remain aligned with the published bytes.
Example ¶
ExampleOpenDurableStore demonstrates application-owned checkpoint loading, publication, and restart. It is distinct from ExampleStore's in-memory mode.
package main
import (
"fmt"
qi "github.com/iamseth/qi"
)
// applicationCheckpoint is an application-owned byte authority. A real
// implementation would load and atomically replace the same database row or
// object; qi supplies complete checkpoint bytes to commit.
type applicationCheckpoint struct {
bytes []byte
}
func (a *applicationCheckpoint) load() []byte {
return append([]byte(nil), a.bytes...)
}
func (a *applicationCheckpoint) CommitCheckpoint(checkpoint []byte) (bool, error) {
a.bytes = append(a.bytes[:0], checkpoint...)
return true, nil
}
func main() {
authority := new(applicationCheckpoint)
identity := qi.DocumentIdentity{
DocumentID: "preferences",
Generation: qi.InitialGeneration,
}
store, err := qi.OpenDurableStore("alice", identity, authority.load(), authority)
if err != nil {
panic(err)
}
if err := store.Set("theme", []byte(`"dark"`)); err != nil {
panic(err)
}
// On restart, the application replaces its Store from the same authority.
// The runtime-local causal replica may change without changing document state.
store, err = qi.OpenDurableStore("alice-after-restart", identity, authority.load(), authority)
if err != nil {
panic(err)
}
visible, _ := store.ExportVisibleJSON()
fmt.Println(string(visible))
}
Output: {"theme":"dark"}
func OpenStore ¶
OpenStore uses qi startup precedence: load the complete qi checkpoint <dataDir>/document.snapshot.json when present, else seed from visiblePath when present, else create an empty document. replica is always the continuing runtime-local author; checkpoint data never supplies it. A non-empty dataDir makes every subsequent mutation durable. An empty dataDir keeps the Store in-memory.
visiblePath is used only while opening. The returned Store retains no file paths; callers own subsequent visible-projection reads and writes.
func OpenStoreForDocument ¶
func OpenStoreForDocument(replica yin.ReplicaID, identity DocumentIdentity, dataDir, visiblePath string) (*Store, error)
OpenStoreForDocument opens or seeds a Store with an explicit stable document identity. An existing checkpoint may have a later generation, but its stable document ID must match identity.DocumentID. Checkpoint decode uses replica for future local writes rather than adopting a replica identity from the payload. A non-empty dataDir installs an atomic filesystem committer before the Store is returned; opening visible or empty state alone does not create a checkpoint.
visiblePath is startup input only. Callers own later projection I/O through ExportVisibleJSON and ImportVisibleJSON.
func (*Store) ApplyCatchUp ¶
ApplyCatchUp is the single ingest boundary for synchronization artifacts. changed reports causal state acceptance. In durable mode no-op artifacts do not publish; unpublished errors leave state unchanged, while ErrDurabilityUncertain returns changed true for the adopted publication.
func (*Store) ApplyDeltaArtifact ¶
func (s *Store) ApplyDeltaArtifact(artifact DeltaArtifact) (bool, error)
ApplyDeltaArtifact rejects identity mismatches before decoding or invoking yin. changed is false for stale or duplicate deltas. In durable mode unpublished errors leave state unchanged; ErrDurabilityUncertain returns changed true and means the published delta was adopted.
func (*Store) Bootstrap ¶
func (s *Store) Bootstrap(artifact SnapshotArtifact) (bool, error)
Bootstrap installs a yin snapshot from a newer generation, or merges one from the current generation, after validating the qi artifact's document identity. Older generations are rejected. Snapshot decode always uses the receiving Store's fixed ReplicaID; sender or artifact identity never authors later writes. changed is false for a causal no-op. In durable mode unpublished errors leave state unchanged; ErrDurabilityUncertain returns changed true and means the published snapshot was adopted. In non-durable mode, a newer generation is accepted in memory.
func (*Store) CatchUp ¶
CatchUp validates a generation-scoped cursor before extracting a yin delta. An older generation receives a current snapshot and must bootstrap.
func (*Store) Delete ¶
Delete removes key from the visible JSON object. In durable mode it returns only after checkpoint publication; an unpublished error leaves state unchanged, while ErrDurabilityUncertain means the deletion was accepted.
func (*Store) ExportVisibleJSON ¶
ExportVisibleJSON returns the visible top-level JSON object projection.
func (*Store) Identity ¶
func (s *Store) Identity() DocumentIdentity
Identity returns the current document generation identity.
func (*Store) ImportVisibleJSON ¶
ImportVisibleJSON diffs data as a top-level JSON object against the current visible projection. Added or changed top-level keys are set, and removed keys are deleted. Nested values are treated as whole JSON values. The changed result is false for a no-op. Invalid input wraps ErrInvalidVisibleJSON. In durable mode an unpublished error returns changed false and leaves state unchanged; ErrDurabilityUncertain returns changed true and means the published change was adopted.
func (*Store) ReplicaID ¶
ReplicaID returns the Store's immutable runtime-local yin replica used to author writes. It is distinct from document identity and transport peer IDs.
func (*Store) RotateCheckpoint ¶
func (s *Store) RotateCheckpoint() (DocumentIdentity, error)
RotateCheckpoint starts the next generation from the current live projection. The Store mutex serializes projection, reconstruction, publication, and swap. Durable Stores publish through their configured committer. In-memory Stores cannot rotate. An unpublished error leaves the old generation unchanged. An ErrDurabilityUncertain error returns the next identity and means Store state changed to match the known-published checkpoint.
func (*Store) Set ¶
func (s *Store) Set(key string, value json.RawMessage) error
Set stores value at key after validating that value is a complete JSON value. In durable mode it returns only after checkpoint publication; an unpublished error leaves the value unchanged, while ErrDurabilityUncertain means it changed.
func (*Store) Snapshot ¶
func (s *Store) Snapshot() (SnapshotArtifact, error)
Snapshot returns the current generation's qi checkpoint envelope. Its payload is a yin snapshot, not visible JSON or a delta.
func (*Store) Version ¶
func (s *Store) Version() yin.VersionVector
Version returns the current document version vector. Prefer Cursor at sync boundaries.