docengine

README

简体中文

Docengine is a format-neutral Go engine for reading, editing, recovering, and organizing large local UTF-8 documents. It provides immutable snapshots, transactional edits, crash recovery, atomic persistence, coordinate mapping, virtualization, search, revision-bound ranges, and multi-source composition without requiring the entire document to be loaded into memory.

The engine understands bytes, UTF-8 boundaries, revisions, ranges, and abstract measures. It deliberately does not understand Markdown, JSON, programming languages, or any other document format. Parsing, rendering, layout, file-type policy, and product workflows belong to the host application.

Docengine is pre-1.0. It is suitable for evaluation and integration where API changes can be accommodated, but its public API and persistent formats do not yet carry a 1.0 compatibility guarantee.

Installation

Docengine currently targets Go 1.26:

go get github.com/moresleep512/docengine/doc

The module path is:

github.com/moresleep512/docengine

Quick start

The doc package is the recommended application-facing API. It separates read-only and read-write permissions through return types.

package main

import (
	"fmt"
	"log"

	"github.com/moresleep512/docengine/doc"
)

func main() {
	document, err := doc.New("notes.txt")
	if err != nil {
		log.Fatal(err)
	}
	defer document.Close()

	if _, err := document.Write("Hello, 世界\n"); err != nil {
		log.Fatal(err)
	}
	if _, err := document.Insert(7, "local "); err != nil {
		log.Fatal(err)
	}
	if _, err := document.Save(); err != nil {
		log.Fatal(err)
	}

	result, err := document.Search("世界")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("revision=%d matches=%d\n",
		result.Revision, len(result.Matches))
}

Use doc.Read for an existing file without mutation methods, doc.Open for an existing writable file, and doc.New for exclusive creation. Context-aware variants are available for open, reads, mutations, search, persistence, maintenance, and cleanup.

All offsets are byte offsets and all ranges are half-open [start, end). Mutations and structural ranges that must preserve valid UTF-8 require rune boundaries; byte-oriented reads may address any in-bounds byte range. Use Coordinates when a host needs line, rune, and column mapping.

Capabilities

• A disk-backed Piece Tree with immutable, revision-bound snapshots.
• Ordered edit batches that commit atomically and form one undo/redo unit.
• An append-only recovery v2 journal with atomic batch replay and damaged-tail repair.
• Full-file UTF-8 validation, content identity, external-change detection, and symbolic-link target pinning.
• Streaming, conflict-checked atomic save with POSIX directory synchronization and Windows write-through replacement.
• Explicit dirty, durability-uncertain, recovery-uncertain, and persistence fault states.
• Bounded change history, affinity-aware anchors, ranges, and incremental byte/line/rune coordinate indexes.
• Immutable snapshots that remain readable across edits and parent closure until their leases are released.
• Bounded resumable event streams and lifecycle, recovery, history, and persistence diagnostics.
• UTF-8-safe logical pages, opaque fragments, host-defined fixed-point measures, overscan, caching, and revision/generation checks.
• Streaming literal and RE2 search with optional pure-Go SQLite/FTS5 persistent candidate indexes. Every indexed candidate is verified against its source snapshot.
• Immutable revision-bound interval sets for host annotations and decorations.
• Immutable multi-source compositions with bidirectional source mapping.

Every subsystem has explicit resource limits. Zero-valued options select bounded defaults rather than unbounded operation.

Architecture

flowchart TB
    Host["Host application<br/>editor · CLI · service · format adapter"]
    Native["Native host · language binding"]
    CAPI["C ABI"]
    Doc["doc<br/>permission-separated public API"]
    Session["document.Session<br/>revision · transaction · lifecycle"]
    Store["store<br/>Piece Tree · Snapshot"]
    Recovery["recovery<br/>v2 journal"]
    Save["save<br/>atomic persistence"]
    Coordinate["coordinate<br/>index · ChangeMap"]
    Virtual["virtual<br/>Page · Fragment"]
    Search["search<br/>scan · persistent index"]
    Interval["interval<br/>immutable ranges"]
    Composition["composition<br/>multi-source snapshot"]
    OS["files · io.ReaderAt · operating system"]

    Host --> Doc
    Native --> CAPI
    CAPI --> Doc
    Doc --> Session
    Doc --> Interval
    Doc --> Composition
    Session --> Store
    Session --> Recovery
    Session --> Save
    Session --> Coordinate
    Session --> Virtual
    Session --> Search
    Store --> OS
    Recovery --> OS
    Save --> OS
    Search --> OS

Most applications should depend only on doc. The lower-level document.Session API is available for hosts that need direct control over storage directories, journal policy, snapshot leases, checkpoints, or subsystem lifetimes. Lower packages remain independently usable over immutable io.ReaderAt-style sources.

Native applications and foreign-language bindings can use the versioned, coarse-grained C API. It exposes the main document lifecycle without passing Go pointers or Go-owned objects across the ABI boundary.

See MODULES.md for package responsibilities and dependency constraints.

Persistence and recovery

Opening a document validates the complete file as UTF-8, computes its content identity, records its requested and resolved paths, and replays a matching recovery journal. A symbolic link is resolved once when opened; later saves continue to target that resolved file even if the link is redirected.

An edit becomes visible only after its complete recovery batch is appended. An incomplete or corrupt journal tail is discarded without exposing a partial batch. A mismatched or ambiguous journal is isolated and causes open to fail instead of applying edits to an uncertain base.

Save streams a selected immutable revision to a temporary file, checks for external changes, atomically replaces the destination, and establishes a new recovery generation. A replacement may have committed even when a later durability or rebind step reports an error; State preserves that distinction. If the session cannot safely resume mutation after commit, it becomes permanently read-only until reopened.

Concurrency and ownership

Reads and immutable snapshots may run concurrently. Mutations are serialized and revision checked. Save may run concurrently with later edits: it commits the selected revision while newer edits remain dirty and recoverable.

Snapshots, coordinate indexes, pagers, search indexes, event subscriptions, and compositions own bounded leases or resources. Close each value that exposes a Close method. Session closure stops accepting new work, cancels background work, waits for accepted operations, and then releases owned resources.

Read-only doc.Reader is an API permission boundary, not a promise that opening the document performs no recovery or runtime-directory writes.

Documentation

• API.md — high-level API guide and lower-level Session guidance.
• C API — native ABI, ownership rules, build instructions, and C examples.
• MODULES.md — package architecture and invariants.
• TODO.md — known limitations and work required before 1.0.
• Go Reference — generated package and symbol documentation.

English documentation is canonical. README_zh.md provides the Chinese project overview.

Testing

CI runs on Windows and Ubuntu and requires:

• module verification, formatting, vet, and the complete test suite;
• 100% statement coverage for all eleven tested Go packages, including the C bridge;
• race-enabled tests on Linux with randomized ordering and repeated runs;
• state-machine and property fuzz smoke tests for the facade, store, recovery, Session, coordinates, virtualization, search, intervals, and composition;
• shared-library builds plus strict C and C++ consumer compilation;
• additional Windows stress coverage for journal durability event ordering.

The tests include fault injection around open, recovery, journaling, save, directory synchronization, atomic replacement, snapshot generations, compaction, persistent search, close barriers, and resource budgets.

Project history

Docengine was extracted from the backend core of TypeMD. During extraction, Markdown-specific scanning, application metadata, Wails bindings, editor layout policy, and product-coupled persistence were removed. The current module is developed independently and TypeMD does not automatically receive changes from this repository.

The canonical repository is GitHub. A mirror is maintained on Codeberg.

License

Docengine is available under the MIT License.