sop

README

⚡ Joltrin ⚡

From milliseconds to microseconds: durable memory and verification infrastructure for AI agents.

⚡ Verified by benchmark: <0.3ms latency & ~6.8µs B-Tree operations →

Joltrin (formerly SOP / Scalable Object Persistence) is a unified in-process state engine providing transactional persistence, durable agent memory, distributed storage primitives, explicit-state verification, and WebAssembly persistence. It combines a sector-aligned copy-on-write B-Tree, checkpointed episodic agent memory, vector similarity search, and a deterministic safety verification barrier for MCP and A2A runbooks into one library.

Instead of managing separate vector databases, message brokers, caching tiers, distributed lock managers, and fragile external checkpoint stores, Joltrin lets your AI agents maintain crash-resilient memory and enforce operational invariants directly within the execution boundary.

Why "from milliseconds to microseconds"? Traditional multi-tier architectures incur an estimated 15-50ms network round-trip penalty across external services (Redis, message queues, relational databases). Joltrin runs embedded in-process, shifting latency from milliseconds to microseconds: empirical benchmarks measure <0.3ms end-to-end in-process execution, ~6.87µs per B-Tree write (>145,000 ops/sec), and ~6.95µs per read (>143,000 ops/sec) with full ACID consistency.
🔗 Proof & Benchmark Reference: View Detailed Benchmarks & Microsecond Breakdown → · Benchmark suite: tools/benchmark · Live client-side run: Technical WASM Demo

🧠 Launch Technical Demo →   |   🎮 Play Joltrin Arena →   |   🔌 Launch Agent Barrier →

Experience	Description	Live Interactive Link
🧠 Joltrin Technical Demo	Client-Side Zero-Server WebAssembly Engine Execute live ACID transactions, 128-dimensional vector cosine searches, microsecond benchmarks, and durable AI agent memory checkpoints (kill the agent mid-task, watch a successor resume from the B-Tree) running 100% in your browser with 0 runtime HTTP network calls after initial load.	Launch Technical Demo →
🎮 Joltrin Arena	Distributed Systems Survival Simulation Command a live digital cluster. Scale worker swarms, crash storage nodes, trigger transaction storms, and watch Joltrin automatically redistribute tasks and rebuild parity in real-time.	Play Joltrin Arena →
🔌 Joltrin Agent Verification Barrier	The MCP/A2A Safety Check, Clickable The same ai/verify barrier gating tools/mcpserver and tools/a2aagent, compiled to WASM. Try dropping a database before validating a backup and watch it get blocked, in your browser, with the trace persisted to OPFS.	Launch Agent Barrier →

---

⚡ Try It in 30 Seconds

Clone the repository and run the unified interactive demo:

git clone https://github.com/sharedcode/joltrin.git && cd joltrin && ./scripts/demo.sh

The interactive script lets you execute and verify each workflow shown on this page:

1. Verification Barrier: Safety precedence check gating destructive operations (examples/verify_barrier).
2. AI Agent Memory: B-Tree reasoning checkpoints with mid-task worker failure and sub-15ms recovery (examples/agent_memory).
3. Core Test Suite: Sanity check running core storage, filesystem, and server unit tests.
4. Local Protocol Probe: JSON-RPC over stdio (cmd/sop-mcp-server) and live A2A agent-card probe (cmd/sop-a2a-agent).

You can also run any step directly with flags:

./scripts/demo.sh --barrier    # Option 1: Precedence barrier check
./scripts/demo.sh --memory     # Option 2: AI agent memory failover
./scripts/demo.sh --test       # Option 3: Core engine test suite
./scripts/demo.sh --protocol   # Option 4: Local MCP and A2A reachability probe
./scripts/demo.sh --all        # Run all 4 stages sequentially

Prefer raw Go commands without scripts? Run them directly:

go run ./examples/verify_barrier  # Option 1
go run ./examples/agent_memory    # Option 2
go test ./...                     # Option 3

No local Go toolchain? Run the exact same demo suite inside Docker:

# Run the interactive demo suite via Docker
docker run --rm -it -v "$PWD":/src -w /src golang:1.26-alpine ./scripts/demo.sh

# Or run the published quickstart container from GHCR
docker run --rm ghcr.io/sharedcode/joltrin-quickstart

📉 Engineering ROI, Verified in This Repo

No revenue or customer numbers exist yet for this project (see For Investors for the honest version of that). What is verified today, in this repo, is the infrastructure cost this architecture removes:

What collapses	From	To
Network hops per operation	3-4 hops across Redis, a queue, and Postgres/Cassandra (estimated 15-50ms network round-trip overhead)	1 embedded in-process call (<0.3ms measured latency, >145k ops/sec)
Stateful services to operate, patch, and page on	Redis + Kafka/RabbitMQ + Postgres/Cassandra + ZooKeeper (4+)	1 embedded library
Language surfaces shipped	N/A	Go (native), Python (sop4py on PyPI), C# (Sop on NuGet); Java and Rust bindings exist in-repo with tests, not yet published
CI rigor on every change	N/A	govulncheck clean on every push; race detector on the core engine packages (btree, common, fs, inmemory); 3-OS build and test matrix (Linux, macOS, Windows)
Deployment footprint of the technical demo	A server-backed demo stack	WASM build running ACID transactions, vector search, and agent-memory checkpointing 100% client-side, 0 runtime HTTP calls after page load (live)

Every row above is something you can run yourself, not a projection. See Performance Benchmarks for the throughput numbers behind the latency claim, and What Has Not Yet Been Proven for what this table deliberately leaves out.

---

🚀 Experience Joltrin

You can test Joltrin directly in your browser without installing anything via the live interactive experiences above (Technical Demo, Joltrin Arena, and Agent Verification Barrier). The technical demo demonstrates the engine's core power directly: safe, ACID-transactional storage running on web storage itself (OPFS), with zero server and zero network calls after the initial page loads the WASM binary. Everything else on this page, including the agent verification barrier below, is built on top of that same engine, a reference implementation showing one concrete use case.

The technical demo persists across reloads now, to Origin Private File System, via the browser's async File System Access API. The diagram below is the real tradeoff behind that choice, not a benchmark; no throughput numbers are shown because none have been measured for either path in this repo.

That same WASM-compiled engine is what the Agent Verification Barrier row above actually runs on, not a separate reimplementation: it's the concrete reference implementation this repo ships to answer "what do you actually build with a durable, transactional engine running client-side?" It is an AI agent safety check an agent cannot talk its way around, with the trace itself durable in OPFS across reloads. The next section is that barrier in depth, plus the same check reachable server-side over MCP and A2A.

---

🔌 Agent Protocols: MCP, A2A, and a Real Verification Barrier

Joltrin runbooks are reachable from two agent protocols, Model Context Protocol and Agent2Agent, both gated by the same safety-and-reachability check before a step is allowed to commit. Real, tested code (ai/verify, tools/mcpserver, tools/a2aagent), not a diagram of an idea; see MCP, A2A, and the Verification Engine for the full audit and design writeup.

Try the barrier yourself, live: sharedcode.github.io/joltrin/agents. GitHub Pages can't run a real MCP or A2A network server (no backend), so this page runs the actual ai/verify check compiled to WASM, wired to buttons instead of protocol calls, the same logic those servers call before committing a step. Click "Drop Prod DB" first and watch it block; the trace persists to OPFS, so a reload picks up where you left off. This is a real recording of that page, not a mockup:

The same scenario also runs as a terminal program, examples/verify_barrier, and the servers themselves are one command away:

# Run the barrier demo yourself
go run ./examples/verify_barrier

# Serve the same runbook over MCP (stdio)
# Note: this speaks JSON-RPC over stdin/stdout for an MCP client (Claude
# Desktop, an SDK, etc). Run bare in a terminal, it'll print "Parse error"
# for every line you type, since your keystrokes aren't valid JSON-RPC -
# that's expected, not a bug. Point an MCP client at this command instead.
go run ./cmd/sop-mcp-server

# Serve it over A2A instead, then fetch its agent card
go run ./cmd/sop-a2a-agent &
curl localhost:8087/.well-known/agent-card.json

# Claude has no native A2A client, so bridge the two: sop-a2a-bridge
# resolves the agent card above and re-exposes execute_step as an MCP tool
go run ./cmd/sop-a2a-bridge -agent-url http://localhost:8087

Wiring sop-mcp-server into Claude

cmd/sop-mcp-server speaks JSON-RPC over stdio and evaluates the barrier policies below (ai/verify's CheckSafety) before execute_step is allowed to commit; a blocked step comes back as input-required, not a crash. Point either Claude client at the command:

Claude Desktop (claude_desktop_config.json, stdio transport):

{
  "mcpServers": {
    "joltrin": {
      "command": "go",
      "args": ["run", "./cmd/sop-mcp-server"],
      "cwd": "/absolute/path/to/joltrin"
    }
  }
}

Swap "command"/"args" for a prebuilt binary once you've run go build -o sop-mcp-server ./cmd/sop-mcp-server:

{
  "mcpServers": {
    "joltrin": {
      "command": "/absolute/path/to/joltrin/sop-mcp-server"
    }
  }
}

Claude Code (CLI):

claude mcp add --transport stdio joltrin -- go run ./cmd/sop-mcp-server

Wiring sop-a2a-agent into Claude (via sop-a2a-bridge)

Claude doesn't speak A2A natively, MCP is the protocol its clients actually implement, so reaching an A2A agent means bridging the two, not writing an A2A client into Claude itself. tools/a2abridge is that bridge: an MCP server that resolves a running sop-a2a-agent's card and re-exposes its execute_step skill as an MCP tool of the same name, translating each call into a real A2A task delegation over the wire and translating the resulting task state (completed / input-required / failed) back into an MCP tool result. It's built on the official a2aclient SDK package, not a hand-rolled JSON-RPC client, and it's covered by its own integration tests (tools/a2abridge/bridge_test.go) that drive the full MCP -> bridge -> real A2A wire protocol -> executor round trip, including the blocked, allowed, and remote-failure paths.

Start the agent, then point the bridge at it:

go run ./cmd/sop-a2a-agent &
go run ./cmd/sop-a2a-bridge -agent-url http://localhost:8087

Claude Desktop:

{
  "mcpServers": {
    "joltrin-a2a": {
      "command": "go",
      "args": ["run", "./cmd/sop-a2a-bridge", "-agent-url", "http://localhost:8087"],
      "cwd": "/absolute/path/to/joltrin"
    }
  }
}

Claude Code (CLI):

claude mcp add --transport stdio joltrin-a2a -- go run ./cmd/sop-a2a-bridge -agent-url http://localhost:8087

Barrier policies ai/verify enforces

ai/verify is a general-purpose explicit-state precondition/postcondition graph (Step, SafetyRule, ReachabilityRule in ai/verify/verify.go) with no built-in notion of databases, clusters, or money. Every state is an opaque string, so a barrier policy for any category of risky action is defined the same way: name the states that must hold, name the step that establishes the dangerous one, and let CheckSafety gate it. This repo ships three concrete runbooks in tools/runbookstore built on that same generic mechanism, one per risky-action category, plus the generic out-of-order rejection that applies to all of them:

• Destructive operations (DBMaintenanceWorkflow, e.g. dropping a database): drop_prod_db requires backup_validated, which only validate_backup establishes after take_backup. A SafetyRule (no-drop-without-validated-backup) names the barrier explicitly, and a ReachabilityRule guarantees rollback_complete stays reachable even after the drop.
• Resource & topology mutations (ClusterTopologyWorkflow, e.g. draining a node, failing over a cluster): drain_node and failover_cluster both require replica_parity_verified, which requires health_check_passed first. Reinstating the node or cluster (topology_rollback_complete) stays reachable from every state in the graph, including after a worker is terminated post-drain.
• Financial / ledger-mutating actions (LedgerTransferWorkflow, e.g. balance updates, account transfers): commit_transfer requires zero_sum_verified, which only verify_zero_sum_invariant establishes after balances are mutated inside a transaction_serialized scope (begin_serializable_transaction -> snapshot_balances -> apply_debit_credit). Reversal (ledger_rollback_complete) stays reachable both before and after commit.
• Unverified / out-of-order execution: this is the same mechanism underlying all three, not a separate check. CheckSafety rejects any step whose Requires states haven't been established yet in the current Trace, and rejects any step that would establish a Forbidden state without its paired Requires state already holding. An agent (or a client bug) trying to call drain_node or commit_transfer before its preconditions land gets a named, actionable violation back, never a silent no-op.

Only DBMaintenanceWorkflow is registered by the example binaries (cmd/sop-mcp-server, cmd/sop-a2a-agent) today; ClusterTopologyWorkflow and LedgerTransferWorkflow are available in tools/runbookstore (with tests in tools/runbookstore/examples_test.go) as worked examples of modeling the other two categories on the same engine. Register them with store.RegisterWorkflow in your own server to serve them.

What this checker is, precisely, matters more than what it sounds like it might be: explicit-state safety and reachability checking over a finite workflow graph, the "P is preceded by Q" precedence pattern from Dwyer/Avrunin/Corbett's property specification patterns (ICSE 1999), not general-purpose LTL/CTL model checking. No formula parser, no Büchi automata, no neural component translating natural language into the graph today. The full accounting of what's built versus proposed is in the linked doc, not summarized rosily here.

---

💡 What Problem Does Joltrin Solve?

Most distributed applications require two fundamentally different operations:

1. Storing state reliably (databases, key-value stores, vector indexes)
2. Coordinating work across machines (task queues, locks, retries, worker failovers)

Today, developers solve this by assembling a multi-component infrastructure stack:

THE FRAGMENTED MULTI-COMPONENT STACK (Without Joltrin):

[ Application ]
       │
       ├──► (TCP Hop 1: 5-15ms)  ──► Redis (Distributed Locks & Leases)
       ├──► (TCP Hop 2: 5-15ms)  ──► RabbitMQ / Kafka (Task Queue)
       ├──► (TCP Hop 3: 10-30ms) ──► PostgreSQL / Cassandra (Persistent Storage)
       └──► (Failover Glue)      ──► ZooKeeper / Custom Retry & Outbox Daemons

⚠️ 6+ infrastructure boundaries | Estimated 15-50ms network latency tax | High split-brain failure risk | High maintenance overhead

When an application worker crashes between releasing a lock in Redis and committing to PostgreSQL, state can enter an inconsistent split-brain condition. Engineering teams end up spending substantial time writing and maintaining outbox listeners, lock renewers, and compensating retry logic.

---

⚡ Why Joltrin?

Joltrin takes a different approach: co-locate storage and compute inside the same engine boundary.

THE UNIFIED DATA & COMPUTE PLATFORM (With Joltrin):

[ Application ]
       │
       └──► (Embedded In-Process Call: < 0.3ms latency)
            ┌─────────────────────────────────────────────────────────────┐
            │                        JOLTRIN ENGINE                       │
            │  • Persistent B-Tree Storage (Sector-aligned Direct I/O)    │
            │  • Strict Serializable ACID Transactions (WAL + 2PC)       │
            │  • Swarm Compute & Autonomous Task Redistribution           │
            │  • High-Dimensional Vector Similarity Indexing (SIMD)       │
            │  • Reed-Solomon Erasure Coding & Partition Resilience       │
            └─────────────────────────────────────────────────────────────┘

✓ 1 Single Engine | Sub-millisecond execution | 100% ACID consistency | Automated failover

Because compute workers, task queues, and storage partitions share the same transaction boundary, a worker failure triggers an automatic rollback of uncommitted work and re-assigns the task in milliseconds with zero orphan locks.

---

⏱️ Why Now?

Three industry shifts make this architecture increasingly relevant:

1. The Explosion of Autonomous AI Agents: Multi-agent swarms require frequent context checkpointing, vector similarity searches, and task coordination. Assembling this across Postgres, Pinecone, Redis, and Celery creates high failure surface area.
2. Edge and Local-First Computing: Devices in factory automation, vehicles, and retail branches cannot rely on constant connections to central cloud databases. They need full ACID storage and local coordination that works offline.
3. Infrastructure Simplification: Engineering organizations are seeking to reduce the operational overhead and cloud bills associated with running dozens of discrete microservices just to manage state and queues.

---

🔍 What Makes Joltrin Different?

Joltrin is built on five core technical principles:

1. Embedded Storage Engine: Operates in-process in Go, Python, and C#, eliminating TCP network hops for local reads and writes.
2. ACID Transactions without Database Servers: Implements Write-Ahead Logging (WAL) and Two-Phase Commit (2PC) with copy-on-write page isolation.
3. Swarm Compute Coordination: Workers coordinate task execution using storage-anchored sector claims and heartbeat leases without requiring global consensus bottlenecks (like Paxos or Raft) on the hot path.
4. Reed-Solomon Erasure Coding: Protects storage shards from hardware failure by striping parity blocks across drives rather than paying the 3x disk storage cost of full replication.
5. Integrated Vector & Structured Storage: Stores high-dimensional vector embeddings in the same B-Tree segments as structured metadata, allowing single-transaction memory commits.

---

⚖️ Joltrin vs. Alternatives

Every architecture involves tradeoffs. Here is an honest comparison of where Joltrin fits relative to industry standards:

Capability	PostgreSQL	Redis	Kafka	Temporal	Pinecone	SQLite	Joltrin
ACID Transactions	✓	△	✗	✗	✗	✓	✓
Ordered B-Tree Range Scans	✓	△	✗	✗	✗	✓	✓
Embedded In-Process	✗	✗	✗	✗	✗	✓	✓
Swarm Work Coordination	✗	△	△	✓	✗	✗	✓
Vector Similarity Search	△ (pgvector)	△	✗	✗	✓	✗	✓
Erasure Coding (N+K)	✗	✗	✗	✗	✗	✗	✓
Zero Standalone Daemons	✗	✗	✗	✗	✗	✓	✓

Legend: ✓ First-class native capability | △ Partial or requires plugin/extension | ✗ Not designed for this capability

Detailed Tradeoffs by Competitor:

• PostgreSQL: Industry standard for general relational databases. Choose Postgres when you need complex relational schemas, advanced SQL aggregations, or standard ecosystem tooling. Joltrin is better suited when you want an embedded storage engine inside your application process without database server management.
• Redis: Industry standard for ultra-low-latency in-memory key-value caching. Choose Redis when all data fits in RAM and you need simple cache operations. Joltrin provides durable B-Tree disk persistence, multi-account ACID transactions, and erasure coding.
• Kafka / RabbitMQ: Industry standards for high-volume streaming and pub/sub. Choose Kafka when you need multi-datacenter event streams and log retention. Joltrin provides transactional task queues co-located with storage state for local swarms.
• NATS (optional, adapters/nats): not a replacement for anything joltrin embeds, and not on the hot path. If a team already runs NATS as part of their own architecture, adapters/nats.VerifyBridge will publish ai/verify barrier decisions to it, fire-and-forget, after the decision is already made, so another service outside joltrin's process can observe it without polling. Nothing imports this by default and a publish failure can never change the barrier's own answer. See the addendum in docs/MCP_A2A_AND_VERIFICATION_ENGINE.md for the full reasoning on why this doesn't reverse the embedded design.
• Temporal: Industry standard for long-running durable workflows spanning external microservices. Choose Temporal for multi-week human-in-the-loop workflows across disparate clouds. Joltrin is designed for local-to-cluster co-located data and task execution.
• SQLite: Industry standard for embedded single-file relational databases. Choose SQLite for client desktop/mobile apps needing SQL. Joltrin is designed for high-concurrency multi-threaded workers, clustered coordination, partitioned vector stores, and erasure coding.

---

🎯 When Joltrin Is a Great Fit

• AI Agent Memory & Swarm Workforces: Autonomous agents requiring durable conversation memory, vector similarity search, and task hand-offs without fragmented external databases. Checkpoints commit directly to B-Tree segments with atomic rollback if a worker crashes mid-reasoning.
• Real-Time Systems & Simulation State: Game servers, robotics, and spatial computing needing sub-millisecond in-process transactional serialization (measured at 100k-145k ops/sec in local benchmarks) without database network hops.
• Financial & Escrow Ledgers: Systems requiring snapshot isolation, optimistic concurrency control (OCC), two-phase commit (2PC), and invariant verification (such as validating zero-sum account deltas before commit).
• Edge & IoT Computing: Devices operating in local or intermittent network environments that need local embedded ACID persistence, with experimental peer coordination.
• Serverless Workloads: Cloud functions and containers that need durable storage without exhausting external database connection pools.

---

🚫 When Joltrin is NOT the Right Tool

To be completely clear on architectural boundaries:

• Massive Analytical Warehousing: If you are running multi-petabyte columnar analytics across billions of historical events, specialized OLAP warehouses (like ClickHouse or Snowflake) are the right choice.
• Global Multi-Region Consensus: If your application requires synchronous commits across continents with multi-region Raft/Paxos quorums, dedicated distributed SQL databases (like CockroachDB or Google Spanner) are designed for that problem.
• Simple Stateless CRUD Apps: If your application is a standard CRUD dashboard with low traffic, standard PostgreSQL or MySQL with an ORM is simpler and has more ecosystem plugins.

---

🎮 See Joltrin in Action (Joltrin Arena Simulation)

In Joltrin Arena, every control maps directly to a real distributed systems concept:

Simulation Control	Distributed Systems Concept	Joltrin Technical Mechanism
Add Worker	Swarm Compute	Dynamic queue rebalancing across peer worker nodes without central master bottlenecks.
Remove Worker	Graceful Degradation	Active tasks drained and re-assigned to healthy nodes with zero dropped writes.
Kill Node / Storage Fault	Fault Tolerance	Reed-Solomon Erasure Coding reconstructs missing B-Tree blocks in-memory from parity chunks.
Transaction Storm	Concurrency & Isolation	Optimistic Concurrency Control (OCC) serializes conflicting writes in microseconds.
Increase Workload (100k TPS)	Scalability	B-Tree node segments partition write load across sector-aligned storage handles.
Automatic Self-Healing	Resilient Coordination	Heartbeat lease detection triggers automated task redistribution in <15ms.

---

👥 Who Joltrin Is For

Joltrin is one codebase, but different people will care about it for different reasons. Jump to the section that matches you:

Investors · Investment Banking & Tech Finance · Potential Customers · CTOs & Engineering Executives · AI Infrastructure Teams · Platform, SRE & Cloud Engineers · Researchers & Distributed Systems Engineers · Students & Learners · Developers · Engineering Leaders & Hiring Managers

💰 For Investors

The problem. Teams building stateful distributed applications, agent systems especially, routinely wire together a database, a cache, a message queue, a lock manager, and a workflow engine just to get durable state and coordinated work. Each boundary between those systems is a place where consistency breaks during a partial failure. That integration tax is paid by every team that builds this kind of system, repeatedly.

What Joltrin uniquely combines. A B-Tree storage engine, ACID transactions, and swarm task coordination live inside one embedded library instead of behind separate network services. That is an architectural bet, not a settled fact: it trades the maturity and ecosystem of specialized tools (Postgres, Kafka, Temporal) for fewer moving parts and a single consistency boundary. Whether that tradeoff wins in a given workload is something a team has to evaluate, which is exactly what the comparison table below is for.

Investment Thesis Joltrin is an open-source bet that "data plus compute in one embedded engine" is a better default for a growing category of workloads (AI agents, edge devices, real-time systems) than assembling that stack from five separate products. If that thesis is right, the project that owns the reference implementation of that architecture has a shot at becoming the default choice for it, the way SQLite became the default embedded relational store. That is a multi-year distribution bet, not a proven outcome.

Why Now

• AI agent systems increasingly need durable memory, checkpointing, and multi-worker coordination, and today that is usually stitched together from a vector database, a cache, and a job queue.
• Edge and local-first computing (factory automation, vehicles, retail devices) need ACID storage that keeps working without a constant connection to a central database.
• Engineering organizations are actively trying to cut the number of discrete stateful services they operate, both for cost and for on-call load.

These are real, observable industry trends. No specific market-sizing figures are cited here because this repository has not commissioned or verified any (see Market Opportunity below).

Market Opportunity Joltrin overlaps several existing categories rather than creating one from nothing: embedded databases (SQLite, RocksDB), distributed coordination (Zookeeper, etcd, Temporal), vector databases (Pinecone, Weaviate, pgvector), and workflow/task systems (Celery, Ray). Plausible buyers are teams building AI agent infrastructure, edge and IoT platforms, real-time/simulation backends, and fintech ledgers with strict transactional invariants. No independently sourced TAM/SAM/SOM figures are presented here; a rigorous estimate would require external market research (for example, from Gartner or IDC) that this project has not commissioned.

Business Model Opportunities The project is MIT-licensed with no commercial product today. The open-core progression and architectural foundations for commercial governance are detailed in Monetization & Editions Architecture below.

What Has Been Proven

• A working Go engine with ACID transactions (WAL plus two-phase commit), a custom B-Tree, and Reed-Solomon erasure coding, each with passing automated tests (18 packages carry tests in the core Go module; run them with go test ./..., while the two WASM-only packages build under GOOS=js GOARCH=wasm, see Performance Benchmarks below for the throughput numbers).
• A real WebAssembly build of the engine running ACID transactions, vector search, and agent-memory checkpointing entirely in-browser with zero runtime network calls after initial page load (live demo).
• Working language bindings for Go (native), Python (sop4py, published to PyPI), and C# (Sop, published to NuGet), plus Java and Rust bindings that exist in-repo with tests but are not yet published to their package registries.
• CI that runs the race detector and govulncheck on every change, and a changelog showing multiple rounds of real dependency and CVE remediation.

What Has Not Yet Been Proven

• No production deployments or paying customers are documented anywhere in this repository.
• No independent, third-party, or peer-reviewed benchmarks exist; the performance numbers below come from this project's own benchmark harness on a single workstation, not a controlled multi-system comparison.
• Joltrin Arena's cluster view is a UI simulation of the underlying concepts for demonstration purposes, not a live multi-node deployment; multi-node swarm clustering itself is real and tested (examples/swarm_clustered, examples/swarm_standalone), but has not been run at meaningful scale or under adversarial network conditions in public.
• No formal third-party security audit has been performed.
• No case studies, design partners, or committed customers exist yet.

🏦 For Investment Banking & Technology Finance

Technology category. Joltrin sits in the embedded data infrastructure layer: a storage and coordination engine that applications link against directly, similar in category placement to SQLite or RocksDB, but extended with distributed ACID transactions and task coordination that those two do not attempt.

Adjacent markets. Embedded/operational databases, distributed coordination and workflow orchestration, vector search infrastructure, and AI agent infrastructure tooling. Each of those adjacent markets has established commercial players (see the comparison table), which is useful context for sizing the competitive landscape Joltrin would need to differentiate against.

Potential strategic relevance. Potential strategic relevance could include: infrastructure vendors looking to add an embedded, agent-friendly storage layer to an existing platform; cloud providers evaluating lightweight alternatives to running separate managed database, cache, and queue services for edge or agent workloads; or AI infrastructure companies needing a durable state layer under an agent runtime. None of this reflects any actual approach, interest, or discussion from any party; it is offered as a way to reason about where the technology could fit strategically.

Open-source distribution. The project is distributed under the MIT license with no dual-licensing or commercial tier today. That maximizes adoption friction reduction (any team can use it in production immediately) at the cost of no current monetization mechanism. See Monetization & Editions Architecture for the open-core progression and architectural separation.

Competitive landscape. Summarized in the Joltrin vs. Alternatives table further down. No competitor is presented as inferior; each is a mature, widely deployed system that Joltrin would need to displace or complement for any given workload.

🏢 For Potential Customers

Is Joltrin Right For Me? Start from the existing When Joltrin is a Great Fit and When Joltrin is NOT the Right Tool sections above, they are the concrete answer. As a quick filter:

• If you are currently running Redis plus Postgres plus a queue just to get durable state and coordinated background work for one application, and that application's data fits comfortably on the machines it runs on, Joltrin is worth evaluating as a replacement for that stack.
• If you already run Postgres or Kafka at scale for reasons unrelated to this problem (complex SQL, multi-datacenter event retention, an existing team's expertise), Joltrin is more likely to complement than replace what you have.
• If your workload is petabyte-scale analytics or requires synchronous multi-region consensus, Joltrin is not the right tool today; see the section above for specifics.

Joltrin is a library you embed, not a managed service you sign up for. There is no hosted offering today; you run it yourself, in-process, in your own infrastructure.

👔 For CTOs & Engineering Executives

Every service you run that exists only to hold state or coordinate work (a cache, a queue, a lock manager) is a service your team has to patch, monitor, upgrade, and page on. Joltrin's bet is that collapsing storage, transactions, and task coordination into one embedded library reduces that surface for the workloads it fits, at the cost of giving up the specialized tooling and operational maturity of dedicated systems your team may already know well.

Concretely, that means: fewer network hops in your hot path (sub-millisecond, in-process calls instead of 15 to 50ms across Redis, a queue, and Postgres), one dependency to patch and upgrade instead of several, and a transaction boundary that spans your data and your background work instead of stopping at the database. It also means your team takes on a less mature, less battle-tested piece of infrastructure than Postgres or Kafka, with a correspondingly smaller ecosystem, smaller hiring pool of people who already know it, and no enterprise support contract available today. Evaluate it the way you would any early infrastructure bet: pilot it on one bounded, non-critical workload before committing a core system to it.

🧠 For AI Infrastructure Teams

What Joltrin already provides. Durable, transactional checkpointing for agent reasoning state: each step an agent commits is a separate, durable B-Tree write, so a killed agent process loses nothing already committed, and a successor process can resume from the last checkpoint. This is not a diagram, it runs today in the browser demo (the "AI Agent Memory" tab) and as a Go example (go run ./examples/agent_memory). Joltrin also provides vector similarity search over embeddings stored in the same B-Tree as structured data (ai/memory, ai/vector), and a real swarm/worker package (ai/swarm) with job and result stores.

What could be built on Joltrin, but is not shipped today. A production multi-agent orchestration framework, a hosted durable-memory-as-a-service for agent frameworks like LangGraph or AutoGen, and distributed MapReduce-style helpers across a live agent swarm are all described as design proposals in ai/SWARM_DESIGN.md (explicitly marked "Proposal / Vision" in that file) but are not implemented and tested the way the checkpointing and vector search primitives are. Treat anything not demonstrated in the linked demo or example as a direction, not a delivered feature.

Protocol interoperability, actually implemented. tools/mcpserver and tools/a2aagent expose Joltrin runbooks to MCP and A2A clients respectively, both gated by a real safety-and-reachability barrier certificate (ai/verify) so a step can't execute out of order regardless of what a calling agent claims. Both protocols share one execution trace store, proven by a test that commits steps via one protocol and confirms the other sees them. See MCP, A2A, and the Verification Engine for the audit, the design, and an honest accounting of what this checker is and is not (it is not general-purpose LTL model checking).

⚙️ For Platform, SRE & Cloud Engineers

Joltrin Engine is a library, not a server: there is no separate database process to provision, patch, or fail over for the embedded case. The optional tools/httpserver Data Manager is a standalone service with its own /metrics endpoint (tested in tools/httpserver/metrics_test.go) if you do want a network-accessible console. Failure recovery is handled by Reed-Solomon erasure coding across storage shards (fs/erasure, 12 passing tests at the time of writing) rather than full N-way replication, which trades some recovery latency for lower disk overhead. A prebuilt quickstart container is published to ghcr.io/sharedcode/joltrin-quickstart. Multi-node swarm clustering exists and is tested (examples/swarm_clustered, examples/swarm_standalone), but has not been documented or proven at production scale.

Supply-Chain Security & Release Provenance: Release builds are secured by an automated pre-publish quality gate (scripts/verify_release.sh), cryptographic SHA-256 manifests (SHA256SUMS), SPDX Software Bill of Materials (SBOM), and cryptographically signed build provenance attestations via GitHub Actions OIDC (actions/attest-build-provenance, SLSA Level 3 compliance). Consumers can independently verify any downloaded artifact using the standalone verification script.

🧪 For Researchers & Distributed Systems Engineers

The interesting parts to read are the B-Tree implementation with copy-on-write page isolation (btree/), the WAL plus two-phase commit transaction protocol (transaction.go, common/), the Reed-Solomon erasure coding layer (fs/erasure/), and the swarm coordination model described in ai/SWARM_DESIGN.md. The Architecture Whitepaper and Architecture vs. Big Tech go deeper into the design tradeoffs than this README does.

🎓 For Students & Learners

Reading this codebase is a reasonable way to see real (not textbook-simplified) implementations of a B-Tree with node splitting and range iteration, optimistic concurrency control, write-ahead logging with two-phase commit, and erasure coding, all in readable Go with test coverage next to the implementation. Start with docs/WHAT_IS_SOP.md for a plain-language overview, then run the zero-dependency quickstart below before reading btree/ and fs/erasure/.

---

💎 Monetization & Editions Architecture

Joltrin follows an Open-Core and Governance architecture. The core database, vector similarity search, and agent memory engine are, and will always remain, 100% free and open-source under the permissive MIT License.

Commercial tiers are focused entirely on enterprise governance, compliance, policy enforcement, multi-tenancy, and managed cloud infrastructure, leaving the open-source core complete, unhindered, and unthrottled.

For deep architectural documentation on package boundaries and code separation, see Monetization & Governance Architecture.

Tier / Edition	What It Provides	Distribution & Licensing	Implementation Status
Free / Open-Source Core	• Embedded copy-on-write B-Tree storage engine • WAL + 2PC strict ACID transactions • Reed-Solomon erasure coding and bitrot healing • Durable AI agent memory & checkpointed buffers • In-memory 128-d cosine vector similarity • Embedded MCP server (cmd/sop-mcp-server) • Embedded A2A agent runtime (cmd/sop-a2a-agent) • Local runbook verification barrier (ai/verify) • Developer GitHub OIDC authentication	Embedded Library & CLI Permissive MIT License ($0)	Available Today
Pro Governance	• Policy-as-Code declarative runtime compiler • Tamper-evident SHA-256 audit lineage & verification • Signed cryptographic audit export • Team-level workspaces and quota management • Priority MCP gateways and traffic shaping • Stripe Checkout, Customer Portal & Webhook Engine	Team Commercial Add-on ($49/team/mo)	Available Today (governance/)
Enterprise Governance	• Enterprise SSO: Okta & Microsoft Entra ID • Multi-tenant RBAC & tenant isolation boundaries • Enterprise audit streaming (real-time SIEM / Kafka) • Fine-grained verification rules & custom safety invariants • Custom invariant enforcement engine • Enterprise compliance guarantees and SLA	Self-Hosted Enterprise Commercial (Custom / Annual)	Foundation Implemented (governance/)
Hosted Cloud	• Managed Joltrin instances (zero-ops) • Cloud-hosted MCP hub & multi-agent routing • Multi-region database replication • Managed agent coordination network • Automated off-site snapshots & backup verification	Managed Cloud SaaS	Planned

Open-Source Guarantees

• No Artificial Paywalls: The open-source core will never cap database size, transaction limits, memory buffers, or local MCP/A2A concurrency.
• Permanent MIT License: Core storage, vector search, and agent safety verification remain permanently open-source under the MIT license.
• Decoupled Architecture: Commercial and governance modules interact via clean, decoupled Go interfaces (governance/) rather than invasive runtime licensing checks.

For a longer strategic view of enterprise defensibility and local-first architecture, see Strategic Architecture & Investor Moat.

---

🗺️ Roadmap

Shipped and tested today: the Go core engine, Python bindings (sop4py, on PyPI), C# bindings (Sop, on NuGet), the WebAssembly browser demo, the standalone HTTP Data Manager, and the interactive AI agent memory checkpointing demo.

In progress, code exists in-repo: Java bindings (sop4j), complete with tests, blocked on Maven Central Portal credential setup rather than on missing functionality (see docs/RELEASE_PROCESS_JAVA_STATUS.md). Rust bindings (sop4rs), with tests and examples in-repo, not yet published to crates.io.

Proposed, not yet implemented: the swarm job distribution, Await, and MapReduce helpers described in ai/SWARM_DESIGN.md, which that document itself labels "Proposal / Vision" rather than shipped.

This list reflects what is actually in the repository at the time of writing. It is not a committed release schedule.

---

🌎 Cross-Platform

CI (.github/workflows/ci.yml) builds, vets, and runs the core unit tests (inmemory, btree, common, cache, encoding, database) on ubuntu-latest, macos-latest, and windows-latest on every push and pull request, as three independent, parallel jobs. macos-latest runs on Apple Silicon (arm64), so that leg also verifies arm64 for free.

The Redis- and Cassandra-backed integration and stress test suites stay Linux-only: GitHub Actions' services: containers require a Linux-hosted runner, so those specific suites are not run on macOS or Windows today. That is a real gap in what is verified there, not a hidden one. All core unit test packages (inmemory, btree, common, cache, encoding, database) run and pass cleanly across all three operating systems (Linux, macOS, and Windows).

---

💻 For Developers

1. In-Memory Quickstart (Zero Dependencies)

# Clone the repository and run the quickstart
git clone https://github.com/sharedcode/joltrin.git
cd joltrin
go run ./examples/quickstart

package main

import (
	"fmt"
	"github.com/sharedcode/joltrin/inmemory"
)

func main() {
	// Create an in-memory B-Tree with unique integer keys and string values
	tree := inmemory.NewBtree[int, string](true)

	// Add records
	tree.Add(101, "Build #101: tests passed")
	tree.Add(102, "Build #102: deployed to staging")

	// Ordered range scan (keys 100 to 105)
	for k, v := range tree.Range(100, 105) {
		fmt.Printf("Key %d -> %s\n", k, v)
	}
}

2. AI Agent Memory & Swarm Task Hand-off

Run the new dedicated Agent Memory demo:

go run ./examples/agent_memory

This demo demonstrates an AI worker creating a context checkpoint, crashing mid-step, rolling back cleanly, and having a healthy peer worker resume the task in <15ms.

---

⚡ Performance Benchmarks

Below are benchmark results from the repository benchmark harness (tools/benchmark) run on a 2015 MacBook Pro (Dual-Core Intel Core i5, 8GB RAM, macOS).

What is measured: these runs benchmark Joltrin Engine's in-memory L2 cache with /tmp storage backing full ACID transactions, not disk-only storage without cache.

Microsecond-Scale Latency Profile

The benchmark measurements confirm sub-millisecond execution down to microsecond item lookups:

• Embedded In-Process Latency: < 0.3ms (<300µs) per transaction or safety barrier check, versus 15-50ms for multi-tier network round trips.
• Per-Item Write Latency: ~6.87µs (at 145,417 ops/sec with full ACID WAL logging).
• Per-Item Read Latency: ~6.95µs (at 143,770 ops/sec).
• Swarm Failover & Re-assignment: < 15ms heartbeat lease detection with automated rollback.

Exact reproduction command:

go run ./tools/benchmark -count <N> -slotlength <N>

For example:

go run ./tools/benchmark -count 10000 -slotlength 2000
go run ./tools/benchmark -count 100000 -slotlength 4000

Tuning SlotLength (Items per B-Tree Node)

10,000 Items Benchmark

SlotLength	Insert (ops/sec)	Read (ops/sec)	Delete (ops/sec)
1,000	107,652	136,754	40,964
2,000 (Balanced)	132,901	142,907	50,093
3,000	135,066	137,035	49,754
4,000	123,190	122,228	48,094

100,000 Items Benchmark

SlotLength	Insert (ops/sec)	Read (ops/sec)	Delete (ops/sec)
1,000	121,139	145,195	48,346
2,000	132,805	136,684	51,817
3,000	137,296	141,764	50,605
4,000 (Write-Heavy)	145,417	143,770	51,988

---

👥 For Engineering Leaders & Hiring Managers

For technical leaders, CTOs, and hiring managers, this repository serves as a working demonstration of systems engineering across:

• Storage Engine Design: Custom B-Tree implementation with sector-aligned direct I/O, node slot tuning, and multi-tier L1/L2 caching.
• Transactional Systems: Strict ACID guarantees, Write-Ahead Logging (WAL), Two-Phase Commit (2PC), Snapshot Isolation, and Optimistic Concurrency Control.
• Fault Tolerance & Reliability: Reed-Solomon Erasure Coding (N+K striping), active/passive metadata redundancy, and automated partition healing.
• High Concurrency: Lock-free data structures, multi-goroutine worker swarms, and SIMD vector dot-product calculation.
• Polyglot Architecture: Native Go kernel, Python bindings (sop4py), C# bindings (Sop), and browser WebAssembly.
• Production Delivery: GitHub Actions CI/CD matrix, distroless container builds on GHCR, Codecov integration, and static GitHub Pages deployments.

If you are building distributed systems, cloud infrastructure, or AI data platforms and want to discuss architecture, feel free to connect via GitHub Discussions.

---

📦 Language Packages & Tooling

Language	Installation	Description
Go	go get github.com/sharedcode/joltrin	Native high-performance core engine.
Python	pip install sop4py	Python bindings with Data Manager and AI scripts.
C#	dotnet add package Sop	Complete .NET Core integration.
WebAssembly	GOOS=js GOARCH=wasm go build	Browser-sandboxed zero-server execution.
HTTP Data Manager	sop-httpserver	Standalone UI console and AI Copilot interface.
Java (in progress)	source in bindings/java, not yet on Maven Central	sop4j bindings and tests are complete; publishing is blocked on Central Portal credential setup, tracked in docs/RELEASE_PROCESS_JAVA_STATUS.md.
Rust (in progress)	source in bindings/rust, not yet on crates.io	sop4rs bindings, tests, and examples exist in-repo but are not yet published as a crate.

How to Consume Joltrin: Releases vs. In-Repo Source

When integrating Joltrin into your stack, choose between official versioned releases and in-repo source consumption based on your development and operational needs:

Dimension	Official Tagged Releases (Recommended for Production)	In-Repo Source / Submodule (Active Prototyping & Contribution)
Artifacts	go get github.com/sharedcode/joltrin@vX.Y.Z PyPI: pip install sop4py NuGet: dotnet add package Sop	Git clone or submodule linked directly to HEAD or a feature branch
Best For	Production services, reproducible CI/CD builds, audited dependencies	Modifying engine internals, local benchmarking, custom protocol servers
Stability	Semantic versioning, tagged releases, audited dependency graph	Bleeding-edge features, experimental branches, unreleased protocol bridges
Maintenance	Handled by standard language package managers	Requires manual git fetch/rebase and local workspace management

1. Official Tagged Releases (Recommended for Production)

For production deployments, pin your dependency to a tagged release. This guarantees reproducible builds, backward-compatible API guarantees, and security-scanned transitive dependencies:

• Go: go get github.com/sharedcode/joltrin@v5.4.0 (see tags for the latest)
• Python: pip install sop4py==0.1.0
• C# / .NET: dotnet add package Sop --version 0.1.0

2. In-Repo Source / Submodule (Prototyping & Contribution)

If you are extending storage engine internals (btree/, fs/), modifying protocol servers (ai/verify, cmd/sop-mcp-server, cmd/sop-a2a-agent), or benchmarking performance enhancements, consuming from source is recommended:

# Add as a git submodule in your project
git submodule add https://github.com/sharedcode/joltrin.git vendor/joltrin

# Or configure a Go workspace (go.work) for local development
go work use ./vendor/joltrin

Cutting a Release (Maintainers)

Releases are cut from this repo with the scripts in scripts/, then tagged and pushed to GitHub. The full step-by-step for building native bindings and publishing to PyPI/NuGet/Maven is in RELEASE_PROCESS.md; the short version:

# 1. Bump the version everywhere (VERSION file, go.mod-adjacent metadata, bindings)
./scripts/update_version.sh 5.4.0

# 2. Review the diff, then commit the bump
git add -A && git commit -m "chore: bump version to 5.4.0"

# 3. Build release artifacts (native libs for Python/Java/C# bindings)
./scripts/build_release.sh

# 4. Verify checksums, archive integrity, and SBOM before publishing
./scripts/verify_release.sh release

# 5. Tag and push. This is what makes `go get github.com/sharedcode/joltrin@v5.4.0` resolve.
git tag v5.4.0
git push origin master v5.4.0

# 6. Create the GitHub Release from the tag (attaches release notes + artifacts)
gh release create v5.4.0 --generate-notes

Go's package proxy needs no separate publish step: once the tag is pushed, go get ...@v5.4.0 works immediately. Python, C#, and Java bindings still require the explicit twine upload / dotnet nuget push / mvn deploy steps in RELEASE_PROCESS.md.

📚 Technical Reference Guides

• What is SOP, in Plain Words: High-level conceptual overview.
• Architecture Whitepaper: Deep dive into B-Tree layout and transactions.
• Platform Tools & Relational Intelligence: Data Manager, CEL expressions, and AI Copilot.
• AI Copilot & Agent Architecture: Multi-agent memory model and Space partitioning.
• Operations & Failover Guide: Erasure coding, recovery, and cluster management.
• Scalability & Capacity Math: Architectural scaling model for billions of items.

---

🤝 Get Involved

We welcome feedback, issues, and contributions:

1. Fork & Clone: git clone https://github.com/sharedcode/joltrin.git
2. Run Tests: go test -v ./...
3. Join Discussions: GitHub Discussions
4. Submit a PR: Follow Go formatting standards (gofmt) and include test coverage.

---

_{Licensed under the MIT License. Built by SharedCode.}

Documentation

Overview

Package sop defines the core interfaces, types, and helpers used across the SOP codebase. It provides transactions, store options and metadata, key/handle abstractions, and shared error codes. Concrete backends live in subpackages such as fs (filesystem), cassandra, and redis, while higher-level features include B-Trees and streaming data helpers. It is designed to be extensible and modular, allowing for various storage backends to be implemented while sharing a common interface. This package is intended for internal use within the SOP project and is not meant for external use. It is a foundational package that other components build upon. It is not intended to be used directly by end-users, but rather serves as a base for more specific implementations and utilities in the SOP ecosystem. It is a foundational package that other components build upon. It is not intended to be used directly by end-users, but rather serves as a base for more specific implementations and utilities in the SOP ecosystem.

See `infs.package` for a concrete implementation of a File System-based store with built-in Redis caching.

Package sop contains SOP integration code for Redis, Cassandra & Kafka (in_red_c).

Index

• Constants
• Variables
• func Authorize(ctx context.Context, access ResourceAccess, action Action) bool
• func CanPerformAction(ctx context.Context, resourceName string, access ResourceAccess, action Action) bool
• func CheckPolicy(ctx context.Context, resourceName string, access ResourceAccess, action Action) error
• func ConfigureLogging()
• func ContextWithAuth(ctx context.Context, auth AuthContext) context.Context
• func EnforcePolicy(ctx context.Context, resourceName string, access ResourceAccess, action Action) error
• func FlattenForSchema(key any, value any) map[string]any
• func FormatRegistryTable(name string) string
• func FormatSchema(schema map[string]string) string
• func GetAllBlueprints() map[string]AssetBlueprint
• func HandleLockAcquisitionFailure(ctx context.Context, err error, rollbackFunc func(context.Context, UUID) error, ...) error
• func InferSchema(item map[string]any) map[string]string
• func InferSchemaType(v any) string
• func InferType(v any) (string, bool)
• func IsFailoverQualifiedIOError(err error) bool
• func IsSystemReadOnly(resourceName string) bool
• func RandomSleep(ctx context.Context)
• func RandomSleepWithUnit(ctx context.Context, unit time.Duration)
• func RegisterAssetRBAC(blueprint AssetBlueprint)
• func RegisterL2CacheFactory(ct L2CacheType, f L2CacheFactory)
• func Retry(ctx context.Context, task func(ctx context.Context) error, ...) error
• func SetDefaultCacheConfig(cacheDuration StoreCacheConfig)
• func SetJitterRNG(r *rand.Rand)
• func SetLogLevel(level slog.Level)
• func ShouldRetry(err error) bool
• func Sleep(ctx context.Context, sleepTime time.Duration)
• func TimedOut(ctx context.Context, name string, startTime time.Time, maxTime time.Duration) error
• type Action
• type AssetBlueprint
  • func GetAssetBlueprint(assetType string) (AssetBlueprint, bool)
• type AuthContext
  • func GetAuthFromContext(ctx context.Context) AuthContext
• type BlobStore
• type BlobsPayload
• type BundledResponse
• type CloseableCache
• type ContextRBACMap
  • func ResolveRBACMap(ctx context.Context, assetType string, entitlementCtx EntitlementContext, ...) ContextRBACMap
• type DatabaseOptions
  • func (do DatabaseOptions) CopyTo(transOptions *TransactionOptions)
  • func (do DatabaseOptions) GetDatabaseType() DatabaseType
  • func (do DatabaseOptions) IsCassandraHybrid() bool
  • func (do DatabaseOptions) IsEmpty() bool
  • func (do DatabaseOptions) IsReplicated() bool
  • func (do *DatabaseOptions) SetDatabaseType(t DatabaseType)
• type DatabaseType
• type EndpointContext
• type EntitlementContext
• type ErasureCodingConfig
• type ErrTimeout
  • func (e ErrTimeout) Error() string
  • func (e ErrTimeout) Unwrap() error
• type Error
  • func (e Error) Error() string
• type ErrorCode
• type Handle
  • func NewHandle(id UUID) Handle
  • func (h *Handle) AllocateID() UUID
  • func (h *Handle) ClearInactiveID()
  • func (h *Handle) FlipActiveID()
  • func (h Handle) GetActiveID() UUID
  • func (h Handle) GetInActiveID() UUID
  • func (h *Handle) HasID(id UUID) bool
  • func (h Handle) IsAandBinUse() bool
  • func (x *Handle) IsEmpty() bool
  • func (x *Handle) IsEqual(y *Handle) bool
  • func (h *Handle) IsExpiredInactive() bool
• type KeyValuePair
• type KeyValueStore
• type KeyValueStoreItemActionResponse
• type KeyValueStoreResponse
• type L2Cache
  • func GetL2Cache(options TransactionOptions) L2Cache
• type L2CacheFactory
• type L2CacheType
• type LockKey
• type Locker
• type ManageStore
• type RedisCacheConfig
• type Registry
• type RegistryPayload
  • func ExtractLogicalIDs(storeHandles []RegistryPayload[Handle]) []RegistryPayload[UUID]
• type Relation
• type ResourceAccess
• type SchemaInferenceResult
  • func InferSchemaFromTypes(key any, value any) SchemaInferenceResult
• type SinglePhaseTransaction
  • func (t *SinglePhaseTransaction) AddPhasedTransaction(otherTransaction ...TwoPhaseCommitTransaction)
  • func (t *SinglePhaseTransaction) Begin(ctx context.Context) error
  • func (t *SinglePhaseTransaction) Close() error
  • func (t *SinglePhaseTransaction) Commit(ctx context.Context) error
  • func (t *SinglePhaseTransaction) CommitMaxDuration() time.Duration
  • func (t *SinglePhaseTransaction) GetID() UUID
  • func (t *SinglePhaseTransaction) GetMode() TransactionMode
  • func (t *SinglePhaseTransaction) GetPhasedTransaction() TwoPhaseCommitTransaction
  • func (t *SinglePhaseTransaction) GetStores(ctx context.Context) ([]string, error)
  • func (t *SinglePhaseTransaction) HasBegun() bool
  • func (t *SinglePhaseTransaction) OnCommit(callback func(ctx context.Context) error)
  • func (t *SinglePhaseTransaction) Rollback(ctx context.Context) error
• type StoreCacheConfig
  • func GetDefaultCacheConfig() StoreCacheConfig
  • func NewStoreCacheConfig(cacheDuration time.Duration, isCacheTTL bool) *StoreCacheConfig
• type StoreInfo
  • func NewStoreInfo(si StoreOptions) *StoreInfo
  • func (si *StoreInfo) DeleteCustomData(key string) bool
  • func (si *StoreInfo) GetCustomData(key string) (any, bool)
  • func (si *StoreInfo) GetCustomDataMap() map[string]any
  • func (s StoreInfo) IsCompatible(b StoreInfo) bool
  • func (s StoreInfo) IsEmpty() bool
  • func (si *StoreInfo) SetCustomData(key string, value any)
  • func (si *StoreInfo) SetCustomDataMap(data map[string]any)
  • func (si *StoreInfo) ValueDataSize() ValueDataSize
• type StoreOptions
  • func ConfigureStore(storeName string, uniqueKey bool, slotLength int, description string, ...) StoreOptions
• type StoreRepository
• type TaskRunner
  • func NewTaskRunner(ctx context.Context, maxThreadCount int) *TaskRunner
  • func (tr *TaskRunner) GetContext() context.Context
  • func (tr *TaskRunner) Go(task func() error)
  • func (tr *TaskRunner) Wait() error
• type Transaction
  • func NewTransaction(mode TransactionMode, twoPhaseCommitTrans TwoPhaseCommitTransaction) (Transaction, error)
• type TransactionLog
• type TransactionMode
• type TransactionOptions
  • func (to TransactionOptions) GetDatabaseOptions() DatabaseOptions
  • func (to TransactionOptions) IsCassandraHybrid() bool
  • func (to TransactionOptions) IsReplicated() bool
• type TransactionPriorityLog
• type Tuple
• type TwoPhaseCommitTransaction
• type UICapability
  • func ActionToUICapability(action Action) UICapability
• type UUID
  • func NewUUID() UUID
  • func ParseUUID(id string) (UUID, error)
  • func (x UUID) Compare(y UUID) int
  • func (id UUID) IsNil() bool
  • func (id UUID) MarshalText() ([]byte, error)
  • func (id UUID) Split() (uint64, uint64)
  • func (id UUID) String() string
  • func (id *UUID) UnmarshalText(b []byte) error
• type ValueDataSize
• type Visibility

Constants

const (
	// Unknown represents an unspecified error condition.
	Unknown ErrorCode = iota
	// LockAcquisitionFailure indicates failure to acquire a required lock.
	LockAcquisitionFailure
	// FileIOError represents file I/O related errors, e.g. encountered by BlobStore (w/ & w/o EC).
	// This should not generate Failover event because BlobStore errors are either handled internally for no EC
	// or by EC replication feature.
	FileIOError
	// FailoverQualifiedError marks an error that qualifies the operation for failover handling.
	FailoverQualifiedError = 77 + iota
	// FileIOErrorFailoverQualified represents file I/O related errors.
	FileIOErrorFailoverQualified
	// RestoreRegistryFileSectorFailure indicates a failure while restoring a registry file sector.
	RestoreRegistryFileSectorFailure
)

const (
	RoleAdmin = "Admin"
	RoleUser  = "User"
	RoleGuest = "Guest"
)

const ContextPriorityLogIgnoreAge contextKey = "plg_ignore_age"

ContextPriorityLogIgnoreAge signals priority log GetBatch to ignore age filter when true.

const (
	// HandleSizeInBytes is the size, in bytes, of a Handle structure when encoded.
	HandleSizeInBytes = 62
)

const MaxSlotLength = 20000

MaxSlotLength is the maximum number of items a node can accommodate. Enforced both when a StoreInfo is first created (NewStoreInfo) and again at the point a node is allocated (btree.getSlotLength), since a StoreInfo loaded from persisted storage doesn't go through NewStoreInfo and could otherwise carry an unbounded or corrupted value straight into a make().

Variables

var (
	ErrSystemReadOnly = errors.New("system knowledge bases are read-only")
	ErrQuotaExceeded  = errors.New("quota exceeded")
	ErrUnauthorized   = errors.New("unauthorized access")
)

var Now = time.Now

Now returns the current time. It is a var to allow tests to override time.Now for determinism.

var RetryStartDuration = 1 * time.Second

RetryStartDuration is the initial duration for the Fibonacci backoff. It is exported to allow tests to reduce the wait time.

var Version = strings.TrimSpace(versionFile)

Version is the current version of the SOP library/application.

Functions

func Authorize

func Authorize(ctx context.Context, access ResourceAccess, action Action) bool

func CanPerformAction

func CanPerformAction(ctx context.Context, resourceName string, access ResourceAccess, action Action) bool

CanPerformAction checks if the current context has permission to perform the action on the resource. It returns a boolean, making it ideal for UI visibility toggles like IsReadOnly.

func CheckPolicy

func CheckPolicy(ctx context.Context, resourceName string, access ResourceAccess, action Action) error

CheckPolicy evaluates the three-layer RBAC model and returns an error if access is denied. It is useful when you need to know exactly *why* access was denied.

func ConfigureLogging

func ConfigureLogging()

ConfigureLogging sets up the global default logger with a TextHandler and configures the log level based on the SOP_LOG_LEVEL environment variable. It defaults to Info level if not specified.

This function should be called by the application at startup if it wants to use the default SOP logging configuration.

func ContextWithAuth

func ContextWithAuth(ctx context.Context, auth AuthContext) context.Context

func EnforcePolicy

func EnforcePolicy(ctx context.Context, resourceName string, access ResourceAccess, action Action) error

EnforcePolicy checks the policy and returns an error if the action is not allowed. Use CanPerformAction for a boolean result (e.g. for adjusting UI states).

func FlattenForSchema

func FlattenForSchema(key any, value any) map[string]any

FlattenForSchema converts key and value of any type into a flat map[string]any suitable for schema inference. Uses JSON marshaling to handle structs. NOTE: InferSchemaFromTypes is preferred when type information is available.

func FormatRegistryTable

func FormatRegistryTable(name string) string

FormatRegistryTable formats a store name into a registry table name by adding an _r suffix.

func FormatSchema

func FormatSchema(schema map[string]string) string

FormatSchema formats a schema map as a sorted string like "{field: type, ...}".

func GetAllBlueprints

func GetAllBlueprints() map[string]AssetBlueprint

GetAllBlueprints returns a cloned map of the system-wide blueprints

func HandleLockAcquisitionFailure

func HandleLockAcquisitionFailure(ctx context.Context, err error,
	rollbackFunc func(context.Context, UUID) error,
	unlockFunc func(context.Context, []*LockKey) error) error

HandleLockAcquisitionFailure checks if the error is a LockAcquisitionFailure and if so, attempts to rollback the blocking transaction using the provided rollbackFunc. If rollback succeeds, it takes over the lock and releases it using unlockFunc. Returns nil if the failure was handled (lock taken over and released), otherwise returns the original error.

func InferSchema

func InferSchema(item map[string]any) map[string]string

InferSchema inspects a map and returns a simplified type definition (e.g. {"id": "uuid", "age": "number"}).

func InferSchemaType

func InferSchemaType(v any) string

InferSchemaType returns a string representation of the type of a value for schema inference.

func InferType

func InferType(v any) (string, bool)

InferType returns the simplified type name (e.g. "string", "int", "uuid") and whether it's an array. This is used for UI display and loose type checking.

func IsFailoverQualifiedIOError

func IsFailoverQualifiedIOError(err error) bool

IsFailoverQualifiedIOError reports whether an error indicates the active drive/filesystem is unhealthy in a way that warrants immediate failover to the passive drive.

This is distinct from ShouldRetry: retryable/transient errors should be retried first, while this function targets permanent/media/FS/device conditions where staying on the current drive is counterproductive.

Notes:

• It includes common POSIX errno values available on macOS/Linux/BSD.
• For Linux-specific errno that may not exist on other platforms as named constants, numeric values are used via syscall.Errno to remain portable (they will simply never match on platforms that don't produce them).
• EFBIG is intentionally excluded per current SOP usage (registry/store repo use small files).

func IsSystemReadOnly

func IsSystemReadOnly(resourceName string) bool

IsSystemReadOnly returns true if the specified resource is a core system resource that must remain read-only to prevent destructive actions by any user.

func RandomSleep

func RandomSleep(ctx context.Context)

RandomSleep sleeps for a random duration between 20ms and 80ms to stagger retries.

func RandomSleepWithUnit

func RandomSleepWithUnit(ctx context.Context, unit time.Duration)

RandomSleepWithUnit sleeps for a random multiple (1..4) of the provided unit duration. Useful to jitter conflicting transactions and reduce contention.

func RegisterAssetRBAC

func RegisterAssetRBAC(blueprint AssetBlueprint)

RegisterAssetRBAC ensures that new Assets declare their UI footprint and execution logic cleanly

func RegisterL2CacheFactory

func RegisterL2CacheFactory(ct L2CacheType, f L2CacheFactory)

RegisterL2CacheFactory registers a cache factory for a given type.

func Retry

func Retry(ctx context.Context, task func(ctx context.Context) error, gaveUpTask func(ctx context.Context)) error

Retry executes task with Fibonacci backoff up to 5 retries. If retries are exhausted, gaveUpTask is invoked (when not nil) and the final error is returned.

func SetDefaultCacheConfig

func SetDefaultCacheConfig(cacheDuration StoreCacheConfig)

SetDefaultCacheConfig assigns the global default cache configuration used when a store does not override it.

func SetJitterRNG

func SetJitterRNG(r *rand.Rand)

SetJitterRNG overrides the RNG used for sleep jitter. Useful for deterministic tests.

func SetLogLevel

func SetLogLevel(level slog.Level)

SetLogLevel sets the logging level for the logger configured by ConfigureLogging.

func ShouldRetry

func ShouldRetry(err error) bool

ShouldRetry reports whether the error is retryable (non-nil and not a known permanent failure).

func Sleep

func Sleep(ctx context.Context, sleepTime time.Duration)

Sleep blocks for the specified duration or until the context is done, whichever happens first.

func TimedOut

func TimedOut(ctx context.Context, name string, startTime time.Time, maxTime time.Duration) error

TimedOut returns an error if the context is done or if the elapsed time since startTime exceeds maxTime.

Types

type Action

type Action string

const (
	ActionRead     Action = "read"
	ActionWrite    Action = "write"
	ActionDelete   Action = "delete"
	ActionList     Action = "list"
	ActionAISelect Action = "ai_select"
)

type AssetBlueprint

type AssetBlueprint struct {
	AssetType   string   // e.g., "space", "store", "agent"
	Description string   // Human-readable context
	Endpoints   []string // UI API endpoints related to this asset
	Actions     []Action // Capabilities: Read, Write, Delete, Execute

	// Evaluator executes the actual permission check for a specific asset instance or context
	Evaluator func(ctx context.Context, entitlementCtx EntitlementContext, action Action) bool
}

AssetBlueprint defines the RBAC footprint for a specific module or asset type.

func GetAssetBlueprint

func GetAssetBlueprint(assetType string) (AssetBlueprint, bool)

GetAssetBlueprint looks up the evaluator instructions by alias

type AuthContext

type AuthContext struct {
	UserID   string
	Roles    []string
	IsSystem bool
}

func GetAuthFromContext

func GetAuthFromContext(ctx context.Context) AuthContext

type BlobStore

type BlobStore interface {
	// GetOne fetches a blob by ID from a blob table.
	GetOne(ctx context.Context, blobTable string, blobID UUID) ([]byte, error)
	// Add inserts blobs.
	Add(ctx context.Context, blobs []BlobsPayload[KeyValuePair[UUID, []byte]]) error
	// Update modifies existing blobs.
	Update(ctx context.Context, blobs []BlobsPayload[KeyValuePair[UUID, []byte]]) error
	// Remove deletes blobs by ID.
	Remove(ctx context.Context, blobsIDs []BlobsPayload[UUID]) error
}

BlobStore defines CRUD operations for binary blobs that are too large for typical databases and are stored in external systems (e.g., S3, filesystem, Cassandra partitions).

type BlobsPayload

type BlobsPayload[T UUID | KeyValuePair[UUID, []byte]] struct {
	// BlobTable is the blob store table name (or base filesystem path).
	BlobTable string
	// Blobs holds either IDs (for deletes) or ID+data pairs (for upserts).
	Blobs []T
}

BlobsPayload is a request/response envelope for blob operations.

type BundledResponse

type BundledResponse struct {
	Data     interface{}               `json:"data"`
	RBAC     ContextRBACMap            `json:"rbac,omitempty"`
	ItemRBAC map[string]ContextRBACMap `json:"item_rbac,omitempty"`
}

BundledResponse represents the standard JSON payload structure containing both the domain data and the paired RBAC map.

type CloseableCache

type CloseableCache interface {
	L2Cache
	io.Closer
}

CloseableCache is a Cache that also implements io.Closer for explicit lifecycle control.

type ContextRBACMap

type ContextRBACMap map[UICapability]bool

ContextRBACMap represents the UI-consumable map format representing capabilities for a given context: (Capability -> bool)

func ResolveRBACMap

func ResolveRBACMap(ctx context.Context, assetType string, entitlementCtx EntitlementContext, getLocalAccess func() ResourceAccess) ContextRBACMap

ResolveRBACMap evaluates the dynamic AssetBlueprint for a functional context (e.g., current space/store), delegating to the core evaluator.

type DatabaseOptions

type DatabaseOptions struct {
	// StoresFolders specifies the folders for replication.
	// If more than one folder, i.e. - one for Active drive/folder,
	// & another for Passive drive/folder, Registry replication is enabled.
	StoresFolders []string `json:"stores_folders,omitempty"`
	// ErasureConfig specifies the erasure coding configuration for Blob store replication.
	ErasureConfig map[string]ErasureCodingConfig `json:"erasure_config,omitempty"`
	// Keyspace to be used for the transaction (Cassandra).
	Keyspace string `json:"keyspace,omitempty"`
	// CacheType specifies the type of cache to use (e.g. InMemory, Redis).
	CacheType L2CacheType `json:"cache_type"`
	// RedisConfig specifies the Redis configuration when CacheType is Redis.
	RedisConfig *RedisCacheConfig `json:"redis_config,omitempty"`
	// Registry hash modulo value used for hashing.
	RegistryHashModValue int `json:"registry_hash_mod,omitempty"`

	// Type specifies the database type (Standalone or Clustered).
	// This is a convenience field that sets the default CacheType.
	Type DatabaseType `json:"type"`

	// EnableObfuscation specifies if this database should be obfuscated when accessed by AI tools.
	// This is a runtime-only field and is not persisted to JSON.
	EnableObfuscation bool `json:"-"`
}

DatabaseOptions holds the configuration for the database.

func (DatabaseOptions) CopyTo

func (do DatabaseOptions) CopyTo(transOptions *TransactionOptions)

Copy Database Options to Transaction Options.

func (DatabaseOptions) GetDatabaseType

func (do DatabaseOptions) GetDatabaseType() DatabaseType

func (DatabaseOptions) IsCassandraHybrid

func (do DatabaseOptions) IsCassandraHybrid() bool

func (DatabaseOptions) IsEmpty

func (do DatabaseOptions) IsEmpty() bool

IsEmpty returns true if database config is considered empty, i.e. - missing folder is primary. A Database should always have folder(s) where Registry data files are/will be stored.

func (DatabaseOptions) IsReplicated

func (do DatabaseOptions) IsReplicated() bool

func (*DatabaseOptions) SetDatabaseType

func (do *DatabaseOptions) SetDatabaseType(t DatabaseType)

type DatabaseType

type DatabaseType int

const (
	// Standalone mode uses an in-memory cache for coordination (locks, etc.).
	// It is appropriate for standalone or embedded applications running in a single process.
	Standalone DatabaseType = iota
	// Clustered mode uses Redis for coordination (locks, etc.).
	// It allows hosting multiple application instances across a network, properly orchestrated by SOP.
	Clustered
)

type EndpointContext

type EndpointContext string

EndpointContext represents an API endpoint representing a grouping of assets.

const (
	EndpointSpacesList EndpointContext = "/api/spaces"
	EndpointStoresList EndpointContext = "/api/stores"
	EndpointItemsList  EndpointContext = "/api/spaces/items"
)

type EntitlementContext

type EntitlementContext struct {
	AssetID    string
	Database   string
	IsSystemDB bool
	UserRole   string
	UserID     string
}

EntitlementContext holds the request scope parameters necessary for granular RBAC evaluation.

type ErasureCodingConfig

type ErasureCodingConfig struct {
	// DataShardsCount is the number of data shards.
	DataShardsCount int `json:"data_shards_count"`
	// ParityShardsCount is the number of parity shards.
	ParityShardsCount int `json:"parity_shards_count"`
	// BaseFolderPathsAcrossDrives lists the drive base paths where data and parity shard files are stored.
	BaseFolderPathsAcrossDrives []string `json:"base_folder_paths_across_drives"`

	// RepairCorruptedShards indicates whether to attempt automatic repair when corrupted shards are detected.
	// Auto-repair can be expensive; applications can disable it to prioritize throughput and handle drive
	// failures via external workflows.
	RepairCorruptedShards bool `json:"repair_corrupted_shards"`
}

ErasureCodingConfig defines per-blob-table erasure coding settings, including shard counts, storage locations, and optional automatic shard repair.

type ErrTimeout

type ErrTimeout struct {
	// Name is a short label for the operation (e.g., "transaction", "lockFileBlockRegion").
	Name string
	// MaxTime is the maximum duration allowed for the operation when applicable.
	MaxTime time.Duration
	// Cause is the underlying timeout/cancellation cause, typically a context error.
	Cause error
}

ErrTimeout is returned when an operation exceeds its allowed time budget.

Semantics:

• If a context cancellation or deadline triggered the timeout, Cause carries the original context error (context.Canceled or context.DeadlineExceeded). Unwrap() returns that Cause so errors.Is(err, context.DeadlineExceeded) works.
• If the operation-specific maximum duration triggered the timeout, Cause may be nil; MaxTime contains the configured bound for the operation.

This enables callers to branch on timeouts consistently while preserving the original context semantics when applicable.

func (ErrTimeout) Error

func (e ErrTimeout) Error() string

func (ErrTimeout) Unwrap

func (e ErrTimeout) Unwrap() error

Unwrap exposes the underlying cause (e.g., context.DeadlineExceeded) for errors.Is/As.

type Error

type Error struct {
	Code     ErrorCode
	Err      error
	UserData any
}

Error is a SOP-specific error carrying a code, the wrapped error and optional user data.

func (Error) Error

func (e Error) Error() string

Error implements the error interface by formatting the code, user data, and wrapped error details.

type ErrorCode

type ErrorCode int

ErrorCode enumerates SOP error categories used across packages.

type Handle

type Handle struct {
	// LogicalID is the stable identifier of the entity.
	LogicalID UUID
	// PhysicalIDA is one of the two physical IDs supported.
	PhysicalIDA UUID
	// PhysicalIDB is the second physical ID supported.
	PhysicalIDB UUID
	// IsActiveIDB indicates whether PhysicalIDB is currently the active ID.
	IsActiveIDB bool
	// Version is the current state version (active ID, final deleted state).
	Version int32
	// WorkInProgressTimestamp stores the millisecond timestamp of the inactive ID (or non-final deleted state).
	WorkInProgressTimestamp int64
	// IsDeleted marks a logical delete.
	IsDeleted bool
}

Handle holds a logical ID and its two physical IDs (A and B) used to implement ACID-safe swaps. SOP uses Handle to quickly switch between node versions and to support logical deletes.

func NewHandle

func NewHandle(id UUID) Handle

NewHandle creates a new Handle with the provided logical ID. PhysicalIDA is initialized to the same value.

func (*Handle) AllocateID

func (h *Handle) AllocateID() UUID

AllocateID generates a new UUID and assigns it to the available physical slot. If both A and B are already in use, NilUUID is returned.

func (*Handle) ClearInactiveID

func (h *Handle) ClearInactiveID()

ClearInactiveID resets the inactive physical ID to NilUUID and clears the WIP timestamp.

func (*Handle) FlipActiveID

func (h *Handle) FlipActiveID()

FlipActiveID switches the active physical ID from A to B or B to A.

func (Handle) GetActiveID

func (h Handle) GetActiveID() UUID

GetActiveID returns the currently active UUID (either PhysicalIDA or PhysicalIDB).

func (Handle) GetInActiveID

func (h Handle) GetInActiveID() UUID

GetInActiveID returns the currently inactive physical UUID.

func (*Handle) HasID

func (h *Handle) HasID(id UUID) bool

HasID reports whether the provided UUID matches either physical ID A or B.

func (Handle) IsAandBinUse

func (h Handle) IsAandBinUse() bool

IsAandBinUse reports whether both physical IDs A and B are populated.

func (*Handle) IsEmpty

func (x *Handle) IsEmpty() bool

IsEmpty reports whether all Handle fields are zero values (no IDs, not deleted, zero version and timestamps).

func (*Handle) IsEqual

func (x *Handle) IsEqual(y *Handle) bool

IsEqual reports whether two Handle instances are equal ignoring the Version field.

func (*Handle) IsExpiredInactive

func (h *Handle) IsExpiredInactive() bool

IsExpiredInactive reports whether the inactive ID has expired based on a fixed window.

type KeyValuePair

type KeyValuePair[TK any, TV any | []byte] struct {
	// Key is the key part in the pair.
	Key TK
	// Value is the value part in the pair.
	Value TV
}

KeyValuePair represents a pair of key and value, commonly used for blob operations where the key may differ from the blob ID.

type KeyValueStore

type KeyValueStore[TK any, TV any] interface {
	// Fetch retrieves entries by keys from the remote storage subsystem.
	Fetch(context.Context, string, []TK) KeyValueStoreResponse[KeyValuePair[TK, TV]]
	// FetchLargeObject retrieves a single large entry by key.
	FetchLargeObject(context.Context, string, TK) (TV, error)
	// Add inserts entries.
	Add(context.Context, string, []KeyValuePair[TK, TV]) KeyValueStoreResponse[KeyValuePair[TK, TV]]
	// Update modifies existing entries.
	Update(context.Context, string, []KeyValuePair[TK, TV]) KeyValueStoreResponse[KeyValuePair[TK, TV]]
	// Remove deletes entries by keys.
	Remove(context.Context, string, []TK) KeyValueStoreResponse[TK]
}

KeyValueStore defines CRUD operations for a generic key-value backend with optional partial success semantics.

type KeyValueStoreItemActionResponse

type KeyValueStoreItemActionResponse[T any] struct {
	Payload T
	Error   error
}

KeyValueStoreItemActionResponse is the per-item response including payload and error for a CRUD action.

type KeyValueStoreResponse

type KeyValueStoreResponse[T any] struct {
	// Details contains per-item action results.
	Details []KeyValueStoreItemActionResponse[T]
	// Error is a summary error if at least one action failed.
	Error error
}

KeyValueStoreResponse aggregates per-item results and an optional summary error.

type L2Cache

type L2Cache interface {
	// Inherit Locking interface.
	Locker

	// Implement to return the CacheType.
	GetType() L2CacheType

	// Set upserts a value under a key, and specifies when it will expire or disappear from L2 cache.
	Set(ctx context.Context, key string, value string, expiration time.Duration) error
	// Get returns: found(bool), value(string), err(error from backend).
	Get(ctx context.Context, key string) (bool, string, error)
	// GetEx returns found(bool), value(string), err using TTL/sliding expiration semantics.
	GetEx(ctx context.Context, key string, expiration time.Duration) (bool, string, error)

	// IsRestarted reports whether the cache backend (e.g., Redis) has restarted since the last check.
	// Implementations should return true once per backend restart event per-process and false otherwise.
	IsRestarted(ctx context.Context) bool

	// SetStruct upserts a struct value under a key.
	SetStruct(ctx context.Context, key string, value interface{}, expiration time.Duration) error
	// SetStructs upserts multiple struct values under the given keys in a single round trip (pipelined).
	SetStructs(ctx context.Context, keys []string, values []interface{}, expiration time.Duration) error
	// GetStruct fetches a struct value; first return indicates success (false for not found or error).
	GetStruct(ctx context.Context, key string, target interface{}) (bool, error)
	// GetStructEx fetches a struct value with TTL/sliding expiration semantics.
	GetStructEx(ctx context.Context, key string, target interface{}, expiration time.Duration) (bool, error)
	// GetStructs fetches multiple struct values with optional TTL/sliding expiration semantics.
	// If expiration > 0, it behaves like GetStructEx for each key (pipelined).
	// If expiration <= 0, it behaves like GetStruct but batched (e.g. MGET).
	GetStructs(ctx context.Context, keys []string, targets []interface{}, expiration time.Duration) ([]bool, error)
	// Delete removes objects by keys; returns whether all keys were deleted.
	Delete(ctx context.Context, keys []string) (bool, error)
	// Ping checks connectivity to the cache backend.
	Ping(ctx context.Context) error

	// Clear purges the entire cache database.
	Clear(ctx context.Context) error
}

L2Cache abstracts an out-of-process cache (e.g., Redis) and its locking facilities.

func GetL2Cache

func GetL2Cache(options TransactionOptions) L2Cache

GetL2Cache gets the cache (client) for the specified type. It returns nil if no factory is registered for that type.

type L2CacheFactory

type L2CacheFactory func(TransactionOptions) L2Cache

L2CacheFactory defines the function signature for creating a cache client.

type L2CacheType

type L2CacheType int

L2CacheType defines the type of cache to use.

const (
	// Default represents no (L2) caching.
	NoCache L2CacheType = iota
	// InMemory represents an in-memory cache.
	InMemory
	// Redis represents a Redis cache.
	Redis
)

type LockKey

type LockKey struct {
	Key         string
	LockID      UUID
	IsLockOwner bool
}

LockKey represents a lockable cache key along with ownership metadata.

type Locker

type Locker interface {
	// FormatLockKey creates a lock key name from an arbitrary string.
	FormatLockKey(k string) string
	// CreateLockKeys builds LockKey objects from a set of key names.
	CreateLockKeys(keys []string) []*LockKey
	// CreateLockKeysForIDs builds LockKey objects for ID tuples (e.g., Transaction ID scoped locks).
	CreateLockKeysForIDs(keys []Tuple[string, UUID]) []*LockKey

	// IsLockedTTL reports whether all keys are locked and refreshes TTL with the provided duration.
	IsLockedTTL(ctx context.Context, duration time.Duration, lockKeys []*LockKey) (bool, error)

	// Lock attempts to lock all keys; returns success, lock owner UUID, and any error encountered.
	Lock(ctx context.Context, duration time.Duration, lockKeys []*LockKey) (bool, UUID, error)
	// DualLock attempts to lock all keys; returns success, lock owner UUID, and any error encountered.
	// It calls Lock then IsLocked to ensure the lock is acquired and persisted.
	DualLock(ctx context.Context, duration time.Duration, lockKeys []*LockKey) (bool, UUID, error)
	// IsLocked reports whether all keys are currently locked.
	IsLocked(ctx context.Context, lockKeys []*LockKey) (bool, error)
	// IsLockedByOthers reports whether the keys are locked by other processes.
	IsLockedByOthers(ctx context.Context, lockKeyNames []string) (bool, error)
	// IsLockedByOthersTTL reports whether the keys are locked by other processes and refreshes TTL with the provided duration.
	IsLockedByOthersTTL(ctx context.Context, lockKeyNames []string, duration time.Duration) (bool, error)
	// Unlock releases a set of keys.
	Unlock(ctx context.Context, lockKeys []*LockKey) error
}

Locker defines lightweight lock management facade.

type ManageStore

type ManageStore interface {
	// CreateStore creates the store(s) container (e.g., a filesystem folder).
	CreateStore(context.Context, string) error
	// RemoveStore removes the store(s) container (e.g., a filesystem folder).
	RemoveStore(context.Context, string) error
}

ManageStore declares lifecycle operations for creating and removing store containers (e.g., folders).

type RedisCacheConfig

type RedisCacheConfig struct {
	// Address is the host:port of the Redis server/cluster.
	Address string `json:"address"`
	// Password is the password used to authenticate.
	Password string `json:"password"`
	// DB is the database index to select.
	DB int `json:"db"`
	// URL is the connection string (e.g. redis://user:pass@host:port/db).
	// If provided, it overrides Address, Password, and DB.
	URL string `json:"url,omitempty"`
	// DialTimeout specifies the timeout for connecting to Redis.
	DialTimeout time.Duration `json:"dial_timeout,omitempty"`
	// ReadTimeout specifies the timeout for reading from Redis.
	ReadTimeout time.Duration `json:"read_timeout,omitempty"`
	// WriteTimeout specifies the timeout for writing to Redis.
	WriteTimeout time.Duration `json:"write_timeout,omitempty"`
	// MaxRetries is the maximum number of retries before giving up on Redis connection.
	MaxRetries int `json:"max_retries,omitempty"`
}

RedisCacheConfig holds configuration for connecting to a Redis server or cluster.

type Registry

type Registry interface {
	// Get fetches Handles (given logical IDs) from registry table(s).
	Get(context.Context, []RegistryPayload[UUID]) ([]RegistryPayload[Handle], error)
	// Add inserts Handles into registry table(s).
	Add(context.Context, []RegistryPayload[Handle]) error
	// Update modifies Handles across registry table(s) and acquires cache locks for each Handle.
	Update(ctx context.Context, handles []RegistryPayload[Handle]) error
	// UpdateNoLocks updates Handles in an active transaction where locks were pre-acquired by the transaction manager.
	UpdateNoLocks(ctx context.Context, allOrNothing bool, storesHandles []RegistryPayload[Handle]) error
	// Remove deletes Handles (given logical IDs) from registry table(s).
	Remove(context.Context, []RegistryPayload[UUID]) error

	// Replicate performs post-commit replication of blobs/data to passive targets.
	Replicate(ctx context.Context, newRootNodesHandles, addedNodesHandles, updatedNodesHandles, removedNodesHandles []RegistryPayload[Handle]) error
}

Registry provides CRUD and replication operations for virtual ID management that back SOP's ACID workflow. All methods accept and/or return batches.

type RegistryPayload

type RegistryPayload[T Handle | UUID] struct {
	// RegistryTable is the table (or namespace) where the virtual IDs are stored or fetched.
	RegistryTable string

	// BlobTable is the paired blob table (or base filesystem path) used during Rollback and Commit.
	BlobTable string
	// CacheDuration specifies Redis cache duration.
	CacheDuration time.Duration
	// IsCacheTTL enables Redis TTL (sliding expiration) semantics when true.
	IsCacheTTL bool

	// IDs contains the virtual IDs (or Handles) to manage.
	IDs []T
}

RegistryPayload represents a request/response payload to manage or fetch Handles/UUIDs in a registry table. T can be either Handle (for writes) or UUID (for reads/deletes).

func ExtractLogicalIDs

func ExtractLogicalIDs(storeHandles []RegistryPayload[Handle]) []RegistryPayload[UUID]

ExtractLogicalIDs converts a slice of RegistryPayload[Handle] to RegistryPayload[UUID] by mapping LogicalID.

type Relation

type Relation struct {
	SourceFields []string `json:"source_fields"`
	TargetStore  string   `json:"target_store"`
	TargetFields []string `json:"target_fields"`
}

Relation describes a foreign key relationship to another store.

type ResourceAccess

type ResourceAccess struct {
	Visibility Visibility          `json:"visibility"`
	OwnerID    string              `json:"owner_id,omitempty"`
	Roles      map[string][]string `json:"roles,omitempty"`
	Users      map[string][]string `json:"users,omitempty"`
}

type SchemaInferenceResult

type SchemaInferenceResult struct {
	// Flat schema without prefixes for LLM correlation with Relations
	Schema map[string]string
	// Fields that belong to the Key
	KeyFields []string
	// Fields that belong to the Value
	ValueFields []string
}

SchemaInferenceResult contains flat schema with field lists for LLM understanding.

func InferSchemaFromTypes

func InferSchemaFromTypes(key any, value any) SchemaInferenceResult

InferSchemaFromTypes uses reflection to directly inspect the types of key and value. Returns flat schema format without prefixes for better LLM understanding and correlation with Relations.

type SinglePhaseTransaction

type SinglePhaseTransaction struct {
	SopPhaseCommitTransaction TwoPhaseCommitTransaction
	// contains filtered or unexported fields
}

SinglePhaseTransaction wraps a TwoPhaseCommitTransaction providing an end-user friendly API and optional participation of other two-phase commit transactions.

func (*SinglePhaseTransaction) AddPhasedTransaction

func (t *SinglePhaseTransaction) AddPhasedTransaction(otherTransaction ...TwoPhaseCommitTransaction)

AddPhasedTransaction registers additional two-phase commit participants.

func (*SinglePhaseTransaction) Begin

func (t *SinglePhaseTransaction) Begin(ctx context.Context) error

Begin starts the wrapped transaction and any registered participants.

func (*SinglePhaseTransaction) Close

func (t *SinglePhaseTransaction) Close() error

Close calls Close on the wrapped transaction implementation.

func (*SinglePhaseTransaction) Commit

func (t *SinglePhaseTransaction) Commit(ctx context.Context) error

Commit executes phase 1 on all participants and then phase 2; on error, Rollback is invoked.

func (*SinglePhaseTransaction) CommitMaxDuration

func (t *SinglePhaseTransaction) CommitMaxDuration() time.Duration

CommitMaxDuration returns the configured commit duration cap from the underlying implementation.

func (*SinglePhaseTransaction) GetID

func (t *SinglePhaseTransaction) GetID() UUID

GetID returns the transaction ID.

func (*SinglePhaseTransaction) GetMode

func (t *SinglePhaseTransaction) GetMode() TransactionMode

GetMode returns the transaction mode.

func (*SinglePhaseTransaction) GetPhasedTransaction

func (t *SinglePhaseTransaction) GetPhasedTransaction() TwoPhaseCommitTransaction

GetPhasedTransaction returns the wrapped two-phase commit transaction.

func (*SinglePhaseTransaction) GetStores

func (t *SinglePhaseTransaction) GetStores(ctx context.Context) ([]string, error)

GetStores delegates to the wrapped transaction to list available stores.

func (*SinglePhaseTransaction) HasBegun

func (t *SinglePhaseTransaction) HasBegun() bool

HasBegun reports whether the transaction has started.

func (*SinglePhaseTransaction) OnCommit

func (t *SinglePhaseTransaction) OnCommit(callback func(ctx context.Context) error)

OnCommit registers a callback to be executed after a successful commit.

func (*SinglePhaseTransaction) Rollback

func (t *SinglePhaseTransaction) Rollback(ctx context.Context) error

Rollback aborts the transaction and attempts to rollback all participants, returning the last error if any.

type StoreCacheConfig

type StoreCacheConfig struct {
	// RegistryCacheDuration controls caching for registry objects.
	RegistryCacheDuration time.Duration `json:"registry_cache_duration"`
	// IsRegistryCacheTTL enables sliding TTL for registry cache.
	IsRegistryCacheTTL bool `json:"is_registry_cache_ttl"`
	// NodeCacheDuration controls caching for nodes.
	NodeCacheDuration time.Duration `json:"node_cache_duration"`
	// IsNodeCacheTTL enables sliding TTL for node cache.
	IsNodeCacheTTL bool `json:"is_node_cache_ttl"`
	// ValueDataCacheDuration controls caching for the item Value part when globally cached.
	ValueDataCacheDuration time.Duration `json:"value_data_cache_duration"`
	// IsValueDataCacheTTL enables sliding TTL for value data cache.
	IsValueDataCacheTTL bool `json:"is_value_data_cache_ttl"`
	// StoreInfoCacheDuration controls caching for StoreInfo records.
	StoreInfoCacheDuration time.Duration `json:"store_info_cache_duration"`
	// IsStoreInfoCacheTTL enables sliding TTL for store info cache.
	IsStoreInfoCacheTTL bool `json:"is_store_info_cache_ttl"`
}

StoreCacheConfig declares cache durations and TTL flags for store artifacts.

func GetDefaultCacheConfig

func GetDefaultCacheConfig() StoreCacheConfig

GetDefaultCacheConfig returns the global default cache configuration.

func NewStoreCacheConfig

func NewStoreCacheConfig(cacheDuration time.Duration, isCacheTTL bool) *StoreCacheConfig

NewStoreCacheConfig returns a StoreCacheConfig with uniform cache durations and TTL settings applied. If cacheDuration is between 1ns and 5 minutes, it will be clamped to 5 minutes. TTL is disabled when duration is zero.

type StoreInfo

type StoreInfo struct {
	// Name is the short store name.
	Name string `json:"name" minLength:"1" maxLength:"128"`
	// SlotLength is the number of items per node.
	SlotLength int `json:"slot_length" min:"2" max:"20000"`
	// IsUnique enforces uniqueness on the key of key/value items.
	IsUnique bool `json:"is_unique"`
	// Description optionally describes the store.
	Description string `json:"description" maxLength:"1000"`
	// RegistryTable is the registry table name.
	RegistryTable string `json:"registry_table" minLength:"1" maxLength:"140"`
	// BlobTable defines the target Erasure Coding (EC) configuration to use.
	// The Database Options contain an EC configuration map keyed by a name, and
	// this field's value is used to look up a match. If no matching entry is found,
	// it falls back to the default EC config (which uses an empty string key "").
	BlobTable string `json:"blob_table" minLength:"1" maxLength:"300"`
	// RootNodeID is the root B-Tree node identifier.
	RootNodeID UUID `json:"root_node_id"`
	// Count is the total number of items persisted.
	Count int64 `json:"count"`
	// CountDelta is used internally to reconcile Count updates; it should not be persisted.
	CountDelta int64 `json:"-"`
	// Timestamp is the add/update time in milliseconds.
	Timestamp int64 `json:"timestamp"`
	// IsValueDataInNodeSegment stores the Value within the node segment when true.
	IsValueDataInNodeSegment bool `json:"is_value_data_in_node_segment"`
	// IsValueDataActivelyPersisted persists Value separately on Add/Update when true.
	IsValueDataActivelyPersisted bool `json:"is_value_data_actively_persisted"`
	// IsValueDataGloballyCached enables Redis caching of Value data when true.
	IsValueDataGloballyCached bool `json:"is_value_data_globally_cached"`
	// LeafLoadBalancing enables distribution to sibling nodes when capacity allows.
	LeafLoadBalancing bool `json:"leaf_load_balancing"`
	// CacheConfig overrides global cache settings per store.
	CacheConfig StoreCacheConfig `json:"cache_config"`

	// MapKeyIndexSpecification contains a CEL or index specification used by the comparer.
	MapKeyIndexSpecification string `json:"mapkey_index_spec"`

	// CELexpression specifies the CEL expression used as comparer for keys.
	CELexpression string `json:"cel_expression,omitempty"`

	// IsPrimitiveKey hints the Python binding which JSON B-Tree to instantiate on open.
	// This is an internal feature and only needed to be managed by code when using (dynamic typed) languages like Python.
	IsPrimitiveKey bool `json:"is_primitive_key"`

	// Relations describes foreign key relationships to other stores.
	Relations []Relation `json:"relations,omitempty"`

	// Schema stores field types without prefixes for LLM correlation with Relations.
	// Format: {"key": "string", "first_name": "string", "age": "number"}
	Schema map[string]string `json:"schema,omitempty"`

	// KeyFields lists the field names that are part of the Key.
	// Example: ["key"] for primitive keys, ["id", "timestamp"] for composite keys
	KeyFields []string `json:"key_fields,omitempty"`

	// ValueFields lists the field names that are part of the Value.
	// Example: ["first_name", "age", "email"] for the value object
	ValueFields []string `json:"value_fields,omitempty"`

	// CustomData stores optional arbitrary configuration for the store.
	// It is intentionally flexible for integration-specific extensions.
	CustomData map[string]any `json:"custom_data,omitempty"`

	// For internal use only. Code can use this as hint.
	NeedsMetaDataSave bool `json:"-"`

	// Version allows versioning of the store info payload for future upgrades.
	Version string `json:"version,omitempty"`
}

StoreInfo describes a B-Tree store configuration and runtime state persisted in the backend.

func NewStoreInfo

func NewStoreInfo(si StoreOptions) *StoreInfo

NewStoreInfo creates and normalizes a StoreInfo based on StoreOptions, applying default naming and cache policy.

func (*StoreInfo) DeleteCustomData

func (si *StoreInfo) DeleteCustomData(key string) bool

DeleteCustomData removes a value for the given key from CustomData.

func (*StoreInfo) GetCustomData

func (si *StoreInfo) GetCustomData(key string) (any, bool)

GetCustomData returns the value for the given key from CustomData.

func (*StoreInfo) GetCustomDataMap

func (si *StoreInfo) GetCustomDataMap() map[string]any

GetCustomDataMap returns a copy of the CustomData map.

func (StoreInfo) IsCompatible

func (s StoreInfo) IsCompatible(b StoreInfo) bool

IsCompatible reports whether two StoreInfo configurations are compatible for merge/attach semantics.

func (StoreInfo) IsEmpty

func (s StoreInfo) IsEmpty() bool

IsEmpty reports whether the StoreInfo has zero values; an empty StoreInfo means the B-Tree does not yet exist.

func (*StoreInfo) SetCustomData

func (si *StoreInfo) SetCustomData(key string, value any)

SetCustomData stores a value under the given key in CustomData.

func (*StoreInfo) SetCustomDataMap

func (si *StoreInfo) SetCustomDataMap(data map[string]any)

SetCustomDataMap replaces the full CustomData map with a copy.

func (*StoreInfo) ValueDataSize

func (si *StoreInfo) ValueDataSize() ValueDataSize

Interpolates back into ValueDataSize

type StoreOptions

type StoreOptions struct {
	// Name is the short name of the store.
	Name string
	// SlotLength is the number of items that can be stored in a node.
	SlotLength int
	// IsUnique enforces uniqueness on keys.
	IsUnique bool
	// IsValueDataInNodeSegment stores Value data within the B-Tree node segment when true.
	// Smaller Value data benefits from this for locality; bigger data should be stored separately.
	// If true, IsValueDataActivelyPersisted and IsValueDataGloballyCached are ignored.
	IsValueDataInNodeSegment bool
	// IsValueDataActivelyPersisted persists Value data to a separate partition on Add/Update and expects
	// IsValueDataInNodeSegment to be false.
	IsValueDataActivelyPersisted bool
	// IsValueDataGloballyCached enables Redis caching for Value data when IsValueDataInNodeSegment is false.
	IsValueDataGloballyCached bool
	// LeafLoadBalancing allows distributing items to sibling nodes when there is capacity to avoid splits.
	LeafLoadBalancing bool
	// Description is an optional text describing the store.
	Description string
	// BlobStoreBaseFolderPath specifies a base folder path when using the filesystem blob store.
	BlobStoreBaseFolderPath string
	// DisableBlobStoreFormatting uses the store name directly as the blob store name (useful for S3-like systems).
	DisableBlobStoreFormatting bool
	// DisableRegistryStoreFormatting uses the store name directly as the registry store name.
	DisableRegistryStoreFormatting bool
	// CacheConfig overrides global cache durations and TTL behavior per store.
	CacheConfig *StoreCacheConfig

	// CELexpression specifies the CEL expression used as comparer for keys.
	CELexpression string
	// MapKeyIndexSpecification contains a CEL or index specification used by the comparer.
	MapKeyIndexSpecification string
	// IsPrimitiveKey hints Python bindings which JSON B-Tree type to instantiate during Open.
	IsPrimitiveKey bool
	// Relations describes foreign key-like relationships.
	Relations []Relation
	// CustomData stores optional arbitrary configuration for the store.
	CustomData map[string]any
}

StoreOptions contains configuration fields used when creating a B-Tree store.

func ConfigureStore

func ConfigureStore(storeName string, uniqueKey bool, slotLength int, description string, valueDataSize ValueDataSize, blobStoreBaseFolderPath string) StoreOptions

ConfigureStore returns StoreOptions tuned according to the expected ValueDataSize. Choose carefully: mismatched size can hurt performance by over/under caching or persisting. blobStoreBaseFolderPath is used only for filesystem blob storage as a base directory.

type StoreRepository

type StoreRepository interface {
	// Get retrieves store info by name(s).
	Get(context.Context, ...string) ([]StoreInfo, error)
	// GetWithTTL retrieves store info using TTL/sliding cache semantics.
	GetWithTTL(context.Context, bool, time.Duration, ...string) ([]StoreInfo, error)
	// GetAll lists all store names available in the backend.
	GetAll(context.Context) ([]string, error)
	// Add creates new store info entries and related tables (registry/blob).
	Add(context.Context, ...StoreInfo) error
	// Remove deletes store info by name and drops related tables.
	Remove(context.Context, ...string) error

	// Update modifies store info and reconciles Count using CountDelta.
	Update(context.Context, []StoreInfo) ([]StoreInfo, error)
	// Replicate performs post-commit replication of updated data managed by the repository.
	Replicate(context.Context, []StoreInfo) error
}

StoreRepository specifies CRUD and replication methods for StoreInfo records.

type TaskRunner

type TaskRunner struct {
	// contains filtered or unexported fields
}

TaskRunner is a thin wrapper around errgroup.Group that carries a context for convenience. Consider using errgroup directly in new code.

func NewTaskRunner

func NewTaskRunner(ctx context.Context, maxThreadCount int) *TaskRunner

NewTaskRunner creates a new TaskRunner. maxThreadCount > 0 limits the number of concurrent goroutines.

func (*TaskRunner) GetContext

func (tr *TaskRunner) GetContext() context.Context

GetContext returns the TaskRunner's context.

func (*TaskRunner) Go

func (tr *TaskRunner) Go(task func() error)

Go runs the provided task function in a new goroutine managed by the underlying errgroup.

func (*TaskRunner) Wait

func (tr *TaskRunner) Wait() error

Wait waits for all launched tasks to complete and returns the first encountered error, if any.

type Transaction

type Transaction interface {
	// Begin starts the transaction.
	Begin(ctx context.Context) error
	// Commit finalizes the transaction.
	Commit(ctx context.Context) error
	// Rollback aborts the transaction.
	Rollback(ctx context.Context) error
	// HasBegun reports whether the transaction has started.
	HasBegun() bool

	// GetPhasedTransaction returns the underlying two-phase commit transaction for orchestration with other systems.
	GetPhasedTransaction() TwoPhaseCommitTransaction
	// AddPhasedTransaction registers external two-phase commit participants.
	AddPhasedTransaction(otherTransaction ...TwoPhaseCommitTransaction)

	// GetStores lists all available B-Tree stores from the backend.
	GetStores(ctx context.Context) ([]string, error)

	// Close releases any resources associated with the transaction.
	Close() error

	// GetID returns the transaction ID.
	GetID() UUID

	// CommitMaxDuration returns the configured maximum duration for commit operations.
	// Effective runtime limit is min(ctx deadline, CommitMaxDuration()).
	CommitMaxDuration() time.Duration

	// OnCommit registers a callback to be executed after a successful commit.
	OnCommit(callback func(ctx context.Context) error)
}

Transaction defines end-user-facing transactional operations.

func NewTransaction

func NewTransaction(mode TransactionMode,
	twoPhaseCommitTrans TwoPhaseCommitTransaction) (Transaction, error)

NewTransaction constructs a Transaction wrapper around a TwoPhaseCommitTransaction. mode controls permissions. When logging is true, lower layers may record commit steps to aid recovery and cleanup of expired resources.

type TransactionLog

type TransactionLog interface {
	// PriorityLog returns the priority logger implementation.
	PriorityLog() TransactionPriorityLog
	// Add appends a transaction log entry.
	Add(ctx context.Context, tid UUID, commitFunction int, payload []byte) error
	// Remove deletes all logs for a transaction.
	Remove(ctx context.Context, tid UUID) error

	// GetOne returns the oldest hour bucket (older than 1 hour) and its logs for cleanup distribution.
	GetOne(ctx context.Context) (UUID, string, []KeyValuePair[int, []byte], error)

	// GetOneOfHour returns the available cleanup logs for a specific hour bucket.
	GetOneOfHour(ctx context.Context, hour string) (UUID, []KeyValuePair[int, []byte], error)

	// NewUUID generates a UUID suitable for the logging backend (e.g., time-based in Cassandra).
	NewUUID() UUID
}

TransactionLog persists transaction steps and provides job-distribution accessors for cleanup tasks.

type TransactionMode

type TransactionMode int

TransactionMode enumerates the supported transaction behaviors.

const (
	// NoCheck disallows any changes and skips read-version checks during commit.
	NoCheck TransactionMode = iota
	// ForWriting allows modifications to B-Tree stores within the transaction.
	ForWriting
	// ForReading disallows modifications; read-only.
	ForReading
)

type TransactionOptions

type TransactionOptions struct {
	// StoresFolders specifies the folders for replication.
	StoresFolders []string `json:"stores_folders,omitempty"`
	// ErasureConfig specifies the erasure coding configuration for replication.
	ErasureConfig map[string]ErasureCodingConfig `json:"erasure_config,omitempty"`
	// Keyspace to be used for the transaction (Cassandra).
	Keyspace string `json:"keyspace,omitempty"`
	// CacheType specifies the type of cache to use (e.g. InMemory, Redis).
	CacheType L2CacheType `json:"cache_type"`
	// RedisConfig specifies the Redis configuration when CacheType is Redis.
	RedisConfig *RedisCacheConfig `json:"redis_config,omitempty"`
	// Registry hash modulo value used for hashing.
	RegistryHashModValue int `json:"registry_hash_mod,omitempty"`

	// Transaction Mode can be Read-only or Read-Write.
	Mode TransactionMode `json:"mode"`
	// Transaction maximum "commit" time. Acts as the commit window cap and lock TTL.
	MaxTime time.Duration `json:"max_time"`
}

func (TransactionOptions) GetDatabaseOptions

func (to TransactionOptions) GetDatabaseOptions() DatabaseOptions

GetDatabaseOptions returns the DatabaseOptions subset from TransactionOptions.

func (TransactionOptions) IsCassandraHybrid

func (to TransactionOptions) IsCassandraHybrid() bool

func (TransactionOptions) IsReplicated

func (to TransactionOptions) IsReplicated() bool

type TransactionPriorityLog

type TransactionPriorityLog interface {
	// IsEnabled reports whether priority logging is enabled.
	IsEnabled() bool
	// Add appends a priority log for a transaction.
	Add(ctx context.Context, tid UUID, payload []byte) error
	// Remove deletes priority log file of a transaction.
	Remove(ctx context.Context, tid UUID) error
	// Get retrieves priority log details for a transaction.
	Get(ctx context.Context, tid UUID) ([]RegistryPayload[Handle], error)

	// GetBatch fetches up to batchSize of the oldest (older than 2 minutes) priority logs for processing.
	GetBatch(ctx context.Context, batchSize int) ([]KeyValuePair[UUID, []RegistryPayload[Handle]], error)

	// ProcessNewer iterates over all priority logs newer than 5 mins and invokes the processor callback for each.
	ProcessNewer(ctx context.Context, processor func(tid UUID, payload []RegistryPayload[Handle]) error) error

	// LogCommitChanges writes a special commit-change log used during drive reinstate for replication.
	LogCommitChanges(ctx context.Context, stores []StoreInfo, newRootNodesHandles, addedNodesHandles, updatedNodesHandles, removedNodesHandles []RegistryPayload[Handle]) error
}

TransactionPriorityLog records prioritised transaction logs used for recovery and replication workflows.

type Tuple

type Tuple[T1 any, T2 any] struct {
	// First is the first element of the pair.
	First T1
	// Second is the second element of the pair.
	Second T2
}

Tuple represents an ordered pair of two generic values when Key/Value semantics are not desired.

type TwoPhaseCommitTransaction

type TwoPhaseCommitTransaction interface {
	// Begin starts the transaction.
	Begin(ctx context.Context) error
	// Phase1Commit performs the first phase (prepare) of the commit.
	Phase1Commit(ctx context.Context) error
	// Phase2Commit performs the second phase (finalize) of the commit.
	Phase2Commit(ctx context.Context) error
	// Rollback aborts the transaction and may be provided an error cause.
	Rollback(ctx context.Context, err error) error
	// HasBegun reports whether the transaction has started.
	HasBegun() bool
	// GetMode returns the configured TransactionMode.
	GetMode() TransactionMode

	// GetStores lists all available B-Tree stores from the backend.
	GetStores(ctx context.Context) ([]string, error)

	// Close releases any resources associated with the transaction implementation.
	Close() error

	// GetID returns the transaction ID.
	GetID() UUID

	// CommitMaxDuration returns the configured maximum duration for commit operations.
	// Effective runtime limit is min(ctx deadline, CommitMaxDuration()).
	CommitMaxDuration() time.Duration

	// OnCommit registers a callback to be executed after a successful commit.
	OnCommit(callback func(ctx context.Context) error)
}

TwoPhaseCommitTransaction defines infrastructure-facing two-phase commit operations.

type UICapability

type UICapability string

UICapability represents the UI-friendly permission key (e.g., "can_edit", "can_delete")

const (
	UICapabilityRead     UICapability = "can_read"
	UICapabilityEdit     UICapability = "can_edit"
	UICapabilityDelete   UICapability = "can_delete"
	UICapabilityAISelect UICapability = "can_ai_select"
)

func ActionToUICapability

func ActionToUICapability(action Action) UICapability

ActionToUICapability maps an internal Action to the UI consumable Capability string

type UUID

type UUID uuid.UUID

UUID is a thin wrapper over github.com/google/uuid.UUID to keep SOP decoupled from the external package.

var NilUUID UUID

NilUUID is the zero-value UUID.

func NewUUID

func NewUUID() UUID

NewUUID returns a new randomly generated UUID. It retries on error with a 1ms backoff up to 10 times and panics only if all attempts fail (which should never happen under normal conditions).

func ParseUUID

func ParseUUID(id string) (UUID, error)

ParseUUID converts a string to a UUID. It returns an error if the input is not a valid UUID.

func (UUID) Compare

func (x UUID) Compare(y UUID) int

Compare compares two UUIDs and returns -1 if x < y, 1 if x > y, and 0 if they are equal.

func (UUID) IsNil

func (id UUID) IsNil() bool

IsNil reports whether the UUID equals the zero-value UUID.

func (UUID) MarshalText

func (id UUID) MarshalText() ([]byte, error)

MarshalText implements the encoding.TextMarshaler interface.

func (UUID) Split

func (id UUID) Split() (uint64, uint64)

Split returns the high and low 64-bit parts of the UUID.

func (UUID) String

func (id UUID) String() string

String returns the canonical string representation of the UUID.

func (*UUID) UnmarshalText

func (id *UUID) UnmarshalText(b []byte) error

UnmarshalText implements the encoding.TextUnmarshaler interface.

type ValueDataSize

type ValueDataSize int

ValueDataSize categorizes the expected size of Value data to guide configuration helpers.

const (
	// SmallData indicates small Value data that can be stored within the node segment.
	SmallData ValueDataSize = iota
	// MediumData indicates medium Value data that should be stored in a separate segment.
	MediumData
	// BigData indicates large Value data stored separately, actively persisted and typically not globally cached.
	BigData
)

type Visibility

type Visibility string

const (
	VisibilityPublic  Visibility = "public"
	VisibilityPrivate Visibility = "private"
	VisibilitySystem  Visibility = "system"
)