README
¶
Netsy
Netsy is a replicated key-value database which stores data in object storage. It implements a subset of the etcd API (Range, Txn, Watch) and can be used as a drop-in etcd replacement for Kubernetes or as a general-purpose KV store for applications that need durable, low-latency persistence without the operational complexity of running a traditional database.
Unlike etcd which uses the Raft consensus algorithm, Netsy's multi-node replication is inspired by PostgreSQL synchronous streaming replication, and by modern architectures of systems like Loki/Mimir and OpenObserve which use object storage for data persistence.
Netsy is an Open Source project, created by Nadrama.
Goals
Netsy was created to reduce operational complexity by providing durable, replicated key-value storage with the cost and reliability advantages of object storage, without the latency trade-offs.
-
Object storage MUST be the permanent data store.
-
Data MUST be durably stored — either synchronously in object storage, or via quorum-based replication — before being acknowledged
-
Netsy MUST maintain compatibility with the subset of the etcd API used by Kubernetes.
-
It is a non-goal to fully support the entire etcd API.
Project Status
Netsy supports multi-node clusters with quorum-based replication, automatic leader election via s3lect, and graceful failover.
Current Features:
- KV ranges, KV transactions, Watches - supporting all options used by Kubernetes
- Snapshots - full copies of records from leader SQLite database stored in object storage
- Chunking - delta of records from last snapshot stored in object storage chunk files
- Backfill from object storage - starting a fresh server restores from snapshots+chunks
- Compaction - cluster-wide protocol to remove values from historical revisions
Roadmap:
- Conformance Tests - VCR-style acceptance test suite (in development)
- Encryption - supporting value field encryption, with rolling key rotation
- Leases - used by Kubernetes.
Documentation
Check out the comprehensive documentation in ./docs or read it rendered on the official Netsy website at netsy.dev
Development
See docs/development.md for development environment setup and usage.
License
Netsy is licensed under the Apache License, Version 2.0. Copyright 2026 Nadrama Pty Ltd.
See the LICENSE file for details.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
dev-s3
command
Command dev-s3 runs a local fake S3 server for development.
|
Command dev-s3 runs a local fake S3 server for development. |
|
netsy
command
|
|
|
read-netsy-file
command
|
|
|
internal
|
|
|
bootstrap
Package bootstrap implements node startup loading and backfill.
|
Package bootstrap implements node startup loading and backfill. |
|
buildvars
Package buildvars exposes build info (version, commit hash, build date) injected via build-time linker flags.
|
Package buildvars exposes build info (version, commit hash, build date) injected via build-time linker flags. |
|
clientapi
Package clientapi implements the etcd-compatible gRPC Client API, including KV (Range, Txn), Watch, Lease, Cluster, and Maintenance services.
|
Package clientapi implements the etcd-compatible gRPC Client API, including KV (Range, Txn), Watch, Lease, Cluster, and Maintenance services. |
|
cmd
Package cmd wires up the main netsy command and server startup.
|
Package cmd wires up the main netsy command and server startup. |
|
commonapi
Package commonapi provides shared request handling logic used by both the Client API and Primary domain layer, such as Range queries.
|
Package commonapi provides shared request handling logic used by both the Client API and Primary domain layer, such as Range queries. |
|
config
Package config loads, parses, and validates per-node settings (from environment variables) and per-cluster settings (from a JSONC config file).
|
Package config loads, parses, and validates per-node settings (from environment variables) and per-cluster settings (from a JSONC config file). |
|
datafile
Package datafile reads and writes the .netsy data file format
|
Package datafile reads and writes the .netsy data file format |
|
datastore
Package datastore manages chunk and snapshot files in object storage, including upload, download, listing, and revision parsing.
|
Package datastore manages chunk and snapshot files in object storage, including upload, download, listing, and revision parsing. |
|
discovery
Package discovery manages node registration files and the members file in object storage.
|
Package discovery manages node registration files and the members file in object storage. |
|
elector
Package elector integrates s3lect leader election into the Netsy node lifecycle.
|
Package elector integrates s3lect leader election into the Netsy node lifecycle. |
|
healthserver
Package healthserver provides an unencrypted HTTP /health endpoint driven by the Node's HealthState.
|
Package healthserver provides an unencrypted HTTP /health endpoint driven by the Node's HealthState. |
|
heartbeat
Package heartbeat manages the outbound heartbeat sender goroutine.
|
Package heartbeat manages the outbound heartbeat sender goroutine. |
|
localdb
Package localdb implements SQLite database operations including schema management, inserts, finds, transactions, replication, and compaction.
|
Package localdb implements SQLite database operations including schema management, inserts, finds, transactions, replication, and compaction. |
|
metrics
Package metrics provides centralized Prometheus metric registration, role-gated collection, and always-on node state metrics for Netsy.
|
Package metrics provides centralized Prometheus metric registration, role-gated collection, and always-on node state metrics for Netsy. |
|
mtls
Package mtls provides mutual TLS configuration, certificate validation, and identity extraction for client and peer gRPC connections.
|
Package mtls provides mutual TLS configuration, certificate validation, and identity extraction for client and peer gRPC connections. |
|
node
Package node implements the Node gRPC service hosted by every Node.
|
Package node implements the Node gRPC service hosted by every Node. |
|
nodestate
Package nodestate tracks per-node runtime state: the Health, Elector, and Primary state triple, the current Cluster State, and the Committed Revision and Compaction Revision used to gate client-visible range requests and watches.
|
Package nodestate tracks per-node runtime state: the Health, Elector, and Primary state triple, the current Cluster State, and the Committed Revision and Compaction Revision used to gate client-visible range requests and watches. |
|
peerclient
Package peerclient manages outbound gRPC client connections to peer Nodes.
|
Package peerclient manages outbound gRPC client connections to peer Nodes. |
|
primary
Package primary implements the Primary gRPC service and domain layer, including transaction processing, the Follow replication stream, chunk buffering for async object storage writes, Replica heartbeat and receipt tracking, and heartbeat-based degradation.
|
Package primary implements the Primary gRPC service and domain layer, including transaction processing, the Follow replication stream, chunk buffering for async object storage writes, Replica heartbeat and receipt tracking, and heartbeat-based degradation. |
|
replication
Package replication implements the Replica-side Follow stream client.
|
Package replication implements the Replica-side Follow stream client. |
|
snapshot
Package snapshot implements the snapshot creation worker that periodically writes complete Netsy snapshot files to object storage based on record count, size, and age thresholds.
|
Package snapshot implements the snapshot creation worker that periodically writes complete Netsy snapshot files to object storage based on record count, size, and age thresholds. |
|
storage
Package storage provides an abstraction layer to object storage.
|
Package storage provides an abstraction layer to object storage. |
|
watch
Package watch manages watch subscriptions and distribution.
|
Package watch manages watch subscriptions and distribution. |
|
pkg
|
|
|
datafile
Package datafile provides a public API for external tools that need to read and write Netsy .netsy data files.
|
Package datafile provides a public API for external tools that need to read and write Netsy .netsy data files. |
Click to show internal directories.
Click to hide internal directories.