keystone

README

Keystone — Lightweight Edge Orchestration in Go

📖 Documentation: https://carlosprados.github.io/keystone

Keystone is a minimal, robust, and secure edge orchestration agent written in Go. It manages local components (native processes and containers), executes deployments atomically with rollback, and keeps devices converging to a desired state — even with flaky networks.

Why Keystone? Because edge fleets need something that is lightweight, predictable, and operable without dragging a full container stack everywhere. Keystone embraces "processes first, containers when needed," with a clean, pluggable design inspired by Greengrass. When you need containers, Keystone supports containerd natively or falls back to Docker/nerdctl/podman CLI.

Highlights

• Lightweight: idle CPU ~0%, ~23 MB RAM baseline, ~22 MB binary
• Solid: atomic deployments, checkpoints, rollback, exponential backoff
• Secure by default: loopback API with bearer-token auth, mandatory recipe & artifact signatures (RSA/ECDSA/Ed25519), fail-closed verification — see docs/security.md
• Portable: Linux x86/ARM, single binary, no mandatory Docker/CRI
• Frugal on the link: downloads resume, and an artifact can opt into being patched instead of re-downloaded — 13.4 MB becomes 1.0 MB between two adjacent releases, with a full-download fallback whenever patching is not possible
• Fresh data, no restarts: datasets (an OUI list, a vulnerability feed) refresh on their own schedule from a signed manifest, activate atomically and roll back if the component cannot live with them — see datasets
• Connected: HTTP REST, NATS (+ JetStream), MQTT adapters
• Operable: structured logs, Prometheus metrics, health endpoints, persistence

Keystone vs. AWS Greengrass

Feature	AWS Greengrass	Keystone
Runtime	Java (JVM) / C (Lite)	Go (Native)
RAM Baseline	~100MB+	~23 MB
Complexity	High (Cloud-first)	Low (Lean & Simple)
Setup	Heavy Bootstrap	Single Binary
Control Plane	AWS IoT Core only	HTTP, NATS, MQTT
Offline Mode	Limited	Full (JetStream jobs)

Documentation

📖 Read the documentation → carlosprados.github.io/keystone

Concepts, internals, the security model, worked examples for HTTP/MQTT/NATS, and a generated OpenAPI reference — written to be read start to finish. Source in site/.

Document	What it covers
docs/security.md	Security model: threat model, secure-by-default controls, auth, signing, the --insecure-skip-verify escape hatch, config reference
docs/component-state.md	Component states, liveness guarantees of GET /v1/components, and when a re-apply reuses a running component
docs/adapters.md	Control-plane adapters: HTTP (auth), NATS (+ JetStream), MQTT — comparison and configuration
docs/containers.md	Container recipe syntax and examples
docs/containerrunner-design.md	containerd integration design notes
configs/trust/README.md	CA setup and signing recipes/artifacts
KeyStone.md	Original architecture proposal and delivery plan

Quick Look: How it Works

Keystone uses simple TOML recipes to define components and deployment plans to keep your devices in sync.

1. Define a Recipe

Describe how to install and run your process in com.example.hello.recipe.toml:

[metadata]
name = "com.example.hello"
version = "1.0.0"

[[artifacts]]
uri = "https://example.com/artifacts/hello.tar.gz"
sha256 = "..."
unpack = true

[lifecycle.run.exec]
command = "./hello"
args = ["--interval", "30s"]

1b. Or Define a Container Recipe

Run containerized workloads when needed (uses containerd or docker/nerdctl/podman):

[metadata]
name = "com.example.nginx"
version = "1.0.0"

[lifecycle.run]
type = "container"
restart_policy = "always"

[lifecycle.run.container]
image = "docker.io/library/nginx:alpine"
pull_policy = "if-not-present"
network_mode = "bridge"

[[lifecycle.run.container.mounts]]
source = "/data/nginx/html"
target = "/usr/share/nginx/html"
read_only = true

[[lifecycle.run.container.ports]]
host_port = 8080
container_port = 80

[lifecycle.run.container.resources]
memory_mb = 256
cpu_shares = 512

[lifecycle.run.health]
check = "http://localhost:8080/"
interval = "10s"

For complete container documentation, see docs/containers.md.

2. Apply a Plan

List the components you want to run in plan.toml:

[[components]]
name = "hello"
recipe = "com.example.hello.recipe.toml"

Deploy it with a single command:

./keystonectl apply plan.toml

3. Check Status

See everything running at a glance:

./keystonectl status

Project Status

Released and in production evaluation. Latest release: v0.4.0 (August 2026) — see releases. Each release publishes a tar.gz per architecture (Linux amd64, arm64, armv7) containing all three binaries: keystone, keystonectl and keystoneserver.

What is done: supervisor with DAG deployments and rollback, process and container runners, artifact manager with resume and optional delta downloads, signature verification that fails closed, privilege dropping, and three control-plane adapters (HTTP, NATS, MQTT). Documentation is published at carlosprados.github.io/keystone and the OpenAPI document is generated from the route table, so it cannot drift from the code.

Known limitations, stated plainly:

• Released binaries are not signed. No cosign signature and no SBOM yet. Verify the published checksums and fetch over HTTPS.
• No built-in TLS for the HTTP API. Terminate at a reverse proxy, use a VPN, or tunnel with keystonectl --ssh. NATS and MQTT do support TLS natively.
• cgroups are a no-op placeholder. ProcessRunner applies RLIMIT_NOFILE only; container resource limits do work.
• The NATS and MQTT adapters still accept planPath, which the HTTP API deliberately rejects. Treat those transports as trusted.
• Self-update and canary rings are not implemented (Phase 7 below).

Install

Every release publishes one archive per architecture, containing all three binaries:

tar xzf keystone_0.4.0_linux_arm64.tar.gz
sudo install -m 0755 keystone keystonectl keystoneserver /usr/local/bin/
keystone --version

Binary	What it is for
keystone	The agent. This is what runs on the device
keystonectl	CLI client, talks to an agent over its HTTP API
keystoneserver	Tiny static file server, for serving test artifacts while developing

Binaries are not signed yet — check the published checksums and fetch over HTTPS. From source: task build.

Quick Start

go run ./cmd/keystone --http 127.0.0.1:8080

For development with live reload (requires air):

task dev

Probe the health endpoint:

curl -s localhost:8080/healthz | jq

You should see a JSON response with status and uptime.

Run the built-in mock demo stack (db -> cache -> api):

task demo            # or: go run ./cmd/keystone --demo --insecure-skip-verify

The mock demo uses synthetic components (no recipes or artifacts). To see the real, secure flow — a signed recipe verified before it runs — use the signing helper:

# Generates a dev CA and signs the example recipe, then tells you how to run it.
task demo-signed

See docs/security.md for the full signing model.

Scrape metrics (Prometheus format):

curl -s localhost:8080/metrics | head

Roadmap (MVP → v1.0)

• Phase 0: Agent skeleton, config base, /healthz, persistent state snapshotting
• Phase 1: Supervisor + ProcessRunner, lifecycle hooks, health checks (HTTP/TCP/Shell)
• Phase 2: DAG-based deployments, layer-wise rollback, Prometheus metrics
• Phase 3: Security hardening — mTLS adapters, artifact signatures (ECDSA/RSA)
• Phase 4: Control plane adapters — HTTP REST, NATS (+ JetStream), MQTT
• Phase 5: Robustness — download resume, exponential backoff, graceful shutdown
• Phase 6: ContainerRunner — containerd client, CLI fallback (docker/nerdctl/podman)
• Phase 6.5: Delta (patch) artifact downloads, privilege dropping, published documentation site
• Phase 7: Self-update and canary rings
• Phase 8: Signed releases (cosign + SBOM)

See KeyStone.md for the architecture proposal and delivery plan.

Implemented Features

Category	Features
Supervisor	DAG execution, parallel layer startup, FSM lifecycle, dependency ordering
ProcessRunner	Process management, log streaming, health probes (HTTP/TCP/cmd), restart policies, exponential backoff
ContainerRunner	containerd client, CLI fallback (docker/nerdctl/podman), image pull, mounts, ports, resource limits
Deployment Engine	TOML plans and recipes, environment variable substitution, dry-run mode
Artifact Manager	Secure download with resume, SHA-256 verification, detached signatures, GC, cache limits, optional delta (patch) updates
Security	Trust bundles (PEM), ECDSA/RSA signature verification, mTLS support
Observability	Prometheus metrics, structured logging, health endpoints, per-process metrics
Persistence	Automatic state snapshotting, recovery on restart, atomic writes
Control Plane	HTTP REST API, NATS adapter (+ JetStream jobs), MQTT adapter (QoS, LWT)
CLI	keystonectl: full parity with the HTTP API, self-documenting help with examples, SSH tunnelling, shell completion
Robustness	Download resume (HTTP Range), exponential backoff with jitter, context propagation, graceful shutdown

Apply a Deployment Plan (End-to-End Preview)

Use a minimal TOML plan to run real processes via the ProcessRunner.

Example plan (see configs/examples/plan.toml):

[[components]]
name = "keystone-server"
recipe = "configs/examples/com.keystone.server.recipe.toml"

Apply it:

# 1. Start the agent (now remote-first).
#    --insecure-skip-verify is needed here because the example recipe is
#    unsigned; drop it once your recipes/artifacts are signed (see below).
task build
./keystone --http 127.0.0.1:8080 --insecure-skip-verify

# 2. Apply the plan remotely using the CLI
./keystonectl apply configs/examples/plan.toml

Notes:

• The example recipe uses the built-in keystoneserver binary and declares no downloadable artifacts.
• Recipes loaded from a file must be signed by default (see "Signed Recipes and Artifacts"). Recipes pushed through the authenticated API (keystonectl upload-recipe) are trusted by that authentication and are not re-verified. For local experiments with unsigned files, run the agent with --insecure-skip-verify (or KEYSTONE_INSECURE_SKIP_VERIFY=true).
• ProcessRunner applies basic RLIMIT_NOFILE; cgroups integration is a safe no-op placeholder for now.

Quick Usage

• Start agent:
  • ./keystone --http 127.0.0.1:8080
• Apply plan:
  • ./keystonectl apply configs/examples/plan.toml
• Health and discovery:
  • curl -s localhost:8080/healthz | jq
  • curl -s localhost:8080/v1/components | jq
  • curl -s localhost:8080/v1/plan/status | jq
• Stop all components:
  • curl -X POST localhost:8080/v1/plan/stop -i
• Stop or restart a single component:
  • curl -X POST localhost:8080/v1/components/hello:stop -i
  • curl -X POST localhost:8080/v1/components/hello:restart -i
• Metrics (Prometheus):
  • curl -s localhost:8080/metrics | head

Signed Recipes and Artifacts

Verification is mandatory by default and fails closed; --insecure-skip-verify (or KEYSTONE_INSECURE_SKIP_VERIFY=true) relaxes all of it for dev/demo only.

Recipes loaded from a filesystem path must carry a detached signature before any lifecycle hook (install/run/shutdown) runs:

• Place a sibling <recipe>.sig next to the recipe, and a cert as <recipe>.crt or via KEYSTONE_LEAF_CERT, chaining to KEYSTONE_TRUST_BUNDLE.
• Sign with OpenSSL: openssl dgst -sha256 -sign signer.key -out com.example.app.recipe.toml.sig com.example.app.recipe.toml.
• Recipes uploaded through the authenticated HTTP API are trusted by that authentication and are not re-verified.

Artifacts declared in a recipe each require a sha256, a sig_uri, and a configured trust bundle, on both apply and restart:

• Provide a trust bundle (PEM) via KEYSTONE_TRUST_BUNDLE and a leaf certificate via KEYSTONE_LEAF_CERT, or include cert_uri in the recipe.
• Add sig_uri to each artifact entry in the recipe.
• Signature format: detached signature over SHA-256 of the artifact, produced with OpenSSL (openssl dgst -sha256 -sign ...).
• See configs/trust/README.md for a quick, dev-friendly CA and signing walkthrough.

Artifact Download Headers (TOML)

• Configure per-artifact HTTP headers directly in the recipe under [[artifacts]].headers.
• For GitHub artifacts (github.com or api.github.com), set github_token (at the same level as uri) to inject Authorization: Bearer <token> when no Authorization header is provided. There is no environment fallback for this: the token is per-artifact, in the recipe.

Example snippet inside a recipe:

[[artifacts]]
uri = "https://api.github.com/repos/org/repo/actions/artifacts/123/zip"
sha256 = "sha256:<...>"
unpack = true
github_token = ""
[artifacts.headers]
Accept = "application/vnd.github+json"   # for the Actions /artifacts/{id}/zip endpoint (302 to S3)

keystonectl (CLI)

keystonectl covers the whole HTTP API and documents itself — every command carries a description, runnable examples and the endpoint it calls, so exploring with help is enough to learn it:

keystonectl help                 # every command, grouped
keystonectl help restart         # one command, with examples
keystonectl restart --help       # the same
source <(keystonectl completion bash)

The commands:

keystonectl health                            # agent liveness and uptime
keystonectl status                            # plan status
keystonectl components                        # state, PID and health of each component
keystonectl graph                             # dependency graph and start order

keystonectl apply plan.toml [--dry]           # upload and apply a plan
keystonectl apply-dry plan.toml               # same as apply --dry
keystonectl stop-plan                         # stop everything

keystonectl stop hello                        # stop one component
keystonectl restart hello [--wait health] [--timeout 120s]
keystonectl restart-dry hello                 # what a restart would touch

keystonectl recipes                           # list the recipe store
keystonectl upload-recipe api.toml [--force]  # add a recipe
keystonectl delete-recipe com.acme.api 1.4.0  # remove one
keystonectl sha256 dist/api.tar.gz            # digest for an artifact entry

Global flags, each with an environment fallback (the flag always wins): --addr / KEYSTONE_ADDR, --token / KEYSTONE_API_TOKEN, --ssh / KEYSTONE_SSH. Exit codes: 0 success, 1 the agent or network failed, 2 the command was used wrongly.

Reaching an agent bound to loopback

The agent listens on 127.0.0.1 by default and demands a token to bind anything else, so on a device its API is only reachable from the device itself. Rather than exposing the port, tunnel the request:

keystonectl --ssh ops@edge-001 components
keystonectl --ssh ops@edge-001:52022 --addr http://127.0.0.1:9180 restart-dry api

# For a device you work on often
export KEYSTONE_SSH=ops@edge-001:52022
export KEYSTONE_ADDR=http://127.0.0.1:9180
keystonectl status

--addr is resolved on the far side, so the default 127.0.0.1:8080 means the agent's own loopback. Under the hood this runs your ssh client with -W, inheriting your ~/.ssh/config — Host aliases, IdentityFile, ProxyJump through a bastion, the agent and known_hosts — with host key verification done by ssh under your own rules. It needs the ssh binary on your machine and AllowTcpForwarding on the device (the default), and is Unix-only; on Windows use ssh -L and point --addr at the forwarded port.

keystoneserver

A simple HTTP server to serve local artifacts for testing:

go run ./cmd/keystoneserver --root ./artifacts --addr :9000

• Accessible at http://localhost:9000/<path>
• Includes a /healthz endpoint.

API Testing with Bruno

We provide a Bruno collection for testing the agent API.

1. Install Bruno.
2. Open the app and select Open Collection.
3. Select the bruno/ folder in this repository.
4. Use the local environment to set the base_url.

Graph and dry-run from API directly:

curl -s localhost:8080/v1/plan/graph | jq
curl -s -X POST 'localhost:8080/v1/components/hello:restart?dry=true' | jq

# The plan is uploaded as content. A JSON body naming a planPath is rejected on
# purpose: it would turn the API into a file-read primitive.
curl -s -X POST 'localhost:8080/v1/plan/apply?dry=true' \
  --data-binary @configs/examples/plan.toml

Releases

Keystone uses GoReleaser for automated builds and releases. Cutting one is two commands with a review in between:

task release:prepare RELEASE=v0.3.1   # bumps the documented version, opens the PR
# review and merge that PR
task release:tag RELEASE=v0.3.1       # tags origin/main and pushes the tag

prepare bumps params.version in site/hugo.toml on a branch, checks the built site really shows it in the footer, and opens the pull request. Merging that PR is what publishes the docs for the new version — the tag cannot, because GitHub Pages discards a second deployment of a commit it has already published (the header of .github/workflows/pages.yml has the details). tag then refuses to tag a main whose documented version disagrees with the tag name, so the two cannot drift.

tag waits for that documentation deployment to finish before tagging, and stops if it failed: releasing binaries beside a site that still names the previous version is not fixable afterwards, because the repository rules forbid moving or deleting a v* tag. A deployment that is merely slow or has not started yet is not treated as a failure — it warns and proceeds. Override with SKIP_DOCS_WAIT=1, or adjust DOCS_APPEAR_TIMEOUT (180 s) and DOCS_FINISH_TIMEOUT (600 s).

Pushing the tag triggers GoReleaser, which builds the binaries for multiple architectures (amd64, arm64, armv7) and creates a GitHub Release with the artifacts. release.yml re-checks the version match server-side, so cutting a tag by hand cannot skip the step either. Pre-release tags (v0.4.0-rc1) are exempt: the site stays on the last stable version, so tag those directly.

Configuration

API Security

The HTTP API can apply plans and run lifecycle hooks, so it is treated as a privileged surface:

• It binds to 127.0.0.1:8080 by default. Binding a non-loopback address (e.g. --http 0.0.0.0:8080) requires a token via --api-token or KEYSTONE_API_TOKEN; otherwise the agent refuses to start.
• When a token is set, every endpoint except /healthz requires Authorization: Bearer <token>. keystonectl sends it automatically from --token or KEYSTONE_API_TOKEN.
• The API accepts plans as uploaded content only; the legacy planPath field (loading an arbitrary server-side file) is rejected.

Environment Variables

Keystone supports loading environment variables from a .env file in the current working directory.

Variable	Description
KEYSTONE_API_TOKEN	Bearer token required for the HTTP API. Mandatory to bind a non-loopback address.
KEYSTONE_INSECURE_SKIP_VERIFY	true disables mandatory artifact integrity (sha256 + signature). Dev/demo only.
KEYSTONE_MAX_REQUEST_BYTES	Max HTTP request body size in bytes (default: 4 MiB).
KEYSTONE_MAX_EXTRACT_BYTES	Max total uncompressed size per archive extraction (default: 2 GiB).
KEYSTONE_ARTIFACT_CACHE_LIMIT_BYTES	Max size of runtime/artifacts (default: 2GiB).
KEYSTONE_DELTA_MAX_BASE_BYTES	Largest artifact the delta path will attempt (default: 256 MiB; 0 = no limit).
KEYSTONE_ARTIFACT_DOWNLOAD_TIMEOUT	Artifact download timeout (default: 30m). Supports "5m", "1h", etc.
KEYSTONE_CLOCK_POLICY	high-water (default) or strict; how certificate validity is judged when the clock cannot be trusted.
KEYSTONE_RECONCILE_INTERVAL	Re-apply the plan in effect on this interval (default: unset, disabled).
KEYSTONE_RECONCILE_JITTER	Fleet spread for reconcile passes (default: 10% of the interval).
KEYSTONE_TRUST_BUNDLE	Path to CA trust bundle (PEM) for signature verification.
KEYSTONE_LEAF_CERT	Default certificate (PEM) for signature verification if not in recipe.
KEYSTONE_IMAGE_VOLUME_DIR	Where container image volumes are materialised.
KEYSTONE_DEVICE_ID	Device ID for NATS/MQTT topics (default: hostname).
KEYSTONE_MQTT_BROKER	MQTT broker URL (enables MQTT if set and --mqtt-broker not passed).
KEYSTONE_MQTT_DEVICE_ID	MQTT device ID (overrides KEYSTONE_DEVICE_ID for MQTT only).
KEYSTONE_MQTT_CLIENT_ID	MQTT client ID.
KEYSTONE_MQTT_TLS_CERT	Path to MQTT client TLS certificate.
KEYSTONE_MQTT_TLS_KEY	Path to MQTT client TLS private key.
KEYSTONE_MQTT_TLS_CA	Path to MQTT CA certificate bundle.
KEYSTONE_MQTT_TLS_VERIFY	Verify MQTT broker certificate (true/false).
KEYSTONE_MQTT_USER	MQTT username.
KEYSTONE_MQTT_PASS	MQTT password.
KEYSTONE_MQTT_QOS	MQTT QoS for command/response (0, 1, 2).
KEYSTONE_MQTT_STATE_INTERVAL	MQTT state event interval (e.g. 10s, 0 to disable).
KEYSTONE_MQTT_HEALTH_INTERVAL	MQTT health event interval (e.g. 30s, 0 to disable).
KEYSTONE_INSTALL_TIMEOUT	Install phase timeout (default: 2m). Supports duration strings.
KEYSTONE_CONTAINERD_SOCKET	containerd socket path (default: /run/containerd/containerd.sock).
KEYSTONE_CONTAINERD_NAMESPACE	containerd namespace for containers (default: keystone).
KEYSTONE_CONTAINER_SNAPSHOTTER	Snapshotter for container images (default: overlayfs).
KEYSTONE_CONTAINER_REGISTRY	Default container registry (default: docker.io).
KEYSTONE_CNI_CONF_DIR	CNI config directory for bridge mode (default: /etc/cni/net.d).
KEYSTONE_CNI_PLUGIN_DIRS	CNI plugin dirs (default: /opt/cni/bin:/usr/lib/cni).
KEYSTONE_CNI_NETNS_DIR	Netns dir for bridge mode (default: /var/run/netns).

Robust Artifact Downloads

Keystone features a robust artifact download system designed for unreliable edge networks:

Feature	Description
Resume Support	Automatic resume via HTTP Range headers if download is interrupted
Exponential Backoff	Retries with jitter to avoid thundering herd (1s-30s, 10 attempts)
Progress Tracking	Real-time download progress with speed and ETA
Timeout Control	Configurable connect, read, and overall timeouts
Atomic Operations	Downloads to .partial file, renamed on completion
SHA-256 Verification	Post-download integrity check
Error Classification	Distinguishes fatal (4xx) from retryable (5xx, network) errors
Rate Limit Handling	Respects Retry-After headers for 429 responses

Control Plane Adapters

Keystone uses a pluggable adapter architecture for control plane communication. Multiple adapters can run simultaneously.

Adapter	Protocol	Use Case	Default
HTTP	REST API	Local management, debugging, Prometheus	Enabled (:8080)
NATS	Pub/Sub	Cloud-scale fleet management	Disabled
MQTT	IoT messaging	AWS IoT Core, edge gateways	Disabled

For complete adapter documentation, see docs/adapters.md.

HTTP Adapter (Default)

The HTTP adapter exposes a REST API for local management:

# Default: enabled on port 8080
./keystone --http 127.0.0.1:8080

# Disable HTTP (use only messaging adapters)
./keystone --http "" --nats-url nats://server:4222 --nats-device-id edge-001

Endpoints (the generated OpenAPI document is in the docs site):

Endpoint	Description
GET /healthz	Health check (exempt from auth)
GET /metrics	Prometheus metrics
GET /v1/components	List components
POST /v1/components/{name}:stop	Stop one component
POST /v1/components/{name}:restart	Restart one component (dry, wait, timeout)
GET /v1/plan/status	Applied plan, status and components
GET /v1/plan/graph	Dependency graph and start order
POST /v1/plan/apply	Apply a plan (body is the plan TOML; dry)
POST /v1/plan/stop	Stop every component
GET/POST /v1/recipes	List or add stored recipes (force)
DELETE /v1/recipes/{name}/{version}	Remove a stored recipe

NATS Adapter

Enable NATS for asynchronous fleet management with optional JetStream persistence:

# Basic NATS
./keystone --http 127.0.0.1:8080 \
  --nats-url nats://control-plane:4222 \
  --nats-device-id edge-001

# With mTLS and JetStream
./keystone --http 127.0.0.1:8080 \
  --nats-url nats://control-plane:4222 \
  --nats-device-id edge-001 \
  --nats-tls-cert /etc/keystone/certs/client.crt \
  --nats-tls-key /etc/keystone/certs/client.key \
  --nats-tls-ca /etc/keystone/certs/ca.crt \
  --nats-jetstream

Key Features:

• mTLS, NKey, Token, User/Pass authentication
• JetStream for durable job queues (survives disconnections)
• Subjects: keystone.{deviceId}.cmd.*, keystone.{deviceId}.events.*

Authentication Priority: NKey > Credentials > Token > User/Pass

MQTT Adapter

Enable MQTT for IoT-friendly communication with brokers like Mosquitto, EMQX, or AWS IoT Core:

# Basic MQTT
./keystone --http 127.0.0.1:8080 \
  --mqtt-broker tcp://broker:1883 \
  --mqtt-device-id edge-001

# With TLS and auth
./keystone --http 127.0.0.1:8080 \
  --mqtt-broker ssl://broker:8883 \
  --mqtt-device-id edge-001 \
  --mqtt-tls-ca /etc/keystone/certs/ca.crt \
  --mqtt-user agent --mqtt-pass secret

Key Features:

• mTLS, User/Pass authentication
• Configurable QoS (0, 1, 2)
• Last Will and Testament for online/offline detection
• Topics: keystone/{deviceId}/cmd/*, keystone/{deviceId}/resp/*, keystone/{deviceId}/events/*

Running Multiple Adapters

# HTTP + NATS + MQTT simultaneously
./keystone --http 127.0.0.1:8080 \
  --nats-url nats://nats.internal:4222 --nats-device-id edge-001 \
  --mqtt-broker tcp://mqtt.internal:1883 --mqtt-device-id edge-001

All CLI Flags

Core & HTTP Adapter Flags

Flag	Default	Description
--http	127.0.0.1:8080	HTTP listen address (empty to disable)
--api-token	(empty)	Bearer token for the API (or KEYSTONE_API_TOKEN); required for a non-loopback bind
--insecure-skip-verify	false	Disable mandatory artifact/recipe verification — dev/demo only (or KEYSTONE_INSECURE_SKIP_VERIFY=true)
--clock-policy	high-water	What to do when the system clock is behind known-good time: high-water verifies against the later of the two, strict refuses to verify
--reconcile-interval	0 (off)	Re-apply the plan in effect on this interval, restarting components that died and ran out of retries
--reconcile-jitter	10% of the interval	Spread reconcile passes across a fleet; the offset is derived from the device ID, not random
--demo	false	Run the built-in mock 3-component stack (db → cache → api)
--version	—	Print version and commit, then exit

NATS Adapter Flags

Flag	Default	Description
--nats-url	(empty)	NATS server URL (empty to disable)
--nats-device-id	hostname	Device ID for subjects
--nats-tls-cert	(empty)	Client TLS certificate path
--nats-tls-key	(empty)	Client TLS key path
--nats-tls-ca	(empty)	CA certificate path
--nats-tls-verify	true	Verify server certificate
--nats-creds	(empty)	Credentials file path (.creds)
--nats-nkey	(empty)	NKey seed file path
--nats-token	(empty)	Authentication token
--nats-user	(empty)	Username
--nats-pass	(empty)	Password
--nats-state-interval	10s	State event interval (0 to disable)
--nats-health-interval	30s	Health event interval (0 to disable)
--nats-jetstream	false	Enable JetStream
--nats-js-stream	KEYSTONE_JOBS	JetStream stream name
--nats-js-workers	1	Job processor workers

MQTT Adapter Flags

Flag	Default	Description
--mqtt-broker	(empty)	MQTT broker URL (empty to disable)
--mqtt-device-id	hostname	Device ID for topics
--mqtt-client-id	keystone-{device-id}	MQTT client ID
--mqtt-tls-cert	(empty)	Client TLS certificate path
--mqtt-tls-key	(empty)	Client TLS key path
--mqtt-tls-ca	(empty)	CA certificate path
--mqtt-tls-verify	true	Verify server certificate
--mqtt-user	(empty)	Username
--mqtt-pass	(empty)	Password
--mqtt-qos	1	QoS level (0, 1, 2)
--mqtt-state-interval	10s	State event interval (0 to disable)
--mqtt-health-interval	30s	Health event interval (0 to disable)

Environment variable equivalents are also supported (flags take precedence): KEYSTONE_MQTT_BROKER, KEYSTONE_MQTT_DEVICE_ID, KEYSTONE_MQTT_CLIENT_ID, KEYSTONE_MQTT_TLS_CERT, KEYSTONE_MQTT_TLS_KEY, KEYSTONE_MQTT_TLS_CA, KEYSTONE_MQTT_TLS_VERIFY, KEYSTONE_MQTT_USER, KEYSTONE_MQTT_PASS, KEYSTONE_MQTT_QOS, KEYSTONE_MQTT_STATE_INTERVAL, KEYSTONE_MQTT_HEALTH_INTERVAL.

Git hooks

Run once after cloning to enable the repo’s versioned hooks:

task hooks   # or: ./scripts/setup-git-hooks.sh

This sets core.hooksPath to .githooks, where the pre-commit hook runs go fmt ./... and stages formatting changes automatically.

Core Concepts

Concept	Description
Recipe	TOML file describing a component: artifacts, lifecycle hooks, health checks, resources
Deployment Plan	TOML file listing components to run, resolved as a DAG with dependencies
Supervisor	Enforces lifecycle (install → start → running → stop) and restart policies
Runner	Executes components: ProcessRunner (native) or ContainerRunner (containerd/CLI)
Artifact Manager	Downloads, verifies (SHA-256 + signatures), caches, and garbage collects artifacts
Adapter	Pluggable control plane interface (HTTP, NATS, MQTT) for remote management

Systemd Unit (example)

See configs/systemd/keystone.service for a hardened example unit file.

Community

We welcome contributors who care about reliability at the edge.

• Issues: bug reports, design discussions, small enhancements
• PRs: focused, well-tested changes; prefer conventional commits
• Security: please report privately first when appropriate

License

📃 Apache-2.0