keystone

README

Keystone — Lightweight Edge Orchestration in Go

📖 Documentation: https://carlosprados.github.io/keystone

Keystone is a minimal, robust, and secure edge orchestration agent written in Go. It manages local components (native processes and containers), executes deployments atomically with rollback, and keeps devices converging to a desired state — even with flaky networks.

Why Keystone? Because edge fleets need something that is lightweight, predictable, and operable without dragging a full container stack everywhere. Keystone embraces "processes first, containers when needed," with a clean, pluggable design inspired by Greengrass. When you need containers, Keystone supports containerd natively or falls back to Docker/nerdctl/podman CLI.

Highlights

• Lightweight: idle CPU ~0%, ~23 MB RAM baseline, ~22 MB binary
• Solid: atomic deployments, checkpoints, rollback, exponential backoff
• Secure by default: loopback API with bearer-token auth, mandatory recipe & artifact signatures (RSA/ECDSA/Ed25519), fail-closed verification — see docs/security.md
• Portable: Linux x86/ARM, single binary, no mandatory Docker/CRI
• Frugal on the link: downloads resume, and an artifact can opt into being patched instead of re-downloaded — 13.4 MB becomes 1.0 MB between two adjacent releases, with a full-download fallback whenever patching is not possible
• Fresh data, no restarts: datasets (an OUI list, a vulnerability feed) refresh on their own schedule from a signed manifest, activate atomically and roll back if the component cannot live with them — see datasets
• Connected: HTTP REST and MQTT adapters; the agent keeps supervising when no control plane is reachable
• Operable: structured logs, Prometheus metrics, health endpoints, persistence
• Verifiable from the outside in: every release is signed with cosign (keyless, recorded in Sigstore's transparency log) and ships an SPDX SBOM per archive — see Verifying a release
• Refuses what it cannot honour: a declaration that a runtime cannot implement is rejected rather than dropped — and a rollback that would drag a component back across a state migration is refused instead of silently corrupting what it reverts to

Keystone vs. AWS Greengrass

Feature	AWS Greengrass	Keystone
Runtime	Java (JVM) / C (Lite)	Go (Native)
RAM Baseline	~100MB+	~23 MB
Complexity	High (Cloud-first)	Low (Lean & Simple)
Setup	Heavy Bootstrap	Single Binary
Control Plane	AWS IoT Core only	HTTP, MQTT
Offline Mode	Limited	Keeps supervising

Documentation

📖 Read the documentation → carlosprados.github.io/keystone

Concepts, internals, the security model, worked examples for HTTP and MQTT, and a generated OpenAPI reference — written to be read start to finish. Source in site/.

Document	What it covers
docs/security.md	Security model: threat model, secure-by-default controls, auth, signing, the --insecure-skip-verify escape hatch, config reference
docs/component-state.md	Component states, liveness guarantees of GET /v1/components, and when a re-apply reuses a running component
docs/adapters.md	Control-plane adapters: HTTP (auth) and MQTT — comparison and configuration
docs/containers.md	Container recipe syntax and examples
docs/containerrunner-design.md	containerd integration design notes
configs/trust/README.md	CA setup and signing recipes/artifacts
KeyStone.md	Original architecture proposal and delivery plan

Quick Look: How it Works

Keystone uses simple TOML recipes to define components and deployment plans to keep your devices in sync.

1. Define a Recipe

Describe how to install and run your process in com.example.hello.recipe.toml:

[metadata]
name = "com.example.hello"
version = "1.0.0"

[[artifacts]]
uri = "https://example.com/artifacts/hello.tar.gz"
sha256 = "..."
unpack = true

[lifecycle.run.exec]
command = "./hello"
args = ["--interval", "30s"]

1b. Or Define a Container Recipe

Run containerized workloads when needed (uses containerd or docker/nerdctl/podman):

[metadata]
name = "com.example.nginx"
version = "1.0.0"

[lifecycle.run]
type = "container"
restart_policy = "always"

[lifecycle.run.container]
image = "docker.io/library/nginx:alpine"
pull_policy = "if-not-present"
network_mode = "bridge"

[[lifecycle.run.container.mounts]]
source = "/data/nginx/html"
target = "/usr/share/nginx/html"
read_only = true

[[lifecycle.run.container.ports]]
host_port = 8080
container_port = 80

[lifecycle.run.container.resources]
memory_mb = 256
cpu_shares = 512

[lifecycle.run.health]
check = "http://localhost:8080/"
interval = "10s"

For complete container documentation, see docs/containers.md.

2. Apply a Plan

List the components you want to run in plan.toml:

[[components]]
name = "hello"
recipe = "com.example.hello.recipe.toml"

Deploy it with a single command:

./keystonectl apply plan.toml

3. Check Status

See everything running at a glance:

./keystonectl status

Project Status

Released and in production evaluation. See releases for the current version — this file deliberately does not name one, because a version written here goes stale the moment a tag is cut. Each release publishes a tar.gz per architecture (Linux amd64, arm64, armv7) containing all three binaries: keystone, keystonectl and keystoneserver, plus a signature and an SBOM (below).

What is done: supervisor with DAG deployments and rollback, process and container runners, artifact manager with resume and optional delta downloads, signature verification that fails closed, privilege dropping, two control-plane adapters (HTTP, MQTT), and a signed release pipeline. Documentation is published at carlosprados.github.io/keystone and the OpenAPI document is generated from the route table, so it cannot drift from the code.

Known limitations, stated plainly:

• Self-update is new. An MQTT cmd/self-update downloads and verifies a new binary, installs it A/B beside the running one, restarts, and a systemd pre-start gate reverts it if it does not confirm (requires --self-update-root and the keystone-ab.service unit). On hardware (a Raspberry Pi, arm64) a full update has been measured end to end in both directions: an update that cannot reach its control plane is reverted after three starts, one that can confirms in seconds, and supervised processes keep their PID throughout (from v0.12.9 on). See MQTT.
• No built-in TLS for the HTTP API. Terminate at a reverse proxy, use a VPN, or tunnel with keystonectl --ssh. MQTT does support TLS natively.
• No memory or CPU limits for process components. ProcessRunner applies RLIMIT_NOFILE only, and a recipe declaring memory_limit or cpu_quota under [resources] is refused rather than run unbounded. Containers get every limit in [lifecycle.run.container.resources], under containerd and the CLI runtimes alike.
• Component re-adoption needs the system manager. Survivors of an agent crash are recognised by being reparented to PID 1; under a subreaper (systemd --user, tini) they are neither adopted nor reaped.
• Canary rings are not implemented (Phase 7b below).

Install

Every release publishes one archive per architecture, containing all three binaries:

tar xzf keystone_<version>_linux_arm64.tar.gz
sudo install -m 0755 keystone keystonectl keystoneserver /usr/local/bin/
keystone --version

Binary	What it is for
keystone	The agent. This is what runs on the device
keystonectl	CLI client, talks to an agent over its HTTP API
keystoneserver	Tiny static file server, for serving test artifacts while developing

Verify before installing: every release from v0.9.0 carries a cosign signature over checksums.txt and an SBOM per archive — see Verifying what you downloaded. From source: task build.

Quick Start

go run ./cmd/keystone --http 127.0.0.1:8080

For development with live reload (requires air):

task dev

Probe the health endpoint:

curl -s localhost:8080/healthz | jq

You should see a JSON response with status and uptime.

Run the built-in mock demo stack (db -> cache -> api):

task demo            # or: go run ./cmd/keystone --demo --insecure-skip-verify

The mock demo uses synthetic components (no recipes or artifacts). To see the real, secure flow — a signed recipe verified before it runs — use the signing helper:

# Generates a dev CA and signs the example recipe, then tells you how to run it.
task demo-signed

See docs/security.md for the full signing model.

Scrape metrics (Prometheus format):

curl -s localhost:8080/metrics | head

Roadmap (MVP → v1.0)

• Phase 0: Agent skeleton, config base, /healthz, persistent state snapshotting
• Phase 1: Supervisor + ProcessRunner, lifecycle hooks, health checks (HTTP/TCP/Shell)
• Phase 2: DAG-based deployments, layer-wise rollback, Prometheus metrics
• Phase 3: Security hardening — mTLS adapters, artifact signatures (ECDSA/RSA)
• Phase 4: Control plane adapters — HTTP REST, MQTT (a NATS adapter existed until v0.13.0)
• Phase 5: Robustness — download resume, exponential backoff, graceful shutdown
• Phase 6: ContainerRunner — containerd client, CLI fallback (docker/nerdctl/podman)
• Phase 6.5: Delta (patch) artifact downloads, privilege dropping, published documentation site
• Phase 6.6: Container service discovery (network aliases), shell-free health probes, container misconfiguration guardrails
• Phase 8: Signed releases (cosign keyless + SPDX SBOM)
• Phase 7a: Self-update (A/B install, systemd gate, rollback, component re-adoption)
• Phase 7b: Canary rings

See KeyStone.md for the architecture proposal and delivery plan.

Implemented Features

Category	Features
Supervisor	DAG execution, parallel layer startup, FSM lifecycle, dependency ordering
ProcessRunner	Process management, logs as journald streams that outlive the agent, health probes (HTTP/TCP/cmd/exec), restart policies, exponential backoff, privilege dropping
ContainerRunner	containerd client, CLI fallback (docker/nerdctl/podman), image pull, mounts, ports, resource limits, network aliases for service discovery, network pre-flight, moving-tag and containerd-namespace warnings
Deployment Engine	TOML plans and recipes, environment variable substitution, dry-run mode, rollback refused across a declared state migration
Artifact Manager	Secure download with resume, SHA-256 verification, detached signatures, GC, cache limits, optional delta (patch) updates
Security	Trust bundles (PEM), Ed25519/ECDSA/RSA signature verification, mTLS support, signed releases (cosign keyless) with an SPDX SBOM per archive
Observability	Prometheus metrics, structured logging, health endpoints, per-process metrics
Persistence	Automatic state snapshotting, recovery on restart, atomic writes
Control Plane	HTTP REST API, MQTT adapter (QoS, LWT, tenant topics), content-only plans on every transport, retained-command refusal and command deduplication, agent version reported over both
CLI	keystonectl: full parity with the HTTP API, self-documenting help with examples, SSH tunnelling, shell completion
Robustness	Download resume (HTTP Range), exponential backoff with jitter, context propagation, graceful shutdown

Apply a Deployment Plan (End-to-End Preview)

Use a minimal TOML plan to run real processes via the ProcessRunner.

Example plan (see configs/examples/plan.toml):

[[components]]
name = "keystone-server"
recipe = "configs/examples/com.keystone.server.recipe.toml"

Apply it:

# 1. Start the agent (now remote-first).
#    --insecure-skip-verify is needed here because the example recipe is
#    unsigned; drop it once your recipes/artifacts are signed (see below).
task build
./keystone --http 127.0.0.1:8080 --insecure-skip-verify

# 2. Apply the plan remotely using the CLI
./keystonectl apply configs/examples/plan.toml

Notes:

• The example recipe uses the built-in keystoneserver binary and declares no downloadable artifacts.
• Recipes loaded from a file must be signed by default (see "Signed Recipes and Artifacts"). Recipes pushed through the authenticated API (keystonectl upload-recipe) are trusted by that authentication and are not re-verified. For local experiments with unsigned files, run the agent with --insecure-skip-verify (or KEYSTONE_INSECURE_SKIP_VERIFY=true).
• ProcessRunner applies basic RLIMIT_NOFILE; cgroups integration is a safe no-op placeholder for now.

Quick Usage

• Start agent:
  • ./keystone --http 127.0.0.1:8080
• Apply plan:
  • ./keystonectl apply configs/examples/plan.toml
• Health and discovery:
  • curl -s localhost:8080/healthz | jq
  • curl -s localhost:8080/v1/components | jq
  • curl -s localhost:8080/v1/plan/status | jq
• Stop all components:
  • curl -X POST localhost:8080/v1/plan/stop -i
• Stop or restart a single component:
  • curl -X POST localhost:8080/v1/components/hello:stop -i
  • curl -X POST localhost:8080/v1/components/hello:restart -i
• Metrics (Prometheus):
  • curl -s localhost:8080/metrics | head

Signed Recipes and Artifacts

Verification is mandatory by default and fails closed; --insecure-skip-verify (or KEYSTONE_INSECURE_SKIP_VERIFY=true) relaxes all of it for dev/demo only.

Recipes loaded from a filesystem path must carry a detached signature before any lifecycle hook (install/run/shutdown) runs:

• Place a sibling <recipe>.sig next to the recipe, and a cert as <recipe>.crt or via KEYSTONE_LEAF_CERT, chaining to KEYSTONE_TRUST_BUNDLE.
• Sign with OpenSSL: openssl dgst -sha256 -sign signer.key -out com.example.app.recipe.toml.sig com.example.app.recipe.toml.
• Recipes uploaded through the authenticated HTTP API are trusted by that authentication and are not re-verified.

Artifacts declared in a recipe each require a sha256, a sig_uri, and a configured trust bundle, on both apply and restart:

• Provide a trust bundle (PEM) via KEYSTONE_TRUST_BUNDLE and a leaf certificate via KEYSTONE_LEAF_CERT, or include cert_uri in the recipe.
• Add sig_uri to each artifact entry in the recipe.
• Signature format: detached signature over SHA-256 of the artifact, produced with OpenSSL (openssl dgst -sha256 -sign ...).
• See configs/trust/README.md for a quick, dev-friendly CA and signing walkthrough.

Artifact Download Headers (TOML)

• Configure per-artifact HTTP headers directly in the recipe under [[artifacts]].headers.
• For GitHub artifacts (github.com or api.github.com), set github_token (at the same level as uri) to inject Authorization: Bearer <token> when no Authorization header is provided. There is no environment fallback for this: the token is per-artifact, in the recipe.

Example snippet inside a recipe:

[[artifacts]]
uri = "https://api.github.com/repos/org/repo/actions/artifacts/123/zip"
sha256 = "sha256:<...>"
unpack = true
github_token = ""
[artifacts.headers]
Accept = "application/vnd.github+json"   # for the Actions /artifacts/{id}/zip endpoint (302 to S3)

keystonectl (CLI)

keystonectl covers the whole HTTP API and documents itself — every command carries a description, runnable examples and the endpoint it calls, so exploring with help is enough to learn it:

keystonectl help                 # every command, grouped
keystonectl help restart         # one command, with examples
keystonectl restart --help       # the same
source <(keystonectl completion bash)

The commands:

keystonectl health                            # agent liveness and uptime
keystonectl status                            # plan status
keystonectl components                        # state, PID and health of each component
keystonectl graph                             # dependency graph and start order

keystonectl apply plan.toml [--dry]           # upload and apply a plan
keystonectl apply-dry plan.toml               # same as apply --dry
keystonectl stop-plan                         # stop everything

keystonectl stop hello                        # stop one component
keystonectl restart hello [--wait health] [--timeout 120s]
keystonectl restart-dry hello                 # what a restart would touch

keystonectl recipes                           # list the recipe store
keystonectl upload-recipe api.toml [--force]  # add a recipe
keystonectl delete-recipe com.acme.api 1.4.0  # remove one
keystonectl sha256 dist/api.tar.gz            # digest for an artifact entry

Global flags, each with an environment fallback (the flag always wins): --addr / KEYSTONE_ADDR, --token / KEYSTONE_API_TOKEN, --ssh / KEYSTONE_SSH. Exit codes: 0 success, 1 the agent or network failed, 2 the command was used wrongly.

Reaching an agent bound to loopback

The agent listens on 127.0.0.1 by default and demands a token to bind anything else, so on a device its API is only reachable from the device itself. Rather than exposing the port, tunnel the request:

keystonectl --ssh ops@edge-001 components
keystonectl --ssh ops@edge-001:52022 --addr http://127.0.0.1:9180 restart-dry api

# For a device you work on often
export KEYSTONE_SSH=ops@edge-001:52022
export KEYSTONE_ADDR=http://127.0.0.1:9180
keystonectl status

--addr is resolved on the far side, so the default 127.0.0.1:8080 means the agent's own loopback. Under the hood this runs your ssh client with -W, inheriting your ~/.ssh/config — Host aliases, IdentityFile, ProxyJump through a bastion, the agent and known_hosts — with host key verification done by ssh under your own rules. It needs the ssh binary on your machine and AllowTcpForwarding on the device (the default), and is Unix-only; on Windows use ssh -L and point --addr at the forwarded port.

keystoneserver

A simple HTTP server to serve local artifacts for testing:

go run ./cmd/keystoneserver --root ./artifacts --addr :9000

• Accessible at http://localhost:9000/<path>
• Includes a /healthz endpoint.

API Testing with Bruno

We provide a Bruno collection for testing the agent API.

1. Install Bruno.
2. Open the app and select Open Collection.
3. Select the bruno/ folder in this repository.
4. Use the local environment to set the base_url.

Graph and dry-run from API directly:

curl -s localhost:8080/v1/plan/graph | jq
curl -s -X POST 'localhost:8080/v1/components/hello:restart?dry=true' | jq

# The plan is uploaded as content. A JSON body naming a planPath is rejected on
# purpose: it would turn the API into a file-read primitive.
curl -s -X POST 'localhost:8080/v1/plan/apply?dry=true' \
  --data-binary @configs/examples/plan.toml

Releases

Keystone uses GoReleaser for automated builds and releases. Cutting one is two commands with a review in between:

task release:prepare RELEASE=v0.3.1   # bumps the documented version, opens the PR
# review and merge that PR
task release:tag RELEASE=v0.3.1       # tags origin/main and pushes the tag

prepare bumps params.version in site/hugo.toml on a branch, checks the built site really shows it in the footer, and opens the pull request. Merging that PR is what publishes the docs for the new version — the tag cannot, because GitHub Pages discards a second deployment of a commit it has already published (the header of .github/workflows/pages.yml has the details). tag then refuses to tag a main whose documented version disagrees with the tag name, so the two cannot drift.

tag waits for that documentation deployment to finish before tagging, and stops if it failed: releasing binaries beside a site that still names the previous version is not fixable afterwards, because the repository rules forbid moving or deleting a v* tag. A deployment that is merely slow or has not started yet is not treated as a failure — it warns and proceeds. Override with SKIP_DOCS_WAIT=1, or adjust DOCS_APPEAR_TIMEOUT (180 s) and DOCS_FINISH_TIMEOUT (600 s).

Pushing the tag triggers GoReleaser, which builds the binaries for multiple architectures (amd64, arm64, armv7) and creates a GitHub Release with the artifacts. release.yml re-checks the version match server-side, so cutting a tag by hand cannot skip the step either. Pre-release tags (v0.4.0-rc1) are exempt: the site stays on the last stable version, so tag those directly, and they are published as pre-releases so nothing asking for the latest version is handed one.

Verifying what you downloaded

Every release from v0.9.0 carries a cosign signature over checksums.txt and an SPDX 2.3 SBOM per archive. The checksum proves integrity; the signature proves the bytes came from this repository's release workflow, which is the property that matters for an agent whose job is installing software on other machines.

cosign verify-blob checksums.txt \
  --bundle checksums.txt.bundle \
  --certificate-identity-regexp '^https://github\.com/carlosprados/keystone/\.github/workflows/release\.yml@refs/tags/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

sha256sum --check --ignore-missing checksums.txt

Do not drop --certificate-identity-regexp: without it the verification accepts a signature from any workflow in any repository, an attacker's fork included. Signing is keyless, so there is no key to hold, and the signature is recorded in Sigstore's transparency log. Full details, including what this does not prove, are in Verifying a release.

Configuration

API Security

The HTTP API can apply plans and run lifecycle hooks, so it is treated as a privileged surface:

• It binds to 127.0.0.1:8080 by default. Binding a non-loopback address (e.g. --http 0.0.0.0:8080) requires a token via --api-token or KEYSTONE_API_TOKEN; otherwise the agent refuses to start.
• When a token is set, every endpoint except /healthz requires Authorization: Bearer <token>. keystonectl sends it automatically from --token or KEYSTONE_API_TOKEN.
• Plans are accepted as uploaded content only, on every transport. The legacy planPath field — which named a file on the device for the agent to read and execute — is rejected by HTTP and MQTT alike.
• Commands that change state are refused if they arrive retained, and are deduplicated by commandId where one is supplied: a retained command is redelivered on every reconnect, and QoS 1 is at-least-once by design.

Environment Variables

Keystone supports loading environment variables from a .env file in the current working directory.

Variable	Description
KEYSTONE_API_TOKEN	Bearer token required for the HTTP API. Mandatory to bind a non-loopback address.
KEYSTONE_INSECURE_SKIP_VERIFY	true disables mandatory artifact integrity (sha256 + signature). Dev/demo only.
KEYSTONE_MAX_REQUEST_BYTES	Max HTTP request body size in bytes (default: 4 MiB).
KEYSTONE_MAX_EXTRACT_BYTES	Max total uncompressed size per archive extraction (default: 2 GiB).
KEYSTONE_ARTIFACT_CACHE_LIMIT_BYTES	Max size of runtime/artifacts (default: 2GiB).
KEYSTONE_DELTA_MAX_BASE_BYTES	Largest artifact the delta path will attempt (default: 256 MiB; 0 = no limit).
KEYSTONE_ARTIFACT_DOWNLOAD_TIMEOUT	Artifact download timeout (default: 30m). Supports "5m", "1h", etc.
KEYSTONE_CLOCK_POLICY	high-water (default) or strict; how certificate validity is judged when the clock cannot be trusted.
KEYSTONE_RECONCILE_INTERVAL	Re-apply the plan in effect on this interval (default: unset, disabled).
KEYSTONE_RECONCILE_JITTER	Fleet spread for reconcile passes (default: 10% of the interval).
KEYSTONE_TRUST_BUNDLE	Path to CA trust bundle (PEM) for signature verification.
KEYSTONE_LEAF_CERT	Default certificate (PEM) for signature verification if not in recipe.
KEYSTONE_IMAGE_VOLUME_DIR	Where container image volumes are materialised.
KEYSTONE_DEVICE_ID	Device ID for MQTT topics (default: hostname).
KEYSTONE_MQTT_BROKER	MQTT broker URL (enables MQTT if set and --mqtt-broker not passed).
KEYSTONE_MQTT_DEVICE_ID	MQTT device ID (overrides KEYSTONE_DEVICE_ID for MQTT only).
KEYSTONE_MQTT_TENANT	MQTT tenant, a lowercase DNS label. Required with MQTT unless KEYSTONE_ENROL_DIR provides it.
KEYSTONE_MQTT_CLIENT_ID	MQTT client ID.
KEYSTONE_MQTT_TLS_CERT	Path to MQTT client TLS certificate.
KEYSTONE_MQTT_TLS_KEY	Path to MQTT client TLS private key.
KEYSTONE_MQTT_TLS_CA	Path to MQTT CA certificate bundle.
KEYSTONE_MQTT_TLS_VERIFY	Verify MQTT broker certificate (true/false).
KEYSTONE_ENROL_DIR	Identity written by keystone enrol: MQTT client certificate, key, broker CA, tenant and device ID, renewed automatically.
KEYSTONE_MQTT_USER	MQTT username.
KEYSTONE_MQTT_PASS	MQTT password.
KEYSTONE_MQTT_QOS	MQTT QoS for command/response (0, 1, 2).
KEYSTONE_MQTT_COMMAND_DEDUPE_TTL	How long a commandId is remembered (default 10m).
KEYSTONE_MQTT_STATE_INTERVAL	MQTT state event interval (e.g. 10s, 0 to disable).
KEYSTONE_MQTT_HEALTH_INTERVAL	MQTT health event interval (e.g. 30s, 0 to disable).
KEYSTONE_INSTALL_TIMEOUT	Install phase timeout (default: 2m). Supports duration strings.
KEYSTONE_CONTAINERD_SOCKET	containerd socket path (default: /run/containerd/containerd.sock).
KEYSTONE_CONTAINERD_NAMESPACE	containerd namespace for containers (default: keystone).
KEYSTONE_CONTAINER_SNAPSHOTTER	Snapshotter for container images (default: overlayfs).
KEYSTONE_CONTAINER_REGISTRY	Default container registry (default: docker.io).
KEYSTONE_CNI_CONF_DIR	CNI config directory for bridge mode (default: /etc/cni/net.d).
KEYSTONE_CNI_PLUGIN_DIRS	CNI plugin dirs (default: /opt/cni/bin:/usr/lib/cni).
KEYSTONE_CNI_NETNS_DIR	Netns dir for bridge mode (default: /var/run/netns).

Robust Artifact Downloads

Keystone features a robust artifact download system designed for unreliable edge networks:

Feature	Description
Resume Support	Automatic resume via HTTP Range headers if download is interrupted
Exponential Backoff	Retries with jitter to avoid thundering herd (1s-30s, 10 attempts)
Progress Tracking	Real-time download progress with speed and ETA
Timeout Control	Configurable connect, read, and overall timeouts
Atomic Operations	Downloads to .partial file, renamed on completion
SHA-256 Verification	Post-download integrity check
Error Classification	Distinguishes fatal (4xx) from retryable (5xx, network) errors
Rate Limit Handling	Respects Retry-After headers for 429 responses

Control Plane Adapters

Keystone uses a pluggable adapter architecture for control plane communication. Multiple adapters can run simultaneously.

Adapter	Protocol	Use Case	Default
HTTP	REST API	Local management, debugging, Prometheus	Enabled (:8080)
MQTT	IoT messaging	AWS IoT Core, edge gateways	Disabled

For complete adapter documentation, see docs/adapters.md.

HTTP Adapter (Default)

The HTTP adapter exposes a REST API for local management:

# Default: enabled on port 8080
./keystone --http 127.0.0.1:8080

# Disable HTTP (use only messaging adapters)
./keystone --http "" --mqtt-broker tls://broker:8883 --mqtt-tenant acme --mqtt-device-id edge-001

Endpoints (the generated OpenAPI document is in the docs site):

Endpoint	Description
GET /healthz	Health check (exempt from auth)
GET /metrics	Prometheus metrics
GET /v1/components	List components
POST /v1/components/{name}:stop	Stop one component
POST /v1/components/{name}:restart	Restart one component (dry, wait, timeout)
GET /v1/plan/status	Applied plan, status and components
GET /v1/plan/graph	Dependency graph and start order
POST /v1/plan/apply	Apply a plan (body is the plan TOML; dry)
POST /v1/plan/stop	Stop every component
GET/POST /v1/recipes	List or add stored recipes (force)
DELETE /v1/recipes/{name}/{version}	Remove a stored recipe

MQTT Adapter

Enable MQTT for IoT-friendly communication with brokers like Mosquitto, EMQX, or AWS IoT Core:

# Basic MQTT
./keystone --http 127.0.0.1:8080 \
  --mqtt-broker tcp://broker:1883 \
  --mqtt-device-id edge-001

# With TLS and auth
./keystone --http 127.0.0.1:8080 \
  --mqtt-broker ssl://broker:8883 \
  --mqtt-device-id edge-001 \
  --mqtt-tls-ca /etc/keystone/certs/ca.crt \
  --mqtt-user agent --mqtt-pass secret

Key Features:

• mTLS, User/Pass authentication
• Configurable QoS (0, 1, 2)
• Last Will and Testament for online/offline detection
• Topics: keystone/{tenant}/{deviceId}/cmd/*, …/resp/*, …/events/*, …/status; the tenant is required

Running Multiple Adapters

# HTTP + MQTT simultaneously
./keystone --http 127.0.0.1:8080 \
  --mqtt-broker tcp://mqtt.internal:1883 --mqtt-tenant acme --mqtt-device-id edge-001

All CLI Flags

Core & HTTP Adapter Flags

Flag	Default	Description
--http	127.0.0.1:8080	HTTP listen address (empty to disable)
--api-token	(empty)	Bearer token for the API (or KEYSTONE_API_TOKEN); required for a non-loopback bind
--insecure-skip-verify	false	Disable mandatory artifact/recipe verification — dev/demo only (or KEYSTONE_INSECURE_SKIP_VERIFY=true)
--clock-policy	high-water	What to do when the system clock is behind known-good time: high-water verifies against the later of the two, strict refuses to verify
--reconcile-interval	0 (off)	Re-apply the plan in effect on this interval, restarting components that died and ran out of retries
--reconcile-jitter	10% of the interval	Spread reconcile passes across a fleet; the offset is derived from the device ID, not random
--demo	false	Run the built-in mock 3-component stack (db → cache → api)
--version	—	Print version and commit, then exit

MQTT Adapter Flags

Flag	Default	Description
--mqtt-broker	(empty)	MQTT broker URL (empty to disable)
--mqtt-tenant	(required)	Tenant for topics, a lowercase DNS label
--mqtt-device-id	hostname	Device ID for topics
--mqtt-client-id	keystone-{tenant}-{device-id}	MQTT client ID
--mqtt-tls-cert	(empty)	Client TLS certificate path
--mqtt-tls-key	(empty)	Client TLS key path
--mqtt-tls-ca	(empty)	CA certificate path
--mqtt-tls-verify	true	Verify server certificate
--mqtt-user	(empty)	Username
--mqtt-pass	(empty)	Password
--mqtt-qos	1	QoS level (0, 1, 2)
--mqtt-state-interval	10s	State event interval (0 to disable)
--mqtt-health-interval	30s	Health event interval (0 to disable)

Environment variable equivalents are also supported (flags take precedence): KEYSTONE_MQTT_BROKER, KEYSTONE_MQTT_TENANT, KEYSTONE_MQTT_DEVICE_ID, KEYSTONE_MQTT_CLIENT_ID, KEYSTONE_MQTT_TLS_CERT, KEYSTONE_MQTT_TLS_KEY, KEYSTONE_MQTT_TLS_CA, KEYSTONE_MQTT_TLS_VERIFY, KEYSTONE_MQTT_USER, KEYSTONE_MQTT_PASS, KEYSTONE_MQTT_QOS, KEYSTONE_MQTT_COMMAND_DEDUPE_TTL, KEYSTONE_MQTT_STATE_INTERVAL, KEYSTONE_MQTT_HEALTH_INTERVAL.

Git hooks

Run once after cloning to enable the repo’s versioned hooks:

task hooks   # or: ./scripts/setup-git-hooks.sh

This sets core.hooksPath to .githooks, where the pre-commit hook runs go fmt ./... and stages formatting changes automatically.

Core Concepts

Concept	Description
Recipe	TOML file describing a component: artifacts, lifecycle hooks, health checks, resources
Deployment Plan	TOML file listing components to run, resolved as a DAG with dependencies
Supervisor	Enforces lifecycle (install → start → running → stop) and restart policies
Runner	Executes components: ProcessRunner (native) or ContainerRunner (containerd/CLI)
Artifact Manager	Downloads, verifies (SHA-256 + signatures), caches, and garbage collects artifacts
Adapter	Pluggable control plane interface (HTTP, MQTT) for remote management

Systemd Unit (example)

See configs/systemd/keystone.service for a hardened example unit file.

Community

We welcome contributors who care about reliability at the edge.

• Issues: bug reports, design discussions, small enhancements
• PRs: focused, well-tested changes; prefer conventional commits
• Security: please report privately first when appropriate

License

📃 Apache-2.0