halos

README

Ship AI coding tools like software.
_{Versioned · signed · ring-deployed · proven by evals}

Start here · Quickstart · Architecture · CLI · Threat model

---

Halos is the open-source control plane for AI coding CLIs: Claude Code, Codex, Gemini CLI, Copilot CLI, Cursor CLI and OpenCode.

• One signed release per ring. A policy repo becomes versioned, content-addressed releases that halod and the gateway enforce on every machine.
• Every change is staged and measured. Dark launch, canary, A/B and shadow behind evals and live telemetry; promotion is a PR a human merges.
• Rollback is a pointer, not a Slack message. A regression measured at the gateway trips a signed kill list that gateways pick up in about 10 s (when halo-server serves it); any other regression opens a rollback PR.

Quickstart

brew install dshakes/tap/halo      # other installs below
halo quickstart                    # the whole stack in Docker, about a minute
halo doctor && halo onboard local  # or pin and manage the CLIs on this machine
halo onboard company               # or: policy repo, gateway, IdP, fleet, first PR

Each path can also be driven by the AI CLI you already use: Start here.

How it works

Your platform team keeps a policy repo: a Git repo that halo init creates. It starts as one file, and developers never touch it.

# halos.yaml: the whole policy, in simple mode
org: acme
tools: {claude-code: 2.1.280, codex: {version: 0.99.0, model: codex}}
provider: anthropic                    # bedrock | vertex | openai | gemini | multi
models: {default: claude-sonnet-5-5, strong: claude-opus-5-5, codex: openai/gpt-5-codex}
gateway: https://ai.acme.example
identity: {issuer: https://login.acme.example, audience: halos, adminGroups: [ai-platform]}
safety: standard                       # strict | standard | relaxed: never bypasses permissions
rollout: standard                      # fast | standard | careful: rings, canary steps, bake times, gates

Presets expand into profiles, rings and gateway routes. Intent commands write the rest for you. The ones that write policy (upgrade start, model switch, enable, eject) support --dry-run, validate their output and list the files they change; halo kill writes to halo-server at once (no release) and takes effect at the next poll, by default 10 s at gateways and 60 s on devices:

# A/B + gated rollout for a CLI upgrade. Builds and signs a candidate release, so it needs a registry
# and key (HALO_REGISTRY / HALO_KEY) and the GA ring's published pointer, or --baseline <digest>.
# --dry-run still reads that pointer.
halo upgrade start claude-code 2.1.300 --registry ghcr.io/acme/halos --key halo.key
halo model switch strong claude-opus-5 --canary
halo enable mcp github --url https://… --for 10%
halo kill <experiment|toggle|rollout> --reason "…"
halo status                                   # rings, rollouts, experiments, toggles
halo explain | halo eject                     # see, or take over, the full policy

A change is a pull request:

1. CI runs halo validate, halo plan and evals.
2. On merge, halo release publish builds and signs a release for each ring.
3. Every environment picks it up: halod on laptops and CI runners, the Dev Container Feature in workspaces, and halo-proxy or Kong for model traffic.

examples/simple is the one-file version and examples/acme-corp the full one; make demo runs it locally; halo quickstart is the same stack (55 s warm, 86 s with a cold build cache, on an arm64 laptop with Docker Desktop 28). Codespaces is UNVERIFIED: CI builds the dev container, but the demo has not been run in a real Codespace. More: how Halos works.

Every rollout technique, as code

A Rollout is an ordered list of steps. Each step sets:

• a strategy and an exposure
• a bake time and a minimum sample size
• gates: metric guardrails, an eval scorecard, a human approval

apiVersion: halos.dev/v1
kind: Rollout
name: opus-5-5-upgrade
axis: traffic                    # client: a CLI/config release · traffic: a model route
experiment: opus-5-5-canary
change: {alias: opus}
steps:
  - name: dark-launch            # mirrored to the candidate, nothing served
    strategy: dark-launch
    percent: 10
    bake: 2d
    gates:
      scorecard: {file: evals/opus-5-5.scorecard.json, variant: opus-5-5, minPass1: 0.75}
  - name: canary-5
    strategy: canary
    percent: 5
    bake: 12h
    minSamples: 500
    gates:
      guardrails: [{metric: halo.api.error_rate, direction: decrease, maxRegression: 0.05}]
  - name: holdout                # 5% stay on the old model for 14 days
    strategy: holdout
    percent: 5
    bake: 14d
    gates: {approval: true}

The controller evaluates every active step on each tick (default every 5 minutes):

• A guardrail breach measured at the gateway trips the signed kill switch on its own (traffic axis, when halo-server serves a kill list). Any other breach opens a rollback PR and notifies.
• Progress always opens a PR that a human merges. Nothing auto-promotes.

Commands: halo rollout plan | status | simulate | advance | rollback (docs).

Experiments and feature toggles

• A/B a CLI version or settings change. Each arm is its own signed release channel, and halod picks the arm with the same hash the gateway uses.
• Canary a model at the gateway, sticky per user and session, so agents never switch model mid-task.
• Shadow real agent sessions to a candidate model, every step or just the first, and compare its next action with the control and with what production actually did (tool agreement, fidelity vs a measured noise floor, invalid calls, premature stops), plus an LLM judge. Nothing from the candidate is served or executed.
• Toggles turn one capability (an MCP server, a hook, a model route) on for a ring, an IdP group or a percentage of users. The payload ships inside the signed release. A kill needs no new release and takes effect at the next poll: about 10 s at gateways, 60 s on devices.

Measure, don't guess

Offline evals	Replay real tasks in containers, N trials each. Reports pass@1, pass@k and pass^k, with paired bootstrap CIs on pass rate, cost, tokens and time. Flaky tasks are detected and excluded. Verdict: ship, hold or block.
Matrix	Runs one suite across harness × model × provider (Claude Code vs Codex vs Gemini, Anthropic vs Bedrock) and produces a PR-ready report.
Online	Shadow pairs are graded against a pinned LLM-judge rubric and land in the same evidence plane as cost, latency and errors.
Sequential stats	mSPRT guardrails stop an experiment as soon as a regression is real, not at an arbitrary horizon.
Upgrade watch	halo upgrade watch spots a new CLI or model release, pins it on a branch, runs the suite and opens a PR with the scorecard.

Six CLIs, any provider

	Claude Code	Codex	Gemini CLI	Copilot CLI	Cursor CLI	OpenCode
Version pin	CLI-enforced	halod	halod	halod	not installable by halod	halod
Model lock	✓	✓	✓	default only	-	✓
MCP allowlist	✓	✓	✓	✓	-	-
Permissions	✓	✓	partial	partial (UNVERIFIED)	-	✓
Gateway	✓	✓	✓	-	-	✓
Telemetry	✓	✓	✓	✓	-	-

A setting a CLI can't enforce produces a warning in the release manifest; it is never silently dropped. halo harnesses prints the capability matrix; the harness reference has every cell and its source.

Traffic goes through halo-proxy to Anthropic, Bedrock (SigV4), Vertex, OpenAI, Azure OpenAI or the Gemini API:

• Model aliases map to weighted splits and priority failover, with a circuit breaker per target.
• Moving a model to another provider is a one-line policy change.
• If you already run Kong, the halo-kong plugin applies the same identity, cohort and experiment decisions. It routes each alias to its first target, with no translation or failover.

Enterprise controls

Control	What it does	Docs
RBAC	IdP groups map to viewer, operator, approver, auditor, admin; every route is deny-by-default and denials are audited	security model
SCIM 2.0	Okta and Entra provisioning; deprovisioning revokes sessions and device tokens at once	SCIM
Org units	Delegated owners and overlays that can tighten, never loosen, enforced fields	org hierarchy
Cost budgets	Per user, group, ring or org caps in USD or tokens, warn or enforce, answered as 429 at the gateway	budgets
Audit to SIEM	Hash-chained audit forwarded to Splunk HEC, signed webhook or syslog TLS, native or OCSF	SIEM export
CI and workload identity	RFC 8693 exchange of a GitHub, GitLab, Buildkite, CircleCI or cloud OIDC token for a credential of at most 1 h	CI and cloud agents
Agent identity	Delegated agent credentials with an act chain, down-only model scope and an A2A allowlist	agent identity
Extension governance	Marketplace, plugin, skill and subagent policy, rendered where the vendor documents a key and warned elsewhere	harness reference
OTel GenAI	gen_ai.* semantic-convention metrics and opt-in content-free spans from the gateway	evidence plane
HA on Postgres	Several halo-server replicas behind one load balancer with a fenced controller lease	production deployment
Vendor admin sync	One policy drives the vendors' cloud admin consoles; halo admin plan fails CI on drift, apply writes what has a public API and never loosens	vendor admin sync
Change governance	Freeze windows and an incident freeze hold advances and promotions, never rollbacks; N-of-M approvals with separation of duties and tickets; halo plan --diff previews who a change reaches	change governance

Runs everywhere

The current release is v0.3.0 (CHANGELOG). Or build from source: git clone https://github.com/dshakes/halos && cd halos && make build (builds halo, halod, halo-shadow, halo-server, halo-proxy and halo-kong into bin/).

brew install dshakes/tap/halo                                                            # macOS / Linux
scoop bucket add dshakes https://github.com/dshakes/scoop-bucket && scoop install halo   # Windows

# The script works everywhere; halod runs as root from a root-owned prefix
curl -fsSLO https://raw.githubusercontent.com/dshakes/halos/v0.3.0/install/install.sh
sh install.sh --version v0.3.0                                        # halo into ~/.local/bin
sudo sh install.sh --version v0.3.0 --prefix /usr/local --with-agent  # plus halod

Also available:

• deb, rpm and apk packages from the release assets, and install.ps1 from the v0.3.0 tag; a winget manifest is built but not yet submitted
• a Dev Container Feature and a Coder module
• Jamf, Kandji and Intune exports
• a GitHub Action and a GitLab template for CI runners
• a Helm chart with signed images on ghcr.io/dshakes

Archives are checksummed and carry SBOMs; checksums.txt is cosign-signed and every archive has a GitHub build-provenance attestation. install.sh verifies the checksums, and the cosign signature when cosign is installed (otherwise it warns). The release workflow re-verifies all of it from the published artifacts before a release counts as done (RELEASING.md, install guide).

Agent-native

halo mcp serve exposes these to any MCP client: validate, plan, rollout status, experiment analysis, toggle evaluation and eval scorecards.

The write tools (propose a rollout step, a rollback or a toggle change, kill a change, run an eval) are opt-in and dry-run by default. Proposals commit to a new local branch; propose_promotion and, on request, the rollout proposals push a review branch and open a PR. None pushes to the base branch, merges or publishes a release. The Claude Code plugin, the Codex config, the Gemini CLI extension and the Copilot CLI config drive the same flows: onboarding (doctor → dry run → approve → write → verify) and rollout (edit → validate → eval → experiment → propose).

Architecture

Deep dive, traced to the code: tech stack, components and low-level design.

Security by construction

• Signed everything. Releases, ring pointers and the kill list are ed25519-signed; releases and pointers can also be co-signed with cosign. Ring pointers carry sequence numbers and an expiry, so a registry can't replay an old one.
• Identity, never headers. The gateway verifies OIDC JWTs (or, in trusted_header mode, accepts identity only from pinned proxy CIDRs), strips client x-halo-* headers and fails closed on unknown models.
• Guardrails twice. bypassPermissions and danger-full-access are rejected at validate time and again on the rendered output.
• Humans own the irreversible step. There is no auto-merge, no auto-promote and no agent (MCP) tool that publishes. Privileged actions land in a hash-chained audit log.

Threat model · SECURITY.md

Status

Every component is implemented and covered by unit, race, e2e (make e2e) and evidence-plane (make obs-e2e) tests. Every pull request also runs three acceptance suites against real software:

Suite	Runs against	Report
make uat-clis	the real Claude Code, Codex, Gemini CLI and Copilot CLI (Cursor CLI and OpenCode are not in this run yet), including the eval runner	CLI-REPORT.md
make uat-k8s	a kind cluster with the Helm chart installed	REPORT.md
make uat-kong	open-source Kong 3.9.3 with the halo-kong plugin	KONG-REPORT.md

Unit tests also run on Windows and macOS in CI. A nightly provider smoke sends real requests through halo-proxy to Anthropic and OpenAI (streaming and non-streaming) and checks the gateway metrics.

Not yet exercised against the real systems:

• AWS Bedrock, Vertex, Azure OpenAI and Gemini API accounts (fixture-tested only; the nightly smoke covers them once their secrets are set)
• Kong Enterprise
• Windows beyond a CI runner: Intune, Authenticode, an interactive logged-in user (CI runs install.ps1, enroll.ps1 and halod as a SYSTEM scheduled task on a hosted Windows runner)
• MDM-managed devices (Jamf, Kandji, Intune)
• Codespaces and Coder workspaces (CI builds the dev container and the Feature with the devcontainer CLI)
• the GitLab CI template on a real GitLab runner (CI runs its script in a container with gitlab-ci-local, an emulation)
• the Helm chart on a managed cloud cluster

Contributing

CONTRIBUTING.md · AGENTS.md for coding agents · ADRs · Apache-2.0