citadel-cli

README

Citadel CLI

The citadel CLI is the on-premise agent and administrator's toolkit for the AceTeam Sovereign Compute Fabric. It allows you to securely connect your own hardware to your AceTeam account, making your resources available to your private workflows.

Quick Start: Spin Up a Worker Node

The fastest way to turn a fresh Ubuntu machine into a Citadel worker node:

curl -fsSL https://get.aceteam.ai/citadel | sudo -E CITADEL_AUTHKEY=xxx bash

This single command:

• Installs NVIDIA drivers (if a GPU is present)
• Installs Docker CE and the NVIDIA Container Toolkit
• Downloads the latest citadel binary
• Creates a systemd service (citadel-worker)
• Pre-pulls the vLLM Docker image
• Joins the AceTeam Network using your authkey
• Starts the worker

Get an authkey from aceteam.ai/fabric. The script is idempotent and safe to run multiple times.

Documentation

Full documentation is available at aceteam-ai.github.io/citadel-cli -- including architecture guides, tutorials, command reference, and development docs.

Core Concepts

• AceTeam: The cloud-based control plane where you design and manage workflows.
• Citadel: The on-premise agent you run on your own hardware (the "node").
• Nexus: The secure coordination server (e.g., nexus.aceteam.ai) that manages the network.
• citadel.yaml: The manifest file that declares a node's identity and the services it provides. This file is automatically generated by the init command.

GPU Capabilities

Citadel auto-detects GPUs and running inference engines at startup, then uses that information to route jobs to the right nodes.

Auto-Detection

When citadel work starts, it:

1. Queries nvidia-smi to discover GPU model, VRAM, and count
2. Inspects running Docker containers for known inference engines (vLLM, Ollama, llama.cpp, LM Studio)
3. Generates capability tags (e.g., gpu:rtx3090, vram:24gb, engine:vllm)

Manual Declaration

You can also declare capabilities explicitly in citadel.yaml. This is useful when auto-detection doesn't fit your setup (e.g., GPUs not yet initialized at startup):

capabilities:
  gpus:
    - name: "NVIDIA GeForce RTX 3090"
      vram_mb: 24576
      count: 1
  engines:
    - vllm

If the capabilities section is present, it takes precedence over auto-detection. If omitted, auto-detection runs every time.

Queue Routing

Capability tags drive job routing. Each tag maps to a Redis Streams queue:

• gpu:rtx3090 subscribes the node to jobs:v1:tag:gpu:rtx3090
• vram:24gb subscribes to jobs:v1:tag:vram:24gb
• engine:vllm subscribes to jobs:v1:tag:engine:vllm

Every node also subscribes to the base fallback queue, so it can pick up untagged jobs.

Viewing Capabilities

citadel status

The CAPABILITIES section shows detected (or declared) GPUs, engines, and the full list of tags the node is advertising.

ACET Earnings

Operators who contribute compute to the AceTeam Fabric earn ACET tokens.

• Revenue split: Operators earn 80% of ACET token revenue generated by jobs that run on their hardware.
• Dashboard: View your earnings at aceteam.ai/fabric.
• How it works: Users purchase ACET tokens and spend them on inference requests. When a job runs on your node, you earn your share automatically.

Proxmox Deployment

For deploying multiple GPU nodes at scale, Citadel includes a Packer template and deployment script for Proxmox.

Building the VM Template

cd packer/
packer init citadel-node.pkr.hcl
packer build citadel-node.pkr.hcl

This produces a citadel-node.qcow2 image with Ubuntu 24.04, NVIDIA drivers, Docker, Citadel CLI, and vLLM pre-installed.

Deploying to Proxmox

./packer/deploy-to-proxmox.sh \
  --host root@pve.local \
  --authkey tskey-auth-xxxxx

The script uploads the image, creates a Proxmox template, clones it to a new VM, injects the authkey via cloud-init, and starts the VM. On first boot, the node joins the network and starts the worker automatically.

See packer/README.md for full options, GPU passthrough, and customization.

Uninstall

To remove Citadel from a node:

curl -fsSL https://get.aceteam.ai/citadel-uninstall | sudo bash

Or run the uninstall script directly:

sudo ./uninstall.sh

This stops the worker service, removes the citadel binary and configuration. It does not uninstall NVIDIA drivers or Docker.

---

Desktop and Computer Use

Citadel can detect desktop capabilities and expose REST APIs for remote desktop automation. This powers Agent Workspaces — AI agents that operate GUIs on your machine.

When running citadel work, the status server automatically detects:

• Operating system and version
• Display availability and screen resolution
• VNC server (ports 5900-5901)

If a token validator is configured (via citadel init), two authenticated endpoints become available:

• GET /api/screenshot — Captures a PNG screenshot of the current display
• POST /api/actions — Executes mouse/keyboard actions (click, type, key, scroll, move)

Linux prerequisites:

sudo apt-get install xdotool imagemagick   # or: sudo apt-get install xdotool scrot

macOS and Windows support is in progress. See docs/desktop-api.md for the full API reference.

---

Installation (CLI Only)

The following methods install just the citadel binary -- useful for development, testing, or managing nodes remotely. For full node provisioning (NVIDIA + Docker + worker service), use the Quick Start one-liner instead.

One-Line Installer

Linux / macOS

curl -fsSL https://get.aceteam.ai/citadel.sh | bash

This installs to ~/.local/bin and automatically configures your PATH. For system-wide install, use sudo bash instead.

macOS (Homebrew)

brew tap aceteam-ai/tap
brew install citadel

Or as a one-liner:

brew install aceteam-ai/tap/citadel

Go Install

If you have Go 1.21+ installed:

go install github.com/aceteam-ai/citadel-cli/cmd/citadel@latest

This installs the citadel binary to your $GOPATH/bin or $GOBIN.

Manual Installation

Linux / macOS

1. Go to the Releases Page.

2. Download the latest .tar.gz archive for your architecture (e.g., citadel_vX.Y.Z_linux_amd64.tar.gz).

3. Extract the archive and place the citadel binary in your PATH.

   tar -xvf citadel_vX.Y.Z_linux_amd64.tar.gz
   mv citadel ~/.local/bin/    # User-local (no sudo)
   # or: sudo mv citadel /usr/local/bin/  # System-wide
   

4. (Optional) Install the man page for man citadel support:

   # User-local
   mkdir -p ~/.local/share/man/man1
   mv citadel.1 ~/.local/share/man/man1/
   
   # System-wide (requires sudo)
   sudo mv citadel.1 /usr/local/share/man/man1/
   

   Homebrew installations include the man page automatically.

Windows

Option 1: One-Line Installer (Recommended)

Open PowerShell and run:

iwr -useb https://raw.githubusercontent.com/aceteam-ai/citadel-cli/main/install.ps1 | iex

This will automatically download, install, and add Citadel to your PATH.

Option 2: Windows Package Manager (winget)

Once published to winget (coming soon):

winget install AceTeam.Citadel

Option 3: Manual Installation

1. Go to the Releases Page.

2. Download the latest .zip archive for Windows (e.g., citadel_vX.Y.Z_windows_amd64.zip).

3. Extract the archive and place citadel.exe in your PATH.

   # Extract the zip file
   Expand-Archive citadel_vX.Y.Z_windows_amd64.zip -DestinationPath C:\Tools\citadel
   
   # Add to PATH (PowerShell as Administrator)
   $env:Path += ";C:\Tools\citadel"
   [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::Machine)
   

Updating

Citadel includes built-in auto-update detection. When running citadel work, it checks for updates once per day and notifies you if a new version is available.

Update Methods by Installation Type

Installation Method	Update Command
One-line installer	citadel update install
Homebrew	brew upgrade citadel
Go install	go install github.com/aceteam-ai/citadel-cli/cmd/citadel@latest
Manual download	Download new release from GitHub

Built-in Update Commands

# Check for updates
citadel update check

# Download and install the latest version
citadel update install

# Show update status and versions
citadel update status

# Restore previous version if update fails
citadel update rollback

# Enable/disable auto-update checks
citadel update enable
citadel update disable

The built-in updater:

• Downloads from GitHub releases
• Verifies SHA256 checksum before installing
• Backs up the current version for rollback
• Validates the new binary before completing
• Auto-rolls back if the new binary fails to start

Command Reference

Interactive Mode

Command	Description
citadel	(Recommended) Launch the interactive control center. Handles login, network, services, and job processing in one place.

Node Setup & Provisioning

Command	Description
citadel init	Join the AceTeam Network (no sudo required). Use --provision for full system provisioning (requires sudo).
citadel init --authkey <key> --service <name> --node-name <name> --test	The non-interactive version of init, perfect for automation. Allows you to specify the service (vllm, ollama, llamacpp, none), set the node name, and run a diagnostic test upon completion.
citadel work	(Primary command) Starts services from manifest AND runs the job worker (Redis Streams). Includes auto-update checks.
citadel run [service]	Starts services only. With no arguments, starts all manifest services. With a service name, adds it to the manifest and starts it.
citadel stop [service]	Stops services. With no arguments, stops all manifest services. With a service name, stops that specific service.
citadel login	Connects the machine to the network. Interactive prompts by default, or use --authkey <key> for non-interactive automation.

Node Operation & Monitoring

Command	Description
citadel status	Provides a comprehensive health check dashboard, showing the CLI version, system vitals (CPU, RAM, Disk), GPU status, network, and services.
citadel test --service <name>	Runs a diagnostic test for a specific service to verify its functionality.
citadel logs <service-name>	Streams the logs for a specific service defined in citadel.yaml. Supports -f to follow the log output.
citadel nodes	Connects to the Nexus API and lists all nodes in your compute fabric.
citadel run --restart	Restarts all services defined in citadel.yaml.
citadel version	Prints the current version of the CLI.
citadel update check	Check for available updates.
citadel update install	Download and install the latest version.
citadel update rollback	Restore the previous version if an update fails.
citadel terminal-server	Starts a WebSocket terminal server for remote browser-based terminal access.

---

Terminal Service

The Citadel Terminal Service provides WebSocket-based terminal access to nodes, enabling browser-based terminal sessions through the AceTeam web application.

Quick Start

# Start the terminal server (requires org-id)
citadel terminal-server --org-id my-org-id

# Start on a custom port with 1-hour idle timeout
citadel terminal-server --org-id my-org-id --port 8080 --idle-timeout 60

Configuration Options

Flag	Description	Default
--org-id	Organization ID for token validation (required)	-
--port	WebSocket server port	7860
--idle-timeout	Session idle timeout in minutes	30
--shell	Shell to use for sessions	Platform default
--max-connections	Maximum concurrent sessions	10

For detailed documentation, see docs/terminal-service.md.

---

Health Checks

Citadel nodes expose HTTP endpoints for health monitoring. Since ICMP ping doesn't work with userspace networking, use HTTP health checks instead.

# Lightweight ping check (minimal overhead)
curl http://<node-ip>:8080/ping
# Returns: {"status":"pong","timestamp":"2024-01-15T10:30:00Z"}

# Full health status
curl http://<node-ip>:8080/health
# Returns: {"status":"ok","version":"1.3.0"}

# Complete node status (system metrics, GPU, services)
curl http://<node-ip>:8080/status

The status server runs when using citadel work --status-port=8080.

---

Example Workflow: Provisioning a New GPU Node

This workflow shows how to take a fresh server and turn it into a fully operational Citadel node.

Quick Start (1 Command)

# Launch the interactive control center
./citadel

That's it! The control center handles login, network connection, service startup, and job processing -- all interactively.

Full Provisioning (with system setup)

For fresh servers that need Docker and dependencies installed:

# Interactive setup with full provisioning
sudo ./citadel init --provision

# Then start the worker
./citadel work

Automated Deployment

For scripted deployments, provide all options as flags:

# Provision and configure in one command
sudo ./citadel init --provision \
  --authkey tskey-auth-k1A2b3C4d5E6f... \
  --service vllm \
  --node-name gpu-node-01

# Start the node
./citadel work --status-port=8080

Verify Status

Check the node's health at any time:

./citadel status

You should see ONLINE for the network connection and RUNNING for your configured service.

Windows E2E Testing

A remote E2E test script validates the full first-time user experience on a Windows machine via WinRM (clean -> install -> init -> verify).

# Prerequisites: pip install pywinrm (or use a venv)
# WinRM must be enabled on the target Windows machine

# Full test (all phases)
PYTHON=~/.venvs/winrm/bin/python3 ./scripts/windows-e2e-test.sh \
  --host 192.168.2.207 --user acewin --password 'P@ssword' \
  --authkey tskey-auth-xxx

# Dry run (show commands without executing)
./scripts/windows-e2e-test.sh --dry-run --host 192.168.2.207 ...

# Run a single phase: clean, install, provision, or verify
./scripts/windows-e2e-test.sh verify --host 192.168.2.207 ...

See scripts/windows-e2e-test.sh for all options and WinRM setup instructions.

---

The citadel.yaml Manifest

The init command generates this file for you. It defines the node's identity and the service it runs.

Example citadel.yaml (generated for a vLLM node):

node:
  name: gpu-node-01
  tags:
    - gpu
    - provisioned-by-citadel
services:
  - name: vllm
    compose_file: ./services/vllm.yml

---

Building from Source

Linux / macOS

If you need to build from the latest source code:

# This will create binaries for your platform in the ./build directory
./build.sh

# Build for all platforms (requires cross-compilation tools)
./build.sh --all

Windows

Quick Setup (Automated):

# One-command setup: installs Go/Git, clones repo, builds, tests
iwr -useb https://raw.githubusercontent.com/aceteam-ai/citadel-cli/main/setup-dev-windows.ps1 | iex

See WINDOWS_QUICKSTART.md for a 5-minute getting started guide.

Manual Build:

# Build for Windows (native PowerShell)
.\build.ps1

# Build for all platforms (requires tar for cross-platform packages)
.\build.ps1 -All

# Quick development build
go build -o citadel.exe ./cmd/citadel

See WINDOWS_DEVELOPMENT.md for detailed Windows development setup instructions.

Releasing (For Maintainers)

Always use release.sh to create releases. Do not manually create tags or GitHub releases.

# Interactive mode - prompts for version
./release.sh

# Non-interactive mode - specify version
./release.sh -v v1.2.0 -y

# Dry run to preview what will happen
./release.sh --dry-run -v v1.2.0

Release Process

The script will:

1. Validate Environment

   • Check for uncommitted changes (working directory must be clean)
   • Verify GitHub CLI (gh) is installed
   • Validate version format (must be vX.Y.Z or vX.Y.Z-rc1)

2. Create and Push Tag

   • Create a git tag with the specified version
   • Push the tag to the remote repository

3. Build Artifacts

   • Run build.sh to create binaries for all platforms (linux/darwin/windows, amd64/arm64)
   • Generate SHA256 checksums

4. Create GitHub Release

   • Generate release notes from commits since the last tag
   • Upload binaries and checksums to GitHub Releases
   • Display the release URL

5. Update Homebrew Tap

   • Automatically updates the aceteam-ai/homebrew-tap formula
   • Commits new version and SHA256 checksums

Version Numbering

Follow semantic versioning (semver):

• Major version (v2.0.0): Breaking changes
• Minor version (v1.1.0): New features, backwards compatible
• Patch version (v1.0.1): Bug fixes, backwards compatible
• Pre-release (v1.1.0-rc1): Release candidates for testing

Manual Release Process

Note: Manual releases will NOT update the Homebrew tap. Always prefer release.sh.

If you absolutely must release manually:

# 1. Create and push tag
git tag -a v1.2.0 -m "v1.2.0"
git push origin v1.2.0

# 2. Build artifacts for all platforms
./build.sh --all

# 3. Create GitHub release
gh release create v1.2.0 \
  --title "v1.2.0" \
  --notes "Release notes here" \
  release/citadel_v1.2.0_*.tar.gz \
  release/citadel_v1.2.0_*.zip \
  release/checksums.txt

# 4. Manually update Homebrew tap (aceteam-ai/homebrew-tap)