panix

README

Panix

Universal NixOS Deployment Tool

Stateless, phase-oriented deployment with real-time visibility across multi-flake fleets

---

[!WARNING] The tool is currently in beta stage. There might be breaking changes.

Demo

mp4 version

Full demo of the bootstrap process with kexec over Arch:

It auto-enrolls Lanzaboote and TPM2 disk encryption bound to Secure Boot. Note that video does not show the steps to enable Audit/Setup mode of Secure Boot in BIOS. Note that it was made with an older version of TUI, and therefore looks slightly different.

---

The Problem

Deploying NixOS at scale introduces operational challenges that existing tools address partially but not holistically:

• Fragmented tooling: nixos-anywhere handles bootstrapping bare-metal machines, while deploy-rs and Colmena manage deployment orchestration. Each tool excels within its domain, but integration between bootstrap and ongoing deployment cycles remains manual.
• Missing visibility: Deployment failures require parsing scrollback logs after execution completes. There's no unified view of what phase failed, which machine is affected, or the current state across a heterogeneous fleet.
• Implicit dependencies: Most deployment tools require modifying your flake to include their module or output, creating a dependency that complicates using alternative tools later.
• No retry mechanism: When a phase fails mid-execution, the typical workflow is to restart from scratch. Partial progress is discarded rather than preserved and recoverable.

Panix addresses these problems by providing deployment orchestration with built-in visibility and interactivity.

---

What Panix Is

Panix is a deployment orchestrator for NixOS flakes.

It provides:

• Stateless operation: No persistent state is maintained between runs. All information is derived from your flake, configuration file, and runtime machine inspection.
• Phase-oriented execution: Six sequential deployment phases - Inspect, Bootstrap, Build, Transfer, Secrets, Activate - execute with defined scopes. The Build phase runs once per configuration, deduplicating work across machines sharing the same nixosConfiguration.
• Remote build mode: Builds can execute on a target machine instead of locally, useful when the target has more resources or a different architecture. The closure is then copied between machines via the Nix remote store protocol.
• Real-time TUI: An interactive interface provides visibility into each phase per machine. You can observe failures as they occur, inspect logs, and retry failed phases without restarting the entire workflow.
• Bootstrap support: machines can be converted to NixOS via NixOS live install or any live install or previusly installed distro using kexec and disko, with full support for custom hooks and secrets at multiple stages.
• Flake-agnostic configuration: The deployment configuration is separate from your flake. No modifications to your flake are required to use Panix.

---

The Phase Pipeline

Panix doesn't just "run a deployment". It executes an ordered pipeline of phases, each with a specific scope and purpose:

Inspect → Bootstrap → Build → Transfer → Secrets → Activate

Scope-aware execution: The build phase runs once per configuration, not per machine. If three machines share the same nixosConfiguration, you build once. The closure is then transferred to all three machines independently in parallel.

Phase-by-phase breakdown:

Phase	Scope	Purpose
Inspect	Per-machine	TCP reachability, SSH authentication, architecture detection, OS detection, generation discovery
Bootstrap	Per-machine	kexec into NixOS installer (if needed), disko partitioning, encryption keys transfer (if provided)
Build	Per-configuration	Build config.system.build.toplevel closure via nix build (local or remote mode)
Transfer	Per-machine	nix copy closure to target
Secrets	Per-machine	Transfer files/directories with proper ownership via rsync
Activate	Per-machine	nixos-install (bootstrap) or switch-to-configuration (deploy)

Standalone phases (in combination with Inpect):

Phase	Scope	Purpose
Rollback	Per-machine	Switch to a previous NixOS generation via switch-to-configuration

Logical diagram:

---
config:
  theme: base
  flowchart:
    subGraphTitleMargin:
      top: 10
      bottom: 30
    nodeSpacing: 10
    rankSpacing: 30
    wrappingWidth: 250
  themeVariables:
    fontSize: 32px
---
flowchart LR
  subgraph Inspect
    direction TB

    A1[TCP reachability] --> B1[SSH echo OK]
    B1 --> C1[uname -m]
    C1 --> D1[id -u]
    D1 --> E1[cat /etc/os-release]
    E1 --> F1[nixos-rebuild list-generations]
  end

  subgraph Bootstrap
    direction TB  
  
    A2[kexec] --> B2[encryption keys]
    B2 --> C2[disko] 
    C2 --> D2[post-bootstrap hooks]
  end
  
  subgraph Per-config
    subgraph Build
      A3[nix build --print-out-paths]
    end
  end

  subgraph Transfer
    A4[nix copy --to target]
  end
  
  subgraph Secrets
    A5[rsync --chmod --chown]
  end
  
  subgraph Activate
    direction TB

    A6[nixos-install --system] --> B6[post install hooks]
    B6 --> C6[reboot]
    C6 --> D6[post boot install hooks]

    E6[nix-env --profile --set] --> F6[switch-to-configuration]
  end

  subgraph Rollback
    A7[readlink] --> B7[nix-env --profile --set]
    B7 --> C7[switch-to-configuration switch]
  end

  Start --> Inspect
  Start --> Build

  Inspect --> Bootstrap
  Bootstrap --> Build
  Build --> Done
  Build --> Transfer
  Transfer --> Secrets
  Secrets --> Activate

  Inspect --> Build
  Build --> Secrets
  Build --> Activate
  Transfer --> Activate
  Activate --> Done

  Inspect --> Rollback
  Rollback --> Done


  classDef cmd fill:#6B7280,color:#fff
  classDef cmdBootstrap fill:#4F4F3A,color:#fff

  class A1,B1,C1,D1,E1,F1 cmd
  class A2,B2,C2,D2 cmdBootstrap
  class A3 cmd
  class A4 cmd
  class A5 cmd
  class A6,B6,C6,D6 cmdBootstrap
  class E6,F6 cmd
  class A7,B7,C7 cmd

  style Inspect fill:#cde8f5,stroke:#6272A4,stroke-width:2px,color:#333,fontWeight:bold
  style Bootstrap fill:#cde8f5,stroke:#6272A4,stroke-width:2px,color:#333,fontWeight:bold
  style Per-config fill:#ffddaa,stroke:#d79b00,stroke-width:2px,color:#333,fontStyle:italic
  style Build fill:#cde8f5,stroke:#6272A4,stroke-width:2px,color:#333,fontWeight:bold
  style Transfer fill:#cde8f5,stroke:#6272A4,stroke-width:2px,color:#333,fontWeight:bold
  style Secrets fill:#cde8f5,stroke:#6272A4,stroke-width:2px,color:#333,fontWeight:bold
  style Activate fill:#cde8f5,stroke:#6272A4,stroke-width:2px,color:#333,fontWeight:bold
  style Rollback fill:#e5e6e8,stroke:#6B7280,stroke-width:2px,color:#333,fontWeight:bold

  style Start fill:#007da7,stroke:#007da7,color:#fff,fontWeight:bold
  style Done fill:#14532D,stroke:#14532D,color:#fff,fontWeight:bold

  linkStyle default stroke:#6272A4,stroke-width:1.5px

The TUI shows this unfolding in real-time:

When a phase fails, you don't have to restart everything. You inspect the logs, understand the failure, maybe do a quick fix in code or on remote, and press r to retry just the failed phases.

Or if it changed beyond a phase, just restart the whole workflow with ctrl+r without leaving the TUI.

---

Features

Configuration & Templating

How you define, parameterize, and target your fleet.

Hierarchical Configuration Inheritance

Your infrastructure has natural hierarchies: flakes contain configurations, configurations contain machines. Panix's configuration model reflects this:

Fleet → Flake → Configuration → Machine

Attributes at each level cascade down, without overriding child attributes. Slices (tags, secrets, disk encryption keys) append. Define once at the fleet, extend at any level:

fleet:
  tags: [production]              # Inherited by all descendants
  secrets:                        # Inherited by all descendants
    - local_path: ./secrets/common.key
      remote_path: /var/secrets/common.key
  
  flakes:
    infrastructure:
      tags: [critical]            # Accumulated: [production, critical]
      secrets:                    # APPENDED to inherited secrets
        - local_path: ./secrets/prod/api.key
          remote_path: /var/secrets/api.key
      
      configurations:
        webserver:
          tags: [web]             # Accumulated: [production, critical, web]
          
          machines:
            web-01:               # Accumulated: [production, critical, web, web-01]
            web-02:
              secrets:            # APPENDED to all inherited secrets
                - local_path: ./secrets/web-02/cert.pem
                  remote_path: /etc/ssl/cert.pem

Multi-Flake Deployments

Your infrastructure may span multiple flakes/repositories. Panix treats this as a first-class concern:

fleet:
  flakes:
    infrastructure:
      url: path:../infra-flake
      configurations:
        servers:
          machines:
            server-01:
            server-02:
    
    monitoring:
      url: github:myorg/monitoring-flake
      configurations:
        prometheus:
          machines:
            prom-01:
    
    secrets-management:
      url: git+ssh://git@github.com/myorg/vault-nixos
      configurations:
        vault:
          machines:
            vault-01:

With this you get complete visibility across your entire infrastructure. Each flake builds independently, each configuration deduplicates builds across its machines.

Tag-Based Filtering

Every name (flake, configuration, machine) is automatically a tag. Tags accumulate through inheritance. Deploy subsets of your infrastructure:

panix --tags production         # All production-tagged machines
panix --tags webserver          # All machines under webserver config
panix --tags server-01          # Single machine
panix --tags server-01,special  # Single machine and tag special

Template Engine & Evaluation

Panix includes a powerful template engine for dynamic YAML configuration. Templates use standard Go template syntax with {{ and }} delimiters and support 100+ built-in functions from the Sprout library.

Basic Syntax

Environment variables and dynamic values:

fleet:
  flakes:
    my-flake:
      url: |
        {{ env "MY_FLAKE_URL" }}
      configurations:
        server:
          machines:
            server-01:
              ssh:
                hostname: |
                  {{ env "SERVER_HOST" | default "192.168.1.100" }}
                port: |
                  {{ env "SSH_PORT" | default "22" }}

Conditional Logic

Use conditionals for environment-specific configurations:

fleet:
  flakes:
    {{if eq (env "ENV") "production"}}
    prod-flake:
      url: github:myorg/prod-config
    {{else}}
    dev-flake:
      url: github:myorg/dev-config
    {{end}}

Template Definitions

Define reusable templates with {{define}} and invoke them with {{template}}:

# Define template (in YAML comment to avoid LSP errors)
# {{define "cryptenroll"}}systemd-cryptenroll --unlock-key-file=/tmp/disko-encryption-password.txt --tpm2-device=auto --tpm2-with-pin=no --wipe-slot=all {{.}}{{end}}

fleet:
  flakes:
    infrastructure:
      configurations:
        server:
          machines:
            server-01:
              bootstrap:
                post_bootstrap_hooks:
                  - |
                    {{template "cryptenroll" "/dev/sda2"}}
            
            server-02:
              bootstrap:
                post_bootstrap_hooks:
                  - |
                    {{template "cryptenroll" "/dev/nvme0n1p2"}}

Built-in Functions

Panix provides 100+ functions from Sprout, documentaion is available in Sprout docs.

YAML Anchors

Use YAML anchors for reusable blocks alongside templates:

anchor_bootstrap: &bootstrap_defaults
  disk_encryption_keys:
    - local_path: /tmp/disko-encryption-password.txt
      remote_path: /tmp/disko-encryption-password.txt

fleet:
  flakes:
    infrastructure:
      configurations:
        server:
          machines:
            server-01:
              bootstrap:
                <<: *bootstrap_defaults
                post_bootstrap_hooks:
                  - systemd-cryptenroll --tpm2-device=auto /dev/sda2

⚠️ Warning
When specifying YAML anchor keys, you have to prefix them with `anchor_` for them not to be rejected by the parser.

Template Command

Preview the processed YAML with templates evaluated and anchors resolved (no validation):

panix template                    # Output to stdout (colorized)
panix template -o processed.yaml  # Output to file (plain YAML)

The template command:

• Resolves all {{...}} template expressions
• Merges YAML anchors
• Outputs only flags and fleet top level keys

Eval Command

Fully evaluate and validate the configuration for execution:

panix eval                    # Output to stdout (colorized)
panix eval -o processed.yaml  # Output to file (plain YAML)

The eval command:

• Everything the template command does
• Loads and validates the full configuration
• Validates flake URLs and configuration keys (--validate.flakes)
• Validates bootstrap disk encryption key local paths (--validate.bootstrap-secrets)
• Resolves inherited attributes, tags, and secrets
• Filters out disabled and untagged parts of the configuration
• Outputs the fully resolved configuration

Notes

• Template definitions ({{define}}) work inside YAML comments
• Standard Go template syntax with {{ and }} delimiters
• YAML LSP may complain about template syntax - this is cosmetic
• For complex multiline templates, use block scalars (|)

IDE Support

You can directly reference the YAML schema for autocompletion and validation:

# yaml-language-server: $schema=https://raw.githubusercontent.com/mihakrumpestar/panix/main/gen/panix-schema.yaml

...

Or you can generate it locally:

panix schema

and reference it:

# yaml-language-server: $schema=./panix-schema.yaml

...

Bootstrap & Provisioning

Converting bare metal or existing systems to NixOS.

Bootstrap: From Nothing to NixOS

The bootstrap flow is where Panix distinguishes itself most clearly:

1. Inspect phase detects the target OS; if not NixOS, kexec boots into a NixOS installer image
2. Disko partitions disks according to your configuration
3. Transfer system closure
4. nixos-install lays down the system
5. Reboot into your new NixOS system

machines:
  bare-metal:
    ssh:
      hostname: 192.168.1.100
    bootstrap:
      disk_encryption_keys:       # Transferred BEFORE disko runs
        - local_path: ./secrets/luks.key
          remote_path: /tmp/luks-key
      post_bootstrap_hooks:       # Run after disko partitioning
        - systemd-cryptenroll --tpm2-device=auto /dev/nvme0n1p2

Bootstrap Hooks

Panix provides multiple hook points during bootstrap:

Hook	When it runs	SSH used
post_bootstrap_hooks	After disko partitioning	Bootstrap SSH
post_bootstrap_install_hooks	After nixos-install, before reboot	Bootstrap SSH
post_bootstrap_provisioned_hooks	After reboot into new system	Regular SSH

Special hook commands:

• waitForOnline - Wait for machine to become reachable (useful after reboot)
• waitForOffline - Wait for machine to become unreachable (useful during reboot)

machines:
  server:
    bootstrap:
      post_bootstrap_hooks:
        - systemd-cryptenroll --tpm2-device=auto /dev/sda2
      post_bootstrap_install_hooks:
        - echo "Installation complete, preparing for reboot"
      post_bootstrap_provisioned_hooks:
        - reboot # second reboot
        - waitForOffline
        - waitForOnline
        - systemctl enable --now my-service

Bootstrap SSH

For machines that need initial provisioning (eg. bootstrap) you must specify SSH credentials.

machines:
  server:
    ssh:                          # Regular SSH (after bootstrap)
      hostname: 192.168.1.100
      port: 9999
      identity_file: ./keys/prod.key
    bootstrap:
      ssh:                        # Bootstrap SSH (during bootstrap)
        hostname: 192.168.1.100
        identity_file: ./keys/temp.key

Important requirements:

• Unbootstrapped machines: must have bootstrap SSH configured (unless force_bootstrap: true)
• Bootstrapped machines: must not have bootstrap SSH configured (unless force_bootstrap: true)
• The force_bootstrap option explicitly allows bypassing these requirements
• Inspect phase will error if you will try to use bootstrap SSH config on an already bootstrapped machine and vice versa (regular SSH config will error on an un-bootstrapped machine)

Workflow:

1. During Inspect phase: Panix validates SSH configuration matches machine state

2. Later phases:

   • If bootstrapping: Uses bootstrap SSH for all bootstrap operations; after reboot: automatically switches to regular SSH

   • If already bootstrapped: uses regular SSH

SSH Key Checking Options

ssh:
  # Disable strict host key checking (default: false for both regular and bootstrap SSH)
  disable_strict_key_checking: false
  # Disable auto-adding host keys (default: false for both regular and bootstrap SSH)
  disable_auto_add_host_key: false
  # Path to known_hosts file (default: user's ~/.ssh/known_hosts; bootstrap SSH uses a temporary file)
  known_hosts_file: ""

Defaults:

• disable_strict_key_checking: false: strict host key checking enabled
• disable_auto_add_host_key: false: automatically adds new host keys to known_hosts
• known_hosts_file: "": uses the user's default ~/.ssh/known_hosts or temporary file if using bootstrap SSH

Behavior:

• Regular SSH: uses the user's ~/.ssh/known_hosts with StrictHostKeyChecking=accept-new (trusts new machines, verifies against existing entries)
• Bootstrap SSH: Panix automatically creates a temporary known_hosts file and uses it for the duration of the bootstrap session. The first SSH connection records the installer's host key (trust on first use / TOFU). All subsequent commands during bootstrapping verify against that recorded key, preventing man-in-the-middle attacks mid-session. If known_hosts_file is explicitly set in bootstrap SSH config, that file is used instead of the temporary one. After bootstrapping completes, the temporary file (but not the user set one) is removed.
• After kexec: the kexec installer preserves SSH host keys from the original system on remote, so the recorded key remains valid. If the kexec SSH port differs from the bootstrap SSH port, a new known_hosts entry is recorded (due to StrictHostKeyChecking=accept-new) for the new port (same key, different [host]:port entry).
• Setting disable_strict_key_checking: true disables all host key checking (UserKnownHostsFile=/dev/null, StrictHostKeyChecking=no)

⚠️ Warning
Changing `disable_strict_key_checking` or `disable_auto_add_host_key` from their defaults has significant security implications:

• Setting disable_strict_key_checking: true disables all host key verification. This allows man-in-the-middle attacks on every SSH connection. Only use this in fully trusted networks (e.g., local VMs with no external access).
• Setting disable_auto_add_host_key: true prevents new host keys from being recorded. Combined with disable_strict_key_checking: false (the default), this enforces strict checking: connections will be rejected if the host key is not already in the known_hosts file. This is the most secure option but requires the key to be pre-provisioned (e.g., via known_hosts_file or manually adding entries).
• The defaults (disable_strict_key_checking: false, disable_auto_add_host_key: false) provide StrictHostKeyChecking=accept-new behavior: new hosts are trusted on first connection and verified on subsequent ones. This is the standard SSH trust model and is secure for most use cases.

Disable Automatic Reboot

To prevent automatic reboot after nixos-install (useful for manual inspection or custom reboot handling):

machines:
  server:
    bootstrap:
      disable_automatic_reboot: true

Reinstalling a Live NixOS Installation

Panix can force a reinstall of an already running NixOS system. This is useful when you want to completely wipe and reinstall a machine from scratch.

machines:
  existing-nixos:
    ssh:
      hostname: 192.168.1.100
    bootstrap:
      allow_destructive_actions: true    # Required for force_bootstrap
      force_bootstrap: true              # Force bootstrap even if NixOS detected
      force_bootstrap_kexec: true        # Use kexec to boot into installer first
      kexec:
        ssh_port: 22                     # SSH port for kexec installer (default: 22)

How it works

Kexec allows detaching from your running NixOS by loading a new kernel and initramfs directly into memory, bypassing the BIOS/UEFI boot process. This means Panix can boot into a NixOS installer image without requiring physical access to reboot the machine. Once in the installer environment, disko can repartition the disks, and the standard bootstrap flow continues - transfer the closure (kexec does not reuse the previous one), run nixos-install, and reboot into the freshly installed system.

SSH after kexec starts

After kexec starts, Panix reconnects using:

• Same SSH settings from previously used SSH method (hostname, username, identity_file)
• Port from kexec.ssh_port (default: 22 for the default kexec image)

If your custom kexec installer uses a different SSH port, configure it:

bootstrap:
  kexec:
    ssh_port: 22222  # Custom SSH port for kexec installer

Custom kexec image

You can provide a custom kexec tarball:

bootstrap:
  kexec:
    image: https://example.com/custom-kexec-<arch>.tar.gz #  Optional custom image tarball (default: https://github.com/nix-community/nixos-images/releases/latest/download/nixos-kexec-installer-noninteractive-<arch>-linux.tar.gz);
    # <arch> placeholder replaced with detected architecture
    extra_flags: # Optional flags passed to kexec (default: "")
      - "--no-sync"
    ssh_port: 22 # Optional kexec ssh port (default: 22)

Local kexec tarball

For machines without internet access or for faster deployments, download the kexec tarball locally and Panix will transfer it instead of downloading it remotely.

Download locally:

export ARCH=x86_64  # or aarch64

curl -L -o ./kexec-$ARCH.tar.gz \
  https://github.com/nix-community/nixos-images/releases/latest/download/nixos-kexec-installer-noninteractive-$ARCH-linux.tar.gz

Configure in panix.yaml:

machines:
  my-machine:
    ssh:
      hostname: 192.168.1.100
    bootstrap:
      kexec:
        image: ./kexec-<arch>.tar.gz  # <arch> placeholder replaced with detected architecture

The <arch> placeholder is automatically replaced with the detected architecture (eg. x86_64). Panix detects whether url is a local path or HTTP URL - local paths are transferred via rsync, URLs trigger remote download with curl.

Secrets Management

Deploy sensitive files and directories to your machines with proper ownership and permissions.

Transfer, ownership, and timing

The secrets phase handles the transfer of plain files/directories via rsync with configurable user/group ownership and file permissions:

fleet:
  secrets:                              # Inherited by all machines
    - local_path: ./secrets/common.key
      remote_path: /var/secrets/common.key
      permissions: 0600                # File permissions (default: 0700)
  
  flakes:
    my-flake:
      secrets:                         # APPENDED to inherited secrets
        - local_path: ./secrets/api.key
          remote_path: /var/secrets/api.key
      
      configurations:
        webserver:
          machines:
            web-01:
              secrets:                  # Further appended secrets
                - local_path: ./secrets/web-01/cert.pem
                  remote_path: /etc/ssl/cert.pem
                  uid: 1000            # User ID (default: SSH user's uid)
                  gid: 1000            # Group ID (default: SSH user's gid)
                  permissions: 0644

Key features:

• Inheritance: Secrets defined at fleet/flake/configuration levels accumulate down to machines
• Ownership control: Optionaly set uid and gid for remote file ownership. Default are the SSH user's uid and gid.
• Permission control: Optionaly set permissions using octal notation (e.g., 0600, 0644). Default is 0700.
• Directory support: Transfer entire directories by pointing local_path to a directory
• Bootstrap awareness: Secrets are transferred to the correct path whether targeting a running NixOS system or a bootstrapped machine. During bootstrap, the target root is mounted at /mnt, so Panix automatically prefixes paths (e.g., /var/secrets/key becomes /mnt/var/secrets/key) to place files in the correct location.

Security advantage: Secrets are transferred directly to the target machine via rsync and are never committed to the Nix store. This means you can safely deploy unencrypted secrets without them being stored in /nix/store (which is world-readable by default). This is fundamentally different from tools like agenix or sops-nix, which require secrets to be encrypted before they enter the Nix store.

When secrets are transferred:

Regular secrets are transferred during the Secrets phase (after Transfer, before Activate). Disk encryption keys have a special timing - they're transferred during Bootstrap, before disko runs, so they're available for disk encryption setup.

SSH & Connectivity

How Panix connects to your machines.

SSH Config Integration

Panix reads ~/.ssh/config. If your machine name matches a host alias:

machines:
  production-server-01:     # Uses SSH config: Host, Port, User, IdentityFile

No duplication. Your SSH config is the source of truth for connection parameters.

If you don't use SSH config or need to add/change something temporarily you can specify SSH options directly:

machines:
  my-server:
    ssh:
      hostname: 192.168.1.100
      port: 2222
      username: admin
      identity_file: ./keys/server.key
      disable_strict_key_checking: false  # Enable strict host key checking
      disable_auto_add_host_key: false    # Allow auto-adding host keys

Local Machine Detection

Deploying to the machine you're on? If the machine name matches the system hostname, Panix skips SSH entirely - executing commands directly via local shell:

# On a machine with hostname "workstation"
machines:
  workstation:              # Detected as local, no SSH

This can be overriden (if you want to prevent this, or if hostname detecton does not work for you):

flags:
  local_machine_hostname: my-local-machine

Observability & Output

Viewing, recording, and controlling deployment output.

Real-Time TUI

The stats table shows you what matters:

• architecture (for cross-compilation awareness)
• generation (for rollback context)
• NixOS version
• kernel version

Click and navigate (left/right keys) to any machine to filter build logs. This also works for phase status.

Keybinds

Key	Action
r	Retry failed phases
ctrl+r	Restart entire workflow (this does not reread the yaml config)
m	Toggle logs fullscreen (make any build logs label or command output in build logs fullscrean for easier reading)
ctrl-c	Copy active build logs label or command output to clipboard
c	Toggle labels between descriptions and raw commands
h	Toggle to include inspect and secrets phases in the build logs
a	Show only active/errored build logs
left/right	Navigate between stats table or phase status entrys
mouse click	Select and entry from stats table, phase status, build logs command label or command output
mouse scroll/up/down	Allows scrolling main view or an inner view when selected (eg. command output)
s	Take a snapshot of current workflow state
q	Quit

Snapshots

Snapshots capture the full state of a deployment workflow at a point in time and save it to a JSON file. They include machine states, phase results, command outputs, inspect metadata, and timing information.

Why snapshots? When a deployment fails, a snapshot preserves the exact state for later inspection, debugging, sharing with others or to give them for context to LLMs. Instead of scrolling through TUI logs or parsing terminal scrollback, you get a structured record of everything that happened.

An example can be found in examples.

Taking snapshots

Three ways to take a snapshot:

Method	Description
Press s in TUI	Manual snapshot at any time
--snapshot.on-retry	Automatic snapshot before retrying failed phases
--snapshot.on-exit	Automatic snapshot when exiting TUI

Snapshot files are written to --snapshot.dir (default: current directory) with the naming format:

panix-snapshot-<start_epoch>-<snapshot_epoch>-<reason>.json

Where <reason> is manual, retry, or exit.

Viewing snapshots

Replay any snapshot in the TUI:

panix snapshot --path panix-snapshot-1776379281-1776379290-manual.json

This opens the familiar TUI view with all phase statuses, build logs, command outputs, and machine stats frozen at the time the snapshot was taken. Phases and commands that were running at the time of capture will also appear as running (loading spinners) in TUI replay.

ℹ️ Note
`r` (retry) and `ctrl+r` (restart) keybinds are disabled in snapshot view since the workflow is not running.

Configuration

flags:
  snapshot:
    dir: .                    # Directory to save snapshots
    on_retry: true            # Take snapshot before retry
    on_exit: true             # Take snapshot on exit

Or via CLI flags:

panix deploy --snapshot.dir=./snapshots --snapshot.on-retry --snapshot.on-exit

Maximizing Visibility for Large Fleets

When deploying to many machines, maximize visibility with:

• Show only active/errored build logs: a keybind or --tui.show-active-only flag
• Reduce build log viewport height: --tui.command-output-max-height=N (default: 8)

Example for minimal build logs, maximum space for machine stats:

panix deploy --tui.show-active-only --tui.command-output-max-height=2

Or in config:

flags:
  tui:
    show_active_only: true
    command_output_max_height: 2

Output Modes, Logging, and CI/CD

Panix supports three output modes via the --output flag:

Mode	Description
tui	Interactive TUI with real-time visibility (default, requires TTY)
console	Human-readable log output to stdout (auto-selected when no TTY present)
json	JSON-structured log output to stdout

# Interactive TUI (default)
panix deploy

# Human-readable output (no TUI)
panix deploy --output console

# JSON output (for piping to tools)
panix deploy --output json

# JSON output with file logging
panix deploy --output json --log

When no TTY is present, --output console is automatically selected since the TUI cannot render without a terminal.

File logging is enabled with --log and writes structured JSON to a file with an epoch timestamp in the name (e.g. panix.1746565600.log):

panix deploy --log                        # Log to panix.<epoch>.log
panix deploy --log --log-file deploy.log  # Log to deploy.<epoch>.log

Console/JSON output includes:

• Structured fields: xpath, phase, description, command, duration, status, output
• command_start / command_end events per command with duration in seconds
• phase_start / phase_end events per phase
• workflow_end with final state of all machines (status, phase, total duration, error)
• Failed events are logged at ERROR level, successful ones at INFO

Example:

...
{"level":"info","xpath":"fleet/infrastructure/personal-workstation/personal-workstation","phase":"activate","description":"activate","command":"ssh -q -o StrictHostKeyChecking=accept-new personal-workstation /nix/store/pari6y3138wfwhc5ayyhs4m28h7qyhv2-nixos-system-personal-workstation-26.05.20260405.68d8aa3/bin/switch-to-configuration switch","event":"command_start","status_running":"activating configuration","time":"2026-04-08T23:58:59+02:00","message":"command started"}
{"level":"info","xpath":"fleet/infrastructure/personal-workstation/personal-workstation","phase":"activate","description":"activate","command":"ssh -q -o StrictHostKeyChecking=accept-new personal-workstation /nix/store/pari6y3138wfwhc5ayyhs4m28h7qyhv2-nixos-system-personal-workstation-26.05.20260405.68d8aa3/bin/switch-to-configuration switch","event":"command_end","duration":1.941539447,"output":"Checking switch inhibitors... done\nInstalling Lanzaboote to \"/boot\"...\nCollecting garbage...\nSuccessfully installed Lanzaboote.\nactivating the configuration...\nsetting up /etc...\nreloading user units for krumpy-miha...\nreloading user units for root...\nrestarting sysinit-reactivation.target\nthe following new units were started: NetworkManager-dispatcher.service","status":"success","time":"2026-04-08T23:59:01+02:00","message":"command finished"}
{"level":"info","phase":"activate","xpath":"fleet/infrastructure/personal-workstation/personal-workstation","event":"phase_end","duration":2.074027147,"status":"success","time":"2026-04-08T23:59:01+02:00","message":"Finished activate of fleet/infrastructure/personal-workstation/personal-workstation"}
{"level":"error","event":"workflow_end","machines":{"fleet/infrastructure/personal-workstation/fake":{"status":"failed","phase":"inspect","duration":0.065123731,"error":"reachability check failed: regular SSH is unreachable"},"fleet/infrastructure/personal-workstation/personal-workstation":{"status":"done","phase":"activate","duration":6.746481573}},"status":"failed","error":"one or more machines failed","time":"2026-04-08T23:59:43+02:00","message":"workflow completed"}

There are 2 additional failure modes:

# Fail immediately on first error
panix deploy --require-all-success

# In TUI mode (console mode does this automatically), you can force it exit when workflow finishes (by default it stays open)
panix deploy --exit-on-complete

Deployment Options

Fine-tuning how builds and deployments run.

Build Modes (Local vs Remote)

Panix supports two build modes: local (default) and remote. This controls where the Nix closure is built.

Local mode (default)

Builds run on the machine executing Panix:

nix build --no-link --print-out-paths <flake-url>#<installable>

The resulting closure is then transferred to each target machine via nix copy.

Remote mode

Builds run on the first machine of the configuration via the Nix remote store protocol:

nix build --eval-store auto --store ssh-ng://<first-machine> --option builders "" --no-link --print-out-paths <flake-url>#<installable>

This is useful when the target machine has more resources (CPU/RAM) than the deployment machine, or when deployment and target machine are not same architecture.

For configurations with multiple machines, the closure is built on the first machine, then copied from the first machine to the remaining machines via nix copy --from ssh-ng://<first-machine> --to ssh-ng://<other-machine>. Single-machine configurations in remote mode skip the transfer phase entirely since the closure is already on the target.

Requirements:

• The first machine must be remote (not the local machine running Panix)
• The first machine must have Nix installed and the flake inputs available

Configuration:

fleet:
  flakes:
    my-flake:
      url: <url>
      configurations:
        my-server:
          nix:
            build_mode: remote    # Build on the first machine instead of locally
          machines:
            builder-and-target:              # First machine = remote builder and target
              ssh:
                hostname: 192.168.1.100
            target-2:               # Closure copied from builder to target-2
              ssh:
                hostname: 192.168.1.101

build_mode can be set at fleet, flake, or configuration level (inherited like other nix options). It cannot be set at machine level, as a configuration's machines share the same build mode (since build is a configuration level phase).

Advanced

For more advanced remote/distributed builds, you can use additional nix command flags or set it up in your NixOS config.

Some documentation is available at:

• https://wiki.nixos.org/wiki/Distributed_build
• https://nix.dev/manual/nix/latest/advanced-topics/distributed-builds
• https://nix.dev/tutorials/nixos/distributed-builds-setup.html
• https://docs.nixbuild.net/remote-builds/
• https://docs.nixbuild.net/use-as-substituter/

Nix Command Flags

Pass additional flags to nix commands (nix build, nix copy, nixos-install) through configuration:

fleet:
  nix:
    extra_flags: ["--option", "sandbox", "false"]  # Applied to both build and copy
  
  flakes:
    my-flake:
      configurations:
        webserver:
          nix:
            build_flags: ["--max-jobs", "8"]       # nix build only
            copy_flags: ["--compress"]             # nix copy only
            extra_flags: []                        # Inherits + appends from parent
          
          machines:
            web-01:
              nix:
                copy_flags: ["--compress"]         # Inherits + appends from parent
                nixos_install_flags: ["--no-bootloader"]  # nixos-install only

Inheritance: Flags accumulate down the hierarchy (fleet → flake → configuration → machine). Slices are appended, not replaced.

Scope matters:

• build_flags and extra_flags for nix build should be set at configuration level (build runs once per configuration)
• copy_flags for nix copy can be set at machine level (transfer runs per machine)
• nixos_install_flags for nixos-install can be set at machine level (bootstrap runs per machine)

Dry Run

# Dry run: show what would happen
panix --dry-run

# Dry run with real status queries (only inspects machines so that it presents what would happen based on machines state)
panix --dry-run-with-inspect

---

Installation

Run directly:

nix run github:mihakrumpestar/panix -- deploy

Add to your flake:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";

    panix.url = "github:mihakrumpestar/panix";
  };

  outputs = { self, nixpkgs, panix, ... }@inputs: {
    nixosConfigurations.my-server = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [
        ./configuration.nix
        {
          environment.systemPackages = [
            panix.packages.${system}.default
          ];
        }
      ];
    };
  };
}

---

Quick Start

Note: Remote requires SSH key authentication (key file must be without password, unless you are using an SSH agent). Password authentication is not supported.

1. Remote

Boot into NixOS installer, any Linux live ISO (note that it might be missing packages needed to start kexec) or your already provisioned NixOS.

2. SSH Authentication

If you have only password auth, create and add a temporary key to remote with the following commands:

# On remote (set password for root user)
sudo passwd

export REMOTE=<host>

# Generate key pair
ssh-keygen -t ed25519 -f ./temp_key -C "temporary_deployment_key" -N ""

# Copy key to remote (with disabled SSH agent to prevent trying to auth with keys in agent)
SSH_AUTH_SOCK="" ssh-copy-id -i ./temp_key.pub -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@$REMOTE

You now may test the login with:

ssh -i ./temp_key -o IdentitiesOnly=yes root@$REMOTE

Now you can get (for example) the hardware config:

nixos-generate-config --no-filesystems --show-hardware-config

# or on non-NixOS installs
nix-shell -p nixos-install-tools --command "nixos-generate-config --no-filesystems --show-hardware-config"

You can also create an encryption password:

# `-n` prevents adding newline, which is a problem for password enrolnment
echo -n "test" > /tmp/disko-encryption-password.txt

3. Create panix.yml

# yaml-language-server: $schema=https://raw.githubusercontent.com/mihakrumpestar/panix/main/gen/panix-schema.yaml

fleet:
  flakes:
    # An arbitrary name for your flake
    my-config:
      url: path:./my-nixos-flake # Path to your NixOS flake or Git repo link
      configurations:
        # Matches "my-server" in nixosConfigurations
        my-server:
          machines:
            # Matches "my-server" in SSH config
            my-server:
              # Or if it is not in SSH config, specify SSH parameters
              ssh:
                hostname: my-hostname

4. Deploy

panix deploy

---

Configuration Reference

CLI

> panix --help
Usage: panix <command> [flags]

Universal NixOS Deployment Tool

Flags:
  -h, --help       Show context-sensitive help.
      --version    Show version ($PANIX_VERSION)

Commands:
  init [flags]
    Initialize a new panix configuration file

  schema [flags]
    Generate YAML schema for configuration files

  template [flags]
    Process templates and anchors, output the result

  eval [flags]
    Fully evaluate and validate configuration (including templating) for
    execution, output the result

  snapshot --path=STRING [flags]
    View snapshot in TUI

  inspect [flags]
    Inspect machines

  build [flags]
    Build NixOS closures

  deploy [flags]
    Do full workflow (inspect -> bootstrap -> build -> transfer -> secrets ->
    activate)

  secrets [flags]
    Deploy secrets to machines

  rollback [flags]
    Rollback to a previous generation, use optional --gen=NUMBER flag (default
    is -1)

Run "panix <command> --help" for more information on a command.

Deploy command flags

> panix deploy --help
Usage: panix deploy [flags]

Do full workflow (inspect -> bootstrap -> build -> transfer -> secrets ->
activate)

Flags:
  -h, --help                       Show context-sensitive help.
      --version                    Show version ($PANIX_VERSION)

  -c, --config="panix.yml"         Config file ($PANIX_CONFIG)
      --validate.flakes            Validate flake URLs and configuration keys
                                   ($PANIX_VALIDATE_FLAKES)
      --validate.bootstrap-secrets
                                   Validate that bootstrap disk encryption
                                   key local paths exist on disk
                                   ($PANIX_VALIDATE_BOOTSTRAP_SECRETS)
  -t, --tags=TAGS,...              Filter machines by tags (flakes,
                                   configurations and machine names are already
                                   registered as tags) ($PANIX_TAGS)
  -s, --skip-phases=SKIP-PHASES,...
                                   Declare phases to skip (not all phases can be
                                   skipped) ($PANIX_SKIP_PHASES)
      --timeout=2h                 Timeout per command (eg. '1h', '1m15s')
                                   ($PANIX_TIMEOUT)
      --activation-mode=ACTIVATION-MODE
                                   Activation mode: check, switch, boot, test,
                                   dry-activate (overrides machine specific
                                   ones) ($PANIX_ACTIVATION_MODE)
      --output="tui"               Output mode: tui, console, json
                                   ($PANIX_OUTPUT)
      --require-all-success        Abort if any task fails, primarily for CI/CD
                                   ($PANIX_REQUIRE_ALL_SUCCESS)
      --exit-on-complete           Exit TUI on completion ('retry' and
                                   'restart' are disabled in this mode)
                                   ($PANIX_EXIT_ON_COMPLETE)
      --local-machine-hostname=STRING
                                   Hostname of the machine that is local
                                   (won't use ssh to connect to it) (default:
                                   your deployment machine hostname)
                                   ($PANIX_LOCAL_MACHINE_HOSTNAME)
      --dry-run                    Show what would be done without executing
                                   ($PANIX_DRY_RUN)
      --dry-run-with-inspect       Show what would be done without
                                   executing, but with real inspect query
                                   ($PANIX_DRY_RUN_WITH_INSPECT)
  -l, --log                        Enable logging to file ($PANIX_LOG)
      --log-file="panix.log"       Log file path (epoch timestamp appended
                                   before .log) ($PANIX_LOG_FILE)
  -d, --debug                      Debug mode (enables logging) ($PANIX_DEBUG)
      --snapshot.dir="."           Directory to save snapshots
                                   ($PANIX_SNAPSHOT_DIR)
      --snapshot.on-retry          Take snapshot before retry
                                   ($PANIX_SNAPSHOT_ON_RETRY)
      --snapshot.on-exit           Take snapshot on exit
                                   ($PANIX_SNAPSHOT_ON_EXIT)
      --tui.show-all-build-logs    Show all build logs in TUI (keybind h)
                                   ($PANIX_TUI_SHOW_ALL_BUILD_LOGS)
      --tui.show-active-only       Show only running or errored logs
                                   in TUI build logs (keybind a)
                                   ($PANIX_TUI_SHOW_ACTIVE_ONLY)
      --tui.show-commands-in-labels
                                   Show raw commands instead of descriptions
                                   as labels in build logs (keybind c)
                                   ($PANIX_TUI_SHOW_COMMANDS_IN_LABELS)
      --tui.command-output-max-height=8
                                   Maximum height for command labels
                                   and outputs viewports in TUI
                                   ($PANIX_TUI_COMMAND_OUTPUT_MAX_HEIGHT)
      --profile.cpu=STRING         Path for CPU profile output (enables CPU
                                   profiling) ($PANIX_PROFILE_CPU)
      --profile.mem=STRING         Path for memory profile output (enables
                                   memory profiling) ($PANIX_PROFILE_MEM)
      --profile.block=STRING       Path for block profile output (enables block
                                   profiling) ($PANIX_PROFILE_BLOCK)
      --profile.mutex=STRING       Path for mutex profile output (enables mutex
                                   profiling) ($PANIX_PROFILE_MUTEX)
      --profile.goroutine=STRING
                                   Path for goroutine profile output
                                   (enables goroutine profiling)
                                   ($PANIX_PROFILE_GOROUTINE)

YAML

For the complete schema, see gen/panix-schema.yaml.

Minimal Example

# yaml-language-server: $schema=https://raw.githubusercontent.com/mihakrumpestar/panix/main/gen/panix-schema.yaml

fleet:
  flakes:
    # An arbitrary name for your flake
    my-config:
      url: path:./my-nixos-flake # Path to your NixOS flake or Git repo link
      configurations:
        # Matches "my-server" in nixosConfigurations
        my-server:
          machines:
            # Matches "my-server" in SSH config
            my-server:
              # Or if it is not in SSH config, specify SSH parameters
              ssh:
                hostname: my-hostname

Full Example

# yaml-language-server: $schema=https://raw.githubusercontent.com/mihakrumpestar/panix/main/gen/panix-schema.yaml

flags: # Listed are default values, all also overridable using CLI arguments
  activation_mode: switch              # Activation mode: check, switch, boot, test, dry-activate (overrides machine specific ones)
  tags: []                             # Filter machines by tags (flakes, configs, machine names are auto-registered as tags)
  timeout: 2h                          # Workflow timeout (e.g. '1h', '1m15s')
  exit_on_complete: false              # Exit TUI on completion (disables retry/restart)
  require_all_success: false           # Abort if any task fails (for CI/CD)
  skip_phases: []                      # Phases to skip (not all phases can be skipped)
  local_machine_hostname: my-laptop    # Override which machine is considered local based on machine name (no SSH)
  dry_run: false                       # Show what would happen without executing
  dry_run_with_inspect: false          # Dry run but with real inspect queries
  output: tui                          # Output mode: tui (interactive), console (human-readable), json (machine-readable)
  rollback_generation: -1              # 0=current generation, -N=Nth before current, N=specific generation

  tui:
    show_all_build_logs: false         # Show inspect/secrets phases in build logs (keybind h)
    show_active_only: false            # Show only running/errored logs (keybind a)
    show_commands_in_labels: false     # Show raw commands instead of descriptions (keybind c)
    command_output_max_height: 8       # Max height for command output viewports
  logging:
    log: false                         # Enable logging to file (epoch timestamp appended before .log)
    log_file: panix.log                # Log file path
    debug: false                       # Enable debug output (enables logging)
  snapshot:
    dir: .                             # Directory to save snapshots
    on_retry: false                    # Take snapshot before retry
    on_exit: false                     # Take snapshot on exit
  profile:
    cpu: ""                            # Path for CPU profile output (enables CPU profiling)
    mem: ""                            # Path for memory profile output (enables memory profiling)
    block: ""                          # Path for block profile output (enables block profiling)
    mutex: ""                          # Path for mutex profile output (enables mutex profiling)
    goroutine: ""                      # Path for goroutine profile output (enables goroutine profiling)

fleet:
  activation_mode: switch              # Default activation mode for fleet
  disabled: false                      # Disable this entire fleet
  tags: [production]                   # Tags inherited by all descendants
  hardware_config_path: ./hardware     # Path for hardware config generation
  sudo_program: doas                  # Override sudo program (default: sudo)
  nix:                                 # Nix command flags inherited by all descendants
    build_mode: local                  # Build mode: local (default) or remote
    extra_flags: []                    # Flags for both nix build and nix copy
    build_flags: []                    # Flags for nix build only
    copy_flags: []                     # Flags for nix copy only
    nixos_install_flags: []            # Flags for nixos-install only
  ssh:                                 # SSH config inherited by all machines (machine-level overrides)
    hostname: ""                       # SSH hostname or IP address
    port: 22                           # SSH port number
    username: root                     # SSH username
    identity_file: ./keys/default.key  # Path to SSH private key
    disable_strict_key_checking: false  # Enable strict host key checking
    disable_auto_add_host_key: false     # Disable auto-adding host keys on first connection
    extra_flags: []                    # Extra flags passed to ssh (e.g. '-o', 'StrictHostKeyChecking=no')
  secrets:                             # Secrets transferred to all machines
    - local_path: ./secrets/common.key
      remote_path: /var/secrets/common.key
      uid: 0                           # User ID for remote file
      gid: 0                           # Group ID for remote file
      permissions: 0600                # File permissions (default: 0700)
  
  flakes:
    infrastructure:
      url: path:../infra-flake         # Flake path or URL (e.g. 'github:...', 'git+ssh://...')
      activation_mode: switch           # Activation mode for this flake
      disabled: false                  # Disable this flake
      tags: [critical]                 # Additional tags (accumulated: [production, critical])
      hardware_config_path: ./hw-config
      sudo_program: sudo
      nix:
        build_mode: local               # Build mode for this flake
        extra_flags: []
        build_flags: []
        copy_flags: []
        nixos_install_flags: []
      ssh:                             # SSH config for all machines in this flake
        hostname: ""
        port: 22
        username: admin
        identity_file: ./keys/infra.key
        disable_strict_key_checking: false
        disable_auto_add_host_key: false
        known_hosts_file: ""            # Path to known_hosts file (empty = default ~/.ssh/known_hosts)
        extra_flags: []
      secrets:                         # APPENDED to inherited secrets
        - local_path: ./secrets/infra.key
          remote_path: /var/secrets/infra.key
          uid: 0
          gid: 0
          permissions: 0600
      bootstrap:                       # Bootstrap config inherited by machines
        ssh:                           # Bootstrap SSH (used during initial provisioning)
          hostname: ""
          port: 22
          username: root
          identity_file: ./keys/bootstrap.key
          disable_strict_key_checking: false  # Default: false for bootstrap SSH
          disable_auto_add_host_key: false    # Default: false for bootstrap SSH
          known_hosts_file: ""              # Empty = auto-generated temp file during bootstrap
        disable_disko: false           # Disable disko tool build/transfer/execution
        kexec:                         # Kexec configuration for non-NixOS machines
          image: ""                    # Custom kexec tarball image (default: nix-community image)
          extra_flags: []              # Extra flags for kexec (e.g. '--no-sync')
          ssh_port: 22                 # SSH port for kexec installer (default: 22)
        disk_encryption_keys:          # Transferred BEFORE disko runs
          - local_path: ./secrets/luks.key
            remote_path: /tmp/luks-key
            uid: 0
            gid: 0
            permissions: 0700
        allow_destructive_actions: false  # Required for force_bootstrap options
        force_bootstrap: false         # Force bootstrap even if already NixOS
        force_bootstrap_kexec: false   # Force kexec even if in NixOS installer (requires force_bootstrap)
        disable_automatic_reboot: false  # Disable auto-reboot after nixos-install
        post_bootstrap_hooks: []       # Commands after disko partitioning
        post_bootstrap_install_hooks: []  # Commands after nixos-install, before reboot
        post_bootstrap_provisioned_hooks: []  # Commands after reboot (uses regular SSH)
      
      configurations:
        webserver:
          activation_mode: switch
          disabled: false
          tags: [web]                  # Accumulated: [production, critical, web]
          flake_output: nixosConfigurations.webserver.config.system.build.toplevel  # Override flake output
          hardware_config_path: ./hardware
          sudo_program: sudo
          nix:                         # Nix flags for this configuration
            build_mode: local           # Build mode for this configuration
            extra_flags: []            # Inherits + appends from parent
            build_flags: ["--max-jobs", "4"]  # Flags for nix build
            copy_flags: []             # Flags for nix copy
            nixos_install_flags: []    # Flags for nixos-install
          ssh:
            hostname: ""
            port: 22
            username: root
            identity_file: ./keys/web.key
            disable_strict_key_checking: false
            disable_auto_add_host_key: false
            known_hosts_file: ""
            extra_flags: []
          secrets:
            - local_path: ./secrets/web.key
              remote_path: /var/secrets/web.key
              uid: 0
              gid: 0
              permissions: 0600
          bootstrap:
            ssh:
              hostname: ""
              port: 22
              username: root
              identity_file: ./keys/web-bootstrap.key
              disable_strict_key_checking: false
              disable_auto_add_host_key: false
              known_hosts_file: ""
              extra_flags: []
            kexec:
              image: ""
              extra_flags: []
              ssh_port: 22
            disk_encryption_keys: []
            allow_destructive_actions: false
            force_bootstrap: false
            force_bootstrap_kexec: false
            disable_automatic_reboot: false
            post_bootstrap_hooks: []
            post_bootstrap_install_hooks: []
            post_bootstrap_provisioned_hooks: []
          
          machines:
            web-01:
              activation_mode: switch
              disabled: false
              tags: [web-01]           # Accumulated: [production, critical, web, web-01]
              hardware_config_path: ./hardware/web-01
              sudo_program: sudo
              ssh:
                hostname: 10.0.0.1
                port: 22
                username: root
                identity_file: ./keys/web-01.key
                disable_strict_key_checking: false
                disable_auto_add_host_key: false
                known_hosts_file: ""
                extra_flags: []
              secrets:
                - local_path: ./secrets/web-01.key
                  remote_path: /var/secrets/web-01.key
                  uid: 0
                  gid: 0
                  permissions: 0600
              bootstrap:
                ssh:
                  hostname: 10.0.0.1
                  port: 22
                  username: root
                  identity_file: ./keys/web-01-bootstrap.key
                  disable_strict_key_checking: false
                  disable_auto_add_host_key: false
                  known_hosts_file: ""
                  extra_flags: []
                kexec:
                  image: ""
                  extra_flags: []
                  ssh_port: 22
                disk_encryption_keys: []
                allow_destructive_actions: false
                force_bootstrap: false
                force_bootstrap_kexec: false
                disable_automatic_reboot: false
                post_bootstrap_hooks: []
                post_bootstrap_install_hooks: []
                post_bootstrap_provisioned_hooks: []
            
            web-02:                    # Minimal machine entry
              ssh:
                hostname: web-02.example.com
        
        database:
          machines:
            db-01:
              activation_mode: switch
              ssh:
                hostname: 10.0.1.50
              bootstrap:
                ssh:
                  hostname: 10.0.1.50
                  identity_file: ./keys/bootstrap.key
                disk_encryption_keys:
                  - local_path: ./secrets/db/luks.key
                    remote_path: /tmp/luks-key
                post_bootstrap_hooks:
                  - systemd-cryptenroll --tpm2-device=auto /dev/sda2
    
    monitoring:
      url: github:myorg/monitoring#main
      configurations:
        prometheus:
          machines:
            prom-01:

The one used for testing to deploy infrastructure is at examples/panix.deploy.yml.

---

Requirements & Caveats

• Nix: Panix uses nix that it finds in PATH, it also uses commands like uname, id, echo, cat, readlink, curl and tar
• rsync: Required on both local and remote for file transfers (included in kexec images)
• kexec memory: Minimum 1GB RAM without swap for kexec bootstrap
• Nix store location: Panix expects Nix store to be in standard location
• Nix store locking: Nix does not allow writing to store by more than one at a time, so some builds may have a waiting for store lock warning for a brief time until the lock is lifted
• ssh key auth: only ssh key auth is supported, no password auth
• flake only: only flakes are supported

---

Testing

Unit Tests

task go:test

E2E Tests

End-to-end tests verify the full deployment pipeline against real QEMU VMs, covering both the NixOS ISO boot and the kexec boot, for both local and remote build modes, followed by a re-deploy to the installed systems.

Prerequisites: KVM (/dev/kvm), QEMU, Nix, cdrtools

task go:test:e2e

Test scope selection:

task go:test:e2e -- --test=local    # Only local build mode (2 VMs)
task go:test:e2e -- --test=remote   # Only remote build mode (2 VMs)
task go:test:e2e -- --test=both     # Both modes (4 VMs, default)

The whole test usually takes only ~1.5 min (local only), or ~2.4 min (both modes).

What it does:

1. Generates SSH keys, downloads kexec and Debian images, builds a NixOS installer ISO, preconfigures Debian image
2. Starts QEMU VMs per test scope: NixOS ISO VM + Debian/kexec VM for local, and/or the same pair for remote
3. Runs panix deploy (bootstrap) against all VMs (disko, nixos-install, reboot). Remote mode builds on the first machine via --store ssh-ng://
4. Runs panix deploy (re-deploy) against all VMs (switch-to-configuration)
5. Verifies NixOS installation on all VMs via SSH

What gets cached (tests/e2e/.cache/): SSH keys, kexec tarball, Debian image (with rsync pre-baked), NixOS installer ISO, disk images. Reuse across runs avoids redundant downloads/builds.

Logs (tests/e2e/log/): Recreated each run. Per-VM console logs with timestamps, panix logs per deploy, snapshot JSONs.

---

Custom TUI Rendering Engine

Panix uses a custom-built terminal rendering engine zeroterm (the pkg/tui and pkg/buffer packages) instead of existing TUI frameworks. The engine is designed for speed and near zero-allocation rendering.

Full-pipeline benchmarks (render + diff, composite layout with viewports, tree, and table at terminal size of 200×50) against Bubble Tea and cview:

Feature	Ours	Bubbletea +181×	Cview +496×
Pipeline_EveryFrameUpdate	24.7 µs (11.4 KB, 178 allocs)	1.3 ms (222.5 KB, 4382 allocs) +54×	2.5 ms (1.1 MB, 28140 allocs) +100×
Pipeline_NoChange	2.8 µs (0 B, 0 allocs)	668.5 µs (173.0 KB, 3535 allocs) +236×	2.1 ms (860.0 KB, 22535 allocs) +748×
Pipeline_QuarterFrameUpdate	2.8 µs (0 B, 0 allocs)	732.4 µs (167.2 KB, 2959 allocs) +257×	1.8 ms (818.4 KB, 20474 allocs) +642×

goos: linux goarch: amd64 cpu: AMD Ryzen 9 5900HX with Radeon Graphics, benchtime: 1s, ran: 2026-05-15

Benchmark source code

How it works

• Contiguous line buffer (LinesBuf): all lines stored in one []byte with []int offsets. Line(i) is zero-copy. Bulk ops (AppendFrom, WritePaddedView) copy visible regions in a single append. Pooled via sync.Pool, reset with buf[:0] retaining capacity.
• In-place styling (AppendStyledLine/AppendStyledPad): writes prefix + content + reset directly into the caller's buffer, no intermediate slices or per-cell allocations. Pre-rendered ANSI prefixes and 1000-space padding pool.
• Double-buffered frame diffing: two LinesBufDiff buffers alternate as current/previous. Diff(prev) returns indices of changed lines via bytes.Equal. Only changed lines get terminal output (\x1b[y;1H + content + clear-to-EOL). Unchanged lines: zero output.
• Incremental caching: Table re-renders only dirty rows (changed content or selection change). Viewport appends to its contiguous padded buffer when only new lines are added, avoiding O(n) rebuild. All components cache rendered output keyed on content/version/width/selection.
• Pre-rendered everything: border bytes, connector chars, scrollbar cells, zone markers (\x1b[<id>z), ANSI prefixes. Computed once, reused every frame.
• Stack-allocated scratch: Tree prefixes use var pfxBuf [1024]byte. Table uses a single partitioned widthsBuf []int. No heap scratch allocations.
• Zero-CGO: Pure Go, no C dependencies.

Diagram on how signals and rendering are realized in Panix:

---
config:
  theme: base
  themeVariables:
    fontSize: 32px
---
sequenceDiagram
  participant W as Workflow
  participant F as Fleet
  participant T as TUI
  participant U as User

  Note over W,U: EXECUTION
  loop per machine goroutine
    W->>W: run phase sequence
    W->>F: update machine state + phase logs
    W-->>T: notifyUpdate (sync.Cond)
  end

  Note over W,U: RENDER
  loop
    T-->>T: WaitForUpdate (block)
    T->>F: Recalculate (caches + stats)
    T->>T: render header + table + flow + logs + footer
  end

  Note over W,U: INTERRUPT
  U-x T: r (retry)
  T->>W: re-execute failed phases

  U-x T: ctrl+r (restart)
  T->>W: cancel + new workflow

  U-x T: s (snapshot)
  T->>F: capture full state to JSON

  U-x T: q (quit)
  T->>F: snapshot on exit
  T->>T: render final + quit

---

Contributing

Contributions are welcome! Whether it's bug reports, feature requests, constructive criticism, or pull requests - all feedback is appreciated. See CONTRIBUTING.md.

---

License

Panix is licensed under AGPL-3.0. Packages under pkg are licensed under MIT.

For more details about licenses, see choosingalicense.com/licenses.

---

If Panix has improved your deployment workflow, consider giving it a star.