atago

README

atago tests real CLI behavior from plain YAML: commands, files, snapshots, services, and interactive terminals. It runs your actual binary — in any language — and asserts what a user observes. No test code, no shell DSL.

Documentation: https://nao1215.github.io/atago/

Try it in 30 seconds

No install, no fictional binary — if you have Go, paste this. It records a real run of a command you already have, then replays it as a test:

go run github.com/nao1215/atago@latest record --out demo.atago.yaml -- git --version
go run github.com/nao1215/atago@latest run demo.atago.yaml

.

PASSED  1 scenario: 1 passed, 0 failed, 0 errored, 0 skipped

record runs git --version once and writes a spec from what it observed — the exit code, the version line on stdout, an empty stderr. run replays it. Open demo.atago.yaml and you have a real test you can tighten, not YAML you wrote from scratch. (Swap git --version for any command you have: go version, jq --version, ls -la.)

Then point it at your own tool:

atago record --out mytool.atago.yaml -- mytool convert input.txt  # turn a real run into a spec
atago run mytool.atago.yaml                                       # replay it as a test
atago run --report junit specs/                                   # or run a whole suite in CI

Why atago?

Pick the tool that owns your layer:

You are testing	Use
An HTTP/gRPC API server — scenario-based API testing	runn
A whole platform — integration suites across HTTP, gRPC, Kafka, databases, and more	venom
Shell functions and scripts — BDD-style unit tests	ShellSpec
Bash scripts — TAP-style tests	Bats
A CLI product — exit codes, output, generated files, snapshots, interactive prompts and TUIs	atago

If the server or the platform is the system under test, use runn or venom. atago points the other way: the CLI is the product, and HTTP, database, SSH, gRPC, browser, and mock servers appear only as peers your CLI talks to.

Install

go install github.com/nao1215/atago@latest

On macOS, Homebrew works too:

brew install --cask nao1215/tap/atago

On Arch Linux, install the atago-bin package from the AUR:

yay -S atago-bin   # or: paru -S atago-bin

The release page contains prebuilt binary archives for Linux, macOS, and Windows (amd64/arm64; .tar.gz, or .zip on Windows), plus .deb, .rpm, and .apk packages for Linux. Requires Go 1.26 or later when building from source.

Runs on Linux, macOS, and Windows (CI tests all three).

Getting started

Start from a real run

You don't write the first spec — your tool does. atago record -- <command> runs it once and generates a spec from what it observed (exit code, output, created files):

$ atago record --out mytool.atago.yaml -- mytool convert input.txt
recorded: exit 0, 2 stdout line(s), 1 file(s) created
wrote mytool.atago.yaml
$ atago run mytool.atago.yaml
.

PASSED  1 scenario: 1 passed, 0 failed, 0 errored, 0 skipped (12ms)

Interactive tools record too: atago record --pty -- <command> runs it in a real terminal, lets you drive one session by hand, and writes a pty: step that replays your keystrokes as expect/send pairs. It works on Linux, macOS, and Windows (a ConPTY); on POSIX a password prompt becomes an ${env:...} placeholder automatically, while on Windows — where a ConPTY exposes no echo state — you convert a secret send to ${env:...} by hand. A --pty session is bounded by --timeout (default 30s): if the program never exits, atago kills it, writes whatever was captured, and fails instead of hanging forever:

$ atago record --pty --out wizard.atago.yaml -- mytool init

Prefer a blank template? atago init scaffolds one. Either way, the shape is always the same: declare a command, run it, assert on what it produced.

1. Check exit code, stdout, and stderr

version: "1"
suite:
  name: example
scenarios:
  - name: echo greets the world
    steps:
      - run:
          shell: true            # portable: echo is a shell builtin on Windows
          command: echo "hello atago"
      - assert:
          exit_code: 0
          stdout:
            contains: atago
          stderr:
            empty: true

atago run accepts spec files and directories (searched recursively for *.atago.yaml; the *.atago.yml spelling is accepted too). Each scenario runs in its own temporary directory, and progress streams as a dot per scenario (. pass, F fail, E error, s skip):

$ atago run ./specs
...............................................

PASSED  47 scenarios: 47 passed, 0 failed, 0 errored, 0 skipped (1.2s)

Scenarios run concurrently by default (--parallel N, defaulting to your CPU count; set --parallel 1 to serialize). Workdirs are isolated, but the host network is shared — so if two scenarios each start a background service:, give them distinct ports, or one scenario's requests can reach the other's server.

When a check fails, atago prints exactly what was expected and what happened; multi-line mismatches render a colorized unified diff:

FAILED: demo / greeting matches its golden

Step:
  assert stdout snapshot

Diff (-expected +actual):
  --- snapshot (golden)
  +++ actual
  @@ -1,3 +1,3 @@
   hello
  -WORLD
  +world
   bye

Hint:
  stdout did not match snapshot "snaps/greeting.txt" (update with --update-snapshots if intended)

2. Check generated files and snapshots

fixture: writes input files into the isolated workdir; file:/dir: assertions check what the command produced, and snapshot: pins output to a committed golden file (volatile details like temp paths, UUIDs, and timestamps are normalized). A fixture's source is one of content: (inline text), base64: (inline bytes), from: (copy an existing file), or symlink: (link to a target):

scenarios:
  - name: the generator writes the expected files
    steps:
      - run:
          command: mytool generate --out site
      - assert:
          file:
            path: site/index.html
            contains:
              - "<html"
      - assert:
          stdout:
            snapshot: snapshots/generate.txt   # record/refresh with `atago snapshot update`

See files_and_fixtures, snapshot, and dir_tree for whole-tree golden manifests.

3. Drive interactive prompts and TUIs

A pty step runs the command in a real pseudo-terminal and drives it with a declarative expect/send session — wizards, REPLs, and TTY-detection branches, no expect(1) scripting:

scenarios:
  - name: the init wizard completes
    steps:
      - pty:
          command: mytool init
          session:
            - expect: "Project name:"
            - send: "demo\n"
            - expect: "created demo/"
      - assert:
          exit_code: 0

Named keys (send: {key: enter}) and asserts on the RENDERED terminal screen cover full TUIs — see pty, pty_screen, and the cross-platform pty_portable. pty steps and atago record --pty run on Linux, macOS, and Windows (where they drive a ConPTY pseudo-console); only signal: stays POSIX-only. The pty/pty_screen examples skip on Windows because their inner commands ([ -t 0 ], cat -v, a SIGINT trap) are POSIX-specific, not because the pty mechanism is.

When your CLI talks to a server

The same YAML also drives HTTP, database, SSH, gRPC, headless-browser, and offline mock-server peers — as dependencies of the CLI under test. atago init --template <name> scaffolds each:

$ atago init --list-templates
browser   drive a headless Chrome; assert page content (needs Chrome on PATH)
cli       run a command; assert exit code/stdout/stderr (runs as-is)
db        run SQL; assert on rows via bundled SQLite (runs as-is)
grpc      call a unary gRPC method via server reflection (edit target first)
http      call an HTTP API; assert status and JSON body (edit base_url first)
mock      stub an HTTP API offline and assert what the client sent (needs curl on PATH)
services  test against a background server: readiness, retry, teardown (runs as-is)
ssh       run a command on a remote host over SSH (edit host/user first)

Examples

Every feature has a commented, runnable spec under examples/, tested in CI on Linux, macOS, and Windows. The cookbook collects 50+ copyable recipes for common jobs — converting images, driving prompts and TUIs, simulating API failures offline, proving idempotency — and indexes every spec by task and by feature.

Selection flags compose with any spec: --filter NAME (repeatable, and comma-separated for OR — --filter a,b or --filter a --filter b runs scenarios whose name contains a or b), --tag T and --skip-tag T (tags match exactly, not by substring — atago list shows the available tags), --parallel N, --fail-fast, and --rerun-failed. While authoring, --verbose traces every command, capture, and assertion verdict — for passing scenarios too.

Use it in CI

Real E2E suites flake (timing, ports, external tools). --retry-failed N re-runs failed scenarios in a fresh workdir and reports recovered ones as flaky — green for the exit code, but loud in every report format; silent retries are explicitly a non-goal. --repeat N does the opposite job: run each scenario N times to detect flakiness before it reaches CI.

atago run --ci --retry-failed 2 ./specs          # keep CI green, report instability loudly
atago run --repeat 20 --filter "race prone" ./specs   # flake detection

setup-atago installs a released binary:

name: behavior-specs
on: [push, pull_request]
jobs:
  atago:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: nao1215/setup-atago@v0
      - run: atago run --ci --report gha ./specs

On GitLab CI (or any CI that starts from a container image), use the published GHCR image:

image: ghcr.io/nao1215/atago:latest

stages: [test]

behavior-specs:
  stage: test
  script:
    - atago run --ci --report junit ./specs > junit.xml
  artifacts:
    when: always
    reports:
      junit: junit.xml

The image contains atago and ca-certificates; if your scenarios drive git, jq, a browser, or your own CLI binary, build FROM ghcr.io/nao1215/atago:latest and layer those tools on top.

• --report json|junit|gha|tap picks the report format; the JSON shape is stable and versioned.
• --ci enables deterministic, color-free output. It also turns an empty selection into a hard error: a --filter/--tag/--skip-tag that matches no scenario fails the run (exit 3) instead of passing an empty suite, so a typo cannot silently disable your specs. Without --ci the same case is a warning that still exits 0.
• --artifacts-dir DIR persists the exact payloads a failed assertion compared — plus, for a failed scenario, its background services' logs and each mock server's recorded requests — so a failure stays reviewable after the job ends.
• Environment variable names listed under secrets: are masked as *** in all reports and snapshots.

Review specs without running them

explain describes what a spec does, doc generates Markdown (with fixtures, expected payloads, and golden files inlined), manifest emits a stable JSON summary for tooling, and list shows scenarios, tags, and artifacts. All of them load and validate the spec first — exit code 2 on a schema error — so any of them doubles as a lint step in CI:

atago explain spec.atago.yaml
atago doc --out docs/specs.md ./specs
atago manifest ./specs
atago list ./specs

Snapshot testing

snapshot matchers compare output against committed golden files; ANSI colors, temp paths, UUIDs, timestamps, ports, and CRLF are normalized so snapshots stay stable across machines. Record or refresh them with:

atago snapshot update spec.atago.yaml

For volatile patterns the built-ins do not cover — auto-increment IDs, request identifiers, epoch times — declare spec-wide scrub: rules that rewrite each regex match to a placeholder before the compare (applied after secrets: masking):

scrub:
  - {pattern: 'id=\d+', placeholder: 'id=<ID>'}

See scrub.

Editor support (JSON Schema)

A JSON Schema lives at schema/atago.schema.json. With the YAML language server you get completion and validation as you type — step types, every matcher, and the ${workdir} / ${env:NAME} / ${name} / $${...} expansion rules. atago init and atago record already emit this header as the first line of every generated spec, so scaffolded specs get completion out of the box. To add it to an existing spec, use the absolute URL (it resolves in any project, unlike a repo-relative path):

# yaml-language-server: $schema=https://raw.githubusercontent.com/nao1215/atago/main/schema/atago.schema.json
version: "1"

Shell completion

atago completion <bash|zsh|fish|powershell> prints a completion script for your shell.

Exit codes

Code	Meaning
0	all scenarios passed
1	one or more failed
2	spec error (YAML syntax or schema/semantic validation)
3	CLI-invocation error (unknown subcommand, bad flag, or no matching spec files)
4	execution error
5	internal error
6	security policy violation

Ctrl-C/SIGTERM stops the run cleanly: in-flight processes, services, and sessions are torn down, partial results are reported, and the run exits 4.

Real CLIs tested with atago

These suites run real programs of every shape: the author's Go tools (atago tests itself) and unmodified third-party binaries — git and jq, interactive TUIs (fzf, htop), the python3 REPL, servers driven as scenario services (redis, gitea, grafana, prometheus), cloud and IaC CLIs tested offline (aws-cli, terraform, ecspresso), crypto tools (openssl, age, sops), and document/media pipelines (pandoc, ffmpeg). Most were migrated from ShellSpec. Real CLIs tested with atago lists all 40+ with specs and generated behavior docs.

The name

atago (愛宕) is named for Mount Atago in Kyoto, whose shrine enshrines a deity of fire prevention. A test runner should do the same job: catch the sparks so a project never catches fire.

Contributing

Issues and pull requests are welcome; see CONTRIBUTING.md. Contributions are not only about code: a GitHub Star also motivates development.

LICENSE

The atago project is licensed under the terms of MIT LICENSE.

Contributors ✨

Thanks goes to these wonderful people (emoji key):

CHIKAMATSU Naohiro 💻 📖

Documentation

Overview

Command atago is a YAML-based black-box behavior spec runner for CLIs, APIs, and generated artifacts. See the README for the format and CLI contract, and schema/atago.schema.json for the machine-readable spec file schema.