atago

README

atago tests real CLI behavior from plain YAML: commands, files, snapshots, services, and interactive terminals. It runs your actual binary — in any language — and asserts what a user observes. No test code, no shell DSL.

atago record --out mytool.atago.yaml -- mytool convert input.txt  # turn a real run into a spec
atago run mytool.atago.yaml                                       # replay it as a test
atago run --report junit specs/                                   # or run a whole suite in CI

Why atago?

Pick the tool that owns your layer:

You are testing	Use
An HTTP/gRPC API server — scenario-based API testing	runn
Shell functions and scripts — BDD-style unit tests	ShellSpec
Bash scripts — TAP-style tests	Bats
A CLI product — exit codes, output, generated files, snapshots, interactive prompts and TUIs	atago

If the server is the system under test, use runn. atago points the other way: the CLI is the product, and HTTP, database, SSH, gRPC, browser, and mock servers appear only as peers your CLI talks to.

Install

go install github.com/nao1215/atago@latest

On macOS, Homebrew works too:

brew install --cask nao1215/tap/atago

The release page contains prebuilt binary archives for Linux, macOS, and Windows (amd64/arm64; .tar.gz, or .zip on Windows). Requires Go 1.26 or later when building from source.

Runs on Linux, macOS, and Windows (CI tests all three).

Verifying release integrity

Every release ships supply-chain metadata so you can verify what you download:

• Signed checksums: checksums.txt is signed with cosign (keyless), producing checksums.txt.sigstore.json.
• SBOM: an SPDX Software Bill of Materials is attached to each release archive.
• Build provenance: SLSA build provenance is attested via GitHub OIDC.

cosign verify-blob \
  --bundle checksums.txt.sigstore.json \
  --certificate-identity-regexp 'https://github.com/nao1215/atago/\.github/workflows/release\.yml@refs/tags/.*' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  checksums.txt
sha256sum --check --ignore-missing checksums.txt

gh attestation verify atago_<version>_<os>_<arch>.tar.gz --repo nao1215/atago  # .zip on Windows

Getting started

Start from a real run

You don't write the first spec — your tool does. atago record -- <command> runs it once and generates a spec from what it observed (exit code, output, created files):

$ atago record --out mytool.atago.yaml -- mytool convert input.txt
recorded: exit 0, 2 stdout line(s), 1 file(s) created
wrote mytool.atago.yaml
$ atago run mytool.atago.yaml
.

PASSED  1 scenario: 1 passed, 0 failed, 0 errored, 0 skipped (12ms)

Interactive tools record too: atago record --pty -- <command> runs it in a real terminal, lets you drive one session by hand, and writes a pty: step that replays your keystrokes as expect/send pairs (POSIX-only; password prompts become ${env:...} placeholders, never literals):

$ atago record --pty --out wizard.atago.yaml -- mytool init

Prefer a blank template? atago init scaffolds one. Either way, the shape is always the same: declare a command, run it, assert on what it produced.

1. Check exit code, stdout, and stderr

version: "1"
suite:
  name: example
scenarios:
  - name: echo greets the world
    steps:
      - run:
          shell: true            # portable: echo is a shell builtin on Windows
          command: echo "hello atago"
      - assert:
          exit_code: 0
          stdout:
            contains: atago
          stderr:
            empty: true

atago run accepts spec files and directories (searched recursively for *.atago.yaml). Each scenario runs in its own temporary directory, and progress streams as a dot per scenario (. pass, F fail, E error, s skip):

$ atago run ./specs
...............................................

PASSED  47 scenarios: 47 passed, 0 failed, 0 errored, 0 skipped (1.2s)

When a check fails, atago prints exactly what was expected and what happened; multi-line mismatches render a colorized unified diff:

FAILED: demo / greeting matches its golden

Step:
  assert stdout snapshot

Diff (-expected +actual):
  --- snapshot (golden)
  +++ actual
  @@ -1,3 +1,3 @@
   hello
  -WORLD
  +world
   bye

Hint:
  stdout did not match snapshot "snaps/greeting.txt" (update with --update-snapshots if intended)

2. Check generated files and snapshots

fixture: writes input files into the isolated workdir; file:/dir: assertions check what the command produced, and snapshot: pins output to a committed golden file (volatile details like temp paths, UUIDs, and timestamps are normalized):

scenarios:
  - name: the generator writes the expected files
    steps:
      - run:
          command: mytool generate --out site
      - assert:
          file:
            path: site/index.html
            contains:
              - "<html"
      - assert:
          stdout:
            snapshot: snapshots/generate.txt   # record/refresh with `atago snapshot update`

See files_and_fixtures, snapshot, and dir_tree for whole-tree golden manifests.

3. Drive interactive prompts and TUIs

A pty step runs the command in a real pseudo-terminal and drives it with a declarative expect/send session — wizards, REPLs, and TTY-detection branches, no expect(1) scripting:

scenarios:
  - name: the init wizard completes
    steps:
      - pty:
          command: mytool init
          session:
            - expect: "Project name:"
            - send: "demo\n"
            - expect: "created demo/"
      - assert:
          exit_code: 0

Named keys (send: {key: enter}) and asserts on the RENDERED terminal screen cover full TUIs — see pty and pty_screen (POSIX-only).

When your CLI talks to a server

The same YAML also drives HTTP, database, SSH, gRPC, headless-browser, and offline mock-server peers — as dependencies of the CLI under test. atago init --template <name> scaffolds each:

$ atago init --list-templates
browser   drive a headless Chrome; assert page content (needs Chrome on PATH)
cli       run a command; assert exit code/stdout/stderr (runs as-is)
db        run SQL; assert on rows via bundled SQLite (runs as-is)
grpc      call a unary gRPC method via server reflection (edit target first)
http      call an HTTP API; assert status and JSON body (edit base_url first)
mock      stub an HTTP API offline and assert what the client sent (needs curl on PATH)
services  test against a background server: readiness, retry, teardown (runs as-is)
ssh       run a command on a remote host over SSH (edit host/user first)

Examples

Every feature has a commented, runnable spec under examples/. The examples are loaded, validated, and (where they need no external server) executed in CI on Linux, macOS, and Windows, so they cannot drift from the implementation.

Example	Shows
run_and_assert	exit code (exact, not, in: [0, 2] sets), stdout/stderr matchers (contains, equals, matches/not_matches, lists, line), multi-target asserts
shell_and_redirect	shell: true vs direct argv execution, stdout_to/stderr_to redirects
json_and_yaml	JSONPath assertions, numeric bounds (gt/lte), the yaml matcher
files_and_fixtures	input fixtures (text and base64), file and dir assertions
store_and_variables	capturing values into ${name}, ${workdir}, ${env:NAME} host-environment reads, the $${...} literal escape
teardown	cleanup steps that always run — pass or fail — sharing the scenario's variables
hermetic_env	clear_env: true starts commands from an empty environment, pass_env re-admits an allowlist of host variables, sandbox_home: true isolates HOME and per-OS config/cache dirs
timeouts	the built-in 60s default step timeout, suite.timeout, per-step overrides, and the timeout: "0" escape hatch
stdin	stdin sources: inline text, stdin: {file: ...} from a workdir file, and binary input via stdin: {base64: ...}
matrix	one template scenario expanded per parameter row
mock_server	test API-client CLIs offline: mock_servers serve canned routes, record every request, and mock: asserts what the client actually sent
pty	interactive testing in a real pseudo-terminal: expect/send sessions, named keys (send: {key: enter}), TTY-detection (POSIX-only)
pty_screen	TUI testing on the RENDERED terminal screen: vt100 emulation, row-addressed asserts, and screen snapshots (POSIX-only)
retry	polling a command until an assertion passes
snapshot	golden-file testing with normalized output
duration	bound a step's wall-clock time with duration: {lt: 2s, gte: 100ms} (use generous bounds — CI runners are slow)
dir_tree	recursive dir assertions and directory-tree snapshots: pin a generator's whole output tree with one golden manifest
changes	changes: pins the exact workdir delta of a step — which files it created, modified, and deleted, and nothing else
services	background servers: readiness probes, ready.store, teardown
signal	signal: steps deliver SIGTERM/SIGHUP/... to a managed service's process group for graceful-shutdown and reload testing (POSIX-only)
defaults	sharing shell/env/service fragments across scenarios
suite_setup	once-per-suite bootstrap: ordered setup steps, suite-wide service: steps, ${suitedir}, suite env, always-run suite teardown
select_skip_only	tags, and gating scenarios by OS, env var, or a probe command
db	SQL via the bundled SQLite driver, rows assertions, value binding
image_and_pdf	image format/dimension/similarity checks, PDF page/metadata/text checks
http	HTTP requests (json:, raw body:, form/multipart uploads, body_file), status/body assertions, token capture, retry polling, redirect assertions, body_to downloads, network allowlist
ssh	running commands on a remote host
grpc	unary gRPC calls via server reflection
browser	headless-Chrome flows and screenshots

Selection flags compose with any spec: --filter NAME, --tag T, --skip-tag T, --parallel N, --fail-fast, and --rerun-failed. While authoring, --verbose traces every command, capture, and assertion verdict — for passing scenarios too.

Use it in CI

Real E2E suites flake (timing, ports, external tools). --retry-failed N re-runs failed scenarios in a fresh workdir and reports recovered ones as flaky — green for the exit code, but loud in every report format; silent retries are explicitly a non-goal. --repeat N does the opposite job: run each scenario N times to detect flakiness before it reaches CI.

atago run --ci --retry-failed 2 ./specs          # keep CI green, report instability loudly
atago run --repeat 20 --filter "race prone" ./specs   # flake detection

setup-atago installs a released binary:

name: behavior-specs
on: [push, pull_request]
jobs:
  atago:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: nao1215/setup-atago@v0
      - run: atago run --ci --report gha ./specs

• --report json|junit|gha|tap picks the report format; the JSON shape is stable and versioned.
• --ci enables deterministic, color-free output.
• --artifacts-dir DIR persists the exact payloads a failed assertion compared, so a failure stays reviewable after the job ends.
• Environment variable names listed under secrets: are masked as *** in all reports and snapshots.

Review specs without running them

explain describes what a spec does, doc generates Markdown (with fixtures, expected payloads, and golden files inlined), manifest emits a stable JSON summary for tooling, and list shows scenarios, tags, and artifacts. All of them load and validate the spec first — exit code 2 on a schema error — so any of them doubles as a lint step in CI:

atago explain spec.atago.yaml
atago doc --out docs/specs.md ./specs
atago manifest ./specs
atago list ./specs

Snapshot testing

snapshot matchers compare output against committed golden files; ANSI colors, temp paths, UUIDs, timestamps, ports, and CRLF are normalized so snapshots stay stable across machines. Record or refresh them with:

atago snapshot update spec.atago.yaml

Editor support (JSON Schema)

A JSON Schema lives at schema/atago.schema.json. With the YAML language server you get completion and validation as you type:

# yaml-language-server: $schema=./schema/atago.schema.json
version: "1"

Shell completion

atago completion <bash|zsh|fish|powershell> prints a completion script for your shell.

Exit codes

Code	Meaning
0	all scenarios passed
1	one or more failed
2	spec error (YAML syntax or schema/semantic validation)
3	CLI-invocation error (unknown subcommand, bad flag, or no matching spec files)
4	execution error
5	internal error
6	security policy violation

Ctrl-C/SIGTERM stops the run cleanly: in-flight processes, services, and sessions are torn down, partial results are reported, and the run exits 4.

Real CLIs tested with atago

These real programs run their end-to-end suites on atago; most were migrated from ShellSpec. The generated behavior docs live in doc/e2e/.

Programs maintained by the author, starting with atago itself (test/e2e/tools/):

Tool	Feature	Specs	Docs
atago	atago tested by atago: the self-hosted specs run the real built binary in CI (make e2e).	specs	docs
gup	Updates and manages the Go command-line tools in $GOBIN.	specs	docs
sqly	Runs SQL against CSV/TSV/LTSV/JSON/Parquet/Excel/ACH/Fedwire files.	specs	docs
truss	Image transformation (convert/resize/re-encode).	specs	docs
iso8583tool	Debugs and inspects ISO 8583 payment messages.	specs	docs
jose	Signs and encrypts with JOSE.	specs	docs
career	Renders résumé PDFs from a single YAML file.	specs	docs
mimixbox	Packs many Unix commands into one BusyBox-style binary.	specs	docs
mobilepkg	Inspects Android packages for metadata and security findings.	specs	docs

Third-party programs (test/e2e/thirdparty/). atago runs each as an unmodified binary and ships only its own spec YAML — no third-party code is copied or redistributed, which every listed license permits:

Program	Feature	License	Specs	Docs
git	Version control — runs in CI on all three OSes.	GPL-2.0	specs	docs
jq	JSON processor: stdin-driven filters, --arg injection, and its documented exit-code contract (1 for -e false, 3 for a bad program, parse failures).	MIT	specs	docs
fzf	Interactive fuzzy finder driven inside a real pseudo-terminal: expect/send sessions type queries, multi-select, and abort (exit 130); --filter pins the non-TTY contract.	MIT	specs	docs
redis	Server+client pair: redis-server as a scenario service (readiness by log and by port), redis-cli round-trips (PING/SET/GET/INCR/TTL), error contracts, and a polled graceful shutdown.	RSALv2/SSPLv1	specs	docs
hugo	Scaffold + build + serve in one binary: hugo new site tree assertions, a theme-less --minify build, and hugo server bootstrapped in suite.setup then queried over HTTP.	Apache-2.0	specs	docs
openssl	Cryptography toolkit: exact digests, keygen → sign → verify (and tamper detection), encrypt/decrypt round-trips with the wrong-password failure mode, self-signed certificates.	Apache-2.0	specs	docs
sqlite3	Embedded-database shell driven as a real binary: one-shot SQL, -json/-csv output modes, .dump → .read rebuilds, .import from CSV, bad-SQL diagnostics.	Public Domain	specs	docs
caddy	Self-hosted web server, booted from an authored Caddyfile and queried over HTTP.	Apache-2.0	specs	docs
coredns	Self-hosted DNS server: an authored zone queried with real dig — authoritative answers, CNAME chasing, NXDOMAIN/REFUSED, and the health plugin over HTTP.	Apache-2.0	specs	docs
gitea	Self-hosted git service: booted with SQLite, administered via its CLI, driven over the REST API (repos, commits, issues), then cloned with real git.	MIT	specs	docs
gotify	Self-hosted notification server: app provisioning, token-authenticated pushes, and the app icon uploaded as real multipart/form-data, downloaded back, and verified as a PNG.	MIT	specs	docs
grafana	Self-hosted observability platform: health/build info, the login redirect asserted with follow_redirects: false, and a dashboard + datasource lifecycle over the REST API.	AGPL-3.0	specs	docs
mailpit	Self-hosted email testing: messages delivered over real SMTP (stock curl), then asserted via the REST API — capture, search, MIME attachments, teardown.	MIT	specs	docs
minio	Self-hosted S3-compatible object storage: full object lifecycle via mc, versioning, anonymous bucket policies, S3 XML error contract.	AGPL-3.0	specs	docs
aws-cli	The AWS CLI driven against a local MinIO S3 endpoint — a cloud CLI tested offline: bucket/object lifecycle, byte-identical round-trip, head-object JSONPath, presigned-URL fetch, missing-key error contract.	Apache-2.0	specs	docs
python3	The REPL as an interactive pty testbed: prompt detection, multi-exchange expect/send, EOF exit, traceback recovery, and TTY branching — a copy-paste template for testing your own REPL.	PSF-2.0	specs	docs
ssh-keygen	OpenSSH key generation: non-interactive and interactive-passphrase (pty) key pairs, generated-file asserts, the exact fingerprint contract, and verification-failure exit codes.	BSD	specs	docs
ffmpeg	Media pipeline: lavfi video synthesis, ffprobe stream JSON (JSONPath), frame extraction verified with image assertions (format/dimensions/pixel similarity), a webm transcode, and error contracts.	LGPL/GPL	specs	docs
pandoc	Document conversion: markdown→HTML/docx, a stdin→stdout filter (via stdin: {file:}), the JSON AST queried with JSONPath, metadata-driven standalone output, and the unknown-format error contract.	GPL-2.0+	specs	docs
terraform	The multi-exit-code IaC CLI, fully offline via the builtin terraform_data resource: init/validate/plan/apply/destroy, the plan -detailed-exitcode 0/1/2 contract, state JSON, and fmt -check exit 3.	BUSL-1.1	specs	docs
age	Modern file encryption: keygen, binary-safe encrypt/decrypt round-trips, armored (PEM) output, interactive-passphrase mode (pty), and clean failure semantics.	BSD-3-Clause	specs	docs
nats	Self-hosted messaging: request/reply through the real broker, JetStream persistence (create → publish → count → purge), the KV store, and the monitoring endpoint.	Apache-2.0	specs	docs
ntfy	Self-hosted push notifications: publish with headers, poll the JSON feed, topic isolation, and deny-all access control unlocked via the admin CLI.	Apache-2.0	specs	docs
prometheus	Self-hosted monitoring: promtool config/rule checks and rule unit tests, the query API, and a self-scrape polled with http retry.	Apache-2.0	specs	docs
pushgateway	Self-hosted metrics gateway; exercises raw text body: payloads.	Apache-2.0	specs	docs
rclone	Self-hosted file sync: copy/sync/check semantics (including the corruption failure mode), JSON listings, and rclone serve http.	MIT	specs	docs
restic	Self-hosted backup: init → backup → restore round-trip, snapshot JSON, diff, integrity check, retention, wrong-password failure mode.	BSD-2-Clause	specs	docs
transfer.sh	Self-hosted file sharing: a PNG uploaded as the raw request body (body_file), downloaded back to disk (body_to), and verified byte-for-byte; multipart uploads too.	MIT	specs	docs
webhook	Self-hosted webhook receiver driven by a config fixture.	MIT	specs	docs

The name

atago (愛宕) is named for Mount Atago in Kyoto, whose shrine enshrines a deity of fire prevention. A test runner should do the same job: catch the sparks so a project never catches fire.

Contributing

Issues and pull requests are welcome; see CONTRIBUTING.md. Contributions are not only about code: a GitHub Star also motivates development.

LICENSE

The atago project is licensed under the terms of MIT LICENSE.

Contributors ✨

Thanks goes to these wonderful people (emoji key):

CHIKAMATSU Naohiro 💻 📖

Documentation

Overview

Command atago is a YAML-based black-box behavior spec runner for CLIs, APIs, and generated artifacts. See the README for the format and CLI contract, and schema/atago.schema.json for the machine-readable spec file schema.