atago

README

atago

atago is an end-to-end test runner for command-line tools. It runs a real command, in any language, and checks what a user observes: the exit code, stdout, stderr, generated files, JSON output, and snapshots. Specs are plain YAML — no test code, no shell DSL, no embedded scripting. It also drives HTTP, database, SSH, gRPC, and headless-browser peers, and generates Markdown docs from specs.

atago init                       # write a starter example.atago.yaml
atago run example.atago.yaml     # run the spec you just created
atago run --report junit specs/  # emit a JUnit report for CI

Install

go install github.com/nao1215/atago@latest

On macOS, Homebrew works too:

brew install --cask nao1215/tap/atago

The release page contains prebuilt binary archives for Linux, macOS, and Windows (amd64/arm64; .tar.gz, or .zip on Windows). Requires Go 1.26 or later when building from source.

Runs on Linux, macOS, and Windows. CI runs the unit tests on all three, the full E2E suite on Linux and macOS, and a portable E2E subset on Windows; specs that lean on POSIX-only shell tools are the remaining gap on Windows.

Verifying release integrity

Every release ships supply-chain metadata so you can verify what you download:

• Signed checksums: checksums.txt is signed with cosign (keyless), producing checksums.txt.sigstore.json.
• SBOM: an SPDX Software Bill of Materials is attached to each release archive.
• Build provenance: SLSA build provenance is attested via GitHub OIDC.

cosign verify-blob \
  --bundle checksums.txt.sigstore.json \
  --certificate-identity-regexp 'https://github.com/nao1215/atago/\.github/workflows/release\.yml@refs/tags/.*' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  checksums.txt
sha256sum --check --ignore-missing checksums.txt

gh attestation verify atago_<version>_<os>_<arch>.tar.gz --repo nao1215/atago  # .zip on Windows

Getting started

atago init scaffolds a runnable spec. The shape is always the same: declare a command, run it, assert on what it produced.

version: "1"
suite:
  name: example
scenarios:
  - name: echo greets the world
    steps:
      - run:
          shell: true            # portable: echo is a shell builtin on Windows
          command: echo "hello atago"
      - assert:
          exit_code: 0
          stdout:
            contains: atago
          stderr:
            empty: true

atago run accepts spec files and directories (searched recursively for *.atago.yaml). Each scenario runs in its own temporary directory, and progress streams as a dot per scenario (. pass, F fail, E error, s skip):

$ atago run ./specs
...............................................

PASSED  47 scenarios: 47 passed, 0 failed, 0 errored, 0 skipped (1.2s)

When a check fails, atago prints exactly what was expected and what happened:

FAILED: demo / expect Alice but the command prints Bob

Step:
  assert stdout contains "Alice"

Command:
  echo Bob

Expected:
  stdout contains "Alice"

Actual:
  Bob

Hint:
  the substring "Alice" was not present in stdout

atago init --template <name> scaffolds a starter for the other runner families; atago init --list-templates describes each one and says whether it runs as-is or what to edit first:

$ atago init --list-templates
browser   drive a headless Chrome; assert page content (needs Chrome on PATH)
cli       run a command; assert exit code/stdout/stderr (runs as-is)
db        run SQL; assert on rows via bundled SQLite (runs as-is)
grpc      call a unary gRPC method via server reflection (edit target first)
http      call an HTTP API; assert status and JSON body (edit base_url first)
services  test against a background server: readiness, retry, teardown (runs as-is)
ssh       run a command on a remote host over SSH (edit host/user first)

Examples

Every feature has a commented, runnable spec under examples/. The examples are loaded, validated, and (where they need no external server) executed in CI on Linux, macOS, and Windows, so they cannot drift from the implementation.

Example	Shows
run_and_assert	exit code, stdout/stderr matchers (contains, equals, matches/not_matches, lists, line), multi-target asserts
shell_and_redirect	shell: true vs direct argv execution, stdout_to/stderr_to redirects
json_and_yaml	JSONPath assertions, numeric bounds (gt/lte), the yaml matcher
files_and_fixtures	input fixtures (text and base64), file and dir assertions
store_and_variables	capturing values into ${name}, ${workdir}, ${env:NAME} host-environment reads, the $${...} literal escape
teardown	cleanup steps that always run — pass or fail — sharing the scenario's variables
matrix	one template scenario expanded per parameter row
retry	polling a command until an assertion passes
snapshot	golden-file testing with normalized output
services	background servers: readiness probes, ready.store, teardown
defaults	sharing shell/env/service fragments across scenarios
select_skip_only	tags, and gating scenarios by OS, env var, or a probe command
db	SQL via the bundled SQLite driver, rows assertions, value binding
image_and_pdf	image format/dimension/similarity checks, PDF page/metadata/text checks
http	HTTP requests (json:, raw body:, form/multipart uploads, body_file), status/body assertions, token capture, retry polling, redirect assertions, body_to downloads, network allowlist
ssh	running commands on a remote host
grpc	unary gRPC calls via server reflection
browser	headless-Chrome flows and screenshots

Selection flags compose with any spec: --filter NAME, --tag T, --skip-tag T, --parallel N (default: the number of CPUs — scenarios are isolated, so runs are concurrent out of the box), --fail-fast, and --rerun-failed (rerun only what failed last time).

Use it in CI

setup-atago installs a released binary:

name: behavior-specs
on: [push, pull_request]
jobs:
  atago:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: nao1215/setup-atago@v0
      - run: atago run --ci --report gha ./specs

• --report json|junit|gha|tap picks the report format; the JSON shape is stable and versioned.
• --ci enables deterministic, color-free output.
• --artifacts-dir DIR persists the exact payloads a failed assertion compared, so a failure stays reviewable after the job ends.
• Environment variable names listed under secrets: are masked as *** in all reports and snapshots.

Review specs without running them

explain describes what a spec does, doc generates Markdown (with fixtures, expected payloads, and golden files inlined), manifest emits a stable JSON summary for tooling, and list shows scenarios, tags, and artifacts. All of them load and validate the spec first — exit code 2 on a schema error — so any of them doubles as a lint step in CI:

atago explain spec.atago.yaml
atago doc --out docs/specs.md ./specs
atago manifest ./specs
atago list ./specs

Snapshot testing

snapshot matchers compare output against committed golden files; ANSI colors, temp paths, UUIDs, timestamps, ports, and CRLF are normalized so snapshots stay stable across machines. Record or refresh them with:

atago snapshot update spec.atago.yaml

Editor support (JSON Schema)

A JSON Schema lives at schema/atago.schema.json. With the YAML language server you get completion and validation as you type:

# yaml-language-server: $schema=./schema/atago.schema.json
version: "1"

Shell completion

atago completion <bash|zsh|fish|powershell> prints a completion script for your shell.

Exit codes

Code	Meaning
0	all scenarios passed
1	one or more failed
2	spec error (YAML syntax or schema/semantic validation)
3	CLI-invocation error (unknown subcommand, bad flag, or no matching spec files)
4	execution error
5	internal error
6	security policy violation

Ctrl-C/SIGTERM stops the run cleanly: in-flight processes, services, and sessions are torn down, partial results are reported, and the run exits 4.

Real CLIs tested with atago

These real programs run their end-to-end suites on atago; most were migrated from ShellSpec. The generated behavior docs live in doc/e2e/.

Programs maintained by the author, starting with atago itself (test/e2e/tools/):

Tool	Feature	Specs	Docs
atago	atago tested by atago: the self-hosted specs run the real built binary in CI (make e2e).	specs	docs
gup	Updates and manages the Go command-line tools in $GOBIN.	specs	docs
sqly	Runs SQL against CSV/TSV/LTSV/JSON/Parquet/Excel/ACH/Fedwire files.	specs	docs
truss	Image transformation (convert/resize/re-encode).	specs	docs
iso8583tool	Debugs and inspects ISO 8583 payment messages.	specs	docs
jose	Signs and encrypts with JOSE.	specs	docs
career	Renders résumé PDFs from a single YAML file.	specs	docs
mimixbox	Packs many Unix commands into one BusyBox-style binary.	specs	docs
mobilepkg	Inspects Android packages for metadata and security findings.	specs	docs

Third-party programs (test/e2e/thirdparty/). atago runs each as an unmodified binary and ships only its own spec YAML — no third-party code is copied or redistributed, which every listed license permits:

Program	Feature	License	Specs	Docs
git	Version control — runs in CI on all three OSes.	GPL-2.0	specs	docs
caddy	Self-hosted web server, booted from an authored Caddyfile and queried over HTTP.	Apache-2.0	specs	docs
coredns	Self-hosted DNS server: an authored zone queried with real dig — authoritative answers, CNAME chasing, NXDOMAIN/REFUSED, and the health plugin over HTTP.	Apache-2.0	specs	docs
gitea	Self-hosted git service: booted with SQLite, administered via its CLI, driven over the REST API (repos, commits, issues), then cloned with real git.	MIT	specs	docs
gotify	Self-hosted notification server: app provisioning, token-authenticated pushes, and the app icon uploaded as real multipart/form-data, downloaded back, and verified as a PNG.	MIT	specs	docs
grafana	Self-hosted observability platform: health/build info, the login redirect asserted with follow_redirects: false, and a dashboard + datasource lifecycle over the REST API.	AGPL-3.0	specs	docs
mailpit	Self-hosted email testing: messages delivered over real SMTP (stock curl), then asserted via the REST API — capture, search, MIME attachments, teardown.	MIT	specs	docs
minio	Self-hosted S3-compatible object storage: full object lifecycle via mc, versioning, anonymous bucket policies, S3 XML error contract.	AGPL-3.0	specs	docs
nats	Self-hosted messaging: request/reply through the real broker, JetStream persistence (create → publish → count → purge), the KV store, and the monitoring endpoint.	Apache-2.0	specs	docs
ntfy	Self-hosted push notifications: publish with headers, poll the JSON feed, topic isolation, and deny-all access control unlocked via the admin CLI.	Apache-2.0	specs	docs
prometheus	Self-hosted monitoring: promtool config/rule checks and rule unit tests, the query API, and a self-scrape polled with http retry.	Apache-2.0	specs	docs
pushgateway	Self-hosted metrics gateway; exercises raw text body: payloads.	Apache-2.0	specs	docs
rclone	Self-hosted file sync: copy/sync/check semantics (including the corruption failure mode), JSON listings, and rclone serve http.	MIT	specs	docs
restic	Self-hosted backup: init → backup → restore round-trip, snapshot JSON, diff, integrity check, retention, wrong-password failure mode.	BSD-2-Clause	specs	docs
transfer.sh	Self-hosted file sharing: a PNG uploaded as the raw request body (body_file), downloaded back to disk (body_to), and verified byte-for-byte; multipart uploads too.	MIT	specs	docs
webhook	Self-hosted webhook receiver driven by a config fixture.	MIT	specs	docs

The name

atago (愛宕) is named for Mount Atago in Kyoto, whose shrine enshrines a deity of fire prevention. A test runner should do the same job: catch the sparks so a project never catches fire.

Contributing

Issues and pull requests are welcome; see CONTRIBUTING.md. Contributions are not only about code: a GitHub Star also motivates development.

LICENSE

The atago project is licensed under the terms of MIT LICENSE.

Contributors ✨

Thanks goes to these wonderful people (emoji key):

CHIKAMATSU Naohiro 💻 📖

Documentation

Overview

Command atago is a YAML-based black-box behavior spec runner for CLIs, APIs, and generated artifacts. See the README for the format and CLI contract, and schema/atago.schema.json for the machine-readable spec file schema.