stub

Documentation

Overview

Command stub is a fake `extend` binary for the eval runner. It records every invocation into the file named by EXTEND_EVAL_RECORD as JSONL, then returns a canned response shaped like the real CLI would emit.

The stub is the linchpin of deterministic skill evaluation: real evals can run against it on any developer machine without API credentials, without burning credits, and without mutating any real workspace.

The behaviour is selected by EXTEND_EVAL_MODE:

real_responses (default) — small fixture set; list/get/upload return
                            canned realistic data; runs return terminal
                            results.
paginated                — list calls return pages with nextPageToken
                            so pagination-discipline tests have multiple
                            pages to iterate over.
auth_error               — every call exits with a 401 envelope on
                            stderr, mirroring the real CLI's error
                            shape.

The stub is invoked the same way the real CLI is — argv is parsed minimally to dispatch to the right canned response, not exhaustively. Anything we don't model is logged and falls through to a generic success exit.