Documentation
¶
Overview ¶
Package acpcheck is a real ACP client, used to grade a real ACP agent.
The distinction from `internal/acp`'s own tests is the whole reason this exists. Those stand an agent up in this process and speak to it through a pipe made in the same test: they prove the two halves of one package agree with each other, which is worth proving and is not the question anyone asks about a protocol. The question is whether the binary we ship says what somebody else's client will understand — and the only way to answer that is to be somebody else's client: launch the executable, write JSON-RPC on its standard input, read its standard output, and believe nothing that is not on the wire.
So nothing here imports the implementation. The message shapes are written out again by hand, from the protocol, because a client that shares the agent's types cannot catch the agent misnaming a field — both sides would be wrong together and the test would be green. That duplication is the point of the file and not an oversight in it.
Index ¶
- Constants
- func RunAgent(script string) int
- func Transcript(ctx context.Context, bin, dir, dialect string) (string, error)
- func TrueCommand() string
- type AgentReport
- type Ask
- type Client
- func (c *Client) Asks() []Ask
- func (c *Client) Call(ctx context.Context, method string, params any) (json.RawMessage, error)
- func (c *Client) Cancel(session string)
- func (c *Client) Close() error
- func (c *Client) Impure() []string
- func (c *Client) Initialize(ctx context.Context) (Init, error)
- func (c *Client) NewSession(ctx context.Context, dir string) (string, error)
- func (c *Client) Notifications() []Notification
- func (c *Client) Notify(method string, params any)
- func (c *Client) Output(stream string) string
- func (c *Client) Prompt(ctx context.Context, session, text string) (string, error)
- func (c *Client) Status(id string) string
- func (c *Client) Stderr() string
- func (c *Client) ToolCalls() []Update
- func (c *Client) Updates() []Update
- type Comparison
- type Config
- type Init
- type Notification
- type Options
- type RPCError
- type Result
- type Row
- type Update
Constants ¶
const ( AllowOnce = "allow-once" AllowAlways = "allow-always" RejectOnce = "reject-once" RejectAlways = "reject-always" )
The option ids this shell offers. They are the agent's to name and a client must echo one back exactly, which is itself one of the things graded: an id we invent must be refused rather than guessed at.
Variables ¶
This section is empty.
Functions ¶
func Transcript ¶
Transcript drives one session and returns it, annotated.
dialect is which shell the binary should be; empty is its default.
func TrueCommand ¶
func TrueCommand() string
TrueCommand is the absolute path of a program that does nothing and succeeds, or "" if there is none to be found.
Resolved rather than written down, because it was written down and it was wrong. Both the cost rows below and the benchmarks in bench_test.go named `/bin/true`, which does not exist on macOS — it lives in /usr/bin. Nothing said so: the spawn rows discarded the error with `_ =`, so a shell that started, failed to find the program, and exited 127 was timed as "the same command as a fresh sh -c", and the gated turn was timing a permission round trip for an exec that then failed. The numbers were plausible, which is why it survived — a process start dominates either way — but they were not measuring what the table said they were.
An absolute path rather than a bare name because a bare name would be the shell's own builtin on one side of the comparison and a process on the other, which is the same mistake wearing a different hat.
Types ¶
type AgentReport ¶
type AgentReport struct {
Script string `json:"script"`
Asked string `json:"asked"`
Got string `json:"got"`
Refused bool `json:"refused"`
Error string `json:"error,omitempty"`
}
AgentReport is what a scripted agent writes down about how it was treated.
Written to a file rather than to a stream, for a reason the protocol imposes: the agent's standard output is the connection and its standard error belongs to the shell's diagnostics. A file is the only channel out of an agent that is not also part of what is being measured.
type Ask ¶
type Ask struct {
ID int
Title string // the tool call's title, when the update carried one
Kind string // execute, edit, read, …
Options []string // option ids, in the order offered
Answered string // the option id we replied with
}
Ask is a `session/request_permission` the agent sent us, and what we said.
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client is a connection to an agent running as a subprocess.
One goroutine reads, everybody else writes under a lock, and requests are matched to their answers by id — the ordinary shape of a JSON-RPC client, written out rather than taken from a library so that the framing this grades is the framing the protocol describes.
func (*Client) Call ¶
Call sends a request and waits for its answer.
An agent that never answers is a failure of the kind this instrument exists to catch, so the wait is bounded by the context and a timeout is returned rather than hung on.
func (*Client) Cancel ¶
Cancel is the protocol's cancellation, which is a notification: the answer to a canceled turn is the turn's own response, with a stop reason.
func (*Client) Close ¶
Close shuts the connection down the way a client does: close the agent's input, which is the protocol's end-of-connection, and wait for it to go.
func (*Client) Initialize ¶
Initialize performs the handshake and returns what the agent said about itself.
func (*Client) NewSession ¶
NewSession opens a session rooted at dir and returns its id.
func (*Client) Notifications ¶
func (c *Client) Notifications() []Notification
Notifications returns the session updates received so far.
func (*Client) Output ¶
Output is everything the shell printed, by stream.
The streams are separated because merging them is how an instrument stops being able to tell a diagnostic from a result — the mistake the conformance harness made with CombinedOutput and had to be rescued from.
type Comparison ¶
type Comparison struct {
Script string
// What each observer could see of the same run.
PipeLines []string // what came back on the pipe: the lines the script printed
PipeActions int // actions the pipe observer could see: none, by construction
ACPActions []string // the actions the protocol put to the client, in order
Refusable int // how many of them the client could have refused
// What it costs, as medians over a run of each.
TurnPlain time.Duration // a builtin command, in an established session
TurnGated time.Duration // a command that asks permission and is allowed
SpawnOurs time.Duration // the same command as a fresh `sh -c` child
SpawnBash time.Duration // and as a fresh `bash -c` child
Handshake time.Duration // initialize, over the pipe, once
SessionOpen time.Duration // session/new
Samples int
}
Comparison is the whole of what Compare measured.
func Compare ¶
func Compare(ctx context.Context, bin, dir, dialect, acpFlag string) (Comparison, error)
Compare runs the script both ways and times both arrangements.
dialect is which shell the binary should be, and it goes to both sides: timing one dialect against another dialect's pipe would make the ratio a measurement of the two dialects rather than of the two arrangements.
type Config ¶
type Config struct {
Bin string
Root string
Self string
Only string
Dialect string
// ACPFlag is how this binary is told to serve the protocol, and it is a
// field because there are two answers rather than one. The multi-call
// `sh` takes `-acp`; a dialect binary takes `--acp`, because real bash
// accepts `-acp` as the `-a -c -p` bundle and a single-dash spelling
// there would shadow working behavior. Empty means `-acp`.
//
// It is what makes the second table possible at all: the same rows, run
// against the binary a shebang names rather than against the substrate's
// own driver. See #2585, and #1826 for the same split in make sandbox.
ACPFlag string
}
Config is what a run is given beside its context.
Root is where each row's scratch directory is made; the caller owns it, so a run whose files are worth looking at can be pointed at somewhere that survives. Dialect is which shell the binary should be — empty for its default, which for the multi-call binary is the core.
type Init ¶
type Init struct {
ProtocolVersion int `json:"protocolVersion"`
AgentInfo struct {
Name string `json:"name"`
Version string `json:"version"`
} `json:"agentInfo"`
AuthMethods []struct {
ID string `json:"id"`
} `json:"authMethods"`
AgentCapabilities json.RawMessage `json:"agentCapabilities"`
}
Init is the initialize result, in the fields a client acts on.
type Notification ¶
type Notification struct {
Method string
Params json.RawMessage
}
Notification is a `session/update` as it arrived, kept whole.
Whole, because the interesting assertions are about fields a typed struct would drop: the `_meta` that says which stream a chunk came from, a `rawOutput` nobody has a type for yet. A grader that decodes into a struct can only ever check what the struct's author thought of.
type Options ¶
type Options struct {
// Args are the agent's arguments. `-acp` is not added for you: which
// flags precede it is part of what some rows are testing.
Args []string
// Dir is the working directory to start the agent in.
Dir string
// Env, when non-nil, replaces the environment rather than adding to it.
Env []string
// Answer decides permission requests. A nil Answer allows once, which is
// the answer that lets a row get far enough to be about something else.
Answer func(Ask) string
// Trace, when set, sees every line in both directions.
Trace func(dir, line string)
// Serve answers an agent's requests other than permission.
Serve func(method string, params json.RawMessage) (any, *RPCError)
}
Options configure a connection.
type RPCError ¶
type RPCError struct {
Code int `json:"code"`
Message string `json:"message"`
Data json.RawMessage `json:"data,omitempty"`
}
RPCError is the error object of a JSON-RPC response.
type Row ¶
type Row struct {
Name string
Claim string // why anyone should care, in one line
Pass bool
Detail string
Known string // the issue that owns a known gap, if one does
}
Row is one graded property.
type Update ¶
type Update struct {
Kind string // agent_message_chunk, tool_call, tool_call_update, …
ToolCallID string
Title string
ToolKind string // execute, edit, read, other
Status string // pending, in_progress, completed, failed
Text string // an agent message chunk's text
Stream string // stdout or stderr, from the chunk's _meta
Raw json.RawMessage
}
Update is a session update decoded far enough to grade.
Source Files