README
¶
AgentAPI Doctor
200 OK is not compatibility. Check the stream. Keep the evidence.
A small, local-first CLI that checks whether an “OpenAI-compatible” or Anthropic-compatible endpoint behaves like a real client expects—and leaves a redacted report you can reproduce, compare, and share.
Quick Start · What it checks · Real SDK case · Offline report source · 简体中文
From download to an answer
Linux and macOS can install the exact v0.1.1 release without Go:
curl --proto '=https' --tlsv1.2 -fsSL \
https://raw.githubusercontent.com/whyiug/agentapi-doctor/v0.1.1/install.sh | sh
$HOME/.local/bin/doctor demo
The pinned installer verifies the release archive against checksums.txt
before extraction. If you prefer to inspect it first, download
install.sh, then run sh install.sh. Windows users can download
the verified ZIP from GitHub Releases;
the Installation guide includes checksum steps for every
platform.
The demo needs no API key. It starts a random loopback fixture, runs four lifecycle checks, stores local evidence, and stops the fixture automatically:
Result: CHECKS PASSED
Cases: 4 candidate / 4 applicable / 4 executed
Verdicts: PASS 4 | FAIL 0 | WARN 0 | INCONCLUSIVE 0 | SKIPPED 0 | ERRORED 0
Important conditions:
[candidate_interpretations_pending_review] Candidate raw-wire interpretations; not certification.
Demo success validates this exact synthetic fixture and the installed CLI. It does not certify another endpoint, SDK, provider, or deployment.
Check an authorized endpoint
No project initialization or YAML is required:
export DOCTOR_TOKEN='replace-with-a-test-token'
doctor test \
--base-url 'https://your-endpoint.example/v1' \
--protocol openai-chat \
--model 'your-model-id' \
--auth-env DOCTOR_TOKEN
Use openai-responses or anthropic-messages for those API shapes. Omit
--auth-env for an unauthenticated endpoint. The endpoint can be local, on a
private network, or remote; it only needs to be yours or explicitly authorized
for testing.
Each run sends at most four requests under one 60-second deadline. The token is
read from the named environment variable, not a command argument. Requests stay
on the configured origin, redirects are not followed, and evidence remains in
the local .agentapi/ directory. Treat that directory as private local state
and add .agentapi/ to the tested project's .gitignore.
What Doctor checks
| A basic smoke test sees | Doctor also checks |
|---|---|
| HTTP status | Required response envelope |
| First SSE chunk | Stream media type and lifecycle |
| Some generated text | Terminal event presence, status, and exactly-once behavior |
| A transient console log | Content-addressed, secret-redacted evidence bound to the exact run |
Today, each Quick Check selects four executable raw HTTP checks for one of:
- OpenAI Chat Completions;
- OpenAI Responses;
- Anthropic Messages.
It can render the same result as terminal output, JSON, JUnit, SARIF, Markdown, or a self-contained offline HTML report. Named baselines and stable exit codes make the result usable in CI.
See a failure—not just a self-test
When the checked-in synthetic server omits the Responses terminal event, Doctor rejects the stream even though its media type looks correct:
Result: CHECKS FAILED
Cases: 4 candidate / 4 applicable / 4 executed
Verdicts: PASS 3 | FAIL 1 | WARN 0 | INCONCLUSIVE 0 | SKIPPED 0 | ERRORED 0
Important conditions:
[candidate_interpretations_pending_review] Candidate raw-wire interpretations; not certification.
PASS stream media type
PASS required response envelope
FAIL terminal event exactly once
PASS terminal status
Download the offline failure report and open it locally, or reproduce it with the documented Synthetic Fixture. This example is a real, deterministic wire/lifecycle observation. It is not by itself a real SDK run or automatic root-cause attribution.
Why a real SDK changes the answer
A status-only smoke test can accept 200 OK and text/event-stream without
ever asking whether the terminal object is usable. The source tree now includes
one deliberately narrow counterexample using the real, pinned OpenAI Python SDK:
| Observation of the same synthetic stream | Result |
|---|---|
| HTTP/SSE smoke | 200 OK; a response.completed event arrived |
| Raw terminal object | output is null, not the array modeled by the pinned SDK |
| OpenAI Python SDK 2.38.0 | rejects the stream during event iteration |
| Doctor bundle | correlates wire.sse with the sanitized SDK observation and exact dependency lock |
Reproduce it on Linux amd64 with CPython 3.12.12:
doctor reproduce openai-python-responses \
--python .venv/bin/python \
--fixture null-completed-output \
--bundle ./openai-python-null-output.zip
This command uses a random loopback fixture and a synthetic token; it never contacts a provider or reads an API key. See the reproducible case study for the hash-locked install and exact evidence boundary. The case proves one frozen SDK behavior, not compatibility or incompatibility of any vendor endpoint.
Where it fits
| Your goal | Best tool today |
|---|---|
| Check whether one key or endpoint responds | curl or a browser checker |
| Explore models and prompts | A web playground |
| Repeatedly inspect lifecycle behavior and keep diffable evidence | AgentAPI Doctor |
| Reproduce one known Responses/SDK failure | Use Doctor's pinned OpenAI Python case and evidence bundle |
| Prove an arbitrary SDK or Agent compatible | Run that exact client against the authorized endpoint; Doctor does not claim this coverage |
Doctor is not a model-quality benchmark, provider ranking, relay checker, or vendor certification service. The current catalog also contains candidate metadata that is not executable coverage; see Known Limitations.
Evidence and privacy
- Credentials are resolved from an environment or protected file reference and are redacted before persistence.
- Exact endpoint, model, plan, profile, pack, and evidence digests stay bound to the run so two results can be meaningfully compared.
- Structured model content and tool arguments are not necessarily anonymous. Review evidence before sharing it.
- The provider still receives the bounded synthetic prompts and may retain them under its own policy.
- Structural checks request 64 output tokens. The Chat/Responses terminal-status check requests 512 so default-thinking models can reach a natural terminal more often. A provider may reject or ignore either field, so these are not enforced cost ceilings.
Only test systems you are explicitly authorized to assess.
Why this project exists
Real reports repeatedly show the same gap: direct requests work, but an SSE terminal, tool-call delta, strict Responses event, proxy, or client state machine breaks later. Examples include Open WebUI #21768, llama.cpp #20607, and Codex #24973.
The public Roadmap keeps the next work focused on reproducible endpoint checks, report reuse, and verified client cases. Registry, public matrix, and hosted-service candidates remain outside the supported v0.1 scope.
Documentation and community
Quick Start · Installation · CLI reference · Troubleshooting · Known limitations · Real SDK case · Roadmap · All docs
Contributions are welcome. Read CONTRIBUTING.md, propose a real compatibility failure or fixture, and use SECURITY.md for private vulnerability reports. Please do not open a public issue containing a credential or unredacted provider response.
Source and documentation use the Apache License 2.0 unless a file says otherwise. Vendored dependency notices are recorded in THIRD_PARTY_LICENSES.txt.
Documentation
¶
Overview ¶
Package agentapidoctor exposes the versioned public contracts bundled into release binaries.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
This section is empty.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
catalog-check
command
|
|
|
doctor
command
|
|
|
reference-server
command
|
|
|
registry
command
|
|
|
schema-index
command
Command schema-index regenerates the deterministic schema registry.
|
Command schema-index regenerates the deterministic schema registry. |
|
support-check
command
|
|
|
internal
|
|
|
attribution
Package attribution maps bounded differential evidence to versioned fault domains.
|
Package attribution maps bounded differential evidence to versioned fault domains. |
|
budget
Package budget enforces independent hard, reservation, and cleanup ledgers.
|
Package budget enforces independent hard, reservation, and cleanup ledgers. |
|
cas
Package cas implements immutable local content-addressed storage for sanitized payloads and their strongly typed Evidence envelopes.
|
Package cas implements immutable local content-addressed storage for sanitized payloads and their strongly typed Evidence envelopes. |
|
catalog
Package catalog owns the offline, content-addressed Requirement Catalog and ProtocolPack candidate generator.
|
Package catalog owns the offline, content-addressed Requirement Catalog and ProtocolPack candidate generator. |
|
cli
Package cli implements the stable doctor command surface without making a specific command-line framework part of the public contract.
|
Package cli implements the stable doctor command surface without making a specific command-line framework part of the public contract. |
|
executor
Package executor runs an exact ResolvedRunPlan.
|
Package executor runs an exact ResolvedRunPlan. |
|
minimizer
Package minimizer implements bounded, deterministic delta debugging over already captured and redacted artifacts.
|
Package minimizer implements bounded, deterministic delta debugging over already captured and redacted artifacts. |
|
normalizer
Package normalizer builds provider-neutral IR without overwriting the provider-native JSON type or the capture-layer evidence that supports it.
|
Package normalizer builds provider-neutral IR without overwriting the provider-native JSON type or the capture-layer evidence that supports it. |
|
openaisdkcase
Package openaisdkcase runs one bounded, local OpenAI Python SDK Responses streaming case and exports a deterministic maintainer-ready evidence bundle.
|
Package openaisdkcase runs one bounded, local OpenAI Python SDK Responses streaming case and exports a deterministic maintainer-ready evidence bundle. |
|
oracle
Package oracle evaluates schema, relational, and state-machine properties while keeping harness failures and insufficient evidence out of target FAIL.
|
Package oracle evaluates schema, relational, and state-machine properties while keeping harness failures and insufficient evidence out of target FAIL. |
|
planner
Package planner constructs immutable offline IntentPlans and resolves only branches that the IntentPlan authorized in advance.
|
Package planner constructs immutable offline IntentPlans and resolves only branches that the IntentPlan authorized in advance. |
|
productrun
Package productrun assembles the small, client-side candidate execution slice used by the CLI.
|
Package productrun assembles the small, client-side candidate execution slice used by the CLI. |
|
provenance
Package provenance provides deterministic local attestations.
|
Package provenance provides deterministic local attestations. |
|
rawdriver
Package rawdriver executes registry-bound raw HTTP scenarios through the origin-restricted transport and the evidence-first evaluation pipeline.
|
Package rawdriver executes registry-bound raw HTTP scenarios through the origin-restricted transport and the evidence-first evaluation pipeline. |
|
recorder
Package recorder captures bounded application-layer observations and commits only sanitized payloads to the evidence CAS.
|
Package recorder captures bounded application-layer observations and commits only sanitized payloads to the evidence CAS. |
|
redaction
Package redaction sanitizes application-layer observations before any persistence boundary.
|
Package redaction sanitizes application-layer observations before any persistence boundary. |
|
registry/httpapi
Package httpapi implements the Registry's standard-library HTTP contract.
|
Package httpapi implements the Registry's standard-library HTTP contract. |
|
registry/store
Package store defines persistence boundaries for the Registry HTTP layer.
|
Package store defines persistence boundaries for the Registry HTTP layer. |
|
report
Package report renders one immutable result model into human and machine formats.
|
Package report renders one immutable result model into human and machine formats. |
|
runstore
Package runstore persists immutable, canonical local run artifacts.
|
Package runstore persists immutable, canonical local run artifacts. |
|
secret
Package secret resolves explicit secret references without placing secret values in command-line arguments, reports, or configuration documents.
|
Package secret resolves explicit secret references without placing secret values in command-line arguments, reports, or configuration documents. |
|
statemachine
Package statemachine validates strict typed stream and tool lifecycles.
|
Package statemachine validates strict typed stream and tool lifecycles. |
|
statistics
Package statistics contains deterministic statistical helpers used by behavioral and operational-reliability oracles.
|
Package statistics contains deterministic statistical helpers used by behavioral and operational-reliability oracles. |
|
transport
Package transport implements an origin-bound HTTP transport for authorized endpoint testing.
|
Package transport implements an origin-bound HTTP transport for authorized endpoint testing. |
|
pkg
|
|
|
driverprotocol
Package driverprotocol defines AgentAPI Doctor's stable out-of-process driver control protocol.
|
Package driverprotocol defines AgentAPI Doctor's stable out-of-process driver control protocol. |
|
packapi
Package packapi defines and compiles AgentAPI Doctor's declarative Scenario and ProtocolPack authoring contracts.
|
Package packapi defines and compiles AgentAPI Doctor's declarative Scenario and ProtocolPack authoring contracts. |
|
schema
Package schema defines AgentAPI Doctor's stable, provider-neutral public contracts.
|
Package schema defines AgentAPI Doctor's stable, provider-neutral public contracts. |
|
reference
|
|
|
mutant-server
Package mutantserver applies one explicit primary fault to the deterministic reference fixture.
|
Package mutantserver applies one explicit primary fault to the deterministic reference fixture. |
|
server
Package server implements a deterministic local protocol fixture.
|
Package server implements a deterministic local protocol fixture. |
|
synthetic-tools
Package synthetictools defines inert, deterministic tool fixtures.
|
Package synthetictools defines inert, deterministic tool fixtures. |
|
registry
|
|
|
api
Package api contains transport-neutral public Registry contracts.
|
Package api contains transport-neutral public Registry contracts. |
|
web
|
|
|
matrix
Package matrix serves the dependency-free public Registry matrix UI.
|
Package matrix serves the dependency-free public Registry matrix UI. |