README
¶
AgentAPI Doctor
200 OK is not compatibility. Check the stream. Keep the evidence.
A small, local-first CLI that checks whether an “OpenAI-compatible” or Anthropic-compatible endpoint behaves like a real client expects—and leaves a redacted report you can reproduce, compare, and share.
From download to an answer
Linux and macOS can install the exact v0.1.0-rc.1 release without Go:
curl --proto '=https' --tlsv1.2 -fsSL \
https://raw.githubusercontent.com/whyiug/agentapi-doctor/v0.1.0-rc.1/install.sh | sh
$HOME/.local/bin/doctor demo
The pinned installer verifies the release archive against checksums.txt
before extraction. If you prefer to inspect it first, download
install.sh, then run sh install.sh. Windows users can download
the verified ZIP from GitHub Releases;
the Installation guide includes checksum steps for every
platform.
The demo needs no API key. It starts a random loopback fixture, runs four lifecycle checks, stores local evidence, and stops the fixture automatically:
Profile outcome: COMPATIBLE
Cases: 4 candidate / 4 applicable / 4 executed
Verdicts: PASS 4 | FAIL 0 | WARN 0 | INCONCLUSIVE 0 | SKIPPED 0 | ERRORED 0
Demo success validates this exact synthetic fixture and the installed CLI. It does not certify another endpoint, SDK, provider, or deployment.
Check an authorized endpoint
No project initialization or YAML is required:
export DOCTOR_TOKEN='replace-with-a-test-token'
doctor test \
--base-url 'https://your-endpoint.example/v1' \
--protocol openai-chat \
--model 'your-model-id' \
--auth-env DOCTOR_TOKEN
Use openai-responses or anthropic-messages for those API shapes. Omit
--auth-env for an unauthenticated endpoint. The endpoint can be local, on a
private network, or remote; it only needs to be yours or explicitly authorized
for testing.
Each run sends at most four requests under one 60-second deadline. The token is
read from the named environment variable, not a command argument. Requests stay
on the configured origin, redirects are not followed, and evidence remains in
the local .agentapi/ directory.
What Doctor checks
| A basic smoke test sees | Doctor also checks |
|---|---|
| HTTP status | Required response envelope |
| First SSE chunk | Stream media type and lifecycle |
| Some generated text | Terminal event presence, status, and exactly-once behavior |
| A transient console log | Content-addressed, secret-redacted evidence bound to the exact run |
Today, each Quick Check selects four executable raw HTTP checks for one of:
- OpenAI Chat Completions;
- OpenAI Responses;
- Anthropic Messages.
It can render the same result as terminal output, JSON, JUnit, SARIF, Markdown, or a self-contained offline HTML report. Named baselines and stable exit codes make the result usable in CI.
See a failure—not just a self-test
When the checked-in synthetic server omits the Responses terminal event, Doctor rejects the stream even though its media type looks correct:
Profile outcome: INCOMPATIBLE
Cases: 4 candidate / 4 applicable / 4 executed
Verdicts: PASS 3 | FAIL 1 | WARN 0 | INCONCLUSIVE 0 | SKIPPED 0 | ERRORED 0
PASS stream media type
PASS required response envelope
FAIL terminal event exactly once
PASS terminal status
Download the offline failure report and open it locally, or reproduce it with the documented Synthetic Fixture. This is a real, deterministic wire/lifecycle observation. It is not yet a real SDK run or automatic root-cause attribution.
Where it fits
| Your goal | Best tool today |
|---|---|
| Check whether one key or endpoint responds | curl or a browser checker |
| Explore models and prompts | A web playground |
| Repeatedly inspect lifecycle behavior and keep diffable evidence | AgentAPI Doctor |
| Prove compatibility with one SDK or Agent | Run that real client; Doctor's first pinned SDK profile is in progress |
Doctor is not a model-quality benchmark, provider ranking, relay checker, or vendor certification service. The current catalog also contains candidate metadata that is not executable coverage; see Known Limitations.
Evidence and privacy
- Credentials are resolved from an environment or protected file reference and are redacted before persistence.
- Exact endpoint, model, plan, profile, pack, and evidence digests stay bound to the run so two results can be meaningfully compared.
- Structured model content and tool arguments are not necessarily anonymous. Review evidence before sharing it.
- The provider still receives the bounded synthetic prompts and may retain them under its own policy.
- A provider may reject or ignore the requested 64-token output field, so it is not an enforced cost ceiling.
Only test systems you are explicitly authorized to assess.
Why this project exists
Real reports repeatedly show the same gap: direct requests work, but an SSE terminal, tool-call delta, strict Responses event, proxy, or client state machine breaks later. Examples include Open WebUI #21768, llama.cpp #20607, and Codex #24973.
The research, competitor comparison, intentionally reduced roadmap, and stop conditions are recorded in the July 13 execution plan. The next product slice is deliberately narrow: one pinned OpenAI Python SDK / Responses case before any Registry, public matrix, or hosted UI expansion.
Documentation and community
Quick Start · Installation · CLI reference · Troubleshooting · Known limitations · Roadmap · All docs
Contributions are welcome. Read CONTRIBUTING.md, propose a real compatibility failure or fixture, and use SECURITY.md for private vulnerability reports. Please do not open a public issue containing a credential or unredacted provider response.
Source and documentation use the Apache License 2.0 unless a file says otherwise. See DATA_LICENSE.md and THIRD_PARTY_LICENSES.txt for additional terms.
Documentation
¶
Overview ¶
Package agentapidoctor exposes the versioned public contracts bundled into release binaries.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
This section is empty.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
catalog-check
command
|
|
|
doctor
command
|
|
|
reference-server
command
|
|
|
registry
command
|
|
|
schema-index
command
Command schema-index regenerates the deterministic schema registry.
|
Command schema-index regenerates the deterministic schema registry. |
|
support-check
command
|
|
|
internal
|
|
|
attribution
Package attribution maps bounded differential evidence to versioned fault domains.
|
Package attribution maps bounded differential evidence to versioned fault domains. |
|
budget
Package budget enforces independent hard, reservation, and cleanup ledgers.
|
Package budget enforces independent hard, reservation, and cleanup ledgers. |
|
cas
Package cas implements immutable local content-addressed storage for sanitized payloads and their strongly typed Evidence envelopes.
|
Package cas implements immutable local content-addressed storage for sanitized payloads and their strongly typed Evidence envelopes. |
|
catalog
Package catalog owns the offline, content-addressed Requirement Catalog and ProtocolPack candidate generator.
|
Package catalog owns the offline, content-addressed Requirement Catalog and ProtocolPack candidate generator. |
|
cli
Package cli implements the stable doctor command surface without making a specific command-line framework part of the public contract.
|
Package cli implements the stable doctor command surface without making a specific command-line framework part of the public contract. |
|
executor
Package executor runs an exact ResolvedRunPlan.
|
Package executor runs an exact ResolvedRunPlan. |
|
minimizer
Package minimizer implements bounded, deterministic delta debugging over already captured and redacted artifacts.
|
Package minimizer implements bounded, deterministic delta debugging over already captured and redacted artifacts. |
|
normalizer
Package normalizer builds provider-neutral IR without overwriting the provider-native JSON type or the capture-layer evidence that supports it.
|
Package normalizer builds provider-neutral IR without overwriting the provider-native JSON type or the capture-layer evidence that supports it. |
|
oracle
Package oracle evaluates schema, relational, and state-machine properties while keeping harness failures and insufficient evidence out of target FAIL.
|
Package oracle evaluates schema, relational, and state-machine properties while keeping harness failures and insufficient evidence out of target FAIL. |
|
planner
Package planner constructs immutable offline IntentPlans and resolves only branches that the IntentPlan authorized in advance.
|
Package planner constructs immutable offline IntentPlans and resolves only branches that the IntentPlan authorized in advance. |
|
productrun
Package productrun assembles the small, client-side candidate execution slice used by the CLI.
|
Package productrun assembles the small, client-side candidate execution slice used by the CLI. |
|
provenance
Package provenance provides deterministic local attestations.
|
Package provenance provides deterministic local attestations. |
|
rawdriver
Package rawdriver executes registry-bound raw HTTP scenarios through the origin-restricted transport and the evidence-first evaluation pipeline.
|
Package rawdriver executes registry-bound raw HTTP scenarios through the origin-restricted transport and the evidence-first evaluation pipeline. |
|
recorder
Package recorder captures bounded application-layer observations and commits only sanitized payloads to the evidence CAS.
|
Package recorder captures bounded application-layer observations and commits only sanitized payloads to the evidence CAS. |
|
redaction
Package redaction sanitizes application-layer observations before any persistence boundary.
|
Package redaction sanitizes application-layer observations before any persistence boundary. |
|
registry/httpapi
Package httpapi implements the Registry's standard-library HTTP contract.
|
Package httpapi implements the Registry's standard-library HTTP contract. |
|
registry/store
Package store defines persistence boundaries for the Registry HTTP layer.
|
Package store defines persistence boundaries for the Registry HTTP layer. |
|
report
Package report renders one immutable result model into human and machine formats.
|
Package report renders one immutable result model into human and machine formats. |
|
runstore
Package runstore persists immutable, canonical local run artifacts.
|
Package runstore persists immutable, canonical local run artifacts. |
|
secret
Package secret resolves explicit secret references without placing secret values in command-line arguments, reports, or configuration documents.
|
Package secret resolves explicit secret references without placing secret values in command-line arguments, reports, or configuration documents. |
|
statemachine
Package statemachine validates strict typed stream and tool lifecycles.
|
Package statemachine validates strict typed stream and tool lifecycles. |
|
statistics
Package statistics contains deterministic statistical helpers used by behavioral and operational-reliability oracles.
|
Package statistics contains deterministic statistical helpers used by behavioral and operational-reliability oracles. |
|
transport
Package transport implements an origin-bound HTTP transport for authorized endpoint testing.
|
Package transport implements an origin-bound HTTP transport for authorized endpoint testing. |
|
pkg
|
|
|
driverprotocol
Package driverprotocol defines AgentAPI Doctor's stable out-of-process driver control protocol.
|
Package driverprotocol defines AgentAPI Doctor's stable out-of-process driver control protocol. |
|
packapi
Package packapi defines and compiles AgentAPI Doctor's declarative Scenario and ProtocolPack authoring contracts.
|
Package packapi defines and compiles AgentAPI Doctor's declarative Scenario and ProtocolPack authoring contracts. |
|
schema
Package schema defines AgentAPI Doctor's stable, provider-neutral public contracts.
|
Package schema defines AgentAPI Doctor's stable, provider-neutral public contracts. |
|
reference
|
|
|
mutant-server
Package mutantserver applies one explicit primary fault to the deterministic reference fixture.
|
Package mutantserver applies one explicit primary fault to the deterministic reference fixture. |
|
server
Package server implements a deterministic local protocol fixture.
|
Package server implements a deterministic local protocol fixture. |
|
synthetic-tools
Package synthetictools defines inert, deterministic tool fixtures.
|
Package synthetictools defines inert, deterministic tool fixtures. |
|
registry
|
|
|
api
Package api contains transport-neutral public Registry contracts.
|
Package api contains transport-neutral public Registry contracts. |
|
web
|
|
|
matrix
Package matrix serves the dependency-free public Registry matrix UI.
|
Package matrix serves the dependency-free public Registry matrix UI. |