skillci

README

SkillCI

Regression testing for Claude Skills. When a model update silently changes how your skill behaves, SkillCI catches it in CI — and turns the failure into a permanent test case, automatically.

A skill fails against a model it's never been tested on → SkillCI doesn't just report red, it writes the missing test case for you (evals/_generated/...) so skillci accept turns it into permanent coverage. That loop — catch once, covered forever — is the whole point.

What's inside · How this compares · Why · Install · Quick start · GitHub Actions · Dashboard · VS Code · Compliance reports · For AI agents · MCP server · Commands · Status

What's inside

Self-growing eval loop	An uncovered regression writes its own permanent test case instead of just failing once.
Git-native bisect	skillci bisect binary-searches your skill's own commit history with a real git worktree — not a synthetic version store.
Local-only security lint	OWASP Agentic Skills Top 10-mapped static scan plus skill-bloat checks — zero API calls, zero network calls unless you opt into --verify-pinned-sources.
Gradual-adoption lint mode	--mode warn (or .skillci.yaml's lint: mode/rules) reports every finding without failing the build — pilot skillci across a team, then promote specific rules to blocking one at a time instead of an all-or-nothing cutover.
LLM-as-judge, done properly	judge: criteria scored by a separate, non-self-judging model — chain-of-thought reasoning before every verdict, response caching, and optional multi-sample self-consistency voting, not just a bare pass/fail prompt.
Deterministic + LLM-assisted fuzz	Non-LLM mutation testing across all 4 levels of PromptBench's own published attack taxonomy — character (typo, case, whitespace, homoglyph), word (synonym swap), sentence (reorder), and semantic (negation, context-prefix) — for free, plus optional model-generated realistic paraphrases (fuzz_llm) cached so you pay for generation once, ever. A mutation that flips trigger behavior on a case's first run self-grows into a permanent regression case, exactly like a redteam attack does — no competitor persists a fuzzing-discovered failure as permanent CI coverage.
Nondeterminism-aware retries	flake_retries reruns a failed trigger check and majority-votes the verdict instead of trusting one noisy sample; flake_always_sample votes on every case, not just failing ones, to also catch a regression that got lucky on attempt 1.
Adversarial redteam assertions	redteam: runs named attack plugins — prompt injection, instruction leakage (verbatim and semantic), excessive agency, excessive tool scope, SSRF-bait, ASCII/homoglyph smuggling, base64 instruction smuggling, markdown-rendering injection, PII exfiltration (single-turn and multi-turn crescendo), direct jailbreak override, roleplay jailbreak, cross-lingual jailbreak, multi-turn crescendo jailbreak, and harmful content elicitation — against the skill. Deterministic plugins cost one extra call and zero judge calls; judge-graded plugins (including multi-turn ones) reuse the existing judge model. A successful attack self-grows into a permanent regression case exactly like an uncovered model regression does.
Slice-level gating	Tag cases with dimensions: and gate CI strictly on just the segment that matters, independent of the global fail_on policy.
Cost & latency budgets	Fail CI on runaway token count, output length, latency, or estimated dollar cost — not just wrong output.
Live editor linting	VS Code extension lints SKILL.md as you type — including unsaved buffer content, not just what's last saved to disk.
Optional hosted dashboard	Per-skill compatibility history and a public "still passes on this week's model" badge, Postgres-backed, entirely opt-in.
Compliance evidence reports	skillci report --compliance nist-ai-rmf|eu-ai-act maps eval cases + run history onto NIST AI RMF/EU AI Act's own documentation and testing requirements — evidence for an auditor, not a certification claim.
MCP server	skillci mcp-serve exposes every command as a native MCP tool over stdio — each tool call runs the real cobra command in-process, so it can never drift from the CLI's own behavior.

How this compares

Every other tool in this space does one slice of what skillci does — none combine OWASP-mapped skill security scanning, cross-model regression testing, git-native culprit-commit bisection, and a self-growing eval loop in one tool. This table reflects each tool's publicly documented capabilities; if something here is out of date, please open an issue.

	skillci	skillgrade	AgentLinter	promptfoo / deepeval / braintrust
Claude-Skills-native (SKILL.md, frontmatter, referenced files)	Yes	Yes	Yes	No — generic prompts, no skill awareness
OWASP Agentic Skills Top 10-mapped security scan	Yes	No	Partial — scans skill content, not OWASP-mapped	No
Cross-model regression matrix, gated on new regressions only	Yes	No	No	Yes — generic, not skill-aware
Self-growing eval loop (a failure writes its own test case)	Yes	No	No	No
Git-native bisect (binary-search commit history for the culprit)	Yes	No	No	No
LLM-as-judge	Yes — CoT + caching + self-consistency	Yes — rubric-based	No	Yes
Live editor integration	Yes — VS Code	No	No	No
Adversarial/red-team prompt fuzzing	Yes — 16 plugins incl. multi-turn crescendo, local-only generation, self-growing corpus	—	—	Yes — 50+ plugins, cloud-generated, no persistent corpus
Hosted dashboard	Yes — opt-in, self-hosted	—	—	Yes (braintrust)

Where a competitor is genuinely ahead — promptfoo ships 50+ red-team plugins to skillci's 16, and years of production hardening skillci's redteam assertions haven't had. Multi-turn crescendo attacks, one of the two gaps this table used to name specifically, are now covered (crescendo-jailbreak); plugin count is still smaller by design — docs/adding-a-redteam-plugin.md plus skillci scaffold redteam-plugin cover how new ones get added. Where skillci is structurally ahead — no cloud roundtrip to generate attacks, and a successful attack becomes a permanent CI regression test instead of a one-time report — that's a narrower, verifiable claim, not "better overall."

Why

You write a Claude Skill. It works today. Six months from now, Anthropic ships a new model, and nobody tested your skill against it first — because until now, no tool did that automatically. It might stop triggering, ignore instructions it used to follow, or blow past a token budget you never knew it had. You find out by accident, not by CI.

Every other kind of software has a safety net for this — tests that fail the moment behavior changes. Skills have had none. SkillCI is that safety net: write down what a skill should do once, and get an automated answer to "did it survive the last model release" instead of finding out the hard way. Full rationale below.

Install

Homebrew (macOS, Linux):

brew tap kabirnarang39/skillci
brew trust kabirnarang39/skillci  # newer Homebrew requires explicitly trusting third-party taps
brew install --cask skillci

Scoop (Windows):

scoop bucket add skillci https://github.com/kabirnarang39/skillci
scoop install skillci/skillci

Prebuilt binaries (Linux/macOS/Windows, amd64/arm64): download from the latest release — Linux/macOS ship as .tar.gz, Windows as .zip, with a checksums.txt alongside to verify against.

Supply chain: every archive ships with an SPDX SBOM (<archive>.sbom.json), and checksums.txt is signed keylessly via cosign using the release workflow's GitHub Actions OIDC identity — no private key exists to leak. Verify with:

cosign verify-blob --bundle checksums.txt.bundle \
  --certificate-identity-regexp "https://github.com/kabirnarang39/skillci/.*" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  checksums.txt

Go install (any platform with Go 1.25+):

go install github.com/kabirnarang39/skillci/cmd/skillci@v0.4.1

An ANTHROPIC_API_KEY is needed for eval/regress (not for check, which is local-only and free).

Shell completion: skillci is built on cobra, so bash, zsh, fish, and powershell completion scripts are generated for free — no extra install.

# bash (current shell)
source <(skillci completion bash)
# bash (persist across shells)
skillci completion bash > /etc/bash_completion.d/skillci

# zsh (persist across shells — run once)
skillci completion zsh > "${fpath[1]}/_skillci"

# fish (persist across shells)
skillci completion fish > ~/.config/fish/completions/skillci.fish

Run skillci completion --help for the full per-shell instructions (including oh-my-zsh setups).

Quick start

# Scaffold config + an example eval case inside your skill's folder
skillci init path/to/your-skill

# Lint SKILL.md — no API calls, catches malformed frontmatter, missing
# references, description-length issues, committed secrets, a
# first-layer static security scan (OWASP Agentic Skills Top 10:
# malicious payloads, unpinned dependencies, over-privileged access,
# insecure metadata parsing, untrusted external instructions,
# cross-platform format issues), and basic "skill bloat" warnings
# (oversized body, duplicate instructions, too many/too-large referenced
# files)
skillci check path/to/your-skill

skillci check's security rules are mapped directly to 6 of OWASP's Agentic Skills Top 10 categories — AST01 (malicious skills), AST02 (supply chain: an install/pull command or Dockerfile FROM pinned to a floating latest tag instead of an immutable version/hash — OWASP's own AST02 mitigation list names this exact practice), AST03 (over-privileged skills), AST04 (insecure metadata), AST05 (untrusted external instructions: a skill body that fetches instructions/config from a URL and directs the agent to treat it as authoritative), and AST10 (cross-platform reuse). This is a first-layer static scan, not a malware scanner — obfuscated or natural-language-only attacks can bypass pattern matching, a limitation OWASP itself documents (AST08). The other 3 categories — AST06 (weak isolation), AST07 (update drift), AST09 (no governance) — were checked against OWASP's own published mitigation lists for each: every mitigation (container sandboxing, signature verification on registry updates, SOC/CMDB inventory) is a property of the runtime, registry, or organization, never of a single SKILL.md file's text — so there's no honest static check to add for these three, not an omission.

One narrow, opt-in exception, since it's directly checkable without a registry: a skill author can declare pinned_sources in frontmatter — the exact expected SHA-256 of an external URL this skill's instructions depend on — and skillci check --verify-pinned-sources fetches it and confirms the content hasn't silently changed since it was pinned. This is the one flag in this entire command that makes a network call; everything else, including the check above, stays local-only. Off by default, and never runs unless you pass it by name:

---
name: my-skill
description: Fetches shared guidelines from a central doc.
pinned_sources:
  - url: https://example.com/shared-guidelines.md
    sha256: 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08
---

skillci check --verify-pinned-sources path/to/your-skill

A hash mismatch means the pinned content changed since someone last reviewed and pinned it — implementing AST02's "pin dependencies to immutable hashes" mitigation, and catching AST07-style update drift, for the one case that's actually checkable without a registry: content the skill's own author already opted to pin.

skillci check also flags basic skill bloat: an oversized SKILL.md body (over 8000 characters — every extra instruction is loaded on every invocation), exact-duplicate instruction lines (copy-paste bloat), and skills that reference too many files or too much referenced-file content (over 10 files or 100KB combined). These are fixed thresholds, not user-configurable, and — like the security rules — purely local pattern matching, not a judgment call about whether a skill is good, just whether it's carrying more than it needs to.

Add --format json for machine-parseable output — a stable array of {file, line, rule, msg, severity} objects on stdout ([] for a clean skill, never null), with the pass/fail exit code as the only other signal a caller needs. This is what the VS Code extension below runs under the hood; useful for any other editor integration or script too.

Every issue is block severity by default — the long-standing behavior, a lint issue fails the command. For rolling check out to a team gradually instead of an all-or-nothing cutover, --mode warn reports every issue but never fails the command:

skillci check --mode warn path/to/your-skill

Or make it permanent (and promote specific rules back to blocking) via .skillci.yaml:

lint:
  mode: warn                        # default severity for any rule not named below
  rules:
    ast01-pipe-to-shell: block      # promote just this one while piloting the rest

--mode on the command line always wins over .skillci.yaml — including over a per-rule block override — for a quick one-off check without touching the config file.

# Run the eval suite against one model
skillci eval path/to/your-skill --model claude-sonnet-5

# Run the full regression matrix (every model in .skillci.yaml), diffed
# against the last known-good run — fails CI only on a *new* regression
skillci regress path/to/your-skill

--verbose additionally prints, for any judge criterion that used judge_samples > 1, every individual sample's verdict — and flags a non-unanimous vote as ⚠ SPLIT VOTE even when it passed. LLM judges are known to disagree with themselves at close-to-coin-flip rates on borderline calls; a 2/3 pass and a 3/3 pass are not equally trustworthy, and this makes that difference visible instead of collapsing both into the same bare "PASS."

What actually fails CI is controlled by fail_on in .skillci.yaml — three values:

• regression (the default) — fail only on a case+model that used to pass and now doesn't. A case with no prior recorded run never fails CI on its own; it gets proposed as a generated case instead (see below).
• any_fail — fail on any failing case+model, regardless of history. Stricter; useful once a skill's suite is mature enough that every failure should be treated as a real problem.
• triggered_only — fail only when a case's triggered assertion doesn't match (ignores contains/budget/snapshot/fuzz/etc. failures entirely). Useful early on, when trigger accuracy is the only thing you're confident enough in to gate on.

A strict_dimensions match (see below) always fails CI regardless of which fail_on policy is set.

Heads up if you're new here: under the default regression policy, a brand-new eval case that fails on its very first run will NOT fail CI — there's no prior passing run to regress from, so it's proposed as a generated case instead (see below) rather than blocking the build. If you just added a case and expected red CI, check for a proposed new eval case line in the output before assuming something's broken. Switch to any_fail if you want every failure, including a brand-new case's, to gate CI from day one.

An eval case (evals/*.yaml) looks like this:

name: "haiku-request-triggers"
prompt: "Can you write me a haiku about autumn leaves?"
skill_under_test: "haiku-writer"
assert:
  triggered: true
  contains: ["autumn"]
  max_tokens_loaded: 3000

When regress finds a failure with zero prior coverage for that case+model, it writes a proposed case to evals/_generated/ instead of just failing:

[FAIL] unrelated-request-should-not-trigger (claude-sonnet-5)
    triggered = true, want false
proposed new eval case: evals/_generated/unrelated-request-should-not-trigger-generated-claude-sonnet-5.yaml

skillci accept unrelated-request-should-not-trigger-generated-claude-sonnet-5

promotes it into evals/, so it's a tracked regression guard from then on. Pass --open-pr to skip the manual git add/commit step too: skillci commits the generated case(s) onto a new branch, pushes it, and opens a pull request against the branch the run started from, so review happens in your normal PR flow instead of a file someone has to notice in the workspace:

skillci regress --open-pr

Requires GITHUB_REPOSITORY and GITHUB_TOKEN (a token with contents: write and pull-requests: write on the repo — secrets.GITHUB_TOKEN in a GitHub Actions workflow already has both if the job's permissions: block grants them). A failure here (missing token, push rejected, API error) is reported as a warning, never as a CI failure — the generated case file is already written either way.

The generated file also carries the failure context that produced it — model, detection time, and the model's actual response — as a YAML comment header, so a reviewer deciding whether to accept it doesn't have to go dig that back out of CI logs:

# generated by skillci's self-growing eval loop — informational, not part of the case spec
# model: claude-sonnet-5
# detected_at: 2026-07-24T21:30:00Z
# actual_response:
#   I can't help with that request.
name: unrelated-request-should-not-trigger-generated-claude-sonnet-5
prompt: unrelated request
assert:
  triggered: false

That detected_at also backs a staleness check: once a case+model's first failure is recorded, regress never re-proposes or re-flags the same combination again on later runs (see fail_on: regression above) — so a generated case nobody ever ran accept on can otherwise sit under evals/_generated/ forever with CI staying green the whole time. regress warns instead of staying silent once one's been sitting there over 14 days:

warning: 1 unaddressed generated eval case(s) older than 14 days — run `skillci accept <name>` or delete them:
  evals/_generated/unrelated-request-should-not-trigger-generated-claude-sonnet-5.yaml (detected 2026-07-10)

For cases where you want to know if a skill's actual response content drifts between runs — not just whether it triggered or hit the right substrings — add snapshot: true:

name: "haiku-tone-check"
prompt: "Write a haiku about the ocean."
skill_under_test: "haiku-writer"
assert:
  triggered: true
  snapshot: true

The first run captures the response as a per-model golden baseline (evals/<case>.<model>.golden.txt). Every later run word-diffs the new response against it and shows what changed — without failing CI, unless you also set snapshot_strict: true. This is deliberately informational by default: it tells you that something changed, the same way any snapshot-testing tool (Jest, ApprovalTests) does — not whether the change is good or bad. You decide, then run:

skillci diff my-case --path path/to/your-skill --model claude-sonnet-5   # inspect
skillci accept my-case --model claude-sonnet-5                    # promote

To check whether a skill's trigger behavior is robust to rewording — not just whether the exact eval prompt fires it — add fuzz: true to a case that also asserts triggered:

name: "haiku-request-triggers"
prompt: "Can you write me a haiku about autumn leaves?"
skill_under_test: "haiku-writer"
assert:
  triggered: true
  fuzz: true

Every run generates deterministic paraphrases of the prompt — synonym swaps, negation insertion, sentence reordering, and unrelated leading context — and checks whether the skill still triggers (or doesn't) the way triggered expects. No LLM writes the paraphrases; the mutations are fixed, non-random string transformations, so a fuzz run costs nothing beyond the extra model calls it makes. Note that regress fuzzes every model in your configured matrix, so total API calls scale as models × fuzz-enabled cases × (1 + up to 31 mutations) — the worst case is a 3-sentence prompt that hits all eight operators (1 synonym-swap + 2 negation + 5 non-identity 3-sentence reorderings + 3 context-prefix + 5 each from the four character-level operators below = 31 mutations), for 32 calls per model per case including the primary run. Like snapshot, this is informational by default:

[FUZZ] 2/26 mutations flipped trigger behavior
  negation: "Can you don't write me a haiku about autumn leaves?" -> triggered=false (want true)

Add fuzz_strict: true to fail CI on a flip. Run it standalone with:

skillci fuzz path/to/your-skill --model claude-sonnet-5

or let it run automatically as part of skillci regress for any case that sets fuzz: true — no separate invocation needed for full coverage.

Four more operators cover character-level perturbation specifically — typo-perturbation (a single deterministic transposed-letter edit), case-mutation (alternating case), whitespace-obfuscation (a zero-width space inserted mid-word), and unicode-homoglyph (Cyrillic look-alike substitution, the same technique ascii-homoglyph-smuggling uses for security testing, applied here to robustness instead). Together with the original 4 operators, this maps onto all 4 levels of PromptBench's own published LLM-robustness taxonomy (character/word/sentence/semantic) — see microsoft/promptbench, whose paper is titled PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts (arXiv:2306.04528).

A mutation that flips trigger behavior on a fuzz_strict: true case's first-ever run doesn't just print once and disappear — it self-grows into a permanent evals/_generated/*.yaml case pinned to that exact wording, through the same self-growing eval loop redteam: uses. skillci accept promotes it into a normal regression test that fails until the skill genuinely stops mis-handling that phrasing. Capped at 5 self-grown cases per case+model per run — the console [FUZZ] line states the cap explicitly when hit, so nothing is silently dropped from view even though only 5 become files.

The deterministic operators above test a different failure mode than the one that actually burns teams: a fixed synonym dictionary can't anticipate how a real user would reword a request. Add fuzz_llm: true alongside fuzz: true to additionally have the model itself generate 3 realistic paraphrases:

assert:
  triggered: true
  fuzz: true
  fuzz_llm: true

Unlike the deterministic operators, this costs a real API call — but only once per unique prompt, ever. Generated paraphrases are cached to .skillci/fuzz-llm-cache.json (git-committed like history.json), so every subsequent run against the same prompt — including on a different machine or CI runner — reuses the cached set instead of asking the model again. Pay the generation cost once, not per run.

Model responses aren't fully deterministic — even at low temperature, sampling variance can make a triggered/contains/not_contains check fail on an otherwise-healthy skill. For cases where that matters more than the extra API cost, flake_retries reruns a failing case's trigger checks and takes a majority verdict instead of trusting a single sample:

name: "haiku-request-triggers"
prompt: "Can you write me a haiku about autumn leaves?"
skill_under_test: "haiku-writer"
assert:
  triggered: true
  flake_retries: 2

By default, this only fires when the FIRST attempt's trigger checks fail — a passing case never pays the extra cost, but that has a real blind spot: a case that's genuinely flaky but happens to pass on attempt 1 is recorded as a clean pass with no vote at all, so it catches false positives (one unlucky roll) but not false negatives (a real regression that got lucky). Add flake_always_sample: true to vote every time, regardless of the first attempt's result, closing that gap at the cost of running the full vote on every case, not just failing ones — the same symmetric approach judge_samples already uses for the judge layer (see below). Up to 1 + flake_retries total attempts are made either way, stopping early once a majority is mathematically decided (e.g. 2 failing attempts out of 3 possible stops before the 3rd call). Budget assertions (max_tokens_loaded, max_output_tokens, max_latency_ms, max_cost_usd) are never retried — they're checked once, same as always, since rerunning can't change a token-count-derived cost or latency reading into something more "correct."

An odd flake_retries value can tie (e.g. flake_retries: 1 → 2 total attempts, 1-1). A tie is informational only by default:

[RETRY] triggered check unstable after 2 attempts — 1/2 passed (tie), informational only unless flake_strict is set

Add flake_strict: true to fail CI on an unresolved tie instead.

Deterministic assertions can't check response quality — tone, empathy, whether an explanation is actually clear. For that, judge sends the response to a separate model with a rubric of named criteria and takes its verdict, informational by default:

name: "haiku-request-triggers"
prompt: "Can you write me a haiku about autumn leaves?"
skill_under_test: "haiku-writer"
assert:
  triggered: true
judge:
  - name: tone
    criterion: "Is the response warm and encouraging, not clinical?"
  - name: imagery
    criterion: "Does the haiku use at least one concrete visual image?"

Requires judge_model in .skillci.yaml — deliberately a separate model from the ones under test, never the model judging itself, since a model can't reliably judge its own drift:

judge_model: claude-opus-4-8

All criteria must pass for the judge step to pass. By default, every criterion is evaluated together in a single API call (the batched mode). Judging only runs once every other assertion has already passed — it's the last check, not a substitute for triggered/contains. Add judge_strict: true to fail CI on a failing criterion; without it, a failure just prints:

[JUDGE] 1/2 criteria failed
  tone: FAIL — reads as clinical rather than warm

To trade cost for full per-criterion reasoning isolation, set judge_mode: isolated in .skillci.yaml (the global default) or override it per-case with assert.judge_mode. In isolated mode, each criterion gets its own API call and independent reasoning, eliminating cross-criterion bias; in batched mode (the default), all criteria share one call. Judge model's own sampling variance can make verdicts flaky on borderline cases. For cases where that matters, judge_samples draws N independent verdict samples per criterion group and votes unconditionally every time — the same symmetric approach flake_always_sample opts the case model into (see above), just always-on here rather than a flag:

assert:
  judge:
    - name: tone
      criterion: "Is the response warm and encouraging, not clinical?"
  judge_samples: 3

The [JUDGE] output shows sample tallies when judge_samples is set above 1 (e.g. "(2/3 samples passed)"). Judge verdicts are cached to .skillci/judge-cache.json so a repeat call against an unchanged response never re-pays for the API call, matching how .skillci/fuzz-llm-cache.json avoids regenerating paraphrases.

This is deliberately the most opt-in, most secondary assertion type in skillci: the whole premise of this tool is catching model drift deterministically, and an LLM judge is the one technique that can't judge itself reliably when the judge model is also something that might drift. Reach for the deterministic assertions first; add judge only for what genuinely can't be checked any other way.

Deterministic assertions and judge criteria both test whether a skill behaves correctly on intended input. redteam tests whether it holds up against adversarial input, across 16 plugins:

Name	Category	Grading	Calls
prompt-injection-canary	injection	deterministic	1
ascii-homoglyph-smuggling	injection	deterministic	1
base64-instruction-smuggling	injection	deterministic	1
markdown-rendering-injection	injection	deterministic	1
instruction-leakage	pii	deterministic	1
jailbreak-direct-override	jailbreak	judge	1 + 1 judge
crescendo-jailbreak	jailbreak	judge	3 + 1 judge
roleplay-jailbreak	jailbreak	judge	1 + 1 judge
crosslang-jailbreak	jailbreak	judge	1 + 1 judge
harmful-content-elicitation	harmful	judge	1 + 1 judge
system-prompt-leak-semantic	pii	judge	1 + 1 judge
pii-exfiltration	pii	judge	1 + 1 judge
pii-exfiltration-crescendo	pii	judge	3 + 1 judge
excessive-agency	agency	judge	1 + 1 judge
excessive-tool-scope	agency	judge	1 + 1 judge
ssrf-bait	agency	judge	1 + 1 judge

name: "haiku-request-redteam"
prompt: "Can you write me a haiku about autumn leaves?"
skill_under_test: "haiku-writer"
assert:
  triggered: true
  redteam:
    - plugin: prompt-injection-canary
    - plugin: instruction-leakage
    - plugin: jailbreak-direct-override
    - plugin: harmful-content-elicitation
    - plugin: system-prompt-leak-semantic
    - plugin: excessive-agency
    - plugin: ssrf-bait
    - plugin: ascii-homoglyph-smuggling
    - plugin: pii-exfiltration
    - plugin: crescendo-jailbreak
  redteam_strict: true

The deterministic plugins in the table above are fully local — a pure string/token check against the attack response, zero judge calls, zero network calls beyond the one attack request. Every judge-graded plugin reuses the same judge model judge: criteria use, and requires judge_model configured the same way. "Calls" in the table above is the cost of running that one plugin for one case: most are a single extra model call (plus one judge call, for judge-graded plugins), but a multi-turn plugin sends one call per turn before its judge call — budget accordingly if you enable a multi-turn plugin broadly. A successful attack is informational only unless redteam_strict: true — and, same as any other case, a first-time-failing redteam case self-grows into a permanent regression under evals/_generated/ via the existing self-growing eval loop, with zero special-casing: a caught jailbreak becomes a test that runs forever, not a report you read once.

When a case that used to pass starts failing, skillci bisect finds which commit in your skill's own git history broke it — the same binary-search idea as git bisect, aimed at your skill instead of your code, holding the current model fixed throughout:

skillci bisect my-case --path path/to/your-skill --model claude-sonnet-5

With no --good/--bad flags, it looks up the last recorded passing run for that case in .skillci/history.json and the most recent recorded run, and searches the commits between them that touched the skill's files — checking out each candidate into a disposable git worktree (your actual working tree is never touched) and re-running the case against it:

verifying good/bad endpoints...
  9f8e7d6 — fail
  a1b2c3d — pass
good: a1b2c3d (2026-06-01) — passes
bad:  9f8e7d6 (2026-07-20) — fails
7 candidate commits, up to 3 more API calls
bisecting...
  4f3a2b1 — pass
  8c7d6e5 — fail

culprit: 6a5b4c3d8e9f0a1b2c3d4e5f6a7b8c9d0e1f2a3
author:  Kabir Narang <kabirnarang39@gmail.com>
date:    2026-07-10
message: tighten haiku-writer's tone guidance

--- SKILL.md (6a5b4c3d8e9f0a1b2c3d4e5f6a7b8c9d0e1f2a3^)
+++ SKILL.md (6a5b4c3d8e9f0a1b2c3d4e5f6a7b8c9d0e1f2a3)
@@ ...
- Write a haiku about the requested topic.
+ Write a haiku about the requested topic, staying strictly formal in tone.

skillci regress also prints a skillci bisect ... suggestion inline whenever it detects a new regression, so you don't need to remember the command yourself. Pass --auto-bisect to have it run bisect immediately instead of just printing the suggestion — useful in CI, where the run's own logs then already contain the culprit without a second manual invocation:

skillci regress --auto-bisect

Merge commits in the range are detected automatically — bisect falls back from binary search to a full linear scan of every candidate, which correctly finds the actual commit that introduced the regression instead of (incorrectly) landing on the merge commit itself. If the history genuinely has more than one commit where behavior transitions from passing to failing, that's reported as a warning rather than silently picked.

Every verified (case, model, commit) result is persisted to .skillci/bisect-cache.json, so re-running bisect on the same case — after an interruption, or while investigating a related case whose range overlaps commits already tested — never re-checks-out or re-runs a commit it already has an answer for; those show up as (cached) in the output instead of a fresh API call.

For cost and latency budgets, three more assertions are available:

name: "cost-budget-case"
prompt: "Write a haiku about autumn."
skill_under_test: "haiku-writer"
assert:
  triggered: true
  max_output_tokens: 500
  max_latency_ms: 3000
  max_cost_usd: 0.01

max_output_tokens and max_cost_usd are hard caps — like max_tokens_loaded, exceeding either fails the case immediately. max_cost_usd needs a pricing entry in .skillci.yaml — skillci never hardcodes or guesses prices, since Anthropic can reprice without notice:

pricing:
  claude-sonnet-5:
    input_per_million: 3.0
    output_per_million: 15.0

A case asserting max_cost_usd for a model with no pricing entry fails loudly, naming the missing model, rather than silently skipping the check.

max_latency_ms is the one exception to the hard-cap rule: latency reflects network and inference variance, not what the skill actually did, so an exceeded cap is informational only — printed, not failed — unless you also set latency_strict: true.

Eval cases can also carry free-form dimensions for slicing results — e.g. a case representing your enterprise-tier traffic or a specific language variant:

name: enterprise-billing-question
prompt: "..."
assert:
  triggered: true
dimensions:
  segment: enterprise
  language: es

By default this only affects reporting — skillci regress groups its output by dimension (--- by dimension ---) so a cratering segment is visible at a glance instead of buried in a flat case list. To make a specific slice's failures always fail CI regardless of the global fail_on policy, name it in .skillci.yaml:

fail_on: triggered_only
strict_dimensions:
  segment: [enterprise]

Any case tagged segment: enterprise now fails CI on any failure, even though the rest of the suite is gated more loosely.

GitHub Actions

- uses: kabirnarang39/skillci/.github/actions/skillci@v0.4.1
  with:
    path: path/to/your-skill
    anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}

Pin the action itself to a release tag (@v0.4.1 above), not @main — @main floats onto whatever the action definition looks like next, silently changing your CI's behavior with no version control. The version input (defaults to v0.4.1) pins which skillci CLI binary gets installed, separately from the action reference; override it once a newer tag ships, or set it to latest if you deliberately want to float (not recommended for production — every consumer's CI would silently pick up whatever ships next, with no build reproducibility).

Gates CI on new regressions only — a flaky non-deterministic miss won't fail your build every time. Commits both the status badge (passing / partial / regressed) and .skillci/history.json back into the checkout on every run, including runs where a regression is caught — that history is what lets the self-growing eval loop avoid re-proposing the same generated case every run and lets skillci bisect auto-detect its good/bad commits, so it needs to actually reach your repo. The action only commits locally within the checkout; add a push step in your own workflow (or git-auto-commit-action) to land it.

Not on GitHub Actions? skillci is a plain Go binary with no GitHub-specific dependency — see docs/ci/gitlab.md, docs/ci/azure-devops.md, and docs/ci/jenkins.md for copy-paste snippets. No dedicated plugin needed for any of them; a shell step calling the binary is all it takes.

On a pull request, set pr-comment: true to get results posted directly on the PR instead of only in the Action's own log — the same comment gets updated on every subsequent push to that PR rather than piling up a new one each time:

- uses: kabirnarang39/skillci/.github/actions/skillci@v0.4.1
  with:
    path: path/to/your-skill
    anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
    pr-comment: true
permissions:
  pull-requests: write

Requires pull-requests: write on the workflow's GITHUB_TOKEN (the default token from a pull_request-triggered workflow already has this if the job's permissions: block grants it — see above). A failure posting the comment (missing permission, API hiccup) is reported as a warning in the Action's log, never as a build failure — pr-comment is a convenience on top of the exit code, not a replacement for it.

Optional: hosted dashboard

cmd/skillci-server is a small Postgres-backed HTTP server that turns skillci regress --upload results into a public, per-skill compatibility history and leaderboard — the "does my skill still pass on the model shipped this week" trust signal, shareable the way a codecov badge is. Entirely opt-in; the CLI works standalone forever without it.

Each skill's page renders one trend sparkline per model — not one line mixing every model's results together, since adjacent points on a combined timeline could belong to entirely different models and imply a trend that isn't real. A model whose latest run failed after previously passing gets a "regressed" badge; the dashboard can't run skillci bisect itself (it only ever receives what --upload sends it — no git access to your repo), so the badge just tells you it's worth running locally, rather than pretending to trigger it.

export SKILLCI_DATABASE_URL="postgres://..."
export SKILLCI_INGEST_TOKEN="a-shared-secret"
go run ./cmd/skillci-server

SKILLCI_INGEST_TOKEN is a single unscoped token — fine for a single project. If one instance serves more than one project, use SKILLCI_INGEST_TOKENS instead (token=owner/repo pairs, comma-separated) so each token only authorizes results for its own repo — a leaked token from one project can't be used to forge results for another sharing the same instance:

export SKILLCI_INGEST_TOKENS="token-a=myorg/skill-a,token-b=myorg/skill-b"

VS Code extension

editors/vscode runs skillci check --format json on open/save/edit and surfaces the results as inline diagnostics — security findings as Errors, everything else as Warnings — instead of only in a terminal or CI log. Available on the VS Code Marketplace (search "SkillCI" in the Extensions panel), or build and install it locally:

cd editors/vscode
npm install
npm run package                              # builds skillci-<version>.vsix
code --install-extension skillci-*.vsix

Requires the skillci binary on your PATH (same install as above); no ANTHROPIC_API_KEY needed since check is local-only. See editors/vscode/README.md for settings and development instructions.

Publishing (maintainers): .github/workflows/vscode-release.yml publishes to the VS Code Marketplace and Open VSX on a vscode/v* tag (e.g. vscode/v0.1.0) — a separate tag namespace from the CLI's own v* release tags, since the extension versions independently. Requires a VSCE_PAT repo secret (Marketplace) and, optionally, OVSX_PAT (Open VSX; best-effort, won't fail the release if missing) before the first tag push will actually publish anything.

Compliance evidence reports

skillci report --compliance nist-ai-rmf path/to/your-skill
skillci report --compliance eu-ai-act path/to/your-skill

Generates a Markdown report mapping a skill's existing eval cases and .skillci/history.json run history onto a governance framework's own documentation/testing requirements — NIST AI RMF's Measure function (MEASURE 2.1/2.3/2.6/2.7/2.13) or EU AI Act Articles 11 (technical documentation), 12 (automatic record-keeping), and 19/26(6) (six-month minimum log retention). Every citation is quoted from the framework's own primary-source text, not paraphrased from memory.

This is evidence, not certification — skillci cannot certify a skill compliant with anything. It reports what testing/logging artifacts already exist, for an auditor or governance reviewer to inspect.

One gap the report states plainly rather than glossing over: history.json's retention is a run-count cap (default 200, configurable via history_retention_runs in .skillci.yaml), not a wall-clock one. Whether 200 runs actually spans six months depends entirely on how often your CI runs skillci regress — a high-frequency pipeline needs a proportionally higher history_retention_runs to genuinely satisfy the EU AI Act's six-month minimum, and skillci doesn't (yet) offer time-based retention as an alternative.

For AI agents: the skillci-guardrails skill

Most SKILL.md files today are written or edited by an agent, not typed by hand — which means the highest-leverage place to enforce skill quality isn't a human remembering to run a linter, it's the agent's own workflow. skills/skillci-guardrails is a Claude Skill that does both halves of that: teaches an agent to author a SKILL.md defensively in the first place (the same OWASP-mapped patterns and bloat thresholds check enforces, so there's nothing to catch after the fact), a decision table for which eval assertion actually fits a new case, and then runs skillci check — and eval where cases exist — every time it creates or edits a SKILL.md, the same way a pre-commit hook runs a linter after any other code change.

Install it as a plugin (recommended — Claude Code auto-discovers the skill from the plugin, no manual copying):

/plugin marketplace add kabirnarang39/skillci
/plugin install skillci

Or copy the directory directly into a project's own .claude/skills/ if you'd rather not add a marketplace source. Either way, once installed, every skill an agent writes in that project gets verified without you having to ask for it each time.

Dogfooding: skillci tests its own skill

User story: as a maintainer editing skillci-guardrails's description or triggers, I want confidence the edit doesn't silently break which prompts fire the skill, so I use skillci's own eval framework to test it — the exact tool this skill tells every other skill to use.

skills/skillci-guardrails/evals/ has 5 real cases: three prompts that should trigger the skill (writing a new SKILL.md, reviewing a frontmatter description, asking which eval assertion to use) and two that deliberately shouldn't (an unrelated Python/CSV question, an unrelated SQL question) — the negative cases matter because a meta-skill with a broad description risks over-firing on prompts it has no business claiming.

skillci's eval harness tests exactly the trigger-matching problem this implies: internal/runner.RunCase sends the model only the skill's name+description (never the body) and asks whether it would invoke the skill — a direct proxy for Claude's real progressive-disclosure mechanism, where only name+description are in context until a skill actually fires. Run it yourself:

skillci eval skills/skillci-guardrails

Needs your own ANTHROPIC_API_KEY to produce real evidence — that part is deliberately not faked here. What's already verified without one: skillci check skills/skillci-guardrails passes clean, and every case's YAML is schema-valid (evalspec.LoadDir loads all 5 without error). The actual trigger-accuracy signal — does a real model agree with these expectations — is exactly what running eval with a real key gives you, and generates a real .skillci/history.json/badge the same way any other skill's regression history would.

MCP server

skillci mcp-serve runs skillci as a Model Context Protocol server over stdio, so an agent calls every skillci command as a native tool call instead of shelling out to a CLI it has to remember exists — lower friction, and no risk of a model misremembering a flag. This is the mechanism the skillci-guardrails skill above recommends when an MCP client is available; the skill itself still works via the plain CLI if it isn't.

Exposes every skillci command except mcp-serve itself — init, check, eval, regress, fuzz, bisect, accept, diff, badge, report — each tool call runs the real cobra command in-process (not a reimplementation), so it can never silently drift from what the equivalent CLI invocation does as commands gain new flags over time.

Add to an MCP client's config, pointing at the compiled binary:

{
  "mcpServers": {
    "skillci": {
      "command": "skillci",
      "args": ["mcp-serve"]
    }
  }
}

eval/regress/fuzz/bisect need ANTHROPIC_API_KEY in the server process's own environment, same as running those commands from a shell. regress's auto_bisect/open_pr arguments and check's verify_pinned_sources have real side effects (git bisect, opening a pull request, a network call) — off by default, on only if the agent's tool call explicitly sets them.

Commands

Command	What it does
skillci init	Scaffold .skillci.yaml and an example eval case
skillci check	Lint SKILL.md — local only, no API calls. --mode warn reports without failing the build
skillci eval	Run the eval suite against one model
skillci regress	Run the full model matrix, diff vs. last known-good, gate CI
skillci accept	Promote a generated eval case into the permanent suite
skillci diff	Show a case's pending snapshot change against its golden baseline
skillci fuzz	Run mutation-based robustness testing for fuzz-enabled eval cases
skillci bisect	Binary-search a skill's git history for the commit that broke an eval case
skillci report --compliance <framework>	Generate a Markdown evidence report (nist-ai-rmf or eu-ai-act) from eval cases + run history — evidence, not certification
skillci scaffold redteam-plugin <name> --grading ...	Generate a starter internal/redteam.Plugin stub — see docs/adding-a-redteam-plugin.md
skillci badge	Regenerate the SVG badge from recorded history
skillci mcp-serve	Run every command above as MCP tools over stdio, for agent clients

The full case for this

Most tooling around Claude Skills does one slice — lint, or eval, or token-budget scoring — and none of it tracks behavior across model versions over time. That's the actual gap: on day one of a new model release, your skill might stop triggering half the time because the model reads its description differently, still trigger but quietly ignore instructions it used to follow, or still work but now blow past a token budget it never had before. You find out how? By accident — a workflow breaks, a review comes back wrong, and you spend twenty minutes discovering it's the model, not your skill file.

SkillCI's core differentiator is the self-growing eval loop: when a regression run catches a failure with no prior test coverage, it doesn't just fail — it writes a proposed eval case capturing exactly what broke, so you skillci accept it and the same gap can never silently regress twice. Catch once, covered forever.

skillci bisect is real git worktree-based binary search over a skill's own commit history — not marketing language for "narrowing a failing axis." Other tools have converged on similar ideas from different angles (prompt-version stores, internal bisection over non-git state); skillci's is the version that runs against your actual repo, with Claude-Skills-native assertions.

Status

Early — the core CLI (lint/eval/regress/self-growing loop) is stable and tested; the dashboard is functional but newer. Issues and PRs welcome — see CONTRIBUTING.md to get started (a few good first issues are tagged) and CODE_OF_CONDUCT.md for community expectations. See CHANGELOG.md for release notes and SECURITY.md to report a vulnerability.

If SkillCI caught a regression for you or saved you a debugging session, a star helps others find it — and tells me what to prioritize next.

License

Apache License 2.0