README
¶
ajq
ajq adds semantic matching and bounded classification to byte-deterministic
jq pipelines over JSON and NDJSON. For example,
.[] | select(.message =~ "payment failure") keeps records whose message
describes a payment failure even when the wording varies. The surrounding jq
selection and output remain deterministic, and model calls occur only for
explicit semantic operations, with scoring and normalization available only in
limited contexts.
Choose the right tool
| Task | Use |
|---|---|
| Exact fields, regular expressions, structural transforms, or reproducible formatting | jq (or ajq with a pure jq query) |
| Find JSON/NDJSON records by topic, intent, or meaning | ajq with =~ or sem_match |
| Route JSON/NDJSON records into labels you define up front | ajq with sem_classify |
| General-purpose extraction or redaction transforms | Choose a dedicated tool because ajq limits sem_extract and sem_redact to gated control-flow contexts |
Each semantic operation is visible in the query, and backend selection and call limits remain under user control. Validate the query with the deterministic mock backend, inspect its plan, and select a real backend with an explicit call cap when the task needs model judgement.
Usage
Start with ajq --help, then run ajq examples for categorized, copy-pasteable
safe workflows. Coding agents should first run ajq capabilities --json to
inspect the static machine-readable contract. --backend mock is the
deterministic, no-network, no-model path to exercise semantic query syntax
before selecting a real backend.
# Help and version
ajq --help
ajq --version
# Pure jq over JSON stays deterministic
printf '{"users":[{"name":"Ada"}]}' | ajq -r '.users[].name'
# Ada
# Semantic grep for JSON with the deterministic mock backend: safe agent probe
# (no model, network, or API key)
printf '[{"id":1,"msg":"please keep this"},{"id":2,"msg":"drop it"}]' \
| ajq --backend mock -c '.[] | select(.msg =~ "keep") | .id'
# 1
# Inspect semantic plan and estimated backend calls before running an LLM-enhanced jq query
printf '[{"msg":"refund demanded"}]' \
| ajq --backend mock --explain '.[] | select(.msg =~ "angry/frustrated") | .msg'
Run ajq provision once before using --backend local; then the same semantic queries can run against the managed local llama.cpp backend.
Install
Use the published Homebrew cask, the release script for supported prebuilt archives, a manual release archive, or Go source:
brew install --cask ricardocabral/tap/ajq
curl -fsSL https://raw.githubusercontent.com/ricardocabral/ajq/main/scripts/install.sh | sh
# manual download (including the current Windows ZIP): https://github.com/ricardocabral/ajq/releases/latest
go install github.com/ricardocabral/ajq/cmd/ajq@latest
The release workflow publishes the Homebrew cask to the public
ricardocabral/tap tap.
Windows MSI packaging is implemented and CI-validated, but the MSI is not yet released. ajq is not available through WinGet; it remains unavailable until a future MSI release completes Microsoft validation and merge and has public clean-install smoke evidence. Windows users should use the published ZIP in the meantime.
Coding-agent skill
Install the ajq routing skill for Codex from this repository's marketplace:
codex plugin marketplace add ricardocabral/ajq
codex plugin add ajq@ajq
The optional npx plugins add ricardocabral/ajq adapter currently targets
Claude Code and Cursor. See the coding-agent skill installation
guide
for pinned, workspace, CI, and verification flows.
Status
| Area | What works today |
|---|---|
| Backends | Six semantic backends ship: local, mock, ollama, openai, openrouter, and Anthropic via --cloud / --backend anthropic. |
| Cost controls | --explain estimates model calls, --max-calls caps post-dedup judgements, and paid/cloud backends default to a 100-call guardrail. |
| Persistent cache | Semantic judgements are stored on disk under the ajq cache directory; --no-cache disables reads/writes for sensitive runs. |
| Local provisioning | ajq provision downloads or locates the llama.cpp engine and default GGUF model for --backend local on supported platforms. |
| Model management | ajq models list, ajq models pull, and ajq models use manage checksum-pinned local GGUF catalog models. |
| Semantic operators | Fuzzy matching (=~ / sem_match) and bounded sem_classify ship for filters and labels; sem_score and sem_norm are limited to supported contexts. Standalone sem_extract and sem_redact are registered but currently unsupported. |
| Semantic NDJSON execution | Supported semantic streams default to complete-frame, byte-budgeted windows (256 KiB by default) that batch and deduplicate judgements without buffering the complete stream. Use --stream when first-frame latency matters more than window batching and cross-frame pre-resolve deduplication; it keeps cache identity and --max-calls semantics unchanged. --window-bytes, AJQ_WINDOW_BYTES, or TOML window_bytes tune default windows. |
| Determinism contract | Pure jq paths stay byte-reproducible and never contact AI backends; only explicit semantic operators make schema-constrained, cache-keyed model calls by backend/model/spec/value. |
Docs
Everything beyond the quick start lives on the website:
- Install details
- First pipeline tutorial
- Filter JSON by meaning
- Classify JSON and NDJSON streams
- Semantic functions reference
- CLI reference
- Split execution and determinism
Contributor verification
make test
make build
make website-build
License
MIT. See LICENSE.
This project uses gojq, an implementation of jq in Go.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
agent-routing-eval
command
Command agent-routing-eval scores a locally captured blind-agent routing run.
|
Command agent-routing-eval scores a locally captured blind-agent routing run. |
|
ajq
command
|
|
|
internal
|
|
|
backend
Package backend defines the minimal semantic judgment seam used by the split executor.
|
Package backend defines the minimal semantic judgment seam used by the split executor. |
|
backend/anthropicbk
Package anthropicbk implements the official Anthropic Messages API backend.
|
Package anthropicbk implements the official Anthropic Messages API backend. |
|
backend/batchdispatch
Package batchdispatch runs semantic backend judgements with a bounded number of in-flight callbacks while retaining their input order.
|
Package batchdispatch runs semantic backend judgements with a bounded number of in-flight callbacks while retaining their input order. |
|
backend/conformance
Package conformance defines the backend acceptance suite shared by every ajq semantic backend implementation.
|
Package conformance defines the backend acceptance suite shared by every ajq semantic backend implementation. |
|
backend/local
Package local implements a semantic backend that talks to a warm localhost llama-server daemon over HTTP.
|
Package local implements a semantic backend that talks to a warm localhost llama-server daemon over HTTP. |
|
backend/oai
Package oai implements a thin OpenAI-compatible chat/completions backend.
|
Package oai implements a thin OpenAI-compatible chat/completions backend. |
|
backend/ollamabk
Package ollamabk implements a thin client for Ollama's native /api/chat API.
|
Package ollamabk implements a thin client for Ollama's native /api/chat API. |
|
backend/promptkit
Package promptkit provides shared deterministic prompt rendering and result coercion for semantic backends.
|
Package promptkit provides shared deterministic prompt rendering and result coercion for semantic backends. |
|
bench
Package bench is the Phase 2.5 latency and throughput benchmark harness for ajq.
|
Package bench is the Phase 2.5 latency and throughput benchmark harness for ajq. |
|
cache
Package cache provides in-memory semantic result memoization for split execution.
|
Package cache provides in-memory semantic result memoization for split execution. |
|
cachepath
Package cachepath resolves the shared ajq cache root used by provisioning, daemon-managed assets, and persistent judgement cache files.
|
Package cachepath resolves the shared ajq cache root used by provisioning, daemon-managed assets, and persistent judgement cache files. |
|
config
Package config loads ajq's optional TOML configuration file and resolves settings precedence across command-line flags, environment variables, config file values, and built-in defaults.
|
Package config loads ajq's optional TOML configuration file and resolves settings precedence across command-line flags, environment variables, config file values, and built-in defaults. |
|
daemon
Package daemon manages the lifecycle of a local llama-server process used as a warm localhost daemon by ajq.
|
Package daemon manages the lifecycle of a local llama-server process used as a warm localhost daemon by ajq. |
|
desugar
Package desugar rewrites ajq-only surface syntax into jq function-core syntax before gojq parses the query.
|
Package desugar rewrites ajq-only surface syntax into jq function-core syntax before gojq parses the query. |
|
engine
Package engine wires ajq's stdin framing, pure-jq evaluation, and output serialization together.
|
Package engine wires ajq's stdin framing, pure-jq evaluation, and output serialization together. |
|
explain
Package explain renders ajq's stable query explanation output.
|
Package explain renders ajq's stable query explanation output. |
|
jq
Package jq wraps github.com/itchyny/gojq behind the small execution contract ajq needs for deterministic pure-jq queries.
|
Package jq wraps github.com/itchyny/gojq behind the small execution contract ajq needs for deterministic pure-jq queries. |
|
plan
Package plan builds ajq's semantic execution plan from gojq queries.
|
Package plan builds ajq's semantic execution plan from gojq queries. |
|
pricing
Package pricing provides static, dated cost estimates for semantic backend calls.
|
Package pricing provides static, dated cost estimates for semantic backend calls. |
|
provision
Package provision implements first-run provisioning of local inference assets for ajq: it locates or downloads a platform-appropriate llama-server engine and a default GGUF model into the ajq cache directory, verifies their integrity, and reports progress.
|
Package provision implements first-run provisioning of local inference assets for ajq: it locates or downloads a platform-appropriate llama-server engine and a default GGUF model into the ajq cache directory, verifies their integrity, and reports progress. |
|
schema
Package schema turns semantic operator contracts into deterministic, grammar/schema-constrained output shapes for local inference.
|
Package schema turns semantic operator contracts into deterministic, grammar/schema-constrained output shapes for local inference. |
|
semantics
Package semantics defines ajq's semantic operator metadata without any jq parser or executor dependencies.
|
Package semantics defines ajq's semantic operator metadata without any jq parser or executor dependencies. |
|
testharness
Package testharness verifies ajq golden-output fixtures.
|
Package testharness verifies ajq golden-output fixtures. |
Click to show internal directories.
Click to hide internal directories.