ai-concurrency-shaper

command module

v0.1.0 Latest Latest Go to latest Published: Jun 10, 2026 License: GPL-3.0 Imports: 18 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/joeycumines/ai-concurrency-shaper

Links

Open Source Insights

README ¶

ai-concurrency-shaper

Reverse proxy with bounded concurrency for AI/LLM API endpoints.

Sits in front of an upstream HTTP API (e.g. Anthropic, OpenAI) and limits concurrent requests to configured routes. Requests that exceed the limit block until a slot opens — no client-side backoff needed. Non-matching requests pass through unmodified.

Run with -tui for a terminal dashboard with live metrics and request inspection.

TUI dashboard showing live request metrics

Install

go install github.com/joeycumines/ai-concurrency-shaper@latest

Usage

ai-concurrency-shaper -upstream https://api.anthropic.com

Flags

Flag	Default	Description
`-upstream`	(required)	Upstream base URL
`-bind`	`:8080`	Listen address
`-limit`	(repeatable)	Route pattern to limit (defaults to common AI endpoints)
`-concurrency`	`4`	Max concurrent limited requests
`-global-concurrency`	`0`	Global concurrency limit (0 = disabled)
`-queue-timeout`	`30s`	Max wait for a concurrency slot
`-retry`	`-1`	Max retry attempts (-1 = unlimited, 0 = disabled)
`-retry-max-body-mb`	`5`	Max request body size (MB) eligible for retry
`-tui`	`false`	Enable terminal dashboard
`-version`		Print version and exit

Examples

Proxy with default limits and TUI:

ai-concurrency-shaper -upstream https://api.anthropic.com -tui

Custom concurrency with per-route limits:

ai-concurrency-shaper \
  -upstream https://api.openai.com \
  -limit "POST /v1/chat/completions:2" \
  -limit "POST /v1/embeddings:4" \
  -global-concurrency 10

Grouped routes sharing a limiter:

ai-concurrency-shaper \
  -upstream https://api.anthropic.com \
  -limit "POST /v1/messages@messages:3" \
  -limit "POST /v1/messages/batches@messages:3"

Building

make build          # compile
make test           # run tests
make lint           # vet + staticcheck + deadcode
make all            # build, then lint + test

License

GPL-3.0

Documentation ¶

Overview ¶

Command ai-concurrency-shaper is a stealth reverse proxy with bounded concurrency for configured routes.

Source Files ¶

View all Source files

main.go

Directories ¶

Path	Synopsis
internal
journal Package journal provides a thread-safe ring buffer of HTTP request/response pairs for both retry body replay and the TUI's Network inspection panel.	Package journal provides a thread-safe ring buffer of HTTP request/response pairs for both retry body replay and the TUI's Network inspection panel.
metrics Package metrics collects and exposes live proxy metrics via atomics.	Package metrics collects and exposes live proxy metrics via atomics.
proxy Package proxy implements a concurrency-bounded reverse proxy.	Package proxy implements a concurrency-bounded reverse proxy.
queue Package queue provides a single blocking concurrency limiter with best-effort wait semantics.	Package queue provides a single blocking concurrency limiter with best-effort wait semantics.
retry Package retry implements a retrying HTTP transport.	Package retry implements a retrying HTTP transport.
route Package route provides fuzzy HTTP route pattern matching for concurrency-limited LLM API endpoints.	Package route provides fuzzy HTTP route pattern matching for concurrency-limited LLM API endpoints.
tui Package tui provides a Bubble Tea v2 terminal dashboard for the proxy.	Package tui provides a Bubble Tea v2 terminal dashboard for the proxy.
tui/tuitest Package tuitest provides PTY-based integration tests for the TUI dashboard.	Package tuitest provides PTY-based integration tests for the TUI dashboard.

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL