internal/

Package backend tracks a set of inference backends grouped by the models they serve, health-checks them, and hands out the next healthy one for a given model in round-robin order.

Package cache implements semantic response caching backed by NuclaDB: it embeds the request's prompt text, looks up the nearest cached vector, and serves the stored response verbatim on a close-enough match.

Package config loads inferoute's JSON configuration file: the set of inference backends to route to and how often to health-check them.

Package metrics holds the Prometheus collectors inferoute exposes on /metrics: request volume/latency by model and outcome, and cache hit rate.

Package proxy implements the OpenAI-compatible HTTP handler that reads the "model" field from the request body (resolving it through Aliases first, if set), picks a healthy backend for it, and forwards the request, retrying the next backend on failure and streaming the response through unbuffered for SSE.

Package ratelimit provides an HTTP middleware that token-bucket limits requests per API key (falling back to remote IP when no key is present).