btdby4

package module
v0.0.0-...-e67d6be Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 22, 2026 License: MIT Imports: 14 Imported by: 0

README

BTDby4 — Better Than Divide-by-4

One generic estimator for any LLM payload. BTDby4 scores a complete request — system, tools, text, images, visible thinking, encrypted thinking, tool I/O — with ~90% accuracy against any provider or model, in under a millisecond. One pure-Go JSON dependency (goccy/go-json, no CGO — it also builds for js/wasm). No provider config, no model tables, no per-API math: drop in any payload shape and get a breakdown you can enforce budgets, compaction and routing on.

import "github.com/italoalmeida0/btdby4"

n, _ := btdby4.CountText("hello world") // 2 — exact, not "len/4"

How it compares

Three kinds of counters exist: raw BPE encoders (exact text, nothing else), and the /4 guess. BTDby4 is the fourth option — a local, whole-payload estimator:

Capability BTDby4 tiktoken family (Python / Rust / Go) JS counters (js-tiktoken / gpt-tokenizer)
Text ✅ identical counts ✅ reference ⚠️ gpt-tokenizer drifts ~1.3%
Whole request in one call ✅ + per-turn Breakdown ❌ manual recipe ⚠️ chat only (gpt-tokenizer)
Protocols ✅ Anthropic + Chat + Responses ❌ OpenAI text ❌ OpenAI text
Images ✅ 28px-tile model ❌ ❌
Encrypted thinking ✅ auto-estimated ❌ invisible ❌ invisible
Offline, pure Go ✅ ⚠️ downloads BPE on first use ✅
Large-context latency (windows/arm64, p50)

Same realistic payloads per size (EN + code + JSON tool args + SQL + multilingual, ~4 chars/token). Counts identical across btdby4 / tiktoken / tiktoken-rs / js-tiktoken (gpt-tokenizer drifts, own table).

Context btdby4 tiktoken (Python) tiktoken-rs (Rust) js-tiktoken gpt-tokenizer Faster than best rival
256K (~1.0MB) 1ms 59ms 70ms 217ms 25ms ~25×
512K (~2.1MB) 3ms 117ms 137ms 1042ms 128ms ~39×
1M (~4.1MB) 5ms 217ms 799ms 2354ms 407ms ~43×
1.5M (~6.1MB) 9ms 398ms 1342ms 3738ms 478ms ~44×
Native Engine Throughput: Go Native vs Rust (Gigatoken & RS-BPE)

Tested on production conversational payloads (Windows ARM64):

Payload Context Size ⚡ btdby4 (Go Native) 🦀 gigatoken (Rust SIMD) 🦀 rs-bpe (Rust O(n)) Advantage
20K Tokens (~60 KB) 0.080 ms (689.4 MB/s) 0.105 ms (527.5 MB/s) 1.894 ms (29.3 MB/s) 23× faster than rs-bpe
50K Tokens (~150 KB) 0.234 ms (617.2 MB/s) 0.282 ms (511.6 MB/s) 4.966 ms (29.1 MB/s) 21× faster than rs-bpe
100K Tokens (~310 KB) 0.290 ms (764.4 MB/s) 0.543 ms (408.8 MB/s) 8.143 ms (27.3 MB/s) 28× faster than rs-bpe
200K Tokens (~630 KB) 0.655 ms (682.9 MB/s) 1.034 ms (433.2 MB/s) 16.812 ms (26.6 MB/s) 25× faster than rs-bpe
Full (~2.2 MB / 700K Tokens) 1.676 ms (765.2 MB/s) 3.100 ms (413.7 MB/s) 48.935 ms (26.2 MB/s) 1.8× faster than gigatoken, 29× faster than rs-bpe
  • Zero Memory Allocations: BenchmarkCountText runs with 0 B/op and 0 allocs/op.
  • End-to-End JSON Chat Processing: Parsing the entire 2.65 MB raw JSON payload, walking the structure, calculating protocol framing and tokenizing costs takes only 13.2 ms (190.8 MB/s of pure JSON unmarshaling + protocol analysis).

Tails stay flat for btdby4 while rivals fan out at 1.5M (p99): btdby4 10ms vs tiktoken 888ms, tiktoken-rs 1435ms, gpt-tokenizer 650ms, js-tiktoken 3928ms. Full p50/p90/p99 runs: 11 timed iterations each (7 on Node), same machine.

Whole-request cost (80 tool-use/tool-result turns, 7295 tokens): CountAnthropicRequest p50 <1µs (cached), p99 1.05ms — per-turn attribution for compaction is effectively free.

go test -run=NONE -bench=. -benchtime=1000000x -count=1 .

Why faster: single-pass BPE scan over a pre-merged rank table with a sharded count cache — no regex split per call, no encode allocation, no BPE download. tiktoken variants pay the full regex + merge loop every call; js-tiktoken adds JS/WASM overhead on top.

Why you'll use it

The /4 rule is silently eating your context. Code tokenizes near 1 char/token, JSON tool payloads blow up, non-English text drifts — teams either over-truncate (wasting context they paid for) or overflow mid-request (crashing long agent runs). BTDby4 scores the whole payload generically (real BPE for text and tool JSON, 28px-tile model for images, calibrated envelope math for encrypted thinking), so a generic AI harness can pack, meter and compact any request to the edge safely:

  • Context-window packing — fit one more tool result instead of leaving 20% headroom "just in case"
  • Cost metering — bill per estimated token, not per guess
  • Budget enforcement — kill runaway agent loops before the invoice does
  • Automatic compaction — trigger summarization off Total / ByMessage instead of provider-specific counters
  • Pre-flight checks — reject or compress oversized requests before paying for the API call

One call per API

// Anthropic Messages
out, _ := btdby4.CountAnthropicRequest(btdby4.AnthropicRequest{
    System: "You are a helpful assistant.",
    Messages: []btdby4.AnthropicMessage{
        {Role: "user", Content: "hello world"},
        {Role: "assistant", Content: []any{
            map[string]any{"type": "text", "text": "hi there"},
            map[string]any{"type": "tool_use", "id": "toolu_1",
                "name": "get_weather", "input": map[string]any{"city": "SP"}},
        }},
    },
    Tools: []btdby4.AnthropicTool{
        {Name: "get_weather", Description: "Get the current weather",
            InputSchema: map[string]any{"type": "object"}},
    },
}, btdby4.Options{} /* IgnoreImages */)

fmt.Println(out.Total, out.TextTokens, out.Images, out.ByMessage, out.ByTool)

// OpenAI Chat Completions
cout, _ := btdby4.CountChatRequest(btdby4.ChatRequest{
    Messages: []btdby4.ChatMessage{
        {Role: "system", Content: "You are a helpful assistant."},
        {Role: "user", Content: "hello world"},
    },
}, btdby4.Options{})

// OpenAI Responses — reasoning + encrypted thinking auto-estimated
rout, _ := btdby4.CountResponsesRequest(btdby4.ResponsesRequest{
    Instructions: "You are a helpful assistant.",
    Input: []any{
        map[string]any{"type": "message", "role": "user", "content": []any{
            map[string]any{"type": "input_text", "text": "hello world"},
        }},
    },
}, btdby4.Options{})

Every counter returns a Breakdown: per-message and per-tool counts plus separate TextTokens/Images/Total, so you can attribute spend per turn. Each protocol exposes the same granularities (plus *JSON variants that take raw bytes):

Level Anthropic Messages OpenAI Chat OpenAI Responses
Request CountAnthropicRequest CountChatRequest CountResponsesRequest
Message CountAnthropicMessage CountChatMessage CountResponsesItem
Part CountAnthropicBlock CountChatPart CountResponsesPart
Tool CountAnthropicTool CountChatTool CountResponsesTool

Supported content: Anthropic text / thinking (visible) / image (base64, URL, data-URL) / tool_use / tool_result / redacted_thinking; Chat text / reasoning_content (visible thinking) / image_url / input_audio (+ tool_calls); Responses message (input_text, output_text, reasoning_text, input_image), function_call, function_call_output, reasoning (summary + encrypted_content). Images use the standard 28px-tile visual-token model (1568px fit). Encrypted thinking (redacted_thinking, encrypted_content, thought_signature — any provider, any model, any protocol) is detected and estimated automatically; there is nothing to configure.

API Reference

All counters take an Options value (Tight, IgnoreImages) and return (Breakdown, error) for requests/messages/parts or (int, error) for text/tools. *JSON variants accept raw []byte instead of structs. Images-only helpers never fail and return a plain int.

Text
Function Signature Description
CountText CountText(text string) (int, error) Count tokens in a plain string. 
n, _ := btdby4.CountText("hello world") // 2
Anthropic Messages

Types: AnthropicRequest{System any, Messages []AnthropicMessage, Tools []AnthropicTool}, AnthropicMessage{Role string, Content any}, AnthropicTool{Name, Description string, InputSchema any}. Content accepts a plain string or []any of content blocks (text, image, tool_use, tool_result, redacted_thinking).

Function Signature Description
CountAnthropicRequest CountAnthropicRequest(req AnthropicRequest, opts Options) (Breakdown, error) Full request: system + messages + tools.
CountAnthropicRequestJSON CountAnthropicRequestJSON(raw []byte, opts Options) (Breakdown, error) Same, from raw JSON.
CountAnthropicMessage CountAnthropicMessage(msg AnthropicMessage, opts Options) (BlockBreakdown, error) Single message (string or blocks).
CountAnthropicMessageJSON CountAnthropicMessageJSON(raw []byte, opts Options) (BlockBreakdown, error) Same, from raw JSON.
CountAnthropicBlock CountAnthropicBlock(block map[string]any, opts Options) (BlockBreakdown, error) Single content block.
CountAnthropicBlockJSON CountAnthropicBlockJSON(raw []byte, opts Options) (BlockBreakdown, error) Same, from raw JSON.
CountAnthropicTool CountAnthropicTool(tool AnthropicTool) (int, error) Tool definition (name + description + schema).
CountAnthropicToolJSON CountAnthropicToolJSON(raw []byte) (int, error) Same, from raw JSON. 
mb, _ := btdby4.CountAnthropicMessage(
    btdby4.AnthropicMessage{Role: "user", Content: "hello world"},
    btdby4.Options{},
)
fmt.Println(mb.Tokens) // 2
OpenAI Chat Completions

Types: ChatRequest{System any, Messages []ChatMessage, Tools []ChatTool}, ChatMessage{Role string, Content any, ToolCalls []ChatToolCall, ToolCallID, Name string}, ChatToolCall{ID, Type string, Function ChatFunction}, ChatFunction{Name, Arguments string}, ChatTool{Type string, Function ChatFunctionDef}, ChatFunctionDef{Name, Description string, Parameters any, Strict bool}. Content accepts a plain string or []any of parts (text, image_url, input_audio).

Function Signature Description
CountChatRequest CountChatRequest(req ChatRequest, opts Options) (Breakdown, error) Full request: system + messages + tools.
CountChatRequestJSON CountChatRequestJSON(raw []byte, opts Options) (Breakdown, error) Same, from raw JSON.
CountChatMessage CountChatMessage(msg ChatMessage, opts Options) (BlockBreakdown, error) Single message (string, parts, or tool calls).
CountChatMessageJSON CountChatMessageJSON(raw []byte, opts Options) (BlockBreakdown, error) Same, from raw JSON.
CountChatPart CountChatPart(part map[string]any, opts Options) (BlockBreakdown, error) Single content part.
CountChatPartJSON CountChatPartJSON(raw []byte, opts Options) (BlockBreakdown, error) Same, from raw JSON.
CountChatTool CountChatTool(tool ChatTool) (int, error) Tool definition (name + description + parameters).
CountChatToolJSON CountChatToolJSON(raw []byte) (int, error) Same, from raw JSON. 
pb, _ := btdby4.CountChatPart(
    map[string]any{"type": "text", "text": "hello world"},
    btdby4.Options{},
)
fmt.Println(pb.Tokens) // 2
OpenAI Responses

Types: ResponsesRequest{Instructions any, Input any, Tools []ResponsesTool}, ResponsesTool{Type, Name, Description string, Parameters any, Strict bool}. Input accepts a plain value or []any of items (message, function_call, function_call_output, reasoning). Item content parts: input_text, output_text, text, input_image, summary_text. Multi-item reasoning shares one envelope overhead per request (the fitted per-call model); unit-level CountResponsesItem estimates each envelope in isolation.

Function Signature Description
CountResponsesRequest CountResponsesRequest(req ResponsesRequest, opts Options) (Breakdown, error) Full request: instructions + input items + tools.
CountResponsesRequestJSON CountResponsesRequestJSON(raw []byte, opts Options) (Breakdown, error) Same, from raw JSON.
CountResponsesItem CountResponsesItem(item map[string]any, opts Options) (BlockBreakdown, error) Single input item.
CountResponsesItemJSON CountResponsesItemJSON(raw []byte, opts Options) (BlockBreakdown, error) Same, from raw JSON.
CountResponsesPart CountResponsesPart(part map[string]any, opts Options) (BlockBreakdown, error) Single content part.
CountResponsesPartJSON CountResponsesPartJSON(raw []byte, opts Options) (BlockBreakdown, error) Same, from raw JSON.
CountResponsesTool CountResponsesTool(tool ResponsesTool) (int, error) Tool definition (non-function tools counted as JSON).
CountResponsesToolJSON CountResponsesToolJSON(raw []byte) (int, error) Same, from raw JSON. 
ib, _ := btdby4.CountResponsesItem(map[string]any{
    "type": "function_call", "name": "get_weather",
    "arguments": `{"city":"SP"}`,
}, btdby4.Options{})
fmt.Println(ib.Tokens) // name + arguments
Images
Function Signature Description
CountImageSize CountImageSize(width, height int) int Visual tokens for W×H (1568px fit, 28px tiles).
CountImageBytes CountImageBytes(b []byte) int Decode (PNG/JPEG/GIF) then count; 0 if undecodable.
CountImageBase64 CountImageBase64(s string) int Strip optional data: prefix, base64-decode, then count; 0 on error. 
btdby4.CountImageSize(1024, 768) // tiles after fit
Result types

Breakdown (requests) — System, Messages, Tools, Images (image-token subtotal), ByMessage []int, ByTool []int, TextTokens (System + Messages + Tools - Images), Total (System + Messages + Tools), ImageCount (number of image blocks).

BlockBreakdown (single message/item/part) — Tokens, Images, TextTokens (Tokens - Images), ImageCount.

Options
type Options struct {
    IgnoreImages bool // count images as zero
    Tight        bool // opt out of the safety margin: closest point estimate
}
Bias: over or under?

Total ships with a small generic safety margin on top of the content count (BPE text + tile area + encrypted-thinking estimate), so against billed prompt_tokens / input_tokens it lands at or above the real total in the common cases — install-and-use safe for budget enforcement and automatic compaction: compacting at 90% of the window fires before the real context fills up instead of after. The margin is generic (no per-provider or per-model tables): +12 per message, +500 with tools plus +45 per tool, +1100 per image — sized to cover the worst provider measured (tool-harness preamble ~500 fixed + ~40/tool; minimum billable cost per small image up to ~1000), while staying negligible at long context (a dozen tokens against 200k+). TextTokens always keeps the pure content count with no margin; pass Options{Tight: true} to get the closest point estimate in Total as well (display/cost paths). Without the margin the estimator would sit at or below billed usage by that same fixed overhead — fine for estimates, late for compaction triggers.

Encrypted thinking (redacted_thinking data, Responses encrypted_content, Google thought_signature — any provider, any model, any protocol, even a Gemini payload inside Responses or a GPT payload inside Anthropic blocks) is opaque ciphertext, so it is estimated, not decoded, and always on: the base64 envelope is unwrapped to raw bytes and mapped per envelope family (Fernet-style ~4.75 raw bytes/token past a ~752-byte overhead shared per request; Google-style compact signature ~2.76 raw bytes/token; jumbo single envelopes ≥ 8000 chars at 966 + raw/16.9). Calibrated on 14 measured payloads across three provider shapes (median error ~5%).

Helper Signature Description
EstimateThinkingTokens EstimateThinkingTokens(enc string) int Automatic estimate for one envelope (empty → 0).

🧠 KV-Cache Simulation & Multi-Provider Gateway Routing

Modern LLM inference engines (vLLM, SGLang, Anthropic Prompt Caching, OpenAI Prompt Caching, DeepSeek) reuse Key-Value (KV) attention states for common prompt prefixes.

BTDby4 includes a built-in production prefix-cache simulator (KvLookup, KvInit, KvStatsSnapshot, KvClear) designed specifically for AI Gateways (OpenRouter, LiteLLM, Cloudflare AI Gateway, Portkey) and multi-tenant agent systems.

Why this is essential for Multi-Provider Routing:
  • Pre-Flight Cost Estimation: Calculate exactly how many tokens will hit the cache (e.g. Anthropic's 90% prompt cache discount or OpenAI's 50% discount) before dispatching the request.
  • Intelligent Cache-Aware Routing: Route conversational traffic to the specific provider replica or model instance where the conversation prefix is already warm, slashing TTFT (time-to-first-token) by up to 80%.
  • Zero Double-Parsing: Uses a canonical FNV-1a block-level prefix trie with sliding TTL (default 10 minutes) and automatic LRU eviction (default 400 MB memory cap).
// 1. Initialize once at gateway startup (TTL 600s, 400MB memory cap)
btdby4.KvInit(600, 400)

// 2. Namespace your cache like real providers: "provider|model|customer-id"
ns := "anthropic|claude-3-7-sonnet|tenant-123"

// 3. Perform atomic lookup + state update
result, err := btdby4.KvLookup(rawJsonBytes, "anthropic", ns, btdby4.Options{})
if err != nil {
    log.Fatal(err)
}

fmt.Printf("Total: %d, Cached: %d (%.1f%% hit), Fresh: %d\n",
    result.Total, result.Cached, result.HitRatio*100, result.Fresh)

🌐 WebAssembly Bindings (btdby4-wasm)

BTDby4 also ships as a premier universal WebAssembly package on NPM: btdby4-wasm.

It is 10× to 27× faster than existing WASM tokenizers (kitoken, bpe-lite), bundles full offline vocabularies (no 30MB downloads), handles raw Chat JSON bodies directly, and includes the full KV-cache simulator.

import initBTDby4 from "btdby4-wasm";

const btdby4 = await initBTDby4();

// Direct OpenAI Chat Completions JSON token breakdown:
const breakdown = btdby4.countChatRequest(rawJsonString);
console.log(breakdown.total, breakdown.byMessage);

// Real-time KV cache simulation:
const kv = btdby4.kvCache(rawJsonString, "chat", "provider|model|user-1");
console.log(`Cached: ${kv.cached}, Fresh: ${kv.fresh}, Savings: ${kv.hit ? '90%' : '0%'}`);

For complete TypeScript / JavaScript documentation, benchmarks, and edge deployment guides, see wasm/README.md.

Install

go get github.com/italoalmeida0/btdby4

Requires Go 1.18+. One dependency (goccy/go-json, pure Go). Builds on windows/linux/darwin × amd64/arm64.

How it works

  • Counter, not encoder. BPE merges are evaluated only to learn how many tokens a piece produces, never which ones. Zero allocations on hot paths.
  • Hand-written splitter. The pre-tokenizer is a direct scanner — no regex engine at runtime.
  • Daemon-safe caches. Bounded sharded caches for pieces and texts; built for background services handling ever-changing requests for hours.
  • Parallel requests. Large requests fan out per message across all CPUs.

License

MIT — see LICENSE.

Documentation

Overview

Package btdby4 is a pure-Go estimator for whole LLM request payloads. It scores system, tools, text, images, visible and encrypted thinking, and tool I/O generically — ~90% accuracy against any provider or model, with nothing to configure beyond Options{Tight, IgnoreImages}. Counting is all you need for context windows, cost estimation, budget enforcement and automatic compaction — and estimating is much cheaper than encoding. JSON parsing uses goccy/go-json (pure Go, no CGO), which also builds for js/wasm.

Supported request shapes (each protocol has the same granularities: request, request-JSON, message/item, message/item-JSON, part/block, part/block-JSON, tool, tool-JSON): 

Anthropic Messages  CountAnthropicRequest / CountAnthropicMessage / CountAnthropicBlock / CountAnthropicTool (+JSON)
OpenAI Chat         CountChatRequest / CountChatMessage / CountChatPart / CountChatTool (+JSON)
OpenAI Responses    CountResponsesRequest / CountResponsesItem / CountResponsesPart / CountResponsesTool (+JSON)

The *JSON request entry points decode once and then count + linearize the prefix blocks in a single fused walk shared with the KV cache, so gateway calls pay one parse instead of two.

KV-cache simulation (KvLookup / KvInit / KvStatsSnapshot / KvClear): a high-performance prefix-cache simulator for gateways. Input is the raw request payload + protocol + namespace ("provider|model|api-key", assembled by the caller); output is cached/fresh/written token counts for billing cached input even when the provider does not report it. One prefix trie per protocol+namespace (or namespace only with KvSetSeparateProtocol(false)), 10-minute sliding TTL and a memory cap (default 400MB) with automatic LRU eviction — all tunable via KvInit. Token totals stay in stats for observability only.

Text is counted with the real BPE merge table; images follow the standard 28px-tile visual-token model. Pass Options{IgnoreImages: true} to count images as zero. Encrypted thinking (redacted_thinking, encrypted_content, thought_signature — whatever the provider or model hides inside any protocol) is detected and estimated automatically by EstimateThinkingTokens; there is nothing to configure.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CountAnthropicTool 

func CountAnthropicTool(tool AnthropicTool) (int, error)

CountAnthropicTool counts a tool definition.

func CountAnthropicToolJSON 

func CountAnthropicToolJSON(raw []byte) (int, error)

CountAnthropicToolJSON counts a tool definition from raw JSON.

func CountChatTool 

func CountChatTool(tool ChatTool) (int, error)

CountChatTool counts a tool definition.

func CountChatToolJSON 

func CountChatToolJSON(raw []byte) (int, error)

CountChatToolJSON counts a tool definition from raw JSON.

func CountImageBase64 

func CountImageBase64(s string) int

CountImageBase64 strips an optional data: prefix, base64-decodes, and counts visual tokens; 0 on error.

func CountImageBytes 

func CountImageBytes(b []byte) int

CountImageBytes decodes image bytes (PNG/JPEG/GIF) and counts visual tokens; 0 if undecodable.

func CountImageSize 

func CountImageSize(width, height int) int

CountImageSize returns visual tokens for a W×H image (1568px fit, 28px tiles).

func CountResponsesTool 

func CountResponsesTool(tool ResponsesTool) (int, error)

CountResponsesTool counts a tool definition.

func CountResponsesToolJSON 

func CountResponsesToolJSON(raw []byte) (int, error)

CountResponsesToolJSON counts a tool definition from raw JSON.

func CountText 

func CountText(text string) (int, error)

CountText counts tokens in a plain string.

func EstimateThinkingTokens 

func EstimateThinkingTokens(enc string) int

EstimateThinkingTokens estimates billed thinking tokens hidden behind an opaque encrypted payload (redacted_thinking, encrypted_content, thought_signature). Provider-agnostic: unwraps the base64 envelope to raw bytes and maps per envelope family. Empty input returns 0.

func KvClear 

func KvClear(namespace string)

KvClear removes one namespace, or everything when namespace == "". With protocol separation on (default), pass the plain namespace and all its protocol scopes are cleared; use KvClearProtocol for a single protocol scope.

func KvClearProtocol 

func KvClearProtocol(protocol, namespace string)

KvClearProtocol removes a single protocol scope of a namespace.

func KvInit 

func KvInit(ttlSeconds int64, maxMB int64)

KvInit configures the cache: ttlSeconds (default 600 = 10min) and maxMB (default 400). Call once at startup; zero/negative keeps the default for that field. It also clears existing entries, since TTL and size class are global.

func KvSetSeparateProtocol 

func KvSetSeparateProtocol(separate bool)

KvSetSeparateProtocol toggles protocol-scoped keying. Default true: protocol + namespace form the key. When false, only namespace keys the entries (same hash in different protocols shares the prefix).

Types

type AnthropicMessage 

type AnthropicMessage struct {
	Role    string `json:"role"`
	Content any    `json:"content"`
}

type AnthropicRequest 

type AnthropicRequest struct {
	System   any                `json:"system,omitempty"`
	Messages []AnthropicMessage `json:"messages"`
	Tools    []AnthropicTool    `json:"tools,omitempty"`
}

type AnthropicTool 

type AnthropicTool struct {
	Name        string `json:"name"`
	Description string `json:"description,omitempty"`
	InputSchema any    `json:"input_schema,omitempty"`
}

type BlockBreakdown 

type BlockBreakdown struct {
	Tokens     int `json:"tokens"`
	Images     int `json:"images"`
	TextTokens int `json:"text_tokens"`
	ImageCount int `json:"image_count"`
}

func CountAnthropicBlock 

func CountAnthropicBlock(block map[string]any, opts Options) (BlockBreakdown, error)

CountAnthropicBlock counts a single content block.

func CountAnthropicBlockJSON 

func CountAnthropicBlockJSON(raw []byte, opts Options) (BlockBreakdown, error)

CountAnthropicBlockJSON counts a single content block from raw JSON.

func CountAnthropicMessage 

func CountAnthropicMessage(msg AnthropicMessage, opts Options) (BlockBreakdown, error)

CountAnthropicMessage counts a single message.

func CountAnthropicMessageJSON 

func CountAnthropicMessageJSON(raw []byte, opts Options) (BlockBreakdown, error)

CountAnthropicMessageJSON counts a single message from raw JSON.

func CountChatMessage 

func CountChatMessage(msg ChatMessage, opts Options) (BlockBreakdown, error)

CountChatMessage counts a single message.

func CountChatMessageJSON 

func CountChatMessageJSON(raw []byte, opts Options) (BlockBreakdown, error)

CountChatMessageJSON counts a single message from raw JSON.

func CountChatPart 

func CountChatPart(part map[string]any, opts Options) (BlockBreakdown, error)

CountChatPart counts a single content part.

func CountChatPartJSON 

func CountChatPartJSON(raw []byte, opts Options) (BlockBreakdown, error)

CountChatPartJSON counts a single content part from raw JSON.

func CountResponsesItem 

func CountResponsesItem(item map[string]any, opts Options) (BlockBreakdown, error)

CountResponsesItem counts a single input item.

func CountResponsesItemJSON 

func CountResponsesItemJSON(raw []byte, opts Options) (BlockBreakdown, error)

CountResponsesItemJSON counts a single input item from raw JSON.

func CountResponsesPart 

func CountResponsesPart(part map[string]any, opts Options) (BlockBreakdown, error)

CountResponsesPart counts a single content part.

func CountResponsesPartJSON 

func CountResponsesPartJSON(raw []byte, opts Options) (BlockBreakdown, error)

CountResponsesPartJSON counts a single content part from raw JSON.

type Breakdown 

type Breakdown struct {
	System     int   `json:"system"`
	Messages   int   `json:"messages"`
	Tools      int   `json:"tools"`
	Images     int   `json:"images"`
	ByMessage  []int `json:"by_message,omitempty"`
	ByTool     []int `json:"by_tool,omitempty"`
	TextTokens int   `json:"text_tokens"`
	Total      int   `json:"total"`
	ImageCount int   `json:"image_count"`
}

func CountAnthropicRequest 

func CountAnthropicRequest(req AnthropicRequest, opts Options) (Breakdown, error)

CountAnthropicRequest counts a full Anthropic Messages request.

func CountAnthropicRequestJSON 

func CountAnthropicRequestJSON(raw []byte, opts Options) (Breakdown, error)

CountAnthropicRequestJSON counts a full request from raw JSON. The request is decoded once and then counted + linearized in a single walk (kvCountAnthropicRequest) shared with the KV cache path.

func CountChatRequest 

func CountChatRequest(req ChatRequest, opts Options) (Breakdown, error)

CountChatRequest counts a full Chat Completions request. NOTE: chat hot path is fused in kvCountChatRequest (single walk); this stays for struct-based callers.

func CountChatRequestJSON 

func CountChatRequestJSON(raw []byte, opts Options) (Breakdown, error)

CountChatRequestJSON counts a full request from raw JSON. The request is decoded once and then counted + linearized in a single walk (kvCountChatRequest) shared with the KV cache path: KvLookup reuses this so a gateway call pays one parse, not two.

func CountResponsesRequest 

func CountResponsesRequest(req ResponsesRequest, opts Options) (Breakdown, error)

CountResponsesRequest counts a full Responses request.

func CountResponsesRequestJSON 

func CountResponsesRequestJSON(raw []byte, opts Options) (Breakdown, error)

CountResponsesRequestJSON counts a full request from raw JSON. The request is decoded once and then counted + linearized in a single walk (kvCountResponsesRequest) shared with the KV cache path.

type ChatFunction 

type ChatFunction struct {
	Name      string `json:"name"`
	Arguments string `json:"arguments,omitempty"`
}

type ChatFunctionDef 

type ChatFunctionDef struct {
	Name        string `json:"name"`
	Description string `json:"description,omitempty"`
	Parameters  any    `json:"parameters,omitempty"`
	Strict      bool   `json:"strict,omitempty"`
}

type ChatMessage 

type ChatMessage struct {
	Role       string         `json:"role"`
	Content    any            `json:"content,omitempty"`
	ToolCalls  []ChatToolCall `json:"tool_calls,omitempty"`
	ToolCallID string         `json:"tool_call_id,omitempty"`
	Name       string         `json:"name,omitempty"`
}

type ChatRequest 

type ChatRequest struct {
	System   any           `json:"system,omitempty"`
	Messages []ChatMessage `json:"messages"`
	Tools    []ChatTool    `json:"tools,omitempty"`
}

type ChatTool 

type ChatTool struct {
	Type     string          `json:"type,omitempty"`
	Function ChatFunctionDef `json:"function"`
}

type ChatToolCall 

type ChatToolCall struct {
	ID       string       `json:"id,omitempty"`
	Type     string       `json:"type,omitempty"`
	Function ChatFunction `json:"function"`
}

type KvBlock 

type KvBlock struct {
	Hash   uint64
	Tokens int
}

KvBlock is one canonical position of the prefix: the FNV-1a hash identifying the block content plus its content tokens (no safety margin — margin is folded back proportionally at the end so that total == cached + fresh always holds).

type KvConfig 

type KvConfig struct {
	TTLSeconds       int64 `json:"ttl_seconds"`
	MaxMB            int64 `json:"max_mb"`
	SeparateProtocol bool  `json:"separate_protocol"`
}

KvConfig reports the effective tunables (for kvInit echo / stats).

func KvConfigSnapshot 

func KvConfigSnapshot() KvConfig

KvConfigSnapshot returns the effective config.

type KvResult 

type KvResult struct {
	Total        int       `json:"total"`
	Cached       int       `json:"cached"`
	Fresh        int       `json:"fresh"`
	Written      int       `json:"written"`
	Hit          bool      `json:"hit"`
	HitRatio     float64   `json:"hit_ratio"`
	PrefixBlocks int       `json:"prefix_blocks"`
	TotalBlocks  int       `json:"total_blocks"`
	Breakdown    Breakdown `json:"breakdown"`
}

KvResult is the per-request simulation outcome.

func KvLookup 

func KvLookup(raw []byte, protocol, namespace string, opts Options) (KvResult, error)

KvLookup is the atomic check+update: it walks the namespace trie with the linearized blocks, reports cached/fresh, then commits the miss suffix as a new branch so the next request can hit it. Old branches stay alive until TTL/eviction — N conversations can coexist under one namespace.

type KvStats 

type KvStats struct {
	Namespaces int   `json:"namespaces"`
	Nodes      int64 `json:"nodes"`
	Branches   int64 `json:"branches"`
	Tokens     int64 `json:"tokens"`
	Bytes      int64 `json:"bytes"`
	MaxBytes   int64 `json:"max_bytes"`
	// AvailableBytes is MaxBytes - Bytes (clamped at zero): how much
	// cache memory is still free before LRU eviction kicks in.
	AvailableBytes int64 `json:"available_bytes"`
	// TTLSeconds is the configured sliding TTL (KvInit, default 600).
	TTLSeconds int64 `json:"ttl_seconds"`
}

KvStats is a global introspection snapshot for the gateway.

func KvStatsSnapshot 

func KvStatsSnapshot() KvStats

KvStatsSnapshot returns a global snapshot (nodes/tokens/bytes are tracked incrementally; branches are counted by traversal as leaf nodes). It also runs the opportunistic maintenance (expiry sweep + memory-cap eviction), so a 1-minute polling loop doubles as the janitor: idle namespaces are collected even without traffic on them.

type Options 

type Options struct {
	IgnoreImages bool
	// Tight disables the built-in safety margin and returns the
	// closest point estimate. Default (false) keeps the margin.
	Tight bool
}

Options tunes counting. IgnoreImages counts images as zero.

Total always includes a small generic safety margin on top of the content count, so the default estimate lands at or above the provider-billed total in the common cases — install-and-use safe for budget enforcement and automatic compaction (compacting on Total fires before the real context fills up instead of after). TextTokens keeps the pure content count; set Tight: true to get the closest point estimate in Total as well (display/cost paths).

type ResponsesRequest 

type ResponsesRequest struct {
	Instructions any             `json:"instructions,omitempty"`
	Input        any             `json:"input,omitempty"`
	Tools        []ResponsesTool `json:"tools,omitempty"`
}

type ResponsesTool 

type ResponsesTool struct {
	Type        string `json:"type"`
	Name        string `json:"name,omitempty"`
	Description string `json:"description,omitempty"`
	Parameters  any    `json:"parameters,omitempty"`
	Strict      bool   `json:"strict,omitempty"`
}

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.