README
¶
Confidential Inference Router
Tinfoil's confidential inference model router terminates TLS connections (optionally with EHBP), inspects the model name, and directs it to a verified secure inference enclave.
Request bodies
The router accepts OpenAI-compatible bodies on /v1/chat/completions and /v1/responses. A few Tinfoil-specific top-level fields are recognized and stripped before the body is forwarded to the model enclave:
code_execution_options— activates the code-execution tool profile. When code execution is requested, this object carries the per-request credentials (accessToken,encryptionKey,containerAuthToken).web_search_options— activates the web-search tool profile.pii_check_options— activates the PII safety check.
Tool Calling
Client side tool calling is handled by the client. Server-side tools currently supported: web search and code execution.
To handle prompting, we add some instructions about how to call each tool in the system prompt, and we have a short description in each tool.
vLLM handles putting the system prompt + the tool prompts together, using internal templates built for the specific models.
Documentation
¶
There is no documentation for this package.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package cacheroute implements cache-aware replica routing: it derives a routing key from the parsed request body, ranks a model's replicas with rendezvous hashing, and — per pool — either only measures what routing would have done (shadow) or drives replica selection (enforced, via Shadow.Decide feeding the manager's selectors).
|
Package cacheroute implements cache-aware replica routing: it derives a routing key from the parsed request body, ranks a model's replicas with rendezvous hashing, and — per pool — either only measures what routing would have done (shadow) or drives replica selection (enforced, via Shadow.Decide feeding the manager's selectors). |
|
Package cachesalt derives the per-request cache_salt values that partition vLLM's prefix cache between principals.
|
Package cachesalt derives the per-request cache_salt values that partition vLLM's prefix cache between principals. |