Documentation
¶
Overview ¶
Package recovery defines typed recovery recipes that decide what to do when a node fails. The runtime engine consults a recipe per error class to choose between retrying the same node, forcing a compaction, pausing for human intervention, or failing terminally.
The package is deliberately decoupled from the engine: a Recipe returns an Action describing what to do, leaving the actual execution (retry, compact, pause) to the caller. Hosts wire the dispatcher into the engine via runtime.WithRecoveryDispatch and recovery.Dispatch(DefaultRecipes()).
Index ¶
- Constants
- func Classify(err error) runtime.ErrorCode
- func DefaultRecipes() map[runtime.ErrorCode]Recipe
- func Dispatch(recipes map[runtime.ErrorCode]Recipe) runtime.RecoveryDispatch
- type Action
- type ActionKind
- type Recipe
- func AuthFailedRecipe() Recipe
- func BudgetRecipe() Recipe
- func ContextLengthRecipe() Recipe
- func ExecutionFailedRecipe(maxRetries int) Recipe
- func NetworkTransientRecipe(maxRetries int) Recipe
- func PermanentToolRecipe() Recipe
- func RateLimitRecipe(maxRetries int) Recipe
- func TransientToolRecipe(maxRetries int) Recipe
- func UsageLimitRecipe() Recipe
- type RecipeFunc
Constants ¶
const ( // ActionRetrySameNode re-executes the failing node, optionally // after Delay; AttemptsLeft tracks the remaining budget. ActionRetrySameNode = runtime.RecoveryRetrySameNode // ActionCompactAndRetry asks the LLM client to drop older // conversation turns (when supported) and then retry. Falls back // to a plain retry when the executor doesn't implement Compactor. ActionCompactAndRetry = runtime.RecoveryCompactAndRetry // ActionPauseForHuman pauses the run with a synthetic // interaction so an operator can resolve and resume. ActionPauseForHuman = runtime.RecoveryPauseForHuman // ActionFailTerminal surfaces the error as a non-recoverable // failure (still produces a checkpoint via failRunWithCheckpoint). ActionFailTerminal = runtime.RecoveryFailTerminal )
const DefaultMaxRetries = 3
DefaultMaxRetries is the per-class retry budget unless overridden per recipe.
Variables ¶
This section is empty.
Functions ¶
func Classify ¶
Classify inspects err and returns the canonical RuntimeError code for it. It recognises:
- *runtime.RuntimeError → its declared Code
- *delegate.ErrRateLimited → RATE_LIMITED (CLI-backend rate-limit signal raised when the assistant text matches the provider's quota wording; not an api.APIError because the CLI's wire is pre-parsed JSON, so the 429 never reaches the SDK as such)
- *api.APIError with StatusCode 429 → RATE_LIMITED
- *api.APIError with body containing "context_length_exceeded" or "context length" → CONTEXT_LENGTH_EXCEEDED
- any other *api.APIError → EXECUTION_FAILED
Hosts that want richer classification can wrap or replace this function.
func DefaultRecipes ¶
DefaultRecipes maps each well-known error code to its default recipe. Hosts can override individual entries before installing.
func Dispatch ¶
func Dispatch(recipes map[runtime.ErrorCode]Recipe) runtime.RecoveryDispatch
Dispatch turns a recipe map into a runtime.RecoveryDispatch callback. The dispatcher classifies, asks the engine for the prior attempt count under that class, and returns the recipe's decision plus the matched code. Errors that don't classify into a wired code fall through to RecoveryFailTerminal.
Safe for concurrent use as long as the wrapped recipes are.
Types ¶
type Action ¶
type Action = runtime.RecoveryAction
Action is the engine-facing decision returned by a recipe.
type ActionKind ¶
type ActionKind = runtime.RecoveryActionKind
ActionKind is an alias for the engine-facing decision kind so existing callers of the recovery package keep working after the engine wiring.
type Recipe ¶
type Recipe interface {
Apply(ctx context.Context, err *runtime.RuntimeError, attempts int) Action
}
Recipe decides what to do for a given error class. `attempts` is the count of prior retries for this class on this node (zero on first failure).
func AuthFailedRecipe ¶ added in v0.39.0
func AuthFailedRecipe() Recipe
AuthFailedRecipe: the model provider rejected our credentials (expired/invalid token, HTTP 401/403). Retrying the same call can never succeed — only a human re-authenticating can fix it. Pause for human (like BudgetRecipe) rather than burning the retry budget on guaranteed failures; the run is resumable once the credential is refreshed. This also keeps a dispatcher from re-dispatching the run in a tight loop (every cycle would re-spend sandbox + partial-run cost only to hit the same 401).
func BudgetRecipe ¶
func BudgetRecipe() Recipe
BudgetRecipe: always pause for human. There is no automatic retry path for budget exhaustion — operator must extend or terminate.
func ContextLengthRecipe ¶
func ContextLengthRecipe() Recipe
ContextLengthRecipe: compact-and-retry twice, then fail terminal (the conversation can't be made smaller).
func ExecutionFailedRecipe ¶ added in v0.39.0
ExecutionFailedRecipe: the catch-all bucket for unclassified node failures. Tries one retry with a short delay (covers transient subprocess crashes, flaky network blips, momentary fs races), then falls through to FailTerminal so the engine produces a failed_resumable checkpoint that the operator can /resume after fixing the root cause. Without this recipe registered, the first failure of any unclassified node short-circuits to FailTerminal with no retry attempted at all.
func NetworkTransientRecipe ¶ added in v0.39.0
NetworkTransientRecipe: longer exponential-backoff loop for transient network failures reaching the upstream model API (ISP blip, captive portal handoff, DNS flutter, datacenter routing change). Each attempt doubles the delay up to a 60s cap; with the default 6 retries that covers ~10 min of cumulative wait — enough to ride out the kind of outages an operator would expect a long-running pipeline to recover from on its own. Beyond that we surface as failed_resumable so the operator can /resume after the network is verified back.
Why a separate recipe (not just a bigger ExecutionFailedRecipe): non-network execution failures (schema mismatch, missing fs entry, in-sandbox script crash) usually won't fix themselves on retry — burning 6 * 60s on a deterministic bug wastes operator time. Network transients DO routinely fix themselves; rewarding the right pattern matters for unattended overnight runs.
Default cap = 6 attempts, 60s max backoff. Hosts override via DefaultRecipes map mutation if they want a different shape.
func PermanentToolRecipe ¶
func PermanentToolRecipe() Recipe
PermanentToolRecipe: no retry, immediately escalate to terminal.
func RateLimitRecipe ¶
RateLimitRecipe: exponential backoff + jitter, escalates to human pause after maxRetries (operator may rotate credentials).
func TransientToolRecipe ¶
TransientToolRecipe: retry with linear backoff up to maxRetries (model gets the error in its next turn), then fail terminal.
func UsageLimitRecipe ¶ added in v0.50.0
func UsageLimitRecipe() Recipe
UsageLimitRecipe: fail terminal immediately — a subscription/quota WINDOW (forfait 5h / session / weekly cap) cannot clear within a node's retry budget, so in-node retries only burn attempts. The run lands failed_resumable; the run-level auto-resume loop owns the reset-aware wait (pkg/cli/auto_resume.go).
type RecipeFunc ¶
func (RecipeFunc) Apply ¶
func (f RecipeFunc) Apply(ctx context.Context, err *runtime.RuntimeError, attempts int) Action
Source Files