safety

Documentation

Overview

Package safety implements SafetyPolicy and SafetyLayer: the five-step approval-evaluation order (Always → Bulk → RoleOverride → RequireApproval → AutoApprove), rate limiting, token budgets, and tool allowlists.

Delete operations are AlwaysRequireApproval and cannot be overridden by any configuration (TAD §12.1 step 1, PRD §28.1).

See TAD §12 and PRD §28 for the full specification. Implemented in Phase 8.

Package safety implements the Agent Safety Layer: approval policy, rate limiting, token budgets, and the tool allowlist (TAD §12, PRD §25.3, §28).

The load-bearing contract is the approval evaluation order in TAD §12.1 — Always → Bulk → RoleOverride → RequireApproval → AutoApprove, fail-closed default — with the policy_reason values of §12.3 surfaced so the UI can render branch-specific copy (e.g. a bulk delete cannot be dismissed with a "don't ask again" option).

Index

• func SessionFromContext(ctx context.Context) string
• func VerbFor(toolName string, args map[string]any) string
• func WithSession(ctx context.Context, sessionID string) context.Context
• type Approval
• type ApprovalReason
• type Layer
  • func NewLayer(policy SafetyPolicy, store cache.Store) *Layer
  • func (l *Layer) CheckRateLimit(ctx context.Context, id auth.Identity) error
  • func (l *Layer) CheckTokenBudget(ctx context.Context, sessionID string, projected int) error
  • func (l *Layer) FilterTools(names []string) []string
  • func (l *Layer) IsToolAllowed(toolName string) bool
  • func (l *Layer) Policy() SafetyPolicy
  • func (l *Layer) RequiresApproval(ctx context.Context, id auth.Identity, toolName string, args map[string]any) bool
  • func (l *Layer) RequiresApprovalWithReason(ctx context.Context, id auth.Identity, info ToolCallInfo) Approval
  • func (l *Layer) WithUsageSource(s UsageSource)
• type RateLimit
• type SafetyLayer
• type SafetyPolicy
• type ToolCallInfo
• type UsageSource

Functions

func SessionFromContext

func SessionFromContext(ctx context.Context) string

SessionFromContext extracts the session id carried by WithSession.

func VerbFor

func VerbFor(toolName string, args map[string]any) string

VerbFor derives a tool call's semantic verb: the lowercase workflow action for execute_action_* tools, otherwise the verb prefix before the first underscore (create_employee → "create"). The read tool prefix "get" maps to the canonical "read" verb so the AutoApprove/RequireApproval verb sets in TAD §12 read naturally. Unknown verbs fail closed at the evaluation tail, matching PRD §28.1's "custom methods (side effects) require approval".

func WithSession

func WithSession(ctx context.Context, sessionID string) context.Context

WithSession returns a copy of ctx carrying the agent session id, so the layer can key session-scoped rate limits (TAD §12.2) and the executor can attribute audit entries.

Types

type Approval

type Approval struct {
	Required    bool
	Reason      ApprovalReason
	TargetCount int
}

Approval is the outcome of the TAD §12.1 evaluation. Reason is the matching policy_reason (empty when not required).

type ApprovalReason

type ApprovalReason string

ApprovalReason is the policy_reason value of the TAD §12.3 payload.

const (
	ReasonAlwaysRequireApproval ApprovalReason = "AlwaysRequireApproval"
	ReasonBulkLimit             ApprovalReason = "BulkLimit"
	ReasonRoleOverride          ApprovalReason = "RoleOverride"
	ReasonRequireApproval       ApprovalReason = "RequireApproval"
)

policy_reason values — one per branch of TAD §12.1, surfaced verbatim in the approval_required WebSocket payload so the UI can render distinct copy.

type Layer

type Layer struct {
	// contains filtered or unexported fields
}

Layer is the concrete SafetyLayer. Use it when the executor needs the policy_reason: RequiresApprovalWithReason (RequiresApproval delegates to it with a ToolCallInfo derived from the name/args alone).

func NewLayer

func NewLayer(policy SafetyPolicy, store cache.Store) *Layer

NewLayer builds a SafetyLayer with PRD-defaulted policy values. store may be nil (rate limiting disabled). The default policy matches PRD §28.1: AutoApprove for reads/searches/lists, RequireApproval for creates/updates, AlwaysRequireApproval for deletes, MaxBulkOperations = 5.

func (*Layer) CheckRateLimit

func (l *Layer) CheckRateLimit(ctx context.Context, id auth.Identity) error

CheckRateLimit enforces a sliding-window operation counter keyed "ratelimit:{scope}:{id}" in cache.Store. With Scope "session" the id is the session id carried on the context (SessionFromContext), falling back to the user id; with "user" it is the user id. No identity or nil store → no limit.

func (*Layer) CheckTokenBudget

func (l *Layer) CheckTokenBudget(ctx context.Context, sessionID string, projected int) error

CheckTokenBudget rejects an LLM call whose accumulated session usage plus the projected tokens would exceed TokenBudgetPerSession. A nil usage source or a zero budget disables the check.

func (*Layer) FilterTools

func (l *Layer) FilterTools(names []string) []string

FilterTools returns a copy of names with allowlist-blocked tools removed. The runtime applies this to ForIdentity output (TAD §10.3 step 3).

func (*Layer) IsToolAllowed

func (l *Layer) IsToolAllowed(toolName string) bool

IsToolAllowed reports whether toolName is permitted by the allowlist. An empty allowlist permits every generated and custom tool.

func (*Layer) Policy

func (l *Layer) Policy() SafetyPolicy

Policy returns a copy of the effective policy (defaults applied).

func (*Layer) RequiresApproval

func (l *Layer) RequiresApproval(ctx context.Context, id auth.Identity, toolName string, args map[string]any) bool

func (*Layer) RequiresApprovalWithReason

func (l *Layer) RequiresApprovalWithReason(ctx context.Context, id auth.Identity, info ToolCallInfo) Approval

RequiresApprovalWithReason evaluates in the TAD §12.1 order; the first match wins (fail-closed default at the end):

1. AlwaysRequireApproval — delete_* tools are folded in here unconditionally (§10.1 step 5: cannot be bypassed by policy config).
2. Bulk check — TargetCount (or the largest array arg) > MaxBulkOperations.
3. RequireApprovalForRoles[callerRole].
4. RequireApproval — the configured default set.
5. AutoApprove — everything else; an unrecognized verb requires approval.

func (*Layer) WithUsageSource

func (l *Layer) WithUsageSource(s UsageSource)

WithUsageSource wires a token-usage source (e.g. the llm.Gateway) so CheckTokenBudget can see the session's accumulated usage (TAD §12.2).

type RateLimit

type RateLimit struct {
	// OperationsPerMinute is the sliding-window cap. 0 selects the default
	// (60). Backed by cache.Store keys of the form "ratelimit:{scope}:{id}".
	OperationsPerMinute int
	// Scope is "user" or "session" (TAD §12.2).
	Scope string
}

RateLimit configures agent operation rate limiting (TAD §12).

type SafetyLayer

type SafetyLayer interface {
	// RequiresApproval reports whether a tool call needs human approval
	// before execution. Evaluates in the TAD §12.1 order.
	RequiresApproval(ctx context.Context, id auth.Identity, toolName string, args map[string]any) bool
	// CheckRateLimit enforces the sliding-window operation throttle keyed
	// "ratelimit:{scope}:{id}" in cache.Store. Returns an error when the
	// window is exhausted.
	CheckRateLimit(ctx context.Context, id auth.Identity) error
	// CheckTokenBudget rejects a call whose projected tokens (accumulated
	// usage + projected) would exceed the session budget.
	CheckTokenBudget(ctx context.Context, sessionID string, projected int) error
	// IsToolAllowed enforces the allowlist. Empty allowlist = allow all.
	IsToolAllowed(toolName string) bool
}

SafetyLayer is the TAD §12 interface exactly as specified.

type SafetyPolicy

type SafetyPolicy struct {
	// AutoApprove lists verbs that never prompt for approval (e.g. "read",
	// "search", "list").
	AutoApprove []string
	// RequireApproval lists verbs that always prompt (e.g. "create",
	// "update", "submit").
	RequireApproval []string
	// AlwaysRequireApproval lists verbs that can never be overridden by any
	// other policy branch (e.g. "delete", "cancel"). delete_* tools are
	// added to this set unconditionally by the layer itself (TAD §12.1
	// step 1, §10.1 step 5) — config cannot remove them.
	AlwaysRequireApproval []string
	// MaxBulkOperations is the record-count threshold above which approval is
	// required regardless of verb (PRD §28.1). 0 selects the default of 5.
	MaxBulkOperations int
	// RequireApprovalForRoles maps a role to the verbs it must confirm; an
	// empty verb list means every verb (PRD §28.3's "Interns confirming
	// everything").
	RequireApprovalForRoles map[string][]string
	// RateLimit is the per-user/per-session operation throttle.
	RateLimit RateLimit
	// TokenBudgetPerSession caps accumulated prompt+completion tokens per
	// session. 0 disables the budget.
	TokenBudgetPerSession int
	// ToolAllowlist restricts which tools the agent may call; empty = every
	// generated and custom tool (TAD §12). Applied by the runtime as the
	// final filter on ForIdentity output (TAD §10.3 step 3) and re-checked at
	// execution time.
	ToolAllowlist []string
}

SafetyPolicy is the agent-specific security policy (TAD §12, PRD §25.3). Verb lists are matched case-insensitively against a tool's semantic verb (e.g. "create", "update", "submit", "delete", "read", "search", "list").

type ToolCallInfo

type ToolCallInfo struct {
	// Verb is the semantic verb: the tool prefix ("create", "get", ...) or
	// the lowercase workflow action for execute_action_* tools.
	Verb string
	// ToolName is the tool name the model requested (e.g. "create_employee").
	ToolName string
	// Args is the tool's argument map.
	Args map[string]any
	// TargetCount is the estimated number of records the call affects (0 =
	// unknown, derived from array args if any).
	TargetCount int
}

ToolCallInfo carries everything the layer needs to evaluate one tool call. The executor derives Verb (e.g. "submit" for execute_action with action="Submit") and TargetCount (from a prior list/search result in the session transcript, TAD §12.1 step 2); callers using the plain RequiresApproval method get both derived from the tool name and args.

type UsageSource

type UsageSource interface {
	UsageFor(key string) llm.TokenUsage
}

UsageSource provides accumulated token usage per key, so the layer can enforce per-session budgets (TAD §12.2). The llm.Gateway satisfies it.