Documentation
¶
Overview ¶
Package sqlsandbox runs ONE read-only SQL statement against the Parquet copy in a separate child process, with hard caps, killable without touching the parent (#1952, slice 1).
Why a child process: `bintrail-console watch` is the web console AND the capture plane in one process. A query that exhausts memory in that process is a capture outage, not a slow page. DuckDB's memory_limit bounds its buffer manager, not the process, and a runaway query cannot be interrupted from Go without cooperation. A child process can simply be killed: the parent sets a wall-clock deadline, and on timeout or cancel kills the child's whole process group. Nothing the child does can reach the parent's memory, file handles or DuckDB sessions.
Why the session is locked (all verified on DuckDB v1.4.5, the pinned 1.4 LTS line; see lockdownStatements for what each setting blocks): the copy is plain Parquet on the index host's disk, next to credentials, other customers' backups and the operator's home directory. The worker opens an in-memory DuckDB, admits only the copy directories, turns external access off, turns extension autoload off, turns spilling off, installs the views the console already generates, and locks the configuration LAST so the user's SQL cannot undo any of it.
Why a single SELECT with an allowlist of table functions: allowed_directories admits WRITES into the copy (COPY ... TO a path under it succeeds on 1.4.5, verified), so the statement shape is checked with DuckDB's own parser (json_serialize_sql) before anything runs. Only a SELECT-shaped statement (SELECT, FROM-first, VALUES, WITH, set operations, DESCRIBE, SHOW, SUMMARIZE) is admitted, and only one. And a SELECT is not read-only by shape alone: table functions such as enable_logging(...), checkpoint() or query('...') change state or run arbitrary text, and they run AFTER the lock (verified on 1.4.5). So every table function in the statement, at any depth, must be on an allowlist of readers.
v1 reads a LOCAL copy. A copy that lives only on S3 is refused with ErrCopyNotLocal; the worker never loads httpfs or opens the network.
One limit to know: DuckDB checks allowed_directories against the path TEXT, not against where it resolves, so a symbolic link planted INSIDE the copy that points outside is readable through it (verified on 1.4.5). The copy is written by the daemon and owned by the operator, so that link has to be put there on purpose; the parent admits a copy directory that is itself a link by also allowing its resolved path (a linked backup directory is an ordinary setup), and a link below it stays out of scope here.
Why the process is also bounded from outside DuckDB: memory_limit bounds the buffer manager, not the process (a single huge string is built outside it, verified), so on Linux the worker raises its own oom_score_adj to be the kernel's first choice ahead of the console, cells are cut at a fixed size, the result is counted in bytes on both sides of the pipe, and the runner caps how many workers run at once.
The parent API is Runner.Run. Slice 2 (the panel and its HTTP route) calls it with a Job built from the selected server's copy location and the views text from the same generator that serves /api/views.sql; slice 3 adds the permission gate and the audit emission around the call.
Index ¶
Constants ¶
const DefaultMaxInFlight = 2
DefaultMaxInFlight is how many workers a Runner runs at once by default: two, so one heavy query and one quick one can overlap on a host that also captures, and no more. The per-user gate is one per user on top of this.
const MaxCellBytes = 1 << 20
MaxCellBytes is the size a single cell is cut at inside the worker. A longer value is truncated with a marker and counted in Result.TruncatedCells. The cut bounds what travels over the pipe and what the parent holds, NOT the worker's memory: the driver materializes the full value before the cut (an 800 MB repeat() still exists in the worker for a moment), and it is oom_score_adj, not this, that protects the host when that goes wrong. A cut JSON or nested value is no longer valid JSON, and the count says how many cells were cut, not which.
const WorkerCommand = "__sql-worker"
WorkerCommand is the hidden subcommand the console binary registers for the child. The double underscore keeps it out of usage telemetry the same way cobra's own __complete is kept out.
Variables ¶
var ErrBusy = errors.New("a query is already running (yours, or the server is at its limit); wait for it to finish")
ErrBusy: the user already has a query running, or the runner is at its limit of concurrent workers.
var ErrCopyNotLocal = errors.New("running SQL on the copy needs a local copy directory; this copy lives only on S3")
ErrCopyNotLocal: the job names no local copy directory. v1 reads a local copy only; a copy that lives only on S3 is not queryable here.
var ErrResultTooLarge = errors.New("the result is too large to return; narrow the query")
ErrResultTooLarge: the result would exceed Limits.MaxResultBytes. Either side may say so: the worker counts as it collects rows, and the parent counts what it reads from the pipe.
Functions ¶
func IsWorkerInvocation ¶
IsWorkerInvocation reports whether args (os.Args) start a worker: the first argument is WorkerCommand. Embedders that do work in main() before calling consoleapp.Main should check this first (consoleapp.RunSQLWorkerIfInvoked does), so that work is not repeated for every query.
func IsWorkerProcess ¶
func IsWorkerProcess() bool
IsWorkerProcess reports whether this process was started as a worker by a Runner (the marker is in the environment). The console's hidden command is selected by argument; the package's own test binary uses this.
func WorkerMain ¶
WorkerMain is the child's whole life: read one job from stdin, run it, write one result to stdout, exit. The exit code is 0 whenever a result (including a structured error) was written; anything else is a protocol failure the parent reports as a WorkerError with stderr attached.
Types ¶
type Column ¶
Column is one result column with DuckDB's type name (INTEGER, VARCHAR, DECIMAL(10,2), TIMESTAMP, ...).
type Config ¶
type Config struct {
// Exe is the worker executable; empty means the current binary, which
// is what the console uses (on Linux /proc/self/exe, so a package
// upgrade mid-run keeps re-executing the binary this process runs, not
// a newer one with a different protocol; elsewhere os.Executable).
Exe string
// Args are the worker's arguments; nil means []string{WorkerCommand}, the
// console's hidden subcommand. A non-nil empty slice means no arguments
// (the package's tests re-execute the test binary).
Args []string
// Limits are the defaults every Job runs under unless it overrides them;
// zero fields fall back to DefaultLimits.
Limits Limits
// MaxInFlight caps how many workers run at once across all users; 0
// means DefaultMaxInFlight.
MaxInFlight int
}
Config builds a Runner.
type Job ¶
type Job struct {
// User keys the one-query-at-a-time gate. Empty runs ungated.
User string
// CopyDirs are the LOCAL directories the query may read: the archive
// directory and the backup directory of the selected server. Every entry
// must be an absolute local path; an s3:// location or an empty list is
// ErrCopyNotLocal.
CopyDirs []string
// ViewsSQL is the views script the console generates for the same server
// (views.Generate), installed in the worker before the query runs. It
// must reference nothing outside CopyDirs.
ViewsSQL string
// SQL is the user's statement.
SQL string
// Limits override the Runner's defaults field by field.
Limits Limits
}
Job is one query to run.
type Limits ¶
type Limits struct {
// Threads is DuckDB's thread count inside the worker.
Threads int
// MemoryLimit is DuckDB's memory budget inside the worker, e.g. "2GB".
// Spilling is off, so a query past it fails with an out-of-memory error
// instead of slowing down and writing to disk.
MemoryLimit string
// Timeout is the wall clock for the whole child: start, views install and
// query. Past it the process group is killed.
Timeout time.Duration
// MaxRows is how many result rows come back. The worker fetches one more
// to know whether the result was cut, and reports it as Truncated.
MaxRows int
// MaxResultBytes bounds the encoded result the parent will read from the
// child. Past it the child is killed and ErrResultTooLarge is returned.
MaxResultBytes int64
}
Limits are the caps one query runs under. A zero field means "the Runner's default" on a Job and "the package default" on a Config.
func DefaultLimits ¶
func DefaultLimits() Limits
DefaultLimits are the caps the issue asks for: 2 threads, 2 GB, 60 s, 1,000 rows. The byte cap is a defensive bound on the wire, not a product number: 1,000 rows of 16 KiB statement texts fit with room.
type QueryError ¶
type QueryError struct{ Message string }
QueryError: DuckDB ran (or tried to run) the statement inside the sandbox and failed. Message is DuckDB's own text: a Permission Error for a path outside the copy, an Out of Memory Error past the cap, a Binder Error for an unknown column.
func (*QueryError) Error ¶
func (e *QueryError) Error() string
type RefusedError ¶
type RefusedError struct{ Reason string }
RefusedError: the statement was not run at all because of its shape (empty, more than one statement, not a SELECT, a syntax error).
func (*RefusedError) Error ¶
func (e *RefusedError) Error() string
type Result ¶
type Result struct {
Columns []Column `json:"columns"`
Rows [][]any `json:"rows"`
Truncated bool `json:"truncated"`
TruncatedCells int `json:"truncated_cells"`
Elapsed time.Duration `json:"elapsed_ns"`
}
Result is what came back. Cells are JSON-shaped values a browser can show as they are: nil, bool, string, json.Number (every digit kept), []any for a LIST and map[string]any for a STRUCT or JSON. Dates are "2006-01-02", timestamps RFC 3339 in UTC, HUGEINT and DECIMAL are text, a BLOB is its text when valid UTF-8 and "0x.." otherwise, NaN and infinities are text. A text cell longer than MaxCellBytes is cut and ends with a marker; TruncatedCells counts them.
type Runner ¶
type Runner struct {
// contains filtered or unexported fields
}
Runner spawns workers and keeps the per-user and the global gate.
func (*Runner) Run ¶
Run runs one job to completion and returns its result, or one of the typed errors above (RefusedError, QueryError, TimeoutError, WorkerError, ErrCopyNotLocal, ErrBusy, ErrResultTooLarge) or ctx.Err() when the caller cancelled. Refusals that need no worker (empty SQL, no local copy, user busy) return before any process starts.
type TimeoutError ¶
TimeoutError: the worker ran past Limits.Timeout and was killed.
func (*TimeoutError) Error ¶
func (e *TimeoutError) Error() string
type WorkerError ¶
WorkerError: the child could not start, died without a result, or could not prepare its session. Stderr carries the child's last output.
func (*WorkerError) Error ¶
func (e *WorkerError) Error() string
func (*WorkerError) Unwrap ¶
func (e *WorkerError) Unwrap() error
Source Files