Documentation
¶
Overview ¶
Package selection decides which Go packages a CI run tests and how the two long lanes are sharded (epic memql#5476; design record docs/superpowers/specs/2026-09-16-pipelines-program-design.md, section 4, epic 1).
It is pure over values. A Graph, a list of changed files, a class table and a timing table go in; a Plan comes out. The only functions that touch the machine are GoList and NodeTags, and they are what the CLI in scripts/ci/affected wires in. Every decision is tested on a recorded graph fixture (testdata/graph.json) with no toolchain in the loop, and one live test holds the loader to the real tree.
What the graph is a union OF ¶
A package's test binary depends on different packages under different build tags: app/build_agent.go imports integrations/agent only when `-tags agent` is set. CI tests the default build AND every node tag, so a change that reaches app only through an agent-tagged import still has to run app's tests. The graph is therefore the union over every configuration CI tests, which over-approximates each one. Over-approximating is the safe direction: a selection that is too wide costs minutes, one that is too narrow merges a red build.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NodeTags ¶
NodeTags returns the node build tags, read from app/build_<type>.go: the canonical node-type list that scripts/ci/node_type_lists_test.go holds every other copy to, so the graph cannot miss a tag the images are built with.
func ParseGoTestOutput ¶
ParseGoTestOutput collects one duration sample per passing package line, keyed by package directory (the import path with prefix stripped, "." for the root). Lines for packages outside prefix are ignored. Only PASSING lines count: a failed package's duration is cut short by the failure or the timeout, and would teach the table the wrong weight.
Types ¶
type Class ¶
type Class struct {
Lane string // "go" or "db"
Name string // e.g. "memql"; also the shard name's stem
Trees []string // repo-relative dirs; empty for the lane's default class
Timeout string // the `go test -timeout` value, e.g. "600s"
Serial bool // `-p=1`: the class's packages never run concurrently
Uncached bool // `-count=1`: results must never come from the test cache
Shards int // the most matrix entries the class may use
}
Class is one budget class of a sharded lane: the packages under Trees, tested with one per-package timeout and one parallelism, spread over at most Shards matrix entries.
Classes exist because the two lanes carry per-package budgets that must survive sharding unchanged: a `go test -timeout` applies to every package of the invocation, so a shard mixing component/memql (600s, serialised) with the fast db packages (180s) would silently hand the fast ones the slow one's budget. A shard therefore never mixes classes.
func ParseClasses ¶
ParseClasses reads the class table ci.yml's plan step carries: one class per line, six whitespace-separated columns,
lane class trees timeout mode shards
trees is a comma-separated list of repo-relative directories, or "-" for the lane's default class (every package no other class claims); mode is "serial", "uncached", both joined by a comma, or "-". Blank lines and lines starting with # are skipped. Each lane needs exactly one default class, so every package lands in some class.
type Graph ¶
type Graph struct {
// contains filtered or unexported fields
}
Graph is the union package graph. Build one with NewGraph, ReadGraph or LoadGraph; the zero value is not usable.
func LoadGraph ¶
func LoadGraph(ctx context.Context, root, prefix string, tags []string, list ListFunc) (*Graph, error)
LoadGraph builds the union graph over the default build and every tag in tags, keeping only packages under prefix (the first-party import path).
It refuses rather than guesses: a package `go list` could not load, a dependency it could not load, or a package that moves between directories across configurations is an error. A graph built over a broken load would answer confidently about a tree it did not see.
func NewGraph ¶
NewGraph validates pkgs and indexes them. It refuses a graph it could answer wrongly from: a duplicate import path or directory, a package with no directory, or a dependency on a package the graph does not hold.
func (*Graph) Expand ¶
Expand resolves a `go test` style package pattern against the graph: "./" or "." is the root package ONLY, "./x/..." is x and everything under it, "./x" is exactly x. A pattern that matches nothing is an error rather than an empty answer, because an empty answer is how a selector quietly stops selecting. "./..." is refused: it is the whole workspace, which is not a selection.
func (*Graph) ImportPaths ¶
ImportPaths returns every import path, sorted.
func (*Graph) Modules ¶
Modules returns every module directory that holds at least one package, sorted.
type ListFunc ¶
ListFunc returns the JSON stream `go list -e -json <pattern>` prints for one build configuration. tags is "" for the default build.
type Mode ¶
type Mode string
Mode says whether a run tests everything or only what a change reaches.
type Package ¶
type Package struct {
// ImportPath is the package's import path.
ImportPath string `json:"importPath"`
// Dir is the package directory, repo-relative and slash-separated. The
// repository root is ".".
Dir string `json:"dir"`
// ModuleDir is the directory of the go.mod that owns the package,
// repo-relative. The root module is ".".
ModuleDir string `json:"moduleDir"`
// Files are the repo-relative files some BUILD of the package names: its
// non-test Go sources under every configuration CI tests (including the
// ones the default build ignores), other compiled sources, and every
// //go:embed match of its non-test build. A change to one of these
// changes what every importer links, so it reaches the importers too.
Files []string `json:"files"`
// TestFiles are the files only the package's TESTS name: _test.go
// sources and the //go:embed matches of its test builds. A change to one
// of these changes this package's test binary and no other.
TestFiles []string `json:"testFiles,omitempty"`
// TaggedFiles is the subset of Files and TestFiles that the default
// build does not compile and some node-tag build does.
TaggedFiles []string `json:"taggedFiles,omitempty"`
// Deps are the first-party packages the package's TEST BINARY depends
// on, transitively, under the union of every configuration. Transitive
// is what lets the affected set be computed in one step.
Deps []string `json:"deps"`
}
Package is one Go package of the workspace, as far as selection cares.
type Plan ¶
type Plan struct {
Mode Mode
Reason string
Seeds int
Selected int
GoShards []Shard
DBShards []Shard
// Tags says whether the seven node-tag passes run.
Tags bool
// Modules are the module directories module-boundaries checks.
Modules []string
}
Plan is what the plan job hands the lanes.
func MakePlan ¶
MakePlan computes the plan. It returns an error -- which fails the plan job and therefore ci-required -- only for a state in which no plan can be trusted: a graph or a full-mode lane under its floor, a class table that cannot place a package, or a planner that disagrees with db-gated-packages.sh about what the go lane is. A change it merely cannot narrow is a full plan, not an error.
func (Plan) GitHubOutput ¶
GitHubOutput renders the plan as $GITHUB_OUTPUT lines. Every value is on one line; the matrices are JSON objects with an `include` list, exactly what `strategy.matrix: ${{ fromJSON(...) }}` takes.
type PlanInput ¶
type PlanInput struct {
// Event is github.event_name. Only "pull_request" is ever narrowed.
Event string
// Changed is the pull request's changed files, repo-relative. Nil when
// the event is not a pull request.
Changed []string
// DiffError is why Changed could not be computed on a pull request. A
// non-empty value makes the run full: an unreadable diff is never read as
// an empty one.
DiffError string
Graph *Graph
// DBTrees is `db-gated-packages.sh --trees`: the db lane owns every
// package under these directories.
DBTrees []string
// Complement is `db-gated-packages.sh --complement` (import paths), the
// script's own answer for the go lane. The planner computes the same set
// from the graph and refuses to run when the two disagree.
Complement []string
Classes []Class
Timings Timings
// GatePatterns are the gate-input packages (see Select).
GatePatterns []string
// TagTrees are the directories whose packages the node-tag passes test
// (app, component/node, component/server): a selected package under one
// of them means the tag passes run.
TagTrees []string
// CPUs is the parallelism of a class that is not serial.
CPUs int
}
PlanInput is everything a plan is computed from.
type Selection ¶
type Selection struct {
Mode Mode
// Reason says why the run is full, or summarises the narrowing.
Reason string
// Seeds are the packages the changed files map onto, in either scope,
// plus the declared filesystem gate packages (affected mode).
Seeds []string
// Packages is the affected set (affected mode): every seed, plus every
// package whose test binary depends on a package whose BUILD changed.
Packages []string
// TagChange is true when a changed file only compiles under a node tag.
TagChange bool
}
Selection is what a set of changed files reaches.
func Select ¶
Select maps changed files (repo-relative paths) onto the packages whose tests they can affect.
A changed file has a SCOPE, and the scope decides how far it reaches:
- build scope -- a non-test Go source (including one only a node tag compiles, and one that was deleted) or a //go:embed match of the package's own build. It changes what every importer links, so it reaches the package AND every package whose test binary depends on it.
- test scope -- a _test.go source, a test build's embed, or any other file under the package's directory that no build names (testdata, a fixture, a README). Importers do not link it, so the import graph adds only that package. Filesystem gates below may also read the file.
gate lists `go test` package patterns whose tests read repository files by PATH rather than through an import (docs gates, manifest gates, the path-routing guards), which no graph can see. They are always added in test scope: Go source and test files are also inputs to source-scanning contracts. The CI step passes the same list the go-checks gate-inputs step runs.
Every changed path lands on a package or turns the run full; a file is never dropped. The order of the rules:
- an empty list, or a path that is not repo-relative, is full -- an empty diff on a pull request means the diff failed, and narrowing to nothing would skip every lane;
- a full trigger (fullTrigger) is full;
- a file some package's build or tests name maps to that package, in that scope;
- anything else maps to the package owning the deepest directory above it -- in build scope when it is a non-test Go file (a deleted source), in test scope otherwise. The root is a package, so this always finds one.
type Shard ¶
type Shard struct {
Name string `json:"name"`
Packages string `json:"packages"` // space-separated `go test` arguments
Timeout string `json:"timeout"`
Parallel int `json:"parallel"`
Uncached bool `json:"uncached"`
Class string `json:"-"`
Dirs []string `json:"-"`
Seconds float64 `json:"-"` // the measured seconds of the shard's packages
Unknown []string `json:"-"` // packages the timing table has not measured
}
Shard is one matrix entry of a sharded lane. The exported JSON fields are exactly what the workflow step interpolates.
func Partition ¶
func Partition(lane string, dirs []string, classes []Class, seconds map[string]float64, cpus int) ([]Shard, error)
Partition spreads one lane's packages (repo-relative directories) over shards: grouped by class, then, within a class, longest-processing-time first over the measured seconds (ties broken by name, so the answer is deterministic), then each unmeasured package, by name, into the then lightest shard. cpus is the parallelism of a class that is not serial.
It refuses to return shards whose union is not exactly dirs: a package dropped here would be a package no lane ever tests, and nothing downstream could notice.
type Timings ¶
Timings is the checked-in timing table (scripts/ci/shard-timings.tsv): lane -> package directory -> measured `go test` seconds.
It is DATA the planner balances shards with, never a gate: a package absent from it still runs, in the lightest shard, and a stale number only makes the balance worse. That is why a refresh rewrites it wholesale from recent logs (scripts/ci/refresh-shard-timings.sh) rather than anyone editing it by hand.
func ReadTimings ¶
ReadTimings parses the table: `#` comments, then one `lane dir seconds` row per line, tab- or space-separated. A malformed row, an unknown lane or a duplicate refuses the whole table rather than half-reading it.
func (Timings) Merge ¶
Merge replaces lane's rows for every package with samples by the median of those samples, and keeps every other row. A package that stops appearing in the logs keeps its last measurement until it is deleted from the tree; the planner ignores rows for packages the graph does not hold.
Source Files