imagegc

package
v0.2.7 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 26, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Overview

Package imagegc garbage-collects containerd image records under disk pressure.

The model is kubelet's: **disk pressure is the trigger, least-recently-used is the order**. A high watermark starts a pass; the pass evicts LRU-first until a distinctly lower low watermark is reached, then stops. Age (`cache_max_age`) survives only as an optional, off-by-default backstop — evicting by age alone throws away a warm cache while the disk is half empty and forces re-downloads, which is the opposite of what we want on a node whose network budget we are also trying to shrink.

Scope is every namespace that holds long-lived image records:

  • "ephemerd", the main runtime namespace, whose images had NO garbage collector at all before this package existed — every runner image and every workflow `container:` image ever pulled was retained forever along with its extracted overlay layers, which is what filled node disks until QEMU froze;
  • "ephemerd-dind-cache-*", the per-repo dind image caches, which already had a correct LRU pruner (dind.CachePrune) that this package now backs.

Per-job namespaces (ephemerd-dind-<runner>) are deliberately out of scope: they belong to a live job and are torn down by pkg/dind's own cleanup.

Index

Constants

View Source 
const (
	ReasonUsedPercent = "used_percent"
	ReasonMinFree     = "min_free"
)

Trigger reasons reported by Pressure.Reasons.

View Source 
const DefaultExhaustedBackoff = 30 * time.Minute

DefaultExhaustedBackoff is how long the collector stays quiet after a pass that evicted everything it was allowed to and still could not get under the high watermark. See Collector.Collect's failsafe.

View Source 
const LastAccessedLabel = "ephemerd.io/last-accessed"

LastAccessedLabel records the most recent time an image record was pulled or used to start a container. It is the LRU key for every namespace this package collects, and is written in RFC3339 UTC.

pkg/dind has stamped this on its cache records since the per-repo cache landed; pkg/runtime stamps it on the runtime namespace as of this package.

Variables

This section is empty.

Functions

func Delete 

func Delete(ctx context.Context, c *client.Client, cand Candidate, synchronous bool) error

Delete removes one image record. Synchronous deletion makes containerd reclaim the now-unreferenced content and snapshots before returning, which is what lets the caller re-measure real free space between evictions instead of guessing from size estimates.

A NotFound is success: something else already removed it.

func Evict 

func Evict(ctx context.Context, c *client.Client, cands []Candidate, synchronous bool, log *slog.Logger, stop func() bool) int

Evict deletes each candidate in order, logging and skipping failures so one stuck record cannot stall the whole pass. Returns the number deleted.

stop, when non-nil, is consulted after every successful deletion; returning true ends the pass early. The pressure collector uses it to stop the moment a real disk re-measurement shows the low watermark reached.

func EvictReferencing added in v0.1.10 

func EvictReferencing(ctx context.Context, c *client.Client, nss []string, snapshotter, key string, protected map[string]struct{}, log *slog.Logger) (int, error)

EvictReferencing removes the image records in nss that name snapshot key. The auto-heal path calls this when a build has already told us exactly which snapshot is missing, so there is no need to scan the whole store.

protected is honoured for the same reason as in PlanBrokenChains.

func ExistingSnapshots added in v0.1.10 

func ExistingSnapshots(ctx context.Context, c *client.Client, snapshotter string, chains []ImageChain, log *slog.Logger) map[string]struct{}

ExistingSnapshots returns the subset of keys the snapshotter actually has, per namespace. Snapshots are namespaced, so a key is checked in the namespace of the record that named it.

A Stat error that is not NotFound is treated as "exists": under uncertainty the safe answer is to leave the record alone, because the cost of a wrong "missing" verdict is evicting a healthy image.

func LastAccessed 

func LastAccessed(img images.Image) time.Time

LastAccessed returns the time an image record was last used.

Prefers LastAccessedLabel. Falls back to the record's UpdatedAt (then CreatedAt) so records that pre-date the label — every image on a node upgrading into this feature — sort by when containerd last touched them rather than being treated as never-used and evicted first.

func PlanBrokenChains added in v0.1.10 

func PlanBrokenChains(chains []ImageChain, existing, protected map[string]struct{}) (broken []ImageChain, brokenProtected []ImageChain)

PlanBrokenChains selects the image records whose snapshot chain is missing from the snapshotter. Pure — the whole "which records are unusable" rule in one testable function.

A record is broken when it names a snapshot key that is not in existing. Records naming no key are skipped (never unpacked, nothing to break), and so are protected records: a pinned runner image or an image a live container is using must not be evicted even when its chain looks broken, because evicting it cannot help — the container already holds the rootfs — and re-pulling a multi-gigabyte runner image mid-job is strictly worse. Those get reported through brokenProtected so the caller can log them instead of silently doing nothing.

Output is sorted by Key so repeated passes over the same store produce the same plan.

func ProtectPrefixed 

func ProtectPrefixed(candidates []Candidate, prefixes []string, protected map[string]struct{}) int

ProtectPrefixed adds to protected every candidate whose Name starts with one of prefixes, and returns how many were added.

Some records are tied to a live job by NAME rather than by container reference: BuildKit writes each job's build output into the one shared "buildkit" namespace under a name scoped by that job's container ID, and no container ever references those records. Evicting one mid-job breaks the job's subsequent `docker push`, so the live jobs' prefixes are expanded into the protected set before planning. Pure.

func ProtectedSet 

func ProtectedSet(pinned []string, running map[string]struct{}) map[string]struct{}

ProtectedSet builds the never-evict set from a list of pinned refs plus the refs of every existing container. Both inputs are optional.

Pinned refs are the node's configured runner images. Evicting one of those guarantees an immediate re-pull on the very next job — precisely the network thrash this collector exists to reduce — so they are vetoed regardless of how long they have sat idle.

func RepairBrokenChains added in v0.1.10 

func RepairBrokenChains(ctx context.Context, c *client.Client, nss []string, snapshotter string, protected map[string]struct{}, log *slog.Logger) (int, error)

RepairBrokenChains evicts every image record in nss whose snapshot chain is missing, and returns how many it removed.

This is ask (1) of #149's second half — "if a chain is already broken, evict the image record too" — and it is what makes the failure self-limiting from the collector's side: a node that gets its snapshots removed out of band heals on the next sweep instead of failing every job until someone logs in.

func ResolveNamespaces 

func ResolveNamespaces(ctx context.Context, c *client.Client, exact, prefixes []string) ([]string, error)

ResolveNamespaces returns the existing containerd namespaces matching either an exact name in exact or a prefix in prefixes, in the order containerd reports them. Names that do not exist are silently skipped.

func RunningContainers 

func RunningContainers(ctx context.Context, c *client.Client, log *slog.Logger) (ids, refs map[string]struct{}, err error)

RunningContainers returns the IDs of every container that currently exists and the set of image refs they reference, across every containerd namespace.

Scope is deliberately all namespaces, not just the collected ones: content blobs are global, and a per-job dind namespace can hold a live sibling container built from the same ref that a cache namespace records. Deleting only the cache record would not free that content anyway, so protecting the ref everywhere costs nothing and removes a whole class of "we evicted an image a job was using" failure.

The IDs are returned as well because some records are tied to a live job by NAME rather than by container reference — BuildKit export records are prefixed with the job's container ID (see Config.LiveJobPrefixes).

func RunningImageRefs 

func RunningImageRefs(ctx context.Context, c *client.Client, log *slog.Logger) (map[string]struct{}, error)

RunningImageRefs returns just the image refs of existing containers. Thin wrapper over RunningContainers for callers that don't need the IDs.

func SnapshotRefLabel added in v0.1.10 

func SnapshotRefLabel(snapshotter string) string

SnapshotRefLabel is the content-store label that links a config blob to the chain ID of the snapshot chain unpacked from that image.

func Touch 

func Touch(ctx context.Context, c *client.Client, ns, name string, log *slog.Logger)

Touch refreshes LastAccessedLabel on an image record so the LRU ordering reflects use, not just pull time. Best-effort: a missing image or a failed update is logged at debug and swallowed, because failing a job over a bookkeeping label would be absurd.

Types

type Candidate 

type Candidate struct {
	Namespace    string
	Name         string
	LastAccessed time.Time
	SizeBytes    int64
}

Candidate is one image record considered for eviction.

SizeBytes is the image's packed content size and is only ever an estimate of what eviction reclaims: layers shared with another image are counted here but will not actually be freed, and the extracted snapshot (typically larger than the packed layers) is not counted at all. The plan therefore treats it as a budgeting hint; Collector re-reads real disk usage after every deletion and stops the moment pressure clears. A non-positive SizeBytes means "unknown" and contributes nothing to the budget, which makes such candidates always eligible rather than invisible.

func ListCandidates 

func ListCandidates(ctx context.Context, c *client.Client, nss []string, log *slog.Logger) ([]Candidate, error)

ListCandidates enumerates every image record in the given namespaces as an eviction candidate, resolving each one's last-accessed time and packed size.

Size resolution is best-effort. It is a content-store metadata walk, not a disk traversal, but it can still fail — a partially-pulled image, or a manifest for a platform this client does not match. A candidate whose size cannot be determined gets SizeBytes 0, which PlanEviction reads as "unknown" rather than "free".

func PlanByAge 

func PlanByAge(candidates []Candidate, protected map[string]struct{}, cutoff time.Time) []Candidate

PlanByAge selects every candidate last accessed before cutoff, least-recently-used first, skipping the protected set. It is the optional age backstop (`cache_max_age`), not the primary mechanism: age alone discards a warm cache while the disk is half empty and forces re-downloads we are otherwise trying to eliminate. Pure.

A candidate with a zero LastAccessed is never selected — a record with no usable timestamp is treated as unknown, not as ancient.

func (Candidate) Key 

func (c Candidate) Key() string

Key is a stable namespace-qualified identifier, used for deterministic tie-breaking and in log lines.

type Collector 

type Collector struct {
	// contains filtered or unexported fields
}

Collector evicts image records under disk pressure.

A Collector serializes its own passes: the periodic timer and the pre-pull headroom check share one mutex, so a job that starts mid-sweep waits for the sweep instead of racing it.

The nil Collector is a valid no-op, so callers that may not have GC configured can invoke it unconditionally.

func New 

func New(cfg Config) *Collector

New builds a Collector. Returns nil when no mechanism is configured, so the caller's nil check doubles as the "is GC on" check.

func (*Collector) Collect 

func (c *Collector) Collect(ctx context.Context) (Result, error)

Collect runs one full pass: the optional age backstop, then the pressure-driven LRU eviction.

The pressure loop re-reads real disk usage after every deletion rather than trusting the planned size estimates, because a layer shared with a surviving image frees nothing and an extracted snapshot frees more than its packed size. It stops the instant the reading clears both watermarks.

Failsafe: if the pass deletes every record it is allowed to and usage is still above the high watermark, the remaining bytes are live data, not cache. Collect logs that loudly with the numbers and then suppresses further passes for ExhaustedBackoff so the daemon does not spin listing and re-planning a store it cannot shrink.

func (*Collector) EnsureHeadroom 

func (c *Collector) EnsureHeadroom(ctx context.Context)

EnsureHeadroom runs a collection pass if — and only if — the disk is currently over a watermark. It is the cheap guard to call immediately before a large pull or before creating a runner environment.

The periodic timer alone loses the race a single job can win: one CI job pulling a multi-gigabyte toolchain image can cross the high watermark and fill the disk well inside a 60s tick. Checking here costs one syscall on the happy path.

Errors are logged, never returned: a GC failure must not fail the job.

type Config 

type Config struct {
	// Client is the containerd client whose namespaces are collected.
	Client *client.Client

	// Path is any path on the filesystem to measure — in practice the
	// containerd root under the data dir. Capacity is read with one
	// statfs-class syscall; nothing walks the tree.
	Path string

	// Thresholds are the pressure watermarks. A zero Thresholds disables
	// pressure-triggered collection (the age backstop, if configured,
	// still runs).
	Thresholds Thresholds

	// Namespaces are collected by exact name (e.g. "ephemerd").
	Namespaces []string

	// NamespacePrefixes are collected by prefix (e.g.
	// "ephemerd-dind-cache-").
	NamespacePrefixes []string

	// PinnedImages are refs that must never be evicted regardless of age
	// — the node's configured runner images. Evicting one forces an
	// immediate re-pull on the next job.
	PinnedImages []string

	// LiveJobPrefixes maps a live container ID to image-name prefixes
	// whose records belong to that job and must survive the pass.
	//
	// Needed for the shared "buildkit" namespace: a job's `docker build`
	// output is stored there under a name scoped by the job's container
	// ID, and no container references it, so nothing else marks it live.
	// Nil disables the expansion.
	LiveJobPrefixes func(containerID string) []string

	// MaxAge is the OPTIONAL age backstop. Zero (the default) disables
	// it. When set, records idle longer than this are evicted on every
	// pass whether or not the disk is under pressure. This is not the
	// primary mechanism — see the package doc.
	MaxAge time.Duration

	// ExhaustedBackoff is how long to skip passes after the failsafe
	// trips. Zero uses DefaultExhaustedBackoff.
	ExhaustedBackoff time.Duration

	Log *slog.Logger
}

Config constructs a Collector.

type ImageChain added in v0.1.10 

type ImageChain struct {
	Namespace string
	Name      string
	// SnapshotKey is the chain ID from the config blob's GC label. Empty
	// means the record names no snapshot at all — it was never unpacked,
	// which is normal for a pushed-but-not-run image and is NOT a broken
	// chain.
	SnapshotKey string
}

ImageChain is one image record and the snapshot chain it claims.

func ChainsReferencing added in v0.1.10 

func ChainsReferencing(chains []ImageChain, key string) []ImageChain

ChainsReferencing returns the chains whose SnapshotKey equals key. Pure. Used by the auto-heal path, which knows exactly which snapshot went missing and wants the image records that name it.

func ListImageChains added in v0.1.10 

func ListImageChains(ctx context.Context, c *client.Client, nss []string, snapshotter string, log *slog.Logger) ([]ImageChain, error)

ListImageChains resolves every image record in nss to the snapshot chain it names, by reading the GC label off each record's config blob.

Best-effort per record: an image whose manifest is missing, whose platform does not match this node, or whose config blob has been collected simply yields no chain rather than failing the pass. Those records are a different (content-level) kind of broken and are out of scope here.

func (ImageChain) Candidate added in v0.1.10 

func (i ImageChain) Candidate() Candidate

Candidate converts a chain back into an eviction candidate.

func (ImageChain) Key added in v0.1.10 

func (i ImageChain) Key() string

Key is the namespace-qualified identifier, matching Candidate.Key.

type Plan 

type Plan struct {
	Pressure
	// Evict lists the records to delete, least-recently-used first.
	Evict []Candidate
	// Protected counts candidates skipped because they are in the
	// protected set.
	Protected int
	// PlannedBytes is the summed SizeBytes of Evict (unknown sizes count
	// as zero).
	PlannedBytes uint64
	// Shortfall is BytesToFree minus PlannedBytes when every evictable
	// candidate was selected and the estimate still falls short. Non-zero
	// means "there may be nothing left to reclaim" and drives the
	// failsafe warning — see Collector.Collect.
	Shortfall uint64
}

Plan is the outcome of a pure eviction decision.

func PlanEviction 

func PlanEviction(candidates []Candidate, protected map[string]struct{}, usage diskspace.Usage, t Thresholds) Plan

PlanEviction decides which image records to evict, in what order.

This is the whole eviction policy as one pure function: disk pressure is the TRIGGER (via Evaluate), least-recently-used is the ORDER, the protected set is an absolute veto, and the low watermark is the stopping point. It touches no containerd, no filesystem and no clock, so the policy is unit-testable on its own.

protected is keyed by image name (ref), not by Key: an image that a running container references, or that is the node's pinned runner image, must survive in every namespace it appears in.

Candidates sort by LastAccessed ascending, ties broken by Key so repeated runs over the same store produce the same plan.

type Pressure 

type Pressure struct {
	// Over reports whether any arm triggered.
	Over bool
	// Reasons lists the arms that triggered, in a stable order
	// (ReasonUsedPercent before ReasonMinFree).
	Reasons []string
	// BytesToFree is how much must be reclaimed to satisfy every
	// triggered arm's target. Zero when Over is false.
	BytesToFree uint64
}

Pressure is the verdict for one capacity reading.

func Evaluate 

func Evaluate(u diskspace.Usage, t Thresholds) Pressure

Evaluate reports whether a capacity reading is over either trigger and how many bytes a pass must reclaim. Pure — no syscalls, no clock.

A reading with TotalBytes == 0 (an unreadable or stubbed probe) never triggers: guessing under uncertainty would delete a warm cache for no reason.

func (Pressure) ReasonString 

func (p Pressure) ReasonString() string

ReasonString joins Reasons for logging.

type Result 

type Result struct {
	// Before and After are the capacity readings that bracket the pass.
	Before, After diskspace.Usage
	// Pressure is the verdict on Before.
	Pressure Pressure
	// AgeEvicted counts records dropped by the optional age backstop.
	AgeEvicted int
	// Evicted counts records dropped by the pressure pass.
	Evicted int
	// Protected counts candidates the pressure pass refused to consider.
	Protected int
	// Exhausted reports that the pass ran out of evictable records while
	// still over the high watermark.
	Exhausted bool
	// Skipped reports that the pass did not run because the failsafe
	// backoff was still in effect.
	Skipped bool
}

Result summarizes one pass.

type Thresholds 

type Thresholds struct {
	// HighUsedPercent triggers a pass when disk used% reaches it.
	HighUsedPercent float64
	// LowUsedPercent is the used% a triggered pass tries to get back to.
	// Must be below HighUsedPercent to be useful; a value at or above it
	// is clamped to HighUsedPercent (degrading to single-threshold
	// behavior rather than erroring at eviction time).
	LowUsedPercent float64
	// MinFreeBytes triggers a pass when free space falls below it.
	MinFreeBytes uint64
	// TargetFreeBytes is the free-space level a triggered pass tries to
	// reach. Values below MinFreeBytes are clamped up to it.
	TargetFreeBytes uint64
}

Thresholds describes when image garbage collection starts and when it stops. Two independent arms trigger a pass, and whichever demands more reclaimed bytes wins:

  • a percentage arm (HighUsedPercent triggers, LowUsedPercent is the target), and
  • an absolute arm (MinFreeBytes triggers, TargetFreeBytes is the target).

Both arms exist because neither is safe alone. 15% free of a 1 TB node is 150 GB — evicting there is pointless churn. 15% free of a 100 GB node is 15 GB, which three concurrent jobs writing ~5 GB of container layers each can consume before the next tick, so a percentage alone under-protects a small disk. Conversely a fixed 20 GB floor under-protects a large busy node. Setting both and taking the more conservative answer covers both shapes.

Using two watermarks rather than one line is what stops the collector thrashing: a single threshold would evict one image, drop just under, and evict again on the very next tick forever. Freeing down to a distinctly lower low-water mark buys many ticks of headroom per pass. This is kubelet's image-GC model.

A non-positive value disables that arm. Zeroing everything disables pressure-triggered collection entirely.

func (Thresholds) Enabled 

func (t Thresholds) Enabled() bool

Enabled reports whether any arm is configured.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.