scanner-worker

README

scanner-worker

Purpose

scanner-worker runs claim-driven scanner work for security analyzers that are too CPU-heavy or memory-heavy for reducer lanes. The installed binary is eshu-scanner-worker.

Flow

flowchart LR
  workflow["workflow work item"] --> claim["scanner-worker claim"]
  claim --> input["scannerworker.ClaimInput"]
  input --> analyzer["bounded analyzer"]
  analyzer --> facts["scanner source facts\nscanner_worker.* / sbom.* / vulnerability.os_package"]
  facts --> postgres["Postgres fact store"]
  postgres --> reducers["reducers admit findings"]

Runtime Contract

• Selects one enabled, claim-capable scanner_worker collector instance from ESHU_COLLECTOR_INSTANCES_JSON.
• Uses ESHU_SCANNER_WORKER_ANALYZER or the instance configuration.analyzer to choose a scanner-worker analyzer profile.
• Applies analyzer defaults, optional instance resource_limits, then ESHU_SCANNER_WORKER_* resource overrides.
• Emits source facts only and rejects silent clean output before committing.
• Runs the concrete image_unpacking analyzer when configured with image_targets; this source reads configured local image rootfs metadata or ordered OCI layer tar streams, extracts Alpine apk or Debian dpkg installed package databases, and emits scanner_worker.analysis coverage, vulnerability.os_package, vulnerability.warning, or scanner_worker.warning unsupported evidence.
• Runs the concrete sbom_generation analyzer when configured with sbom_targets; this source walks a configured repository root, reads bounded package-lock.json, npm-shrinkwrap.json, go.mod, Cargo.lock, composer.lock, packages.lock.json, Pipfile.lock, poetry.lock, Gemfile.lock, and gradle.lockfile lockfile evidence, and emits sbom.document, sbom.component, and sbom.warning facts.
• Runs the concrete os_package_extraction analyzer when configured with os_package_targets; this parser consumes already-extracted Alpine or Debian rootfs metadata and emits vulnerability.os_package / vulnerability.warning facts.
• Records retry and dead-letter payloads with locator hashes and bounded failure classes.
• Exposes /healthz, /readyz, /metrics, /admin/status, and optional private pprof through ESHU_PPROF_ADDR.

The fallback analyzer emits an explicit scanner_worker.warning source fact (reason=analyzer_not_configured). sbom_generation falls back to reason=sbom_generator_source_not_configured when no sbom_targets are configured. Concrete secret, license, source, and misconfiguration analyzers must plug into this boundary instead of running in reducer lanes.

image_unpacking targets are configured inside the selected scanner_worker collector instance:

{
  "analyzer": "image_unpacking",
  "image_targets": [
    {
      "scope_id": "image://registry.example/team/app@sha256:...",
      "rootfs_path": "/var/lib/eshu/scanner/rootfs/...",
      "layer_paths": ["/var/lib/eshu/scanner/layers/layer.tar.gz"],
      "source_uri": "oci://registry.example/team/app@sha256:...",
      "source_record_id": "sha256:...",
      "image_reference": "registry.example/team/app:1.2.3",
      "image_digest": "sha256:..."
    }
  ]
}

rootfs_path and layer_paths are runtime-local configuration. They must not appear in retry, dead-letter, metric, log, or public documentation payloads. Layer paths are ordered from base to top layer. Unsupported image shapes emit scanner_worker.warning facts with analysis_status=not_scanned, coverage_status=unsupported, and an extraction_reason instead of clean results. Missing image digests are unsupported coverage evidence; tag-only image configuration does not emit package facts.

sbom_generation repository targets are configured inside the selected scanner_worker collector instance:

{
  "analyzer": "sbom_generation",
  "sbom_targets": [
    {
      "scope_id": "scanner-worker://repository/team-api",
      "root_path": "/var/lib/eshu/scanner/repositories/team-api",
      "subject_digest": "sha256:..."
    }
  ]
}

The repository path is runtime-local configuration. It must not appear in retry, dead-letter, metric, log, or public documentation payloads. If no usable components are found, the analyzer emits a document fact plus an sbom.warning fact instead of returning silent clean output. Component facts preserve bounded evidence fields: ecosystem, repository-relative lockfile_path, evidence_source, package name, installed version, PURL, dependency scope/type when the lockfile reports it, and extraction_reason="lockfile_exact_version". Malformed supported lockfiles emit sbom.warning with reason="lockfile_malformed" and the relative lockfile path; they do not leak the runtime-local repository root. NPM package-lock.json package entries may encode dependency edges as version range strings. Those ranges are accepted so the lockfile remains usable, but they are not emitted as installed-version components unless a sibling lockfile entry supplies the exact installed version. The repository walk counts only supported manifest and lockfile inputs toward FileCount and ESHU_SCANNER_WORKER_MAX_FILES; ignored directories and unsupported regular files do not consume the SBOM input-file budget.

os_package_extraction targets are configured inside the selected scanner_worker collector instance:

{
  "analyzer": "os_package_extraction",
  "os_package_targets": [
    {
      "scope_id": "image://registry.example/team/app@sha256:...",
      "rootfs_path": "/var/lib/eshu/scanner/rootfs/...",
      "source_uri": "oci://registry.example/team/app@sha256:...",
      "source_record_id": "sha256:..."
    }
  ]
}

The rootfs path is runtime-local configuration. It must not appear in retry, dead-letter, metric, log, or public documentation payloads.

Environment

Variable	Purpose
ESHU_SCANNER_WORKER_INSTANCE_ID	Select one configured scanner-worker instance when more than one exists.
ESHU_SCANNER_WORKER_ANALYZER	Analyzer override such as source_analysis or image_unpacking.
ESHU_SCANNER_WORKER_POLL_INTERVAL	Claim poll interval.
ESHU_SCANNER_WORKER_CLAIM_LEASE_TTL	Workflow claim lease TTL.
ESHU_SCANNER_WORKER_HEARTBEAT_INTERVAL	Claim heartbeat interval; must be less than lease TTL.
ESHU_SCANNER_WORKER_CPU_MILLIS	Analyzer CPU budget in millicores.
ESHU_SCANNER_WORKER_MEMORY_BYTES	Analyzer memory budget in bytes.
ESHU_SCANNER_WORKER_TIMEOUT	Analyzer timeout.
ESHU_SCANNER_WORKER_MAX_INPUT_BYTES	Maximum analyzer input bytes.
ESHU_SCANNER_WORKER_MAX_FILES	Maximum analyzer input files per claim.
ESHU_SCANNER_WORKER_MAX_FACTS	Maximum source facts emitted per claim.

Evidence

No-Regression Evidence: scanner-worker runtime behavior is covered by go test ./internal/collector/scannerworker ./internal/collector/scannerworker/imageanalyzer ./internal/collector/scannerworker/sbomgenerator ./internal/collector/ospackagevulnerability/osruntime ./cmd/scanner-worker -count=1.

No-Regression Evidence: go test ./cmd/scanner-worker -run 'TestRepositorySBOMSource(ParsesNPMPackageDependencyRanges|ParsesCargoAndComposerLockfiles|ParsesPythonRubyAndGradleLockfiles|EmitsMalformedLockfileWarning)|TestNPMLockDependencyUnmarshalStringResetsPriorObject' -count=1 proved repository SBOM generation accepts npm package dependency range maps, extracts Cargo, Composer, PyPI, RubyGems, Gradle/Maven, and NuGet exact lockfile components with ecosystem, relative path, dependency scope/type, PURL, and extraction reason, while malformed Composer lockfiles produce bounded sbom.warning evidence instead of terminal analyzer failure or silent clean output.

Observability Evidence: the runtime records scanner-worker claim, retry, dead-letter, facts-emitted, queue-wait, scan-duration, target-count, result-count, CPU, and memory metrics, plus scanner_worker.* spans and bounded structured failure logs. Post-analysis persistence failures retry as failure_class=commit_failed and include a bounded commit_failure_class stage in logs, while analyzer source-read failures remain source_unavailable. Configured image_unpacking and sbom_generation sources return measured metadata or manifest input bytes as peak memory usage and CPU seconds from the Go runtime counters, so operators can distinguish queue wait, extraction or manifest read cost, source fact volume, retries, commit-stage failures, and terminal resource-limit failures.

Documentation

Overview

Command scanner-worker runs isolated scanner-worker claims for CPU-heavy or memory-heavy security analyzers.

The binary consumes workflow work items with collector_kind=scanner_worker, builds scannerworker.ClaimInput values with resource limits, commits source facts under the claim fence, and records bounded retry or dead-letter payloads. It can run concrete image_unpacking, sbom_generation repository-manifest, and os_package_extraction rootfs analyzers. Image unpacking emits coverage or unsupported evidence, but the binary does not emit reducer-owned findings.