kubectl-toolbox

README

kubectl-toolbox

A collection of OpenShift/Kubernetes debugging tools in one kubectl plugin. Each tool lives under its own subcommand.

Tool	Answers
toolbox vmsnapshot	where a VM's disks and snapshots actually live, down to the storage array
toolbox nodecheck	which of a pod's scheduling label requirements no node satisfies

toolbox vmsnapshot resolves the full reference chain behind a KubeVirt / OpenShift Virtualization VM's snapshots and disks, down to the objects you can actually look up on the storage array. Built for debugging missing PVCs and missing snapshots.

Every hop is verified. A broken reference is reported with the reason that hop usually breaks, and the process exits non-zero.

Quick start

Download the archive for your platform from the latest release:

tar -xzf kubectl-toolbox_v0.0.1_linux_amd64.tar.gz    # or a darwin/windows asset
install -m 0755 kubectl-toolbox ~/bin/                # any directory on your PATH
oc toolbox vmsnapshot vm <vm-name> -n <namespace>

Or build from source (needs Go 1.26 or newer, which the Kubernetes client libraries require):

go install github.com/mjovanovic0/kubectl-toolbox@latest
# or, from a clone:
make install PREFIX=~/bin

Verify a download against checksums.txt from the same release.

oc toolbox vmsnapshot vm <vm-name> is the main entry point: it walks the VM's disks down to their PVCs/PVs and the array, then resolves every snapshot of that VM down to the array, and merges all findings into one problem list.

The plugin is discovered by both oc and kubectl from the single kubectl-toolbox binary — oc looks for kubectl-* as well as oc-*, so oc toolbox vmsnapshot ..., kubectl toolbox vmsnapshot ... and kubectl-toolbox ... are equivalent.

The two chains

Snapshot chain — kubectl toolbox vmsnapshot snapshot <vm> <snapshot>:

Object	Reference followed
VirtualMachine	VirtualMachineSnapshot.spec.source points back at it
VirtualMachineSnapshot	status.virtualMachineSnapshotContentName
VirtualMachineSnapshotContent	spec.volumeBackups[].volumeSnapshotName
VolumeSnapshot	status.boundVolumeSnapshotContentName
VolumeSnapshotContent	status.snapshotHandle
PureSnapshot	matched on the CSI handle

Disk chain — kubectl toolbox vmsnapshot disks <vm>:

Object	Reference followed
DataImportCron	spec.managedDataSource
DataSource	status.source.pvc / status.source.snapshot
DataVolume	spec.sourceRef, spec.source, status.claimName
PersistentVolumeClaim	spec.volumeName, spec.dataSourceRef
PersistentVolume	spec.csi.volumeHandle
PureVolume	matched on the CSI handle

The two chains interlink: a DataVolume can be sourced from a VolumeSnapshot, and a restored PersistentVolumeClaim records its origin snapshot in spec.dataSourceRef. Both links are followed.

Install

make install PREFIX=~/bin      # anywhere on PATH
oc plugin list                 # should show kubectl-toolbox

Cross-compile release binaries with make cross.

Installing on a host you can only paste text into

make text packages the plugin as a single self-extracting shell script — the binary, gzipped and base64-encoded, with its sha256 embedded — for hosts reachable only by copying text into a terminal.

make text                     # -> dist/kubectl-toolbox-linux-amd64.sh  (~16 MiB of text)
make text TEXT_ARCH=arm64
make text COMP=xz             # ~33% less text, needs unxz on the host
make text CHUNK=4             # also split the payload into 4 MiB .b64 parts
make text-all                 # linux amd64 + arm64

Then on the host:

sh kubectl-toolbox-linux-amd64.sh          # writes ./kubectl-toolbox, verifies sha256
install -m 0755 kubectl-toolbox ~/bin/
oc toolbox vmsnapshot vm <name> -n <namespace>

The script decodes to a staging path and only moves the binary into place after the checksum matches, so a truncated or mangled paste fails loudly and leaves nothing runnable behind. make text verifies its own round-trip before reporting success. If the paste is too large for one go, CHUNK=<MiB> also emits numbered parts to reassemble with cat *.part-* | base64 -d | gunzip > kubectl-toolbox.

Usage

# everything about one VM: disks and all its snapshots
oc toolbox vmsnapshot vm win-db-01 -n virt-prod

# a VM with a long snapshot history: only the newest few
oc toolbox vmsnapshot vm win-db-01 -n virt-prod --latest 3
oc toolbox vmsnapshot vm win-db-01 -n virt-prod --no-snapshots   # disks only
oc toolbox vmsnapshot vm win-db-01 -n virt-prod --no-disks       # snapshots only

# what snapshots exist, and which ones are already suspect
kubectl toolbox vmsnapshot list -n virt-prod
kubectl toolbox vmsnapshot list win-db-01 -n virt-prod

# narrow to one snapshot, or to the disks only
kubectl toolbox vmsnapshot snapshot win-db-01 snap-2026-07-20 -n virt-prod
kubectl toolbox vmsnapshot disks win-db-01 -n virt-prod

# start from a name you found in CSI or storage-operator logs
kubectl toolbox vmsnapshot volumesnapshot vmsnapshot-8f3a-vol-rootdisk -n virt-prod
kubectl toolbox vmsnapshot pvc win-db-01-root -n virt-prod

Standard kubectl flags (-n, --context, --kubeconfig) all apply to the vmsnapshot subcommands.

nodecheck

oc toolbox nodecheck <pod> -n <namespace>     # alias: label-check

Reports which of a pod's nodeSelector and required nodeAffinity requirements no node in the cluster satisfies — the ones that keep it Pending. The AND/OR structure is evaluated faithfully: nodeSelector entries must all match, required nodeAffinity needs at least one term to match, and within a term every expression must match.

preferredDuringScheduling… is ignored because it never blocks scheduling. matchFields selects on node fields rather than labels, so a term containing it is reported as undecidable rather than being silently dropped — dropping it would narrow the OR and make a schedulable pod look stuck.

Only labels are considered. Taints and tolerations, resource requests, topology spread and volume topology can equally block scheduling; the report says so, so a clean result is not mistaken for "scheduling is fine".

Like every tool here it also reads from --dump, so you can ask why a pod was Pending from a must-gather after the fact.

Staying up to date

oc toolbox version            # build stamp, install path, and whether it is writable
oc toolbox version --check    # ask GitHub right now, ignoring the cache
oc toolbox update             # download the newest release and replace this binary
oc toolbox update --yes       # ... without the confirmation prompt

update verifies the download against the checksums.txt published with the release before installing anything, then replaces the binary atomically: the new one is staged alongside the old and moved into place, and it is executed once to prove it runs. A failed or interrupted update leaves the working binary untouched. If the target directory is not writable it says so rather than half-failing.

Every command also starts a background check and prints a one-line notice on stderr after the output if a newer release exists — stderr so it can never interleave with the tree or corrupt a pipe. The check is cached for 12 hours, gives up after 3 seconds, and is skipped entirely for local builds. To turn it off:

export TOOLBOX_NO_UPDATE_CHECK=1

On a host with no route to GitHub — which is where this tool is often used — the check fails silently and update explains that you need the offline install path below.

Chains are always resolved in full, including where the same VolumeSnapshot is reached from more than one branch — a branch you are following should never dead-end in a pointer to somewhere else in the output. When that repetition is more than you want, --trim-repeats collapses any sub-chain already rendered earlier in the same report, keeping the node and its attributes but omitting the repeated subtree. The collapse is tracked against what was actually written out, so it never claims something was "shown above" when it wasn't.

Offline / post-mortem

Resolve from a directory of YAML instead of the cluster. This is how you investigate a chain whose PVCs are already deleted, or work without cluster access:

oc adm must-gather --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9
kubectl toolbox vmsnapshot snapshot win-db-01 snap-2026-07-20 --dump ./must-gather.local.123 -n virt-prod

A hand-rolled capture works too — kind: List wrappers and multi-document files are both unwrapped:

mkdir dump && cd dump
for k in vm vmsnapshot vmsnapshotcontent volumesnapshot dv pvc; do oc get $k -o yaml > $k.yaml; done
oc get volumesnapshotcontent,pv,puresnapshot,purevolume -o yaml > cluster.yaml
kubectl toolbox vmsnapshot snapshot win-db-01 snap-2026-07-20 --dump ./dump -n virt-prod

Exit codes

Code	Meaning
0	chain resolved; at most warnings
1	a reference is missing or errored — a successful diagnosis
2	the tool itself failed (bad kubeconfig, unreadable dump)

Suitable for --no-color use in scripts and CI.

Storage-array CRs

The array layer defaults to Pure/Portworx CSI (PureSnapshot, PureVolume). Portworx documents these CRs' fields but not their API group, so the group is resolved from the Kind through API discovery rather than hardcoded — which also means other vendors' CRs work by passing a different kind:

kubectl toolbox vmsnapshot snapshot vm snap --array-snapshot-kind MyVendorSnapshot
kubectl toolbox vmsnapshot disks vm --array-volume-kind MyVendorVolume
kubectl toolbox vmsnapshot snapshot vm snap --no-array   # stop at the CSI handle

Because the handle-to-CR mapping is not specified by CSI, the object is located by matching the handle against several candidate fields (metadata.name, spec.snapshotName, spec.snapshotID, and for volumes spec.volumeName, spec.volumeID, spec.serial), trying the whole handle and each trailing segment after ::, : and /. The matched field is reported as matchedOn= so the result is checkable, and a failed match says how many objects were scanned rather than silently showing nothing. The CSI handle is always printed even when no CR matches — that string is what you search for on the array.

What else it surfaces when something is broken

This is a tool you reach for when there is a problem, so it gathers the things you would go and check by hand next.

Kubernetes events. The single most useful signal and the one that says why rather than what. Attached automatically to any object the report has already flagged, so a clean run stays readable. --events attaches them to everything; --no-events skips the lookup entirely.

The live runtime view. The VirtualMachineInstance is the authority on what the VM actually has attached right now: status.volumeStatus gives a per-disk phase, target device and reason, which is where hotplug failures and unmountable volumes appear. Underneath it the virt-launcher pod is where mount errors and scheduling failures surface. Neither is reachable from the spec-side chain, and both are where you look when every storage object appears healthy but the VM will not run. A failed live migration is reported too, since it leaves volumes attached to the wrong node. Skip with --no-runtime.

Storage class facts instead of guesses. The volumeBindingMode is read, not assumed, so a Pending PVC is explained correctly: under WaitForFirstConsumer Pending is expected, under Immediate it means provisioning is failing. Whether a VolumeSnapshotClass exists for the provisioner is checked, which definitively answers why a VM snapshot silently skipped a disk — reported as CONFIRMED cause or CONFIRMED NOT the cause.

Where the volume is attached. VolumeAttachment gives the node holding each PV and any attach/detach error — this is what produces Multi-Attach errors and what you correlate against array-side host connections.

Who else holds the PVC. Every pod mounting it, with phase and node. More than one consumer of a ReadWriteOnce volume is flagged, as is a PVC terminating while still mounted, along with the finalizers holding it.

Capacity consistency. A snapshot larger than the PVC it is being restored into cannot bind; that is caught here rather than surfacing later as an opaque CSI error.

--timeline. A chronological view of every dated fact found, deduplicated across branches. Establishing that a restore ran before a PVC vanished, or that a snapshot predates a resize, is often the whole investigation.

What it checks beyond following references

• spec.volumeBackups[] vs status.volumeSnapshotStatus[] consistency, in both directions — a stale status entry and an un-snapshotted disk look nothing alike.
• Whether each captured source PVC still exists. When it doesn't, the PVC spec recorded inside volumeBackups[].persistentVolumeClaim is the only record of its size and storage class, and it is printed.
• status.snapshotVolumes.excludedVolumes — the usual reason a disk was never snapshotted is that its storage class has no VolumeSnapshotClass.
• Crossed back-references: VolumeSnapshotContent.spec.volumeSnapshotRef and PersistentVolume.spec.claimRef are both verified to point back at the object that claims them.
• dataVolumeTemplates declared by the VM with no corresponding DataVolume.
• PVCs stuck terminating on a finalizer, and DataVolumes with importer restarts.

Provenance staleness is deliberately not treated as a failure: a Bound PVC whose origin golden-image snapshot has rotated away is normal, and is reported as lineage rather than driving a non-zero exit.

What sits above a VirtualMachineSnapshot

Unlike DataVolume → PersistentVolumeClaim, a VirtualMachineSnapshot has no single fixed parent kind. It may be created directly, by a VirtualMachineClone (which records it in status.snapshotName and is built on Snapshot+Restore), or by backup tooling such as OADP/Velero via the KubeVirt plugin. --owners walks metadata.ownerReferences upward and also scans VirtualMachineClone status, so whatever produced the snapshot is found without enumerating kinds.

VirtualMachineRestore objects consuming a snapshot are always listed, since a restore is what creates new PVCs and is therefore usually the thing that failed.

Development

make test    # runs against testdata/ fixtures, no cluster needed
make vet

testdata/fixture is a healthy chain with two seeded faults; testdata/broken covers inconsistent back-references and an unmatched array handle.

Contributing

See CONTRIBUTING.md. The project uses Conventional Commits and Semantic Versioning; releases are cut by pushing a vX.Y.Z tag. Notable changes are recorded in CHANGELOG.md.

Licence

Apache-2.0. Copyright 2026 Milan Jovanović.

kubectl-toolbox is an independent tool. It is not affiliated with or endorsed by Red Hat, the KubeVirt project, Pure Storage or Portworx; those names are used only to describe the APIs and resources it reads.

Documentation

Overview

Command kubectl-toolbox resolves KubeVirt / OpenShift Virtualization snapshot and disk reference chains down to storage-array objects.