subset

Documentation

Overview

Command subset writes a smaller corpus in the source format, so that a full pass of the framework can be rehearsed in minutes rather than hours.

The need is a measurement, not a guess. Filtering entities inside the replay engine turned out to buy only about 1.6×, so this command exists to give a run less to read: one pass over the source now, and every run afterwards reads a file a fraction of the size.

A later measurement corrected the reasoning, and the correction is worth recording because it redirects where the time goes. This command's own pass over days 0–13 read 239,471,460 rows in 2 minutes 12 seconds, while the sampled replay of the same window took 55 minutes for 8.7M warmed-and-scored events. Reading and parsing the corpus was never what dominated a run — scoring is, at roughly 2,700 events/s — so a subset saves about two minutes of a fifty-five minute run and no more.

The levers that actually shorten a run are, in order: the scoring window (replay's -maxseconds), dropping the shadow arms, which score every event and are excluded from the combination in any case, and the entity sample. Burn-in is roughly half the events and is frozen, so it sets the floor.

What is sampled

Entities, never events. A per-entity detector is a statement about one entity's own history, so thinning events within an entity would corrupt exactly the histories under test; dropping whole entities leaves every retained history intact and merely shrinks the population. The co-occurrence graph and the population marginals are consequently built from the retained entities only, which is a real difference from a full run and is why a subset is for rehearsal rather than for a headline figure.

Every entity named by a red-team label is kept regardless of the sample, so the labelled population is not itself thinned. That inflates the labelled share of the corpus, and a detection rate measured on a subset is therefore NOT comparable to one measured on the full population. The manifest written alongside the corpus records this in terms, so the file cannot be mistaken for the original later.