rollinghash

Documentation

Overview

Command rollinghash digests the rewritten output as it streams, without holding it.

Two hashes, because they answer different questions and only one of them is rolling.

A whole-stream digest (FNV-1a here) answers "is this the same output as last time" - an ETag, a cache key, a comparison between two rewrites. It depends on the bytes and not on how they arrive.

A rolling hash over a sliding window answers "where should this stream be cut" - content-defined chunking, the thing that makes a delta useful when a page changes in the middle. Boundaries fall where the window's hash has a run of low bits, so inserting a banner near the top shifts one chunk and leaves the rest aligned, which a fixed-size split would not.

The reason this program is worth writing against this library rather than any io.Reader is what wrapping the destination shows. The rewriter does not hand its output over in convenient pieces, and how many pieces there are depends on what the rewrite does rather than on the document:

<div class="row"><a href="/p">link</a></div>

passthrough                     1 write
a handler that matches          3 writes
the handler sets one attribute  12 writes

because a mutated start tag is re-serialised piece by piece: "<", "a", " ", `href="/p"`, " ", "rel", `="`, "noopener", `"`, ">". Measured over 2000 such elements, one 132 KB write becomes 22,001 writes with a median size of one byte. Both hashes here are indifferent to that - which is the point of checking them against it - but a destination that is a socket or a file is not, and belongs behind a bufio.Writer.