Documentation
¶
Overview ¶
Package consistency provides primitives that detect, with overwhelming probability, whether a Parquet snapshot diverges from the MySQL table it was taken from.
The foundation is ConsistentTableChecksum: a point-in-time, order-independent, type-canonical fingerprint of a source table. The fingerprint is a 64-bit non-cryptographic multiset hash — sized to catch accidental corruption and divergence (collision ≈ 2⁻⁶⁴ per comparison), not to resist an adversary who can forge a colliding table. Fidelity is only checkable at a frozen consistent point — you cannot compare a checksum of the Parquet against a live table that has moved on. So the fingerprint is computed inside START TRANSACTION WITH CONSISTENT SNAPSHOT and bound to the @@gtid_executed captured at that same point. The baseline writer (issue #633) and the verify capstone (issue #634) both call this primitive — at dump time and at audit time respectively — and compare the result against a Parquet-derived digest.
Part of the data-consistency guarantee epic (#631).
Index ¶
Constants ¶
const DigestVersion = digestVersion
DigestVersion is the current digest contract tag (see digestVersion). Exported so a consumer comparing a persisted digest against a freshly computed one can tell whether they share a contract.
Variables ¶
This section is empty.
Functions ¶
func DigestVersionOf ¶ added in v0.30.0
DigestVersionOf returns the "vN:" version-tag prefix of a digest produced by this package (ConsistentTableChecksum / Hasher.Digest) — the text up to and including the first ':'. It returns "" when the string carries no recognizable tag (e.g. an empty or pre-tag legacy value). Comparing two digests whose tags differ would byte-differ even on identical data, so a consumer must degrade to a needs-rebaseline signal rather than report a false mismatch.
Types ¶
type Hasher ¶
type Hasher struct {
// contains filtered or unexported fields
}
Hasher is the exported, order-independent content-digest accumulator shared across the consistency epic (#631). ConsistentTableChecksum uses it to fingerprint a live source table; the baseline writer (#633) uses it to fingerprint the rows it ingests from a mydumper dump. Both route through the same internal rowHasher and the same version tag, so a live-source digest and a persisted baseline digest are byte-identical for the same data — which is what lets the verify capstone (#634) compare them meaningfully.
A row is a list of field byte-slices in column order; a nil field is SQL NULL, distinct from a non-nil empty value. The digest is order-independent (rows may be added in any order) — see rowHasher for the construction and its 64-bit, non-cryptographic, accidental-corruption threat model.
func (*Hasher) AddBytes ¶
AddBytes folds one row given its column values already rendered to bytes, in ordinal column order. A nil element is SQL NULL (distinct from a non-nil empty value). This is the form the verify capstone (#634) feeds after rendering a reconstructed row to ConsistentTableChecksum's text-protocol form, so a reconstructed digest can be compared to a live one.
func (*Hasher) AddStrings ¶
AddStrings folds one row given the parallel value/null slices the baseline dump parser yields. A column is NULL when nulls[i] is true or absent.
The string bytes are MySQL's text rendering (mydumper dumps what MySQL's text protocol returns), so the digest matches a live ConsistentTableChecksum, which hashes the same source text via CAST(... AS CHAR). It therefore fingerprints the SOURCE rows, not the Parquet encoding: value transforms the baseline writer applies downstream (e.g. a MySQL zero-date stored as Parquet NULL) are deliberately not reflected here — see the call site for why mirroring them would be wrong.
Byte-identity with the live checksum holds only while the dump's column order equals ordinal order (ConsistentTableChecksum SELECTs by ORDINAL_POSITION) — normally true, since mydumper dumps columns in table order.
type TableChecksum ¶
type TableChecksum struct {
Schema string
Table string
GTIDSet string
RowCount int64
Digest string
// Columns is the ordered set of column names the digest was computed over
// (ordinal order, generated columns excluded). A consumer that recomputes a
// digest to compare (the verify capstone #634) must hash exactly this set in
// this order, rather than re-deriving it — re-deriving from a schema snapshot
// risks a different generated-column membership and a spurious mismatch.
Columns []string
}
TableChecksum is a point-in-time fingerprint of a single source table.
GTIDSet is @@gtid_executed captured just after the consistent snapshot opens — the point against which a Parquet snapshot can later be compared. It is empty on a server with GTIDs disabled (gtid_mode=OFF) or absent (MariaDB); callers that need a position anchor on such servers capture it separately.
@@gtid_executed is global state, not MVCC-filtered, so a commit landing in the brief window between the snapshot opening and this read is reflected in the GTID but not in the snapshot data. This lock-free window is the same one every bintrail baseline carries (mydumper dumps with NO_LOCK and reads its metadata GTID the same way); the digest itself is computed entirely over the snapshot and is unaffected — only the anchor's precision is bounded by this window.
Digest is a version-tagged, order-independent multiset hash of the row contents (see digestVersion). Two tables holding the same rows in any physical or primary-key order produce the same Digest; a single changed byte produces a different one with overwhelming probability; a representation-only difference (e.g. JSON whitespace, which MySQL normalizes on storage) produces the same one.
func ConsistentTableChecksum ¶
func ConsistentTableChecksum(ctx context.Context, db *sql.DB, schema, table string) (TableChecksum, error)
ConsistentTableChecksum computes a TableChecksum for schema.table against the live source db. The whole computation — GTID capture, column introspection, and the table scan — runs on a single pinned connection inside START TRANSACTION WITH CONSISTENT SNAPSHOT, so the digest and the row count describe one snapshot of the data and the captured GTID anchors it (modulo the lock-free window documented on TableChecksum.GTIDSet).
The canonical form of every value is MySQL's text-protocol rendering with the session time zone pinned to UTC. That rendering is already type-exact — UNSIGNED integers print unsigned, DATETIME/TIMESTAMP carry their declared fractional precision, DECIMAL is pre-formatted, JSON is normalized (MySQL; MariaDB stores JSON as LONGTEXT and renders it verbatim) — so no per-type canonicalization is reimplemented here. String columns are read with character_set_results = binary, so their RAW STORED BYTES are hashed (no transcoding to the connection charset) — matching mydumper's `SET NAMES binary` dump contract (issue #792). The Parquet side of the comparison (#634) must reproduce this same contract: "MySQL text rendering, session time zone UTC, raw string bytes". Two digests remain FLOAT/DOUBLE-text-rendering and server-family dependent, which is the natural case since the baseline and its verify run against the same source; the charset dependency is now removed by the binary pin.
Generated columns (VIRTUAL/STORED) are excluded: mydumper does not dump them, so they are absent from the baseline Parquet and must be absent here too.
func ConsistentTableChecksumNormalized ¶ added in v0.26.2
func ConsistentTableChecksumNormalized(ctx context.Context, db *sql.DB, schema, table string, normalize func(raw []byte, dataType string) []byte) (TableChecksum, error)
ConsistentTableChecksumNormalized is ConsistentTableChecksum with an extra per-column hook: normalize, when non-nil, rewrites a scanned column's raw text-protocol bytes before it is folded into the digest.
This exists so a caller comparing this digest against a digest computed by a DIFFERENT renderer (internal/verify's reconstruct, in the live-source verify path) can rewrite away known representation-only gaps between the two — e.g. a MySQL zero-date sentinel the reconstruct side normalizes to NULL, or a JSON-shaped TEXT value the reconstruct side re-serializes in canonical key order — symmetrically on THIS side too, the same way internal/verify's own renderCellNormalized does for its side. Applying the SAME normalization asymmetrically (only one side) would trade one false mismatch for a different one; this package intentionally does not import internal/verify to decide what "normalized" means — the caller supplies that policy as a closure, keeping this package's only knowledge of the concept a raw-bytes-in, raw-bytes-out hook.
normalize receives the column's information_schema.COLUMNS DATA_TYPE (lower-case) and is never called for a genuine SQL NULL (nil raw bytes) — there is nothing to rewrite. It is not called by ConsistentTableChecksum (nil hook, unmodified behavior) — callers that persist this digest independently of internal/verify (e.g. the baseline writer, #633) must not normalize, since their digest is compared against ANOTHER raw, unnormalized rendering of the same contract, not against a reconstruct digest.
Source Files