pdq

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 15, 2026 License: BSD-3-Clause Imports: 6 Imported by: 0

README

pdq

The PDQ perceptual image hash, in pure Go, with no dependencies and no cgo.

PDQ answers one question: are two pictures essentially the same picture? It is a syntactic hasher — it compares similarity without extracting meaning — built for media shared with minimum adversariality: recompressed, resized, lightly blurred, JPEG turned into PNG. It is not built to survive cropping, border addition or deliberate obfuscation.

h, quality, err := pdq.Sum(img)   // 256-bit hash, 0..100 quality
d := h.Distance(other)            // 0 identical, 256 opposite

A distance of 30 or less — better 20 or less — indicates a confident match. Unrelated images average 128.

What it is

hash 256 bits, compared by Hamming distance
quality 0..100; near 0 means featureless, and such a hash should not be trusted
symmetries all eight rotations and flips for the price of one transform
cost the reference states hashing is about as fast as reading the file off disk

SumDihedral returns the hashes of all eight symmetries. They are cheap because they come from the same 16×16 transform by transposition and sign flips — no image is rotated, and no second transform is run.

The algorithm, and where it came from

The algorithm is Facebook's, published as The TMK+PDQF video-hashing algorithm and the PDQ image-hashing algorithm (hashing/hashing.pdf in facebook/ThreatExchange). This package is a port of the reference C++ in pdq/cpp of the same repository.

  1. RGB → luminance (0.299R + 0.587G + 0.114B), not greyscale.
  2. A two-pass Jarosz tent filter — four 1-D box passes — at full resolution, then the centre sample of each block gives a 64×64 image.
  3. A quality metric from quantised neighbour gradients of that 64×64.
  4. A 2-D DCT, keeping slots 1–16 in each direction. Slot 0 is the DC term — overall brightness — and dropping it is what makes the hash indifferent to exposure.
  5. Each of the 256 bits is 1 where its coefficient exceeds the median.

⭐ Because every bit compares against the median, every non-degenerate hash has exactly 128 one-bits. That is an invariant, not a statistic, and the test suite asserts it across image sizes: it is the cheapest proof that the median, the transform and the bit-packing still agree.

⛔ Bit identity is not the contract

The specification states that the downscale to hashing resolution is implementation-specific, and that two implementations hashing the same image are expected to differ by 0, 2, 4 or 6 bits. So this package does not promise, and does not assert, identity with any other implementation. What it asserts is that published bound, against the reference's own regression vectors.

Measured against bridge-1-original.jpg from the reference corpus:

distance 6 from the reference vector, quality 100 vs 100

⚠ Within the bound, but at its edge, and the residual is not yet explained. One hypothesis has been tested and killed: decoding with the fleet's go-images/jpeg fork instead of image/jpeg produces a bit-identical hash, because that fork differs only for four-component pictures and the reference image has three components.

Running the reference vectors

The reference images are not redistributed here. Put them in testdata/ and the oracle test runs; without them it skips and says so out loud, because a test that skips quietly is a test that reports zero.

Licence

BSD-3-Clause.

Documentation

Overview

Package pdq implements the PDQ perceptual image hash.

PDQ answers one question: are two pictures essentially the same picture? It is a SYNTACTIC hasher — it compares similarity without extracting meaning — and it is built for media shared with minimum adversariality: recompressed, resized, lightly blurred, JPEG turned into PNG. It is NOT built to survive cropping, border addition or deliberate obfuscation.

The algorithm is Facebook's, published as "The TMK+PDQF video-hashing algorithm and the PDQ image-hashing algorithm" (hashing/hashing.pdf in github.com/facebook/ThreatExchange). This package is a port of the reference C++ in pdq/cpp of the same repository, read at the commit named in testdata/REFERENCE. Every step below cites the source it came from.

⛔ Bit identity with another implementation is NOT the contract. The specification says so itself: the resize down to the hashing resolution is implementation-specific, and hashes from two implementations of the same image are expected to differ by 0, 2, 4 or 6 bits of Hamming distance. What this package promises is the published algorithm, run at full resolution the way the reference does, with the stated bound asserted against the reference's own regression vectors.

Index

Constants

This section is empty.

Variables

View Source
var ErrEmpty = errors.New("pdq: empty image")

ErrEmpty is returned for an image with no pixels.

View Source
var ErrMalformed = errors.New("pdq: malformed hash")

ErrMalformed is returned for a hash string that is not 64 hex digits.

Functions

func Sum

func Sum(img image.Image) (Hash, Quality, error)

Sum returns the PDQ hash of img and its quality metric.

func SumDihedral

func SumDihedral(img image.Image) (Dihedral, Quality, error)

SumDihedral returns the hashes of all eight symmetries of img, and the quality metric, for the price of one transform.

Types

type Dihedral

type Dihedral struct {
	Original   Hash
	Rotate90   Hash
	Rotate180  Hash
	Rotate270  Hash
	FlipX      Hash
	FlipY      Hash
	FlipPlus1  Hash // transpose, about the main diagonal
	FlipMinus1 Hash // about the anti-diagonal
}

Dihedral holds the hashes of the eight symmetries of one image: the original, three rotations and four flips.

They cost almost nothing: all eight come from the SAME 16x16 DCT matrix by transposition and sign flips, so no image is rotated and no second transform is run.

type Hash

type Hash [16]uint16

Hash is a 256-bit PDQ hash.

The bit at row i, column j of the 16x16 transform output is bit j of word i, which is how the reference stores it (pdqBuffer16x16ToBits sets bit i*16+j, and Hash256 keeps 16 words of 16 bits).

func ParseHash

func ParseHash(s string) (Hash, error)

ParseHash reads the 64-hex-digit form written by String.

func (Hash) Distance

func (h Hash) Distance(o Hash) int

Distance is the Hamming distance between two hashes: 0 when identical, 256 at most.

The specification's guidance on thresholds: distances between unrelated images average 128, and a distance of 30 or less — better 20 or less — has been found to indicate a confident match.

func (Hash) Norm

func (h Hash) Norm() int

Norm is the number of 1 bits.

⭐ It is 128 for every non-degenerate image, and that is a real invariant rather than a statistical one: each bit says whether its coefficient exceeds the MEDIAN of the 256 coefficients, so exactly half are set. It only fails when coefficients tie, which happens for degenerate input such as a solid colour — and those are what the quality metric exists to flag.

func (Hash) String

func (h Hash) String() string

String renders the hash as 64 lowercase hex digits.

⛔ The reference prints the HIGHEST word first (Hash256's format lists w[15] down to w[0]), so the leading hex digits are row 15, not row 0. Printing them the other way round produces a plausible string that matches nothing.

type Quality

type Quality int

Quality is the 0..100 metric PDQ reports beside the hash. It says how much detail the downsample retained: a featureless or solid-colour image scores near 0, and such a hash should not be trusted for matching.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL