pdq

README

gopdq

A Go implementation of Meta's PDQ perceptual hashing algorithm.

PDQ is a perceptual hashing algorithm designed to identify visually similar images. It generates a compact 256-bit hash that remains stable across common image transformations like resizing, compression, and minor edits.

Installation

go get github.com/mayflower-tech/gopdq

Usage

There are two different functions provided in this package: HashFromFile and HashFromImage. While either will work, you should ensure that the input image has been resized to a size no greater than 512x512. See the PDQ paper.

Using two-pass Jarosz filters (i.e. tent convolutions), compute a weighted average of 64x64 subblocks of the luminance image. (This is prohibitively time-consuming for megapixel input so we recommend using an off-the-shelf technique to first resize to 512x512 before converting from RGB to luminance.)

For conveneicne, there is a helper method helpers.ResizeIfNeeded(img image.Image) which will return a resized image.Image that can be passed to HashFromImage.

package main

import (
    "fmt"
    "log"

    "github.com/mayflower-tech/gopdq"
)

func main() {
    // Hash an image file, assuming it has already been resized.
    // NOTE: There is no logic that _guarantees_ an image has been resized, this is up to you to ensure.
    result, err := pdq.HashFromFile("image.jpg")
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("Hash: %s\n", result.Hash)
    fmt.Printf("Quality: %d\n", result.Quality)
}

Using with pre-loaded images

import (
    "image"
    _ "image/jpeg"

    "github.com/mayflower-tech/gopdq"
    "github.com/mayflower-tech/gopdq/helpers"
)

func main() {
    // Open the image and decode it
    file, _ := os.Open("image.jpg")
    img, _, _ := image.Decode(file)

    // Resize if needed
    img = helpers.ResizeIfNeeded(img)

    // Generate hash
    result, _ := pdq.HashFromImage(img)
    fmt.Println(result.Hash)
}

HashResult

Both of the above functions will return a HashResult, which includes both the hash and the quality score.

type HashResult struct {
    Hash                  string
    Quality               int           // Results with a quality score < 50 should be discarded
    ImageHeightTimesWidth int
    HashDuration          time.Duration
}

Command Line Tools

PDQ Hasher

# Build the hasher
go build ./cmd/pdqhasher

# Hash an image
./pdqhasher path/to/image.jpg

# Output:
# Hash: e77b19ca5399466258c656bc4666a7853939a567a9193939e667199856ccc6c6
# Quality: 100
# Binary: 1110011110110001000110011010010100110011100110010100011001100010...

Hamming Distance Helper

# Build the helper
go build ./cmd/helper

# Calculate hamming distance
./helper hamming <hash1> <hash2>

# Output:
# 8

About Distance

Please see https://github.com/facebook/ThreatExchange/tree/main/pdq#matching

Note that outputs from the C++ implementation's example binary and the pdqhasher binary provided here may not return hashes that are exactly the same due to differences in resizing libraries. This is expected, see https://github.com/facebook/ThreatExchange/tree/main/pdq#hashing.

Benchmark

❯ go run ./cmd/benchmark --workers 32 --with-resize --duration 10
CPU:             AMD RYZEN AI MAX+ 395 w/ Radeon 8060S
CPU Cores:       32
Image Directory: testdata/images
Duration:        10s
Workers:         32
With Resize:     true
With I/O:        false

Results
=======

Total Time:       10.011804696s
Total Hashes:     27999
Errors:           0

Throughput:       2796.6 hashes/sec
Avg Time/Hash:    0.36 ms

Per Worker:       875.0 hashes
Per Worker/Sec:   87.4 hashes/sec

References

• PDQ Algorithm (C++ Reference)
• PDQ Hashing Paper
• ThreatExchange

Acknowledgments

This is a Go implementation of Meta's PDQ algorithm. All credit for the algorithm design goes to the original authors.

Documentation

Index

• Constants
• Variables
• type HashResult
  • func HashFromFile(filename string) (*HashResult, error)
  • func HashFromImage(img image.Image) (*HashResult, error)

Constants

const (
	LumaFromRCoeff = 0.299
	LumaFromGCoeff = 0.587
	LumaFromBCoeff = 0.114

	PdqNumJaroszXYPasses = 2

	DownsampleDims = 512

	MinHashableDim = 5
)

Various constants pulled from the reference implementation

Variables

var (
	ErrInvalidFile = errors.New("invalid input file name")
	// ErrImageTooLarge is returned when an input image exceeds MaxImagePixels.
	ErrImageTooLarge = errors.New("image exceeds maximum allowed pixel count")
)

var MaxImagePixels = 100_000_000

MaxImagePixels bounds the total number of pixels (width * height) that will be hashed. It is a backstop against memory-exhaustion / decode-bomb DoS when hashing untrusted input: HashFromImage allocates two float32 buffers of width*height each (8 bytes/pixel total) on top of the decoded bitmap, all of which scale with the source dimensions. PDQ input should be resized to 512x512 or smaller before hashing (see helpers.ResizeIfNeeded), so the default of 100 megapixels is a generous safety ceiling, not a normal operating point. Set to 0 to disable the check.

Types

type HashResult

type HashResult struct {
	Hash                  string
	Quality               int
	ImageHeightTimesWidth int
	HashDuration          time.Duration
}

HashResult contains the output of a PDQ hash operation

func HashFromFile

func HashFromFile(filename string) (*HashResult, error)

Opens a file at the specified file and uses image.Image to decode the image. Returns the result of HashFromImage. This is a convenience wrapper around HashFromImage that handles the IO and decoding for you. Ideally, you should call HashFromImage on your own with a 512x512 or smaller image that you have resized yourself. This function is provided only to match the reference implementation.

func HashFromImage

func HashFromImage(img image.Image) (*HashResult, error)

HashFromImage generates a PDQ hash from an image.Image The image should idealy be pre-resizes to 512x512 or smaller for performance reasons. SEE: https://github.com/facebook/ThreatExchange/blob/main/hashing/hashing.pdf, "More on Downsampling" Returns a HashResult containing the hash and a quality score between 0 and 100. Please reference the evaluation data for selecting a good quality score. From hashing.pdf: "Confident-match distances are up to the system designer, of course, but 30, 20, or less has been found to produce good results on evaluation data."