pdq

README

gopdq

A Go implementation of Meta's PDQ perceptual hashing algorithm.

PDQ is a perceptual hashing algorithm designed to identify visually similar images. It generates a compact 256-bit hash that remains stable across common image transformations like resizing, compression, and minor edits.

Installation

go get github.com/haileyok/gopdq

Usage

There are two different functions provided in this package: HashFromFile and HashFromImage. While either will work, you should ensure that the input image has been resized to a size no greater than 512x512. See the PDQ paper.

Using two-pass Jarosz filters (i.e. tent convolutions), compute a weighted average of 64x64 subblocks of the luminance image. (This is prohibitively time-consuming for megapixel input so we recommend using an off-the-shelf technique to first resize to 512x512 before converting from RGB to luminance.)

For conveneicne, there is a helper method helpers.ResizeIfNeeded(img image.Image) which will return a resized image.Image that can be passed to HashFromImage.

package main

import (
    "fmt"
    "log"

    "github.com/haileyok/gopdq"
)

func main() {
    // Hash an image file, assuming it has already been resized.
    // NOTE: There is no logic that _guarantees_ an image has been resized, this is up to you to ensure.
    result, err := pdq.HashFromFile("image.jpg")
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("Hash: %s\n", result.Hash)
    fmt.Printf("Quality: %d\n", result.Quality)
}

Using with pre-loaded images

import (
    "image"
    _ "image/jpeg"

    "github.com/haileyok/gopdq"
    "github.com/haileyok/gopdq/helpers"
)

func main() {
    // Open the image and decode it
    file, _ := os.Open("image.jpg")
    img, _, _ := image.Decode(file)

    // Resize if needed
    img = helpers.ResizeIfNeeded(img)

    // Generate hash
    result, _ := pdq.HashFromImage(img)
    fmt.Println(result.Hash)
}

HashResult

Both of the above functions will return a HashResult, which includes both the hash and the quality score.

type HashResult struct {
    Hash                  string
    Quality               int           // Results with a quality score < 50 should be discarded
    ImageHeightTimesWidth int
    HashDuration          time.Duration
}

Command Line Tools

PDQ Hasher

# Build the hasher
go build ./cmd/pdqhasher

# Hash an image
./pdqhasher path/to/image.jpg

# Output:
# Hash: e77b19ca5399466258c656bc4666a7853939a567a9193939e667199856ccc6c6
# Quality: 100
# Binary: 1110011110110001000110011010010100110011100110010100011001100010...

Hamming Distance Helper

# Build the helper
go build ./cmd/helper

# Calculate hamming distance
./helper hamming <hash1> <hash2>

# Output:
# 8

About Distance

Please see https://github.com/facebook/ThreatExchange/tree/main/pdq#matching

Note that outputs from the C++ implementation's example binary and the pdqhasher binary provided here may not return hashes that are exactly the same due to differences in resizing libraries. This is expected, see https://github.com/facebook/ThreatExchange/tree/main/pdq#hashing.

References

• PDQ Algorithm (C++ Reference)
• PDQ Hashing Paper
• ThreatExchange

Acknowledgments

This is a Go implementation of Meta's PDQ algorithm. All credit for the algorithm design goes to the original authors.

Documentation

Index

• Constants
• Variables
• type HashResult
  • func HashFromFile(filename string) (*HashResult, error)
  • func HashFromImage(img image.Image) (*HashResult, error)

Constants

const (
	LumaFromRCoeff = 0.299
	LumaFromGCoeff = 0.587
	LumaFromBCoeff = 0.114

	PdqNumJaroszXYPasses = 2

	DownsampleDims = 512

	MinHashableDim = 5
)

Various constants pulled from the reference implementation

Variables

var (
	ErrInvalidFile = errors.New("invalid input file name")
)

Types

type HashResult

type HashResult struct {
	Hash                  string
	Quality               int
	ImageHeightTimesWidth int
	HashDuration          time.Duration
}

HashResult contains the output of a PDQ hash operation

func HashFromFile

func HashFromFile(filename string) (*HashResult, error)

Opens a file at the specified file and uses image.Image to decode the image. Returns the result of HashFromImage. This is a convenience wrapper around HashFromImage that handles the IO and decoding for you. Ideally, you should call HashFromImage on your own with a 512x512 or smaller image that you have resized yourself. This function is provided only to match the reference implementation.

func HashFromImage

func HashFromImage(img image.Image) (*HashResult, error)

HashFromImage generates a PDQ hash from an image.Image The image should idealy be pre-resizes to 512x512 or smaller for performance reasons. SEE: https://github.com/facebook/ThreatExchange/blob/main/hashing/hashing.pdf, "More on Downsampling" Returns a HashResult containing the hash and a quality score between 0 and 100. Please reference the evaluation data for selecting a good quality score. From hashing.pdf: "Confident-match distances are up to the system designer, of course, but 30, 20, or less has been found to produce good results on evaluation data."