Documentation
¶
Overview ¶
Package tmk implements TMK+PDQF, a video-similarity hash.
TMK answers "are these two videos the same video?" for videos that differ in frame rate, container, resolution, light watermarking or a few seconds of leader and trailer. It is NOT built to find a clip inside a longer video, nor to survive cropping or heavy overlays.
Its useful property is that the signature has a FIXED SIZE whatever the video's length — about 256 KB, of which the first kilobyte already separates almost every pair — so a library of videos costs a predictable amount of memory and can be compared without the originals.
The algorithm is published as "The TMK+PDQF video-hashing algorithm and the PDQ image-hashing algorithm" (hashing/hashing.pdf in github.com/facebook/ThreatExchange), with the kernel from Poullot et al. This package is a port of the reference C++ in tmk/cpp of that repository, and of the derivation written in its tmkfv.h.
⛔ This package does NOT decode video. It takes frame features — one [256]float32 per frame, which is what go-perceptual/pdq's Float returns — at a fixed frame rate. Decoding belongs to whoever owns the container and the codec, and keeping it out means this package is testable without a single video file and usable with any decoder.
Index ¶
Constants ¶
const DefaultThreshold = 0.7
DefaultThreshold is the reference's validated threshold for both the level-1 and the level-2 score.
const Dim = 256
Dim is the length of one frame feature: PDQF, the PDQ transform before its binarisation step.
const FPS = 15
FPS is the frame rate frames must be resampled to before hashing.
⛔ It is part of the algorithm, not a tuning knob: the temporal kernel compares phases at integer frame indices, so two videos hashed at different rates are not comparable even though both hashes look valid.
Variables ¶
var ErrIncomparable = errors.New("tmk: hashes were built with different parameters")
ErrIncomparable is returned when two hashes were built with different parameters. ⛔ It is an error rather than a number because a score computed across mismatched periods is meaningless and looks exactly like a real one.
var ErrNoFrames = errors.New("tmk: no frames were ingested")
ErrNoFrames is returned when a hash is finished without any frame.
var PoullotCoefficients = [...]float64{
0.0708041893112, 0.13937789309, 0.132897260304,
0.122765735552, 0.109878684888, 0.09529606433,
0.0800986647852, 0.0652590650356, 0.0515478238322,
0.0394851531195, 0.0293374252025, 0.0211492623679,
0.0147973073245, 0.0100512818746, 0.0066306408014,
0.00424947117334, 0.0026467615764, 0.00160270959695,
0.000943882629639, 0.000540841638603, 0.000301633183798,
0.000163800158855, 8.66454753015e-05, 4.46626303151e-05,
2.24429442235e-05, 1.09982139799e-05, 5.25823487999e-06,
2.45358229988e-06, 1.11781474895e-06, 4.97406489221e-07,
2.16265487234e-07, 9.19087006565e-08,
}
PoullotCoefficients are the 32 Fourier weights, likewise.
var PoullotPeriods = [...]int{2731, 4391, 9767, 14653}
PoullotPeriods are the four periods of the temporal kernel, from Poullot et al. via the reference implementation.
Functions ¶
func Level1 ¶
Level1 is the cosine similarity of the two level-1 features: 1.0 for the same video, -1.0 at the opposite end.
It carries NO timing information — it is the mean frame, and two videos with the same shots in a different order score the same. That is exactly why it is only a filter: the reference computes the expensive level-2 score only when this one passes, because when level-1 scores differ greatly the level-2 scores do too.
func Level2 ¶
Level2 is the TMK pair score: 1.0 for identical hashes, 0.0 at worst.
For each period it slides one video against the other over every whole-frame offset and keeps the best match, then takes the best period. That sliding is what lets it match a video against the same video with a few seconds of leader removed.
The kernel at offset delta is
K = u_c0 . v_c0 + sum_j cos(j d) (u_cj . v_cj + u_sj . v_sj) + sum_j sin(j d) (u_sj . v_cj - u_cj . v_sj)
⭐ The four dot products do not depend on the offset, so they are computed ONCE per coefficient and then combined for each of the thousands of offsets. Recomputing them inside the offset loop — the obvious reading of the formula — multiplies the work by 256 and changes no answer.
Types ¶
type Hash ¶
type Hash struct {
// Average is the level-1 feature: the plain mean of the frame features,
// carrying no timing information at all. Comparing two of these is cheap
// and is what a large index stores.
Average [Dim]float32
// Cos and Sin are the level-2 features, indexed [period][coefficient][k].
Cos [][][]float32
Sin [][][]float32
// contains filtered or unexported fields
}
Hash is one video's signature.
type Hasher ¶
type Hasher struct {
// contains filtered or unexported fields
}
Hasher accumulates frame features into a Hash.
func NewWith ¶
NewWith returns a Hasher with explicit parameters. ⛔ Two hashes are only comparable when they were built with the SAME periods and coefficients; Level1 and Level2 refuse a mismatched pair rather than returning a number that means nothing.