media

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 15, 2026 License: BSD-3-Clause Imports: 13 Imported by: 0

README

media

Decode any media file into what a perceptual hash needs: mono PCM at a chosen rate, or video frames resampled by time.

import "github.com/go-perceptual/media"

kind, err := media.Classify(path)        // image, audio, video — by reading, not by name

pcm, err := media.Mono(path, 11025, 180) // mono int16 at 11025 Hz, first 180 s

err = media.Frames(path, 15, 180, func(i int, img image.Image) error {
    return hasher.Add(i, feature(img))   // one frame per index, resampled to 15 fps
})

It is the plumbing, not the policy. Thresholds, verdicts and what counts as a duplicate belong to the caller. This package turns a path into samples or pictures, and refuses loudly when it cannot.

Used by go-perceptual/perceptual and go-perceptual/dedupe, where it was copied code before.

Why this is a package and not a snippet

Every function here encodes a decision that goes wrong silently if two copies of it drift apart. None of these crash. None fail a test. Each one produces two tools that disagree about whether two files are the same thing — confidently.

the decision what the other way does
Downmix averages the channels taking the first channel compares a one-channel recording against silence, and silence matches other silence
Resample clamps an overshoot converted to int16 wraps to the opposite sign, putting a click into the feature — content that is not in the recording
the PCM width is asserted, then derived PCM read at the wrong width is noise that still hashes
FrameImage refuses 10-bit truncating to 8 gives one film two different fingerprints
Frames resamples by time taking every nth frame puts a 25 fps and a 30 fps copy of one film on different indices
Classify reads bytes believing the extension makes a renamed copy invisible, and an invisible file reads as "nothing found"

That last row is the whole reason the rest matters. For a tool that looks for copies, a file it cannot see is indistinguishable from a clean result.

Classify reads the file

An extension is a claim by whoever named it. A download folder is full of wrong ones, and the copy a deduplicator exists to find is often precisely the one that was renamed.

Two stages, because a tree scan cannot read every file whole:

  1. the first 64 KiB — image.DecodeConfig settles images, container.Sniff settles WAV, FLAC, ADTS, MPEG audio and anything unrecognisable;
  2. only a container that could hold either gets opened properly, and that open streams rather than slurping, which matters when the candidate is a 4 GB film.

A file holding both audio and video is KindVideo: its pictures are the stronger evidence, since two different films can share a soundtrack while one film in two languages shares its pictures.

KindOther is not an error — most of a directory tree is not media. A file that is media but cannot be read is an error, and the two are kept distinct because only one of them should make a caller warn.

Linear PCM

A WAV holds samples that already are PCM, so go-avkit/decode refuses them — rightly, since there is nothing to decode. The consequence is that WAV, the simplest format in its list, is unreachable through the decoder API. This package carries the passthrough, in one place, so the workaround is not copied into every consumer. Reported upstream with an implementation offered.

The sample width is not carried by TrackConfig, so it is derived from a chunk's byte length over its frame count. That is valid only while a sample's Duration counts frames — only while the track's timescale is its sample rate. Measured on go-avkit's WAV reader at 8000, 22050, 44100 and 48000 Hz: the equality holds. It is asserted at run time and refused otherwise.

Every width is tested through a real file rather than only through PCMValue, because the arithmetic and the derivation are different claims:

width worst error on a full-scale ramp
8-bit unsigned 1.5e-2
16-bit signed 6.0e-5
24-bit signed 2.4e-7
32-bit signed 9.2e-10
32-bit float 3.0e-8

The sample count is asserted too, and that is the assertion that catches a bad derivation: a width inferred as 2 instead of 3 returns half again as many samples, all garbage, with no error anywhere.

24-bit gets its own test. Three bytes have no sign bit where Go expects one, and without explicit sign extension every negative sample reads as a large positive one — not noise, but a recording with its bottom half folded over, which hashes consistently and so matches other copies of itself while matching nothing else.

Frames

Frames takes a callback rather than returning a slice, because a decoded video does not fit in memory and no caller wants it there — they hash frame by frame.

Indices are consecutive from zero and each one holds the frame showing at that instant. A test checks that index n at 15 fps and index 2n at 30 fps land on the same picture, which is the property the by-time resampling exists for and which taking every nth frame silently breaks.

Unsupported codecs

VP9, AV1, Vorbis and MP2 are missing from go-avkit's decoders. Frames returns an error for them rather than skipping.

⚠ The message is misleading, and is reported upstream. A VP9 file produces decode: invalid sample: NAL unit of 2185855810 bytes with 2939 left, which is an H.264 NAL-unit parser handed VP9 data: the decoder does not check the codec before parsing, so it calls the file corrupt instead of saying the codec is unsupported. The length is bounds-checked, so this is a wrong diagnosis rather than an unbounded allocation. A fixture is committed and the test asserts an error is returned, not its wording, so it will not silently start passing for the wrong reason when that is fixed.

Formats

Containers and codecs come from go-avkit: MP4, Matroska/WebM, MPEG-TS, Ogg, WAV, AVI, native FLAC, ADTS and MPEG audio; H.264 and H.265 video; FLAC, MP3, Opus, AAC-LC and HE-AAC audio. Images go through Go's own decoders: JPEG, PNG and GIF.

Resampling is go-dsp/signal's polyphase resampler, with SciPy's resample_poly semantics.

Test fixtures

Committed so CI runs the real decoders on a machine with no ffmpeg, and their provenance is recorded rather than remembered. The audio fixtures are four encodings of one four-second melody; the video fixtures are a clip and scaled copies of that clip. WAVs are generated inside the tests, since the PCM path is the one place where the bytes on disk are the thing under test — and writeWAVWidth refuses a sample outside [-1, 1] rather than clamping it, because a fixture that overflows its own width produces garbage that still decodes and reads exactly like a decoder defect.

See go-perceptual/dedupe's README for the ffmpeg commands that produced the committed files, including why the 180p clip is derived from the 360p one rather than rendered at that size.

Licence

BSD-3-Clause.

Documentation

Overview

Package media decodes a media file into what a perceptual hash needs: mono PCM at a chosen sample rate, or video frames resampled to a chosen rate by time.

It is the plumbing, not the policy. Thresholds, verdicts and what counts as a duplicate belong to the caller — this package only turns a path into samples or pictures, and refuses loudly when it cannot.

Why it is a package

Every function here encodes a decision that goes wrong SILENTLY if two copies of it drift apart. A downmix that takes the first channel instead of averaging compares a one-channel recording against silence; PCM read at the wrong sample width is noise that still hashes; a 10-bit frame truncated to 8 gives one film two different fingerprints; frames taken every nth instead of by time put a 25 fps and a 30 fps copy on different indices. None of those crash, and none fail a test. They produce two tools that disagree about whether two files are the same, confidently.

Index

Constants

View Source
const HeaderBytes = 64 << 10

HeaderBytes is how much of a file Classify reads before deciding.

Enough for every signature container.Sniff looks at, and for the configuration header of a JPEG, PNG or GIF. MPEG-TS is the greediest: its sync byte recurs every 188 bytes and a sniffer wants several packets before believing it, so this is generous rather than minimal.

Variables

View Source
var ErrNoFrames = errors.New("media: the video track yielded no frames")

ErrNoFrames means the video track decoded to nothing. Like ErrNoSamples it is a skip rather than a failure.

View Source
var ErrNoSamples = errors.New("media: the audio track yielded no samples")

ErrNoSamples means the track decoded to nothing, or to too little to fingerprint. It is a skip, not a failure: a one-frame file is not broken.

Functions

func Downmix

func Downmix(a decode.Audio) ([]float64, error)

Downmix averages one buffer's channels into samples in [-1, 1].

⛔ AVERAGING, not taking the first channel. A track whose content sits in one channel — an interview, a badly mixed rip, a karaoke version — would otherwise be compared against silence, and silence matches other silence, so such files would all group together as "the same recording".

func FrameImage

func FrameImage(f decode.Frame) (image.Image, error)

FrameImage wraps a decoded frame's planes as an image WITHOUT copying them.

⛔ It refuses what it cannot represent honestly rather than approximating. A 10-bit frame shifted down to 8 bits would hash perfectly well — and would hash DIFFERENTLY from the same film delivered as 8-bit, so one film would carry two fingerprints. For a tool whose whole purpose is deciding whether two files are the same thing, that is the one failure that must never be silent.

func Frames

func Frames(path string, fps float64, maxSeconds int, fn func(index int, img image.Image) error) error

Frames decodes a file's first video track, resampled to fps, and hands each frame to fn as an image.

maxSeconds bounds how much is read (0 reads all of it). If fn returns an error, decoding stops and that error is returned.

⛔ Frames are resampled BY TIME, not by taking every nth frame. A video hash compares phases at integer frame indices, so a 30 fps and a 25 fps copy of one film must land on the SAME indices — which happens only if each index is filled by the frame showing at that instant. Taking every other frame of the 30 fps file and every frame of a 15 fps one produces two valid-looking hashes that cannot be compared, and nothing says so.

⛔ A callback rather than a returned slice, because a decoded video does not fit in memory and no caller wants it there: they hash frame by frame.

func IsPCM

func IsPCM(codec string) bool

IsPCM says whether a track's samples are already PCM and need no decoder.

⛔ These are the codecs a container reports for a WAV: `ipcm` signed integer, `fpcm` float, `raw ` unsigned 8-bit. go-avkit's decoder REFUSES all three — correctly, since the samples already ARE PCM and there is nothing to decode — so a consumer that only knows how to call a decoder cannot read a WAV at all, which is the simplest format in the list.

Reported upstream (go-avkit/decode, 2026-11-15, with a working implementation offered). This package carries the passthrough meanwhile, in one place, so the workaround does not get copied into every consumer.

func Mono

func Mono(path string, rate, maxSeconds int) ([]int16, error)

Mono decodes a file's first audio track to mono samples at rate, reading at most maxSeconds of it (0 reads all of it).

The chain is demux, decode to PCM, downmix to mono, resample. Each step is somebody else's job done properly — the container and decoder are go-avkit's, the resampler is go-dsp's — and this is the plumbing between them.

⛔ The RATE is part of a fingerprint, not a preference. A feature's frame geometry and frequency bounds are expressed in samples at a particular rate, so audio fed at another rate produces a perfectly plausible fingerprint that matches nothing. Callers pass the rate their feature is defined at and do not treat it as a tuning knob.

func MonoFloat

func MonoFloat(path string, maxSeconds int) (samples []float64, rate int, err error)

MonoFloat is Mono without the resampling or the conversion to int16: the track's own samples, downmixed, in [-1, 1], with the rate they are at.

It exists for a caller whose feature wants float input or another rate.

func PCMValue

func PCMValue(codec string, b []byte, width int) (float64, error)

PCMValue turns one PCM sample's bytes into a value in [-1, 1].

It is exported because the sample width cannot be read from the track configuration, so a caller doing its own PCM reading needs the same conversion rather than a second guess at it.

func Resample

func Resample(mono []float64, from, to int) ([]int16, error)

Resample brings mono samples from one rate to another and converts them to int16.

⛔ It CLAMPS. A polyphase resampler rings, and its ringing overshoots a full-scale signal; converting an overshoot to int16 without clamping wraps it to the opposite sign, which puts a click into the feature — content that is not in the recording.

Types

type Kind

type Kind string

Kind is what a file turned out to hold.

const (
	KindImage Kind = "image"
	KindAudio Kind = "audio"
	KindVideo Kind = "video"
	// KindOther is a file this package cannot hand to a perceptual hash. It
	// is not an error: most of a directory tree is not media.
	KindOther Kind = ""
)

func Classify

func Classify(path string) (Kind, error)

Classify says what a file holds, by READING it.

⛔ Not by its extension. An extension is a claim by whoever named the file, and a download folder is full of wrong ones: a .jpg holding a PNG, an .mp4 holding only audio, a film saved with no extension at all. Code that believes the name returns a confident wrong answer — and for a tool that looks for copies, a file it cannot see reads as "nothing found", which is indistinguishable from a clean tree.

It works in two stages, because a tree scan cannot afford to read every file whole. The header alone settles images and the audio-only containers. Only a container that could hold either gets opened properly, and that open STREAMS rather than slurping the file, which matters when the candidate is a 4 GB film.

A file holding BOTH audio and video is KindVideo. Its pictures are the stronger evidence: two different films can share a soundtrack, while one film in two languages shares its pictures.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL