videosift

README

videosift

A Go library and CLI tool that extracts the minimum set of visually unique frames from a video file, optimized for use with image-based machine learning models such as content moderation classifiers.

How it works

videosift runs up to four FFmpeg-based extraction strategies in parallel, then deduplicates the full candidate set using perceptual hashing and Hamming distance comparison:

1. Scene change detection — selects frames where the visual content changes significantly
2. Keyframe extraction — pulls only the video's encoded I-frames (fast, no full decode)
3. Uniform temporal sampling — one frame every N seconds as a coverage baseline
4. MPDecimate — drops near-duplicate consecutive frames during decode

The result is an ordered, deduplicated slice of Frame values with timestamps, extraction method labels, and source video metadata attached.

Prerequisites

FFmpeg and FFprobe must be installed and available on PATH. No CGo or other native dependencies are required.

macOS: brew install ffmpeg
Ubuntu/Debian: apt install ffmpeg
Windows: Download from ffmpeg.org and add to PATH.

Installation

As a library:

go get github.com/matthupy/videosift

CLI tool (pre-built, via Go toolchain):

go install github.com/matthupy/videosift/cmd/extract@latest

Building from source

Prerequisites: Go 1.22+, FFmpeg/FFprobe on PATH, GNU make (optional).

git clone https://github.com/matthupy/videosift.git
cd videosift

Command	Description
make build	Compile the CLI to ./bin/extract
make install	Install the CLI to $GOBIN
make test	Run all tests with the race detector
make vet	Run go vet
make lint	Run go vet
make tidy	Tidy go.mod / go.sum
make clean	Remove ./bin/

Without make, the equivalent Go commands are:

go build -o bin/extract ./cmd/extract   # build
go install ./cmd/extract                # install
go test ./... -race                     # test
go vet ./...                            # vet

The CLI binary embeds the version string from the most recent git tag:

extract -version   # e.g. "v0.1.0" or "v0.1.0-3-gabcd1234-dirty"

Code quality and linting

This project uses go vet, the built-in Go static analysis tool, for code quality checks. This is an intentional dependency-reduction strategy: go vet is part of the standard Go toolchain and requires no external setup.

Linting scope

go vet catches:

• Unreachable code
• Type mismatches and Printf formatting errors
• Unused variables and assignments
• Suspicious code patterns

go vet does not enforce:

• Style conventions (naming, formatting)
• Complex linter rules (dependency checks, unsafe usage patterns)
• Custom project conventions

If you need more comprehensive linting beyond go vet, you can optionally run:

• gofmt — code formatting (go fmt ./...)
• goimports — unused import cleanup (go run golang.org/x/tools/cmd/goimports@latest -w ./...)

Run linting locally with make lint or go vet ./... before submitting pull requests.

Library usage

import "github.com/matthupy/videosift"

cfg := videosift.DefaultConfig()
// cfg.MaxFrames = 30        // cap output
// cfg.SceneThreshold = 0.3  // more sensitive scene detection
// cfg.WorkDir = "./frames"   // persist output (caller cleans up)

frames, err := videosift.Extract(ctx, "input.mp4", cfg)
if err != nil {
    log.Fatal(err)
}

for _, f := range frames {
    fmt.Printf("[%02d] %.3fs  %-12s  %s\n", f.Index, f.TimestampSec, f.Strategy, f.Path)
}

When WorkDir is left empty, extracted PNGs are written to a temporary directory that is removed before Extract returns — copy the files in the returned []Frame if you need them to persist.

CLI usage

extract -i input.mp4 [options]

Options:
  -version           Print version and exit
  -i string          Input video file (required)
  -o string          Output directory for extracted PNGs (default: frames)
  -threshold float   Scene change detection threshold 0.0–1.0 (default: 0.4)
  -interval float    Temporal sampling interval in seconds (default: 2.0)
  -max-frames int    Maximum output frames; 0 = unlimited (default: 0)
  -hamming int       Hamming distance dedup threshold; 0 = disabled (default: 8)
  -algo string       Hash algorithm: phash or dhash (default: phash)
  -no-scene          Disable scene change detection
  -no-keyframe       Disable keyframe extraction
  -no-temporal       Disable temporal sampling
  -no-mpdecimate     Disable MPDecimate
  -ffmpeg string     Path to ffmpeg binary (default: ffmpeg)
  -ffprobe string    Path to ffprobe binary (default: ffprobe)
  -json              Print frame metadata as JSON to stdout

Examples:

# Extract with defaults
extract -i input.mp4 -o ./frames

# Cap to 25 frames, print as JSON
extract -i input.mp4 -o ./frames -max-frames 25 -json

# Scene detection only, lower threshold
extract -i input.mp4 -o ./frames -no-keyframe -no-temporal -no-mpdecimate -threshold 0.25

Configuration reference

Field	Default	Description
Scene	true	Enable scene change detection
Keyframe	true	Enable keyframe (I-frame) extraction
Temporal	true	Enable uniform temporal sampling
MPDecimate	true	Enable MPDecimate
SceneThreshold	0.4	Scene change score threshold [0.0–1.0]
TemporalInterval	2.0	Seconds between temporally sampled frames
MPDecimateHi	768	MPDecimate hi threshold (64×12)
MPDecimateLo	320	MPDecimate lo threshold (64×5)
MPDecimateFrac	0.33	MPDecimate fraction
HashAlgo	phash	Perceptual hash algorithm: phash or dhash
HammingThreshold	8	Max Hamming distance to consider frames duplicates
HashResizeWidth	256	Width frames are rescaled to before hashing
MaxFrames	0	Output frame cap; 0 = unlimited
WorkDir	""	Output directory; empty = temp dir (auto-cleaned)
Concurrency	auto	Number of concurrent strategy goroutines
FFmpegPath	"ffmpeg"	Path to ffmpeg binary
FFprobePath	"ffprobe"	Path to ffprobe binary

Frame fields

type Frame struct {
    Index        int       // 0-based position in the final ordered set
    TimestampSec float64   // presentation timestamp in seconds
    Strategy     Strategy  // "scene", "keyframe", "temporal", or "mpdecimate"
    Path         string    // absolute path to the extracted PNG
    Hash         uint64    // perceptual hash value
    Video        *VideoInfo
}

type VideoInfo struct {
    Duration  float64 // total duration in seconds
    Width     int
    Height    int
    Codec     string  // e.g. "h264"
    FrameRate float64
    BitRate   int64
}

Error handling

var (
    ErrNoBinaries   // ffmpeg or ffprobe not found in PATH
    ErrNoStrategies // all extraction strategies disabled
    ErrNoFrames     // pipeline produced zero frames
)

FFmpegError wraps non-zero ffmpeg/ffprobe exits and includes the captured stderr for debugging.

License

MIT — see LICENSE.

Documentation

Overview

Package videosift extracts the minimum set of visually unique frames from a video file, suitable for processing with image-based machine learning models such as content moderation classifiers.

The package runs up to four FFmpeg-based extraction strategies in parallel (scene change detection, keyframe extraction, uniform temporal sampling, and MPDecimate), then deduplicates the full candidate set using perceptual hashing and Hamming distance comparison.

FFmpeg and FFprobe must be installed and available on PATH (or specified via Config.FFmpegPath / Config.FFprobePath). No CGo is required.

Basic usage:

frames, err := videosift.Extract(ctx, "input.mp4", videosift.DefaultConfig())
for _, f := range frames {
    fmt.Printf("[%s] %.3fs → %s\n", f.Strategy, f.TimestampSec, f.Path)
}

Index

• Variables
• type Config
  • func DefaultConfig() Config
• type FFmpegError
  • func (e *FFmpegError) Error() string
  • func (e *FFmpegError) Unwrap() error
• type Frame
  • func Extract(ctx context.Context, videoPath string, cfg Config) ([]Frame, error)
• type HashAlgo
• type Strategy
• type VideoInfo

Variables

var (
	// ErrNoBinaries is returned when ffmpeg or ffprobe cannot be found.
	ErrNoBinaries = errors.New("videosift: ffmpeg/ffprobe not found in PATH")

	// ErrNoStrategies is returned when all extraction strategies are disabled.
	ErrNoStrategies = errors.New("videosift: all extraction strategies disabled")

	// ErrNoFrames is returned when the pipeline produces zero frames.
	ErrNoFrames = errors.New("videosift: no frames extracted")
)

Types

type Config

type Config struct {
	// Strategy toggles — all enabled by default.
	Scene      bool
	Keyframe   bool
	Temporal   bool
	MPDecimate bool

	// SceneThreshold is the scene-change score [0.0, 1.0] that triggers
	// frame selection. Lower values are more sensitive. Default: 0.4.
	SceneThreshold float64

	// TemporalInterval is the number of seconds between uniformly sampled
	// frames (i.e. fps=1/TemporalInterval). Default: 2.0.
	TemporalInterval float64

	// MPDecimate filter parameters. Defaults map to FFmpeg's built-in defaults.
	MPDecimateHi   int     // default 768  (64*12)
	MPDecimateLo   int     // default 320  (64*5)
	MPDecimateFrac float64 // default 0.33

	// HashAlgo selects pHash or dHash for Hamming-distance deduplication.
	// Default: HashPHash.
	HashAlgo HashAlgo

	// HammingThreshold is the maximum Hamming distance at which two frames
	// are considered duplicates; the later frame is dropped. Default: 8.
	// Set to 0 to disable hash-based deduplication.
	HammingThreshold int

	// HashResizeWidth is the pixel width frames are scaled to before hashing.
	// Smaller values speed up hashing with minimal accuracy loss. Default: 256.
	// Set to 0 to disable rescaling (not recommended for large videos).
	HashResizeWidth int

	// MaxFrames caps the size of the final frame set. When the deduplicated
	// count exceeds MaxFrames, frames are selected by uniform stride to
	// preserve temporal spread. 0 means no cap.
	MaxFrames int

	// WorkDir is the directory where extracted PNGs are written.
	// If empty, Extract creates a temporary directory and removes it on return.
	// If set, the caller owns the directory and is responsible for cleanup.
	WorkDir string

	// FFmpegPath and FFprobePath override binary discovery via PATH.
	FFmpegPath  string
	FFprobePath string

	// Concurrency limits how many strategy goroutines run simultaneously.
	// Defaults to the number of enabled strategies (all run in parallel).
	Concurrency int
}

Config controls every tunable parameter of the extraction pipeline. Use DefaultConfig() to obtain sensible defaults, then override as needed.

Note the zero-value asymmetry: applyDefaults fills in every zero-valued field except HammingThreshold and HashResizeWidth, whose zero value means "disabled" rather than "use default". A from-scratch Config{} therefore runs with hash dedup and resize off; use DefaultConfig() to get 8/256.

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns a Config with all strategies enabled and sensible defaults suitable for most content-moderation workloads.

type FFmpegError

type FFmpegError struct {
	Args   []string
	Stderr string
	Cause  error
}

FFmpegError wraps a non-zero ffmpeg or ffprobe exit with its captured stderr.

func (*FFmpegError) Error

func (e *FFmpegError) Error() string

func (*FFmpegError) Unwrap

func (e *FFmpegError) Unwrap() error

type Frame

type Frame struct {
	// Index is the 0-based position in the final ordered set.
	Index int

	// TimestampSec is the presentation timestamp of this frame in seconds.
	TimestampSec float64

	// Strategy is the extraction method that first produced this frame.
	Strategy Strategy

	// Path is the absolute path to the extracted PNG file.
	Path string

	// Hash is the perceptual hash value, available for downstream use.
	Hash uint64

	// Video holds source video metadata. The same pointer is shared across
	// all frames in a result set.
	Video *VideoInfo
}

Frame is one extracted, deduplicated frame in the final result set.

func Extract

func Extract(ctx context.Context, videoPath string, cfg Config) ([]Frame, error)

Extract runs the full extraction pipeline for videoPath and returns the deduplicated, ordered set of frames.

When cfg.WorkDir is empty, Extract creates a temporary directory for intermediate files and removes it before returning, leaving only the final []Frame paths valid for the lifetime of the returned slice — callers must copy the files if they need them after Extract returns. When WorkDir is set by the caller, the extracted PNGs persist and the caller is responsible for cleanup.

type HashAlgo

type HashAlgo string

HashAlgo selects the perceptual hash function used for deduplication.

const (
	HashPHash HashAlgo = "phash"
	HashDHash HashAlgo = "dhash"
)

type Strategy

type Strategy string

Strategy identifies which FFmpeg extraction method surfaced a frame.

const (
	StrategyScene      Strategy = "scene"
	StrategyKeyframe   Strategy = "keyframe"
	StrategyTemporal   Strategy = "temporal"
	StrategyMPDecimate Strategy = "mpdecimate"
)

type VideoInfo

type VideoInfo struct {
	Duration  float64 // total duration in seconds
	Width     int
	Height    int
	Codec     string  // e.g. "h264"
	FrameRate float64 // average frames per second
	BitRate   int64   // bits per second
}

VideoInfo holds source video metadata populated by ffprobe.