overlay

Documentation

Overview

Package overlay turns analysis results into a debug overlay burnt into a copy of the video: an ASS subtitle script (Write) that libass renders through ffmpeg's subtitles filter, in the same encode that writes the copy (Renderer, through its Burner port, encode.FFmpeg in practice).

ASS rather than frames drawn in Go: libass draws styled text and vector shapes during the encode, at frame-accurate times, so the overlay costs about as much as a plain encode, and the script stays small because an event only starts when what it shows changes.

The overlay shows, for each frame: its timecode, number, size and keyframe flag, the bitrate over the last second, the shot and the camera work (with a vector of the camera move), SI/TI, luma levels, light levels of HDR videos, black, frozen, banded or out-of-range frames, the VMAF and other metrics of a comparison on scored frames, and a timeline of the title (VMAF or bitrate, shot cuts, a playhead).

Index

• Constants
• Variables
• func Write(w io.Writer, in Input, opts Options) error
• type Burner
• type Input
• type Item
  • func Items() []Item
  • func ParseItems(names []string) ([]Item, error)
• type Options
• type Progress
• type RenderOptions
• type Renderer
  • func NewRenderer(burner Burner) *Renderer
  • func (r *Renderer) Render(ctx context.Context, source, output string, in Input, opts RenderOptions) (err error)

Examples

• Renderer.Render
• Write

Constants

const DefaultFont = "DejaVu Sans Mono"

DefaultFont is the overlay's font family: DejaVu Sans Mono, the monospaced font of most Linux distributions (fonts-dejavu-mono on Debian, installed in qc's Docker images). fontconfig substitutes another font when it is missing.

Variables

var ErrNoFrames = errors.New("overlay: the report has no frame timeline")

ErrNoFrames is returned by Write when the report has no frame timeline (no bitstream analysis, or no packet).

var ErrUnknownItem = errors.New("unknown overlay item")

ErrUnknownItem is returned by ParseItems for an unknown item name.

Functions

func Write

func Write(
	w io.Writer,
	in Input,
	opts Options,
) error

Write writes the ASS script of the overlay of in to w. It fails with ErrNoFrames without a frame timeline, and with the error of w.

Example

ExampleWrite writes the ASS script of an overlay, here of a two-frame inspection, for a player (mpv --sub-file) or another encoder.

package main

import (
	"bytes"
	"fmt"
	"log"
	"strings"

	"github.com/eko/qc/analysis"
	"github.com/eko/qc/bitstream"
	"github.com/eko/qc/media"
	"github.com/eko/qc/overlay"
)

func main() {
	report := &analysis.Report{
		Info: &media.Info{Path: "clip.mp4", Video: []media.VideoStream{{Width: 1280, Height: 720}}},
		Bitstream: &bitstream.Report{
			Duration:   media.Seconds(0.08),
			PTS:        []media.Duration{0, media.Seconds(0.04)},
			FrameSizes: []int{12_000, 800},
			KeyFlags:   []bool{true, false},
		},
	}

	var script bytes.Buffer
	if err := overlay.Write(&script, overlay.Input{Report: report}, overlay.Options{Items: []overlay.Item{overlay.ItemTime}}); err != nil {
		log.Fatal(err)
	}

	for line := range strings.Lines(script.String()) {
		if strings.HasPrefix(line, "PlayRes") || strings.Contains(line, "#1") {
			fmt.Print(line)
		}
	}

}

Output:
PlayResX: 1920
PlayResY: 1080
Dialogue: 1,0:00:00.02,0:00:00.08,qc,,0,0,0,,{\an7\pos(42,38)}{\fs36\b1}00:00:00.040{\fs26\b0}{\c&HA8B0B8&}  #1

Types

type Burner

type Burner interface {
	Burn(
		ctx context.Context,
		spec encode.BurnSpec,
	) error
}

Burner burns an ASS script into a copy of a video (*encode.FFmpeg): the port the Renderer encodes through.

type Input

type Input struct {
	// Report is the analysis of the video the overlay is burnt into: its
	// inspection at least (the bitstream gives the frame timeline), with
	// its frame analysis for every item but time, bitrate and timeline.
	Report *analysis.Report
	// Quality, when set, is the comparison of that video against its
	// reference: its per-frame scores (every frame with an exact
	// measurement, the sampled clips otherwise).
	Quality *quality.Result
}

Input is what the overlay shows.

type Item

type Item string

Item is a part of the overlay.

const (
	// ItemTime is the timecode, frame number, frame size and keyframe flag.
	ItemTime Item = "time"
	// ItemBitrate is the bitrate over the last second.
	ItemBitrate Item = "bitrate"
	// ItemShots is the shot number, and a marker at cuts.
	ItemShots Item = "shots"
	// ItemMotion is the camera work of the shot and the camera move of
	// the frame.
	ItemMotion Item = "motion"
	// ItemSITI is the spatial and temporal information of the frame.
	ItemSITI Item = "siti"
	// ItemLevels is the luma range and average of the frame.
	ItemLevels Item = "levels"
	// ItemHDR is the peak and average light level of an HDR frame.
	ItemHDR Item = "hdr"
	// ItemLoudness is the short-term and momentary loudness of the audio
	// (the default track) when the frame is shown.
	ItemLoudness Item = "loudness"
	// ItemFlags are badges for black, frozen, banded and out-of-range
	// frames, and letterboxing.
	ItemFlags Item = "flags"
	// ItemQuality is the VMAF and the other metrics of scored frames.
	ItemQuality Item = "quality"
	// ItemTimeline is a chart of the title with a playhead.
	ItemTimeline Item = "timeline"
)

Items of the overlay. An item without data (motion without the motion analysis, hdr on SDR, quality without a comparison...) is left out.

func Items

func Items() []Item

Items returns every item, in display order.

func ParseItems

func ParseItems(
	names []string,
) ([]Item, error)

ParseItems reads item names ("vmaf" is ItemQuality). Blank names are skipped; no name at all returns nil, which Options reads as every item.

type Options

type Options struct {
	// Items are the parts shown; nil shows every item.
	Items []Item
	// Font is the font family of the overlay (default DefaultFont). A
	// monospaced font keeps the values from shifting as digits change.
	Font string
}

Options configures the overlay. The zero value shows every item.

type Progress

type Progress struct {
	Done  int
	Total int
}

Progress reports the frames of the annotated copy written so far, out of the title's frame count.

type RenderOptions

type RenderOptions struct {
	Options
	// Height scales the annotated copy to that height (0 keeps the
	// source's): 720 renders about twice as fast as 1080.
	Height int
	// Encoder is the H.264 encoder of the copy (encode.BurnAuto:
	// VideoToolbox on macOS, x264 elsewhere).
	Encoder encode.BurnEncoder
	// Workers is how many segments of the video render at once (0: the
	// encoder's default, 1: a single pass). The segments start at
	// keyframes and are joined without re-encoding: the copy has the same
	// frames and timestamps either way.
	Workers int
	// CRF and Preset are the x264 settings (default 20, fast).
	CRF    float64
	Preset string
	// FontsDir, when set, is searched for the overlay's font first.
	FontsDir string
	// Progress, when set, is called as frames are written.
	Progress func(Progress)
}

RenderOptions configures a rendering. The zero value renders every item at the source's size.

type Renderer

type Renderer struct {
	// contains filtered or unexported fields
}

Renderer writes annotated copies of videos.

func NewRenderer

func NewRenderer(
	burner Burner,
) *Renderer

NewRenderer returns a Renderer burning through burner.

func (*Renderer) Render

func (r *Renderer) Render(
	ctx context.Context,
	source, output string,
	in Input,
	opts RenderOptions,
) (err error)

Render writes to output a copy of source, the video in.Report analyses, with the overlay of in burnt into its frames. The ASS script lives in a temporary directory for the time of the encode.

Example

ExampleRenderer_Render writes a copy of an encode with its analysis and the VMAF of every frame burnt in: the exact mode scores every frame, a sampled measurement only some.

package main

import (
	"context"
	"log"

	"github.com/eko/qc/analysis"
	"github.com/eko/qc/bitstream"
	"github.com/eko/qc/decode"
	"github.com/eko/qc/encode"
	"github.com/eko/qc/overlay"
	"github.com/eko/qc/probe"
	"github.com/eko/qc/quality"
	"github.com/eko/qc/vmaf/libvmaf"
)

func main() {
	ctx := context.Background()
	dec := decode.NewFFmpeg("ffmpeg", 0)
	analyzer := analysis.New(nil,
		probe.NewFFprobe("ffprobe"),
		bitstream.NewFFprobeReader("ffprobe"),
		dec,
		quality.NewMeter(dec, libvmaf.NewEngine()),
	)

	report, err := analyzer.Analyze(ctx, "encode.mp4", analysis.Options{})
	if err != nil {
		log.Fatal(err)
	}

	cmp, err := analyzer.Compare(ctx, "reference.mov", "encode.mp4", analysis.CompareOptions{
		Quality:   quality.Options{Exact: true},
		Distorted: report,
	})
	if err != nil {
		log.Fatal(err)
	}

	renderer := overlay.NewRenderer(encode.NewFFmpeg("ffmpeg"))

	err = renderer.Render(ctx, "encode.mp4", "annotated.mp4",
		overlay.Input{Report: report, Quality: cmp.VMAF},
		overlay.RenderOptions{
			Options: overlay.Options{Items: []overlay.Item{overlay.ItemTime, overlay.ItemQuality, overlay.ItemTimeline}},
			Height:  720, // a 720p copy encodes faster; libass scales the overlay
		},
	)
	if err != nil {
		log.Fatal(err)
	}
}