hwmediacodec

package module
v0.5.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 21, 2026 License: Apache-2.0 Imports: 18 Imported by: 3

README

hwmediacodec

English | 日本語

A fireworks show recorded from an Ebitengine game, and the bundled waterfall clip decoded in hardware

Hardware video decoding and encoding (H.264, HEVC, AV1) from Go, without cgo, plus the operating system's audio codecs (AAC, Opus) through the same API. One module for macOS, iOS, Android, Linux, Windows and the browser: the library loads the platform's codec engine at run time (VideoToolbox, MediaCodec, VA-API, Intel VPL, NVDEC/NVENC, Media Foundation, WebCodecs), so CGO_ENABLED=0 go build works and everything cross-compiles from one machine.

The picture above was made with the library itself: the fireworks show was recorded from an Ebitengine game with capture, and both frames were taken out with examples/thumbnails on an AMD Radeon 890M.

What it is for

  • 🎮 Videos in games: play an MP4 or WebM with sound inside an Ebitengine game, decoded by the GPU, with seeking and looping, on the desktop, on phones and in the browser. The frame is an ordinary *ebiten.Image, so a video also works as a texture: on a 3D object, on an in-game screen, or through a shader.
  • 📡 Streaming a game to the browser: encode what the game draws and send it to browsers over WebRTC (about 100 ms of latency) or HLS, or record it to MP4.
  • 🛠️ Format conversion in CLI tools: convert H.264 ↔ HEVC (↔ AV1), extract thumbnails, convert HEIC/AVIF ↔ PNG/JPEG, with the hardware codecs and without shipping ffmpeg.

Status

✅ verified on real hardware · ⚠️ implemented, not verified yet · (blank) not available

Video

Platform Backend Verified on H.264 dec H.264 enc HEVC dec HEVC enc AV1 dec AV1 enc
macOS (Apple Silicon) VideoToolbox M3 ✅ ✅ ✅ ✅ ✅
iOS 15+ VideoToolbox iPhone SE 3 (#31) ✅ ✅ ✅ ✅ ⚠️
Android 14+ MediaCodec Redmi Pad SE ✅ ✅ ✅ ✅ ⚠️
Linux, AMD VA-API Radeon 890M (#38, #39) ✅ ✅ ✅ ✅ ✅ ✅
Linux, Intel Intel VPL / VA-API ⚠️ ⚠️ ⚠️ ⚠️ ⚠️ ⚠️
Linux, NVIDIA NVDEC / NVENC ⚠️ ⚠️ ⚠️ ⚠️ ⚠️
Windows, NVIDIA NVDEC / NVENC RTX 2060 Max-Q (#33, #35) ✅ ✅ ✅ ✅ ⚠️
Windows, Intel Media Foundation Intel iGPU ✅ ✅ ✅ ⚠️ ✅
Windows, AMD Media Foundation Ryzen 4000 iGPU (#35) ✅ ✅ ✅ ✅ ⚠️
Windows, ARM64 Media Foundation ⚠️ ⚠️ ⚠️ ⚠️ ⚠️
Browser: Chrome, Safari WebCodecs macOS (M3) ✅ ✅ ✅ ✅ ✅
Browser: Firefox WebCodecs macOS (M3) ✅ ✅ ⚠️
Browser: other OSes WebCodecs ⚠️ ⚠️ ⚠️ ⚠️ ⚠️

AV1 decoding needs a GPU with an AV1 decoder (Apple M3, Radeon RX 6000, Intel Tiger Lake, GeForce RTX 30 and newer). AV1 encoding is wired up on VA-API only; Apple Silicon has no AV1 encoder.

Audio (the OS's software codecs)

Platform Backend AAC dec AAC enc Opus dec Opus enc
macOS, iOS AudioToolbox ✅ ✅ ✅ ✅
Android MediaCodec ✅ ✅ ✅ ✅
Linux libavcodec, libfdk-aac, libopus (when installed) ✅ ✅ ✅ ✅
Windows Media Foundation ✅ ✅ ⚠️
Browser (macOS) WebCodecs ✅ ✅ (not Firefox) ✅ ✅

On Linux, AAC needs libavcodec or libfdk-aac; on Ubuntu, sudo apt install ubuntu-restricted-extras installs both. The details (devices, drivers, measurements, known quirks of each backend) are in the per-platform reports linked from the tables, and the check reports people sent are the issues titled "Check report".

Check your machine

One command checks what the library can do on your machine and helps fill the ⚠️ cells:

go run github.com/shibukawa/hwmediacodec/cmd/hwmediacodec@latest check

It decodes a bundled clip in H.264, HEVC and AV1, encodes and decodes back in every codec the machine has, plays the clip with sound, streams H.264 and HEVC to your browser over WebRTC, and ends with a report and a link that opens a new issue with it filled in. Please send it! -auto runs the automatic checks only, without a window. See the command's README.

Use cases

🎮 Videos in games

The ebitenvideo module plays a video into an *ebiten.Image, with its sound through Ebitengine's audio:

d, _ := mp4.Open("movie.mp4")                     // or webm.Open for AV1 + Opus
player, err := ebitenvideo.NewPlayerFromSource(d.Video().PacketSource(),
	ebitenvideo.WithAudio(d.Audio().PacketSource()), ebitenvideo.WithLoop())
player.Play()

func (g *game) Update() error { return player.Update() }
func (g *game) Draw(screen *ebiten.Image) {
	if img := player.Image(); img != nil {
		screen.DrawImage(img, nil)
	}
}
  • Video textures: player.Image() is an ordinary *ebiten.Image updated in place, so it can be drawn anywhere an image can: scaled and rotated with DrawImage, mapped onto polygons with DrawTriangles (a TV in the game world, a spinning cube), or fed to a Kage shader with DrawRectShader as one of its source images. examples/texture shows all three.
  • Decoding runs on a background goroutine a few frames ahead, in display order, as RGBA; frames that come late are skipped. Seek, Pause, SetVolume, looping.
  • The sound is the clock: the picture follows the audio device's position. Without sound (a browser before the first click, a platform without an AAC decoder) the film still plays on the game's clock.
  • Browser: the same code builds for GOOS=js GOARCH=wasm and decodes through WebCodecs. Firefox has no HEVC there, so ship H.264 as well.
  • Mobile: ebitenmobile bind builds it for iOS and Android (CGO_ENABLED=1 there, which purego needs; the library itself has no cgo code).
  • Another engine or your own renderer: the playback package is the same player without Ebitengine; it hands out RGBA (or NV12) frames for a clock you drive.
  • Try it: go run github.com/shibukawa/hwmediacodec/examples/player@latest (bundled clip, or pass an MP4 / WebM), and go run github.com/shibukawa/hwmediacodec/examples/texture@latest (keys 1 flat, 2 box, 3 shader).
📡 Streaming a game to the browser

capture reads what the game draws each frame and encodes it on a background goroutine; the packets go to any sink, such as the WebRTC broadcaster of the net/webrtc module:

var rec *capture.Recorder
bc := webrtc.NewBroadcaster(60, nil, func() { rec.RequestKeyframe() })
rec, _ = capture.New(1280, 720, bc, capture.Options{
	FPS: 60, LowLatency: true, Profile: hwmediacodec.ProfileBaseline})
http.Handle("/offer", bc)        // the page POSTs its SDP offer here

func (g *game) Draw(screen *ebiten.Image) {
	g.scene.Draw(screen)
	rec.Capture(screen)          // never blocks on the encoder
}
  • WebRTC (net/webrtc, pion): about 100 ms end to end; signalling is one HTTP POST. H.264 everywhere; HEVC with NewCodecBroadcaster where the browser decodes it (Safari, Chrome with a hardware decoder on macOS and Windows; not Edge without the paid HEVC extension; Chrome on Linux not verified yet).
  • HLS (mp4.Segmenter + net/hls): a few seconds of latency, plays in Safari natively and elsewhere with hls.js.
  • Recording: mp4.CreateVideoFile as the sink writes an MP4; mp4.CreateAVFile adds an AAC or Opus track for the microphone.
  • capture does not import Ebitengine: anything with ReadPixels can be recorded. The endpoints are unauthenticated; serve them on a trusted interface or behind your own authentication.
  • Try it: go run github.com/shibukawa/hwmediacodec/examples/webrtc@latest (or hls, record) streams a fireworks show and opens the browser.
🛠️ Format conversion in CLI tools

A transcoder is a decoder and an encoder between a demuxer and a muxer:

d, _ := mp4.Open("input.mp4")
src := d.Video().PacketSource()
dec, _ := hwmediacodec.NewDecoder(ctx, src.Codec(), hwmediacodec.WithTimeScale(src.TimeScale()))
enc, _ := hwmediacodec.NewEncoder(ctx, hwmediacodec.HEVC, w, h,
	hwmediacodec.WithTimeScale(src.TimeScale()), hwmediacodec.WithBitrate(6_000_000))
out := hwmediacodec.NewEncodeWriter(enc, track) // track: a video track of mp4.Create("output.mp4")
_, err := hwmediacodec.CopyFrames(ctx, out, hwmediacodec.NewDecodeReader(dec, src))
err = out.Flush(ctx)
  • Containers: MP4/MOV (mediacontainer/mp4, also fMP4 segments), WebM with AV1 + Opus (mediacontainer/webm), IVF, ADTS (.aac), raw Annex-B. Images: HEIC and AVIF through the standard image package (image/heif, image/avif).

  • The library stops at frames and samples: no scaling, no resampling, no filters. A filter is a FrameReader or FrameWriter around the other.

  • errors.Is(err, hwmediacodec.ErrUnsupported) tells that the machine has no engine for the job; HasHardware asks in advance, and WithSoftwareFallback lets the OS's software codecs in where they exist.

  • Ready-made tools, each runnable with go run …@latest:

    go run github.com/shibukawa/hwmediacodec/examples/convert@latest -codec hevc -bitrate 6M input.mp4 output.mp4
    go run github.com/shibukawa/hwmediacodec/examples/thumbnails@latest -every 10s -o thumbs movie.mp4
    go run github.com/shibukawa/hwmediacodec/examples/imgconv@latest photo.heic photo.png
    go run github.com/shibukawa/hwmediacodec/cmd/hwmediacodec@latest probe
    

    convert keeps the timestamps and copies the audio; cmd/hwmediacodec also has decode, encode, transcode and the audio counterparts for raw streams (see the guide).

Packages

Import path What it is
github.com/shibukawa/hwmediacodec The codec API: Probe, NewDecoder, NewEncoder, NewAudioDecoder, NewAudioEncoder, packets and frames
.../encoding/annexb H.264/HEVC Annex-B byte streams: split into NAL units and access units, classify NAL units
.../encoding/adts AAC in ADTS framing (.aac files): reader and writer around the codec's packets
.../mediacontainer/mp4 MP4/MOV demuxer and muxer around the codec's packets (video, AAC and Opus audio), fMP4 segmenter
.../mediacontainer/webm WebM demuxer and muxer for AV1 video and Opus audio
.../mediacontainer/ivf IVF reader and writer (AV1)
.../image/heif HEIC still images (HEVC), registered with the standard image package
.../image/avif AVIF still images (AV1), registered with the standard image package
.../net/hls Live HLS playlist and HTTP handler over the fMP4 segments
.../playback Plays a stream against a clock: background decoding, pacing, seeking, looping; frames to show
.../capture Records what a renderer draws (an Ebitengine screen, for example) through the encoder into a sink
.../net/webrtc Separate module: one H.264 or HEVC stream broadcast to browsers over WebRTC (pion)
.../ebitenvideo Separate module: playback with the frames in an *ebiten.Image
.../cmd/hwmediacodec Separate module: the command-line tool and the machine check
.../examples Separate module: complete programs (converter, thumbnails, recorder, HLS and WebRTC servers, player, mobile app); see examples/README.md

The core module depends on purego, golang.org/x/sys and, for the MP4 and image packages, mp4ff; all are pure Go. Ebitengine and pion are only pulled in by the separate modules. Go 1.27 or later.

Documentation

License

Copyright 2026 Yoshiki Shibukawa

Licensed under the Apache License, Version 2.0.

Documentation

Overview

Package hwmediacodec drives the operating system's hardware video decode and encode engines, and its audio codecs, from Go without cgo.

Native libraries are loaded at run time (dlopen on macOS, iOS, Linux and Android, LoadLibrary on Windows), so CGO_ENABLED=0 builds and cross-compilation work on the desktop platforms. On iOS and Android purego needs the cgo runtime, so mobile builds (as made by ebitenmobile) set CGO_ENABLED=1 with the platform's clang; the library itself still contains no cgo code. A platform without a usable engine reports an empty Probe result and ErrUnsupported from NewDecoder and NewEncoder; it never fails at link time.

In a browser (GOOS=js GOARCH=wasm, the target Ebitengine builds for the web) the engine is the WebCodecs API, with the platform's codecs behind it. Its results arrive on the page's event loop, which a WebAssembly program shares: Send blocks while the browser's codec is saturated and Flush until everything is delivered, and that is when the frames and packets Receive returns come in. A loop that only polls Receive, without ever blocking, starves the browser; the loops below do not.

Status

H.264 and HEVC decoding and encoding are implemented on macOS (Apple Silicon) and iOS through VideoToolbox, on Android through the NDK MediaCodec API (the vendor's hardware codecs), on Linux and Windows (amd64) with the NVIDIA driver through NVDEC and NVENC, on Linux with an Intel GPU through Intel VPL (a VPL or Media SDK runtime), on Linux through VA-API (AMD Mesa and Intel drivers), and on Windows (amd64 and arm64) through Media Foundation (decoding with the Microsoft decoder transforms accelerated by Direct3D 11, encoding with the vendor's hardware encoder transforms), and in browsers through WebCodecs (the codecs the browser reports as hardware ones; RGBA and BGRA output converted by the browser, and the Native format for frames that stay on the GPU). Where the NVIDIA driver is installed its backend is tried first. Raw frames are NV12, RGBA or BGRA in CPU memory. VideoToolbox converts to and from the RGB formats itself and NVENC takes them as input; everywhere else the library converts in Go, so both formats work for decoding and encoding on every backend. Decoded frames are returned in display order (see NewDecoder and WithDecodeOrder; the Media Foundation and Intel VPL decoders and the browsers' reorder natively, so WithDecodeOrder has no effect there). AV1 decoding is available where the GPU has an AV1 decoder: on macOS with an Apple M3 or newer and on iOS from the A17 Pro on (8-bit and 10-bit Main profile), and on Linux and Windows through every backend (8-bit Main profile; NVIDIA RTX 30 series, Intel Tiger Lake, AMD RX 6000 series and newer; Media Foundation needs the "AV1 Video Extension" package), on Android where the SoC has an AV1 decoder, and in a browser that reports a hardware AV1 decoder. A Packet is then one temporal unit (an IVF frame or ISOBMFF sample; see package mediacontainer/ivf), each yielding one frame in presentation order. AV1 encoding is available through VA-API on Linux (8-bit Main profile, GPUs with an AV1 encoder such as the AMD RX 7000 series and Radeon 700M/800M); the other backends do not encode AV1 yet. The separate module github.com/shibukawa/hwmediacodec/ebitenvideo plays streams in Ebitengine.

AAC and Opus audio is decoded and encoded through the codecs of the platform (see NewAudioDecoder and NewAudioEncoder): AudioToolbox on macOS and iOS, MediaCodec on Android, Media Foundation on Windows (which has no Opus encoder), WebCodecs in the browser as far as the browser has the codecs, and on Linux, which has no codecs of its own, the libraries of the distribution where they are installed (libopus, libavcodec for AAC decoding, libfdk-aac for AAC encoding). They run in software; the library contains no codec and bundles none. HWMEDIACODEC_AUDIO_BACKENDS restricts and reorders the audio backends ("libopus", "libavcodec", "fdk-aac" on Linux) the way HWMEDIACODEC_BACKENDS does the video ones.

On Linux the VA-API backend opens the first usable DRM render node and the Intel VPL backend the first Intel one; set HWMEDIACODEC_VAAPI_DEVICE or HWMEDIACODEC_VPL_DEVICE to a /dev/dri/renderD* path to choose a GPU. The NVIDIA backend uses CUDA device 0; set HWMEDIACODEC_NVIDIA_DEVICE to another device ordinal on machines with several NVIDIA GPUs.

When several backends can serve a request the first registered one wins (on Linux: nvidia, vpl, vaapi; on Windows: nvidia, mediafoundation; one backend each on macOS, iOS, Android and in the browser); set HWMEDIACODEC_BACKENDS to a comma-separated list of backend names, for example "vaapi" or "vaapi,vpl", to restrict and reorder them.

Decoding

dec, err := hwmediacodec.NewDecoder(ctx, hwmediacodec.H264)
...
r := annexb.NewReader(file, hwmediacodec.H264)
for {
	au, err := r.Next()
	if err == io.EOF { break }
	if err := dec.Send(ctx, hwmediacodec.Packet{Data: au}); err != nil { ... }
	for {
		f, err := dec.Receive(ctx)
		if errors.Is(err, hwmediacodec.ErrAgain) { break }
		... use f ...
		f.Release()
	}
}
dec.Flush(ctx)
// Receive until io.EOF, then Close.

Encoding

enc, err := hwmediacodec.NewEncoder(ctx, hwmediacodec.H264, 1920, 1080,
	hwmediacodec.WithFrameRate(30), hwmediacodec.WithBitrate(4_000_000))
...
for i, f := range frames { // *hwmediacodec.Frame in NV12
	f.PTS = int64(i) * int64(hwmediacodec.DefaultTimeScale) / 30
	if err := enc.Send(ctx, f); err != nil { ... }
	for {
		p, err := enc.Receive(ctx)
		if errors.Is(err, hwmediacodec.ErrAgain) { break }
		out.Write(p.Data) // Annex-B access unit
	}
}
enc.Flush(ctx)
// Receive until io.EOF, then Close.

Audio

dec, err := hwmediacodec.NewAudioDecoder(ctx, track.Format()) // *mp4.AudioTrack
...
if err := dec.Send(ctx, packet); err != nil { ... }
f, err := dec.Receive(ctx) // interleaved samples in f.Data, see f.Int16s

enc, err := hwmediacodec.NewAudioEncoder(ctx, hwmediacodec.AAC, 48000, 2,
	hwmediacodec.WithBitrate(128_000))
...
if err := enc.Send(ctx, hwmediacodec.S16Frame(samples, 48000, 2, pts)); err != nil { ... }
p, err := enc.Receive(ctx) // one raw AAC access unit; enc.Format() describes the stream

Index

Constants

View Source
const (
	AAC  = codec.AAC
	Opus = codec.Opus

	S16 = codec.S16
	F32 = codec.F32
)
View Source
const (
	H264 = codec.H264
	HEVC = codec.HEVC
	AV1  = codec.AV1

	Decode = codec.Decode
	Encode = codec.Encode

	NV12   = codec.NV12
	RGBA   = codec.RGBA
	BGRA   = codec.BGRA
	Native = codec.Native

	VBR = codec.VBR
	CBR = codec.CBR

	ProfileDefault  = codec.ProfileDefault
	ProfileBaseline = codec.ProfileBaseline
	ProfileMain     = codec.ProfileMain
	ProfileHigh     = codec.ProfileHigh
)
View Source
const DefaultTimeScale int32 = 90000

DefaultTimeScale is the PTS unit used when WithTimeScale is not given.

Variables

View Source
var (
	ErrUnsupported = codec.ErrUnsupported
	ErrAgain       = codec.ErrAgain
	ErrClosed      = codec.ErrClosed
	ErrInvalidData = codec.ErrInvalidData
)

Functions

func CopyFrames

func CopyFrames(ctx context.Context, dst FrameWriter, src FrameReader) (int64, error)

CopyFrames writes every frame of src to dst and releases it, until src reports io.EOF, and returns the number of frames copied. With a DecodeReader and an EncodeWriter this is a transcoder; Flush the EncodeWriter afterwards.

func HasHardware

func HasHardware(ctx context.Context, c Codec, dir Direction) bool

HasHardware reports whether Probe lists a hardware engine for codec c in the given direction on this machine, that is, whether NewDecoder or NewEncoder can succeed without WithSoftwareFallback. Use it to choose another path (a software codec, another format) before trying. It is false for the audio codecs, which never run in hardware.

Types

type AudioDecoder

type AudioDecoder = codec.AudioDecoder

Re-exported audio types. See the documentation on each in package codec.

func NewAudioDecoder

func NewAudioDecoder(ctx context.Context, format AudioFormat, opts ...AudioDecoderOption) (AudioDecoder, error)

NewAudioDecoder opens a decoder for the stream format describes on the first backend that supports it. A container track reports its format (mp4.AudioTrack.Format, adts.Reader.Format) and an AudioEncoder reports the one of its output; for a stream from elsewhere fill in the codec, the sample rate and the channel count. The error matches ErrUnsupported (and is an *UnsupportedError) when no backend can serve the request.

The codecs are the operating system's own, which run in software on every platform: WithSoftwareFallback is not needed, and nothing is bundled with the library.

Every packet yields one frame of interleaved samples (S16 unless WithSampleFormat says otherwise) that carries the packet's PTS. The decoder returns all the samples a stream codes: the AudioFormat.Delay samples at its start, which lie before PTS 0 in a stream that starts there, and the padding of its last packet are for the caller to drop.

type AudioDecoderOption

type AudioDecoderOption = codec.AudioDecoderOption

Re-exported audio types. See the documentation on each in package codec.

type AudioEncoder

type AudioEncoder = codec.AudioEncoder

Re-exported audio types. See the documentation on each in package codec.

func NewAudioEncoder

func NewAudioEncoder(ctx context.Context, c Codec, sampleRate, channels int, opts ...AudioEncoderOption) (AudioEncoder, error)

NewAudioEncoder opens an encoder for c that takes interleaved samples at the given rate and channel count (S16 unless WithSampleFormat says otherwise) on the first backend that supports it. The error matches ErrUnsupported (and is an *UnsupportedError) when no backend can serve the request, including a sample rate or a bitrate the codec does not offer.

Without options the encoder uses the backend's default bitrate. PTS and Packet.Duration count in samples (the time scale is the sample rate) unless WithTimeScale sets another unit. The first packets of a stream have a PTS before that of the first frame: an encoder starts AudioFormat.Delay samples early (see AudioEncoder.Format).

type AudioEncoderOption

type AudioEncoderOption = codec.AudioEncoderOption

Re-exported audio types. See the documentation on each in package codec.

type AudioFormat

type AudioFormat = codec.AudioFormat

Re-exported audio types. See the documentation on each in package codec.

type AudioFrame

type AudioFrame = codec.AudioFrame

Re-exported audio types. See the documentation on each in package codec.

func F32Frame

func F32Frame(samples []float32, sampleRate, channels int, pts int64) *AudioFrame

F32Frame wraps interleaved floating point samples as an encoder input frame without copying, for an encoder opened with WithSampleFormat(F32). The opposite direction is AudioFrame.Float32s.

func S16Frame

func S16Frame(samples []int16, sampleRate, channels int, pts int64) *AudioFrame

S16Frame wraps interleaved 16-bit samples as an encoder input frame without copying: the frame shares the slice, which must stay untouched until AudioEncoder.Send has returned. The opposite direction is AudioFrame.Int16s.

type AudioOption

type AudioOption = codec.AudioOption

Re-exported audio types. See the documentation on each in package codec.

func WithSampleFormat

func WithSampleFormat(f SampleFormat) AudioOption

WithSampleFormat sets the format of the samples a decoder returns and an encoder is given: S16 (the default) or F32.

type BackendError

type BackendError = codec.BackendError

Re-exported core types. See the documentation on each in package codec.

type Capability

type Capability = codec.Capability

Re-exported core types. See the documentation on each in package codec.

func Probe

func Probe(ctx context.Context) ([]Capability, error)

Probe reports every codec/direction pair the registered backends can serve on this machine: the hardware video engines first, then the audio codecs of the operating system (Capability.Hardware is false for those; test Codec.IsAudio to tell the two apart). It returns an empty (non-nil) slice, not an error, when nothing is present.

type Codec

type Codec = codec.Codec

Re-exported core types. See the documentation on each in package codec.

type DecodeReader

type DecodeReader struct {
	// contains filtered or unexported fields
}

DecodeReader couples a PacketReader to a Decoder and hands out the decoded frames one at a time: the Send/Receive loop of the Decoder documentation, written once. It does not own the decoder; Close that when done.

Open the decoder with WithTimeScale(src.TimeScale()) so that Frame.PTS is in the reader's units.

func NewDecodeReader

func NewDecodeReader(dec Decoder, src PacketReader) *DecodeReader

NewDecodeReader returns a reader of the frames dec decodes from src.

func (*DecodeReader) ReadFrame

func (r *DecodeReader) ReadFrame(ctx context.Context) (*Frame, error)

ReadFrame implements FrameReader. Frames come in the decoder's output order (display order unless it was opened with WithDecodeOrder).

func (*DecodeReader) Reset

func (r *DecodeReader) Reset(ctx context.Context) error

Reset drops what the decoder holds and makes the reader start afresh from the PacketReader's current position, which must be a keyframe: call it after PacketSeeker.SeekKeyframe.

type Decoder

type Decoder = codec.Decoder

Re-exported core types. See the documentation on each in package codec.

func NewDecoder

func NewDecoder(ctx context.Context, c Codec, opts ...DecoderOption) (Decoder, error)

NewDecoder opens a hardware decoder for c on the first backend that supports it. The error matches ErrUnsupported (and is an *UnsupportedError) when no backend can serve the request.

Frames come back in display order: pictures are reordered by their picture order count, derived from the slice headers in Go, and held back no longer than the stream's reorder bound (num_reorder_frames in the SPS or, for H.264 streams without that VUI field, the DPB size of the level). WithDecodeOrder turns the reordering off.

type DecoderOption

type DecoderOption = codec.DecoderOption

Re-exported core types. See the documentation on each in package codec.

func WithDecodeOrder

func WithDecodeOrder() DecoderOption

WithDecodeOrder makes Receive return frames in decode order, as the backend produces them, instead of display order. It avoids the latency of the reorder buffer for streams without B-frames whose headers do not say so, and is what a transcoder that preserves timestamps wants.

func WithOutputFormat

func WithOutputFormat(f PixelFormat) DecoderOption

WithOutputFormat requests a pixel format for decoded frames: NV12 (the default), RGBA or BGRA, or Native for frames that stay with the decoder (see Native; browsers only, ErrUnsupported elsewhere). RGBA and BGRA frames have one plane with rows of exactly 4*Width bytes, ready for ebiten.Image.WritePixels or image.RGBA. The backend converts where it can; for a backend that returns NV12 only the frames are converted in Go, with the colour matrix and range the stream declares (for a stream that declares none, BT.709 for pictures larger than 704x576 and BT.601 otherwise).

type Direction

type Direction = codec.Direction

Re-exported core types. See the documentation on each in package codec.

type EncodeWriter

type EncodeWriter struct {
	// contains filtered or unexported fields
}

EncodeWriter couples an Encoder to a PacketWriter: every frame written is encoded and the packets that become ready go to the writer, the Send/Receive loop of the Encoder documentation written once. It owns neither side; Flush it at the end, then close the encoder and the writer.

func NewEncodeWriter

func NewEncodeWriter(enc Encoder, dst PacketWriter) *EncodeWriter

NewEncodeWriter returns a frame writer that encodes with enc into dst.

func (*EncodeWriter) Flush

func (w *EncodeWriter) Flush(ctx context.Context) error

Flush makes the encoder emit the frames it is holding and writes their packets. The encoder can be reused afterwards; the next frame is a keyframe.

func (*EncodeWriter) Packets

func (w *EncodeWriter) Packets() int64

Packets is the number of packets written so far.

func (*EncodeWriter) WriteFrame

func (w *EncodeWriter) WriteFrame(ctx context.Context, f *Frame) error

WriteFrame implements FrameWriter. The pixel data is copied before it returns (see Encoder.Send).

type Encoder

type Encoder = codec.Encoder

Re-exported core types. See the documentation on each in package codec.

func NewEncoder

func NewEncoder(ctx context.Context, c Codec, width, height int, opts ...EncoderOption) (Encoder, error)

NewEncoder opens a hardware encoder for c at the given picture size on the first backend that supports it. The error matches ErrUnsupported (and is an *UnsupportedError) when no backend can serve the request, including when a requested control (profile, rate control mode) is not available.

Without options the encoder uses the backend's default rate control, no B-frames and the backend's keyframe placement. Set WithFrameRate so that rate control knows the frame rate, and give every frame an increasing PTS.

type EncoderOption

type EncoderOption = codec.EncoderOption

Re-exported core types. See the documentation on each in package codec.

func WithBFrames

func WithBFrames() EncoderOption

WithBFrames allows the encoder to use B-frames. Packets then arrive in decode order with DTS lagging PTS. B-frames are off by default.

func WithFrameRate

func WithFrameRate(fps float64) EncoderOption

WithFrameRate tells the encoder the expected frame rate. It sets the duration of every frame and lets rate control plan for the rate.

func WithInputFormat

func WithInputFormat(f PixelFormat) EncoderOption

WithInputFormat sets the pixel format of frames given to Encoder.Send: NV12 (the default), RGBA or BGRA. The alpha byte of RGBA and BGRA input is ignored. A backend whose encoder takes NV12 only gets RGBA and BGRA frames converted in Go (BT.709, video range) and declares that matrix in the stream where the encoder lets it.

func WithKeyframeInterval

func WithKeyframeInterval(frames int) EncoderOption

WithKeyframeInterval makes every n-th frame a keyframe, counted from the first frame and from every Flush. The encoder may insert extra keyframes. 0 (the default) leaves keyframe placement to the backend.

func WithLowLatency

func WithLowLatency() EncoderOption

WithLowLatency configures the encoder for live streaming: real-time rate control, no frame reordering and the backend's low-latency mode where it has one. WithKeyframeInterval and Frame.ForceKeyframe keep working.

func WithProfile

func WithProfile(p Profile) EncoderOption

WithProfile selects the coding profile. Combinations the codec does not define (ProfileBaseline or ProfileHigh with HEVC) report ErrUnsupported.

func WithQuality

func WithQuality(q float64) EncoderOption

WithQuality requests constant-quality coding with q in (0, 1] (higher is better) instead of a bitrate target. It cannot be combined with WithBitrate. Backends without a quality mode report ErrUnsupported.

type Frame

type Frame = codec.Frame

Re-exported core types. See the documentation on each in package codec.

func RGBAFrame

func RGBAFrame(img *image.RGBA, pts int64) *Frame

RGBAFrame wraps img as an encoder input frame without copying: the frame shares the image's pixels, which must stay untouched until Encoder.Send has returned. Open the encoder with WithInputFormat(RGBA). The opposite direction is Frame.RGBAImage.

type FrameReader

type FrameReader interface {
	// ReadFrame returns the next frame, or io.EOF at the end. The caller
	// owns the frame and must Release it.
	ReadFrame(ctx context.Context) (*Frame, error)
}

FrameReader is a source of raw frames: a decoder fed from a PacketReader (DecodeReader), or anything that produces or transforms pictures.

type FrameWriter

type FrameWriter interface {
	// WriteFrame consumes f before it returns; the frame stays the
	// caller's, who may release or reuse it afterwards.
	WriteFrame(ctx context.Context, f *Frame) error
}

FrameWriter consumes raw frames: an encoder drained into a PacketWriter (EncodeWriter), or anything that shows or stores pictures.

type Option

type Option = codec.Option

Re-exported core types. See the documentation on each in package codec.

func WithSoftwareFallback

func WithSoftwareFallback() Option

WithSoftwareFallback lets a backend use the operating system's software codec when no hardware engine exists for the codec. By default the request is rejected with ErrUnsupported instead. It applies to the video decoders and encoders; the audio codecs run in software everywhere and take the option without needing it.

func WithTimeScale

func WithTimeScale(unitsPerSecond int32) Option

WithTimeScale sets the number of PTS units per second. The default is 90000 for video and the sample rate for an audio encoder, so that audio timestamps count samples. An audio decoder hands the PTS of each packet on unchanged, whatever its unit, and takes the option without effect.

type Packet

type Packet = codec.Packet

Re-exported core types. See the documentation on each in package codec.

type PacketReader

type PacketReader interface {
	// Codec of the stream.
	Codec() Codec
	// TimeScale is the number of units per second of the packets' PTS and
	// DTS. Open the decoder with WithTimeScale(TimeScale()) so that the
	// frames carry the same clock.
	TimeScale() int32
	// ReadPacket returns the next access unit, whose PTS counts from the
	// start of the presentation. It returns io.EOF at the end of the
	// stream.
	ReadPacket() (Packet, error)
}

PacketReader hands out the access units of a stream in decode order: a demuxed video or audio track, a raw elementary stream. It is what decoders are fed from (see DecodeReader).

type PacketSeeker

type PacketSeeker interface {
	PacketReader
	// SeekKeyframe repositions the reader at the last keyframe whose
	// presentation time is not after t (at the first keyframe when t lies
	// before it) and returns that keyframe's presentation time. The next
	// ReadPacket returns the keyframe.
	SeekKeyframe(t time.Duration) (time.Duration, error)
	// Length returns the stream duration, or 0 when it is not known.
	Length() time.Duration
}

PacketSeeker is a PacketReader that can be repositioned, which is what seeking and looping need.

type PacketWriteCloser

type PacketWriteCloser interface {
	PacketWriter
	io.Closer
}

PacketWriteCloser is a PacketWriter that is finished with Close (the sample tables of a file are written, the viewers disconnected).

type PacketWriter

type PacketWriter interface {
	WritePacket(p Packet) error
}

PacketWriter receives encoded packets in decode order: a file, a segmenter, a network track. It is what encoders are drained into (see EncodeWriter and package capture).

type PacketWriterFunc

type PacketWriterFunc func(p Packet) error

PacketWriterFunc adapts a function to a PacketWriteCloser whose Close does nothing.

func (PacketWriterFunc) Close

func (f PacketWriterFunc) Close() error

Close does nothing.

func (PacketWriterFunc) WritePacket

func (f PacketWriterFunc) WritePacket(p Packet) error

WritePacket calls f.

type PixelFormat

type PixelFormat = codec.PixelFormat

Re-exported core types. See the documentation on each in package codec.

type Profile

type Profile = codec.Profile

Re-exported core types. See the documentation on each in package codec.

type RateControl

type RateControl = codec.RateControl

Re-exported core types. See the documentation on each in package codec.

type SampleFormat

type SampleFormat = codec.SampleFormat

Re-exported audio types. See the documentation on each in package codec.

type SharedEncoderOption

type SharedEncoderOption = codec.SharedEncoderOption

Re-exported audio types. See the documentation on each in package codec.

func WithBitrate

func WithBitrate(bitsPerSecond int) SharedEncoderOption

WithBitrate sets the target bitrate in bits per second, for a video or an audio encoder. The rate control mode is VBR unless WithRateControl(CBR) is also given.

func WithRateControl

func WithRateControl(rc RateControl) SharedEncoderOption

WithRateControl selects VBR (default) or CBR for the bitrate given with WithBitrate, for a video or an audio encoder. Backends that cannot do CBR report ErrUnsupported.

type UnsupportedError

type UnsupportedError = codec.UnsupportedError

Re-exported core types. See the documentation on each in package codec.

Directories

Path Synopsis
Package capture records what a game or any other renderer draws.
Package capture records what a game or any other renderer draws.
cmd
hwmediacodec module
ebitenvideo module
encoding
adts
Package adts reads and writes ADTS streams: AAC access units with a seven-byte header in front of each, which is the format of .aac files and of AAC in MPEG transport streams.
Package adts reads and writes ADTS streams: AAC access units with a seven-byte header in front of each, which is the format of .aac files and of AAC in MPEG transport streams.
annexb
Package annexb splits H.264 and HEVC Annex-B byte streams into NAL units and groups them into access units (one coded picture each), which is the unit a hwmediacodec decoder consumes.
Package annexb splits H.264 and HEVC Annex-B byte streams into NAL units and groups them into access units (one coded picture each), which is the unit a hwmediacodec decoder consumes.
examples module
image
avif
Package avif reads and writes AVIF still images (HEIF with AV1 pictures) with the hardware codecs, in the manner of image/jpeg and image/png: Decode and DecodeConfig take an io.Reader, Encode an image.Image, and importing the package registers the format with the image package, so that image.Decode understands AVIF files:
Package avif reads and writes AVIF still images (HEIF with AV1 pictures) with the hardware codecs, in the manner of image/jpeg and image/png: Decode and DecodeConfig take an io.Reader, Encode an image.Image, and importing the package registers the format with the image package, so that image.Decode understands AVIF files:
heif
Package heif reads and writes HEIC still images (HEIF with HEVC pictures) with the hardware codecs, in the manner of image/jpeg and image/png: Decode and DecodeConfig take an io.Reader, Encode an image.Image, and importing the package registers the format with the image package, so that image.Decode understands HEIC files:
Package heif reads and writes HEIC still images (HEIF with HEVC pictures) with the hardware codecs, in the manner of image/jpeg and image/png: Decode and DecodeConfig take an io.Reader, Encode an image.Image, and importing the package registers the format with the image package, so that image.Decode understands HEIC files:
internal
aac
Package aac reads and writes the two small headers that describe an AAC stream: the AudioSpecificConfig of MPEG-4 (ISO/IEC 14496-3, section 1.6.2.1), which an MP4 file keeps in its esds box, and the ADTS frame header of raw .aac files and MPEG transport streams.
Package aac reads and writes the two small headers that describe an AAC stream: the AudioSpecificConfig of MPEG-4 (ISO/IEC 14496-3, section 1.6.2.1), which an MP4 file keeps in its esds box, and the ADTS frame header of raw .aac files and MPEG transport streams.
audiocmp command
Command audiocmp is the host half of the audio checks that run a codec somewhere else (a phone over adb, a Windows machine): it makes the test signal and compares raw 16-bit samples that came back with a reference.
Command audiocmp is the host half of the audio checks that run a codec somewhere else (a phone over adb, a Windows machine): it makes the test signal and compares raw 16-bit samples that came back with a reference.
audiotoolbox
Package audiotoolbox implements the audio codecs of macOS and iOS on top of the AudioConverter API of Apple's AudioToolbox framework, loaded at run time with purego.
Package audiotoolbox implements the audio codecs of macOS and iOS on top of the AudioConverter API of Apple's AudioToolbox framework, loaded at run time with purego.
audiotoolbox/sys
Package sys binds the AudioToolbox entry points used by the AudioToolbox backend (AudioConverter and AudioFormat) through purego, so that no cgo is required.
Package sys binds the AudioToolbox entry points used by the AudioToolbox backend (AudioConverter and AudioFormat) through purego, so that no cgo is required.
audioutil
Package audioutil holds what the audio backends share: the trial that measures an encoder's delay, the queue that hands a decoder's samples out one frame per packet, and sample format conversion.
Package audioutil holds what the audio backends share: the trial that measures an encoder's delay, the queue that hands a decoder's samples out one frame per packet, and sample format conversion.
av1
Package av1 parses the parts of an AV1 bitstream that a hardware decoder backend needs in order to describe a stream to the operating system: the OBU framing of a temporal unit (Section 5.3 of the AV1 specification), the sequence header (5.5), the frame header (5.9), the tile group layout (5.11.1) and the ISOBMFF AV1CodecConfigurationRecord ("av1C").
Package av1 parses the parts of an AV1 bitstream that a hardware decoder backend needs in order to describe a stream to the operating system: the OBU framing of a temporal unit (Section 5.3 of the AV1 specification), the sequence header (5.5), the frame header (5.9), the tile group layout (5.11.1) and the ISOBMFF AV1CodecConfigurationRecord ("av1C").
bitstream
Package bitstream reads bit fields and Exp-Golomb codes from H.264/HEVC raw byte sequence payloads.
Package bitstream reads bit fields and Exp-Golomb codes from H.264/HEVC raw byte sequence payloads.
browserexec command
Command browserexec runs a GOOS=js GOARCH=wasm test binary in a real browser, where WebCodecs exists (Node.js, which the Go toolchain's own go_js_wasm_exec uses, has none).
Command browserexec runs a GOOS=js GOARCH=wasm test binary in a real browser, where WebCodecs exists (Node.js, which the Go toolchain's own go_js_wasm_exec uses, has none).
codec
Package codec holds the core types shared by the public hwmediacodec API and its platform backends.
Package codec holds the core types shared by the public hwmediacodec API and its platform backends.
fdkaac
Package fdkaac implements AAC decoding and encoding on Linux on top of the system's libfdk-aac (the Fraunhofer FDK AAC library), loaded at run time with purego.
Package fdkaac implements AAC decoding and encoding on Linux on top of the system's libfdk-aac (the Fraunhofer FDK AAC library), loaded at run time with purego.
h264
Package h264 parses the H.264 (Rec.
Package h264 parses the H.264 (Rec.
hevc
Package hevc parses and writes HEVC (ITU-T H.265 / ISO/IEC 23008-2) parameter sets and slice segment headers and keeps the decoded picture buffer bookkeeping of the decoding process: picture order counts, reference picture sets and reference picture lists.
Package hevc parses and writes HEVC (ITU-T H.265 / ISO/IEC 23008-2) parameter sets and slice segment headers and keeps the decoded picture buffer bookkeeping of the decoding process: picture order counts, reference picture sets and reference picture lists.
imageitem
Package imageitem is the ISOBMFF image-item plumbing shared by the image/heif (HEVC) and image/avif (AV1) packages: parsing the "meta" box into items and properties, grids, clean aperture, rotation and mirroring, driving the hardware decoder and encoder one keyframe per item, and writing the file.
Package imageitem is the ISOBMFF image-item plumbing shared by the image/heif (HEVC) and image/avif (AV1) packages: parsing the "meta" box into items and properties, grids, clean aperture, rotation and mirroring, driving the hardware decoder and encoder one keyframe per item, and writing the file.
libavcodec
Package libavcodec implements AAC decoding on Linux on top of the system's libavcodec (FFmpeg), loaded at run time with purego.
Package libavcodec implements AAC decoding on Linux on top of the system's libavcodec (FFmpeg), loaded at run time with purego.
libopus
Package libopus implements Opus decoding and encoding on Linux on top of the system's libopus, loaded at run time with purego.
Package libopus implements Opus decoding and encoding on Linux on top of the system's libopus, loaded at run time with purego.
mediacodec
Package mediacodec implements the Android backend on top of the NDK MediaCodec C API (libmediandk.so), loaded at run time with purego.
Package mediacodec implements the Android backend on top of the NDK MediaCodec C API (libmediandk.so), loaded at run time with purego.
mediacodec/sys
Package sys binds the Android NDK media library (libmediandk.so: the AMediaCodec and AMediaFormat C API) through purego, so that no cgo source is needed.
Package sys binds the Android NDK media library (libmediandk.so: the AMediaCodec and AMediaFormat C API) through purego, so that no cgo source is needed.
mediafoundation
Package mediafoundation implements the Windows backend on top of Media Foundation decoder transforms accelerated through Direct3D 11 (DXVA).
Package mediafoundation implements the Windows backend on top of Media Foundation decoder transforms accelerated through Direct3D 11 (DXVA).
mediafoundation/sys
Package sys binds the Media Foundation, Direct3D 11 and COM entry points used by the Media Foundation backend.
Package sys binds the Media Foundation, Direct3D 11 and COM entry points used by the Media Foundation backend.
mediatest
Package mediatest drives ffmpeg, ffprobe and sips as test oracles for the container and image packages: it generates MP4 inputs, inspects outputs and converts pictures.
Package mediatest drives ffmpeg, ffprobe and sips as test oracles for the container and image packages: it generates MP4 inputs, inspects outputs and converts pictures.
nvidia
Package nvidia implements the NVIDIA backend on Linux and on Windows (amd64): decoding through NVDEC (the cuvid parser and decoder in libnvcuvid / nvcuvid.dll) and encoding through NVENC (libnvidia-encode / nvEncodeAPI64.dll), both on top of the CUDA driver API in libcuda / nvcuda.dll.
Package nvidia implements the NVIDIA backend on Linux and on Windows (amd64): decoding through NVDEC (the cuvid parser and decoder in libnvcuvid / nvcuvid.dll) and encoding through NVENC (libnvidia-encode / nvEncodeAPI64.dll), both on top of the CUDA driver API in libcuda / nvcuda.dll.
nvidia/sys
Package sys binds the NVIDIA driver libraries (libcuda, libnvcuvid and libnvidia-encode on Linux; nvcuda.dll, nvcuvid.dll and nvEncodeAPI64.dll on Windows) through purego so that no cgo is required.
Package sys binds the NVIDIA driver libraries (libcuda, libnvcuvid and libnvidia-encode on Linux; nvcuda.dll, nvcuvid.dll and nvEncodeAPI64.dll on Windows) through purego so that no cgo is required.
opus
Package opus reads and writes the headers around Opus packets: the TOC byte that gives a packet's duration (RFC 6716, section 3.1) and the identification header "OpusHead" (RFC 7845, section 5.1).
Package opus reads and writes the headers around Opus packets: the TOC byte that gives a packet's duration (RFC 6716, section 3.1) and the identification header "OpusHead" (RFC 7845, section 5.1).
pixconv
Package pixconv converts packed RGB pictures to NV12 in Go, for the backends whose encoder takes NV12 only (Media Foundation, VA-API and Intel VPL).
Package pixconv converts packed RGB pictures to NV12 in Go, for the backends whose encoder takes NV12 only (Media Foundation, VA-API and Intel VPL).
reorder
Package reorder turns the decode-order output of a backend decoder into display order.
Package reorder turns the decode-order output of a backend decoder into display order.
testutil
Package testutil generates test bitstreams and reference output with the ffmpeg command-line tool.
Package testutil generates test bitstreams and reference output with the ffmpeg command-line tool.
vaapi
Package vaapi implements the Linux backend on top of VA-API (libva), loaded at run time with purego.
Package vaapi implements the Linux backend on top of VA-API (libva), loaded at run time with purego.
vaapi/sys
Package sys binds libva (VA-API) through purego so that no cgo is required.
Package sys binds libva (VA-API) through purego so that no cgo is required.
videotoolbox
Package videotoolbox implements the macOS backend on top of Apple's VideoToolbox framework, loaded at run time with purego.
Package videotoolbox implements the macOS backend on top of Apple's VideoToolbox framework, loaded at run time with purego.
videotoolbox/sys
Package sys binds the CoreFoundation, CoreMedia, CoreVideo and VideoToolbox entry points used by the VideoToolbox backend through purego, so that no cgo is required.
Package sys binds the CoreFoundation, CoreMedia, CoreVideo and VideoToolbox entry points used by the VideoToolbox backend through purego, so that no cgo is required.
vpl
Package vpl implements the Intel backend on Linux on top of the Intel Video Processing Library (VPL, the successor of the Media SDK and the API behind "Quick Sync Video").
Package vpl implements the Intel backend on Linux on top of the Intel Video Processing Library (VPL, the successor of the Media SDK and the API behind "Quick Sync Video").
vpl/sys
Package sys binds the Intel VPL dispatcher (libvpl) through purego so that no cgo is required.
Package sys binds the Intel VPL dispatcher (libvpl) through purego so that no cgo is required.
webcodecs
Package webcodecs implements the browser backend on top of the WebCodecs API (VideoDecoder, VideoEncoder, VideoFrame, EncodedVideoChunk), reached through syscall/js in a GOOS=js GOARCH=wasm build.
Package webcodecs implements the browser backend on top of the WebCodecs API (VideoDecoder, VideoEncoder, VideoFrame, EncodedVideoChunk), reached through syscall/js in a GOOS=js GOARCH=wasm build.
mediacontainer
ivf
Package ivf reads and writes IVF files, the minimal container for raw AV1 (and VP8/VP9) streams: a 32-byte file header followed by frames that each carry a 12-byte header with the frame size and timestamp.
Package ivf reads and writes IVF files, the minimal container for raw AV1 (and VP8/VP9) streams: a 32-byte file header followed by frames that each carry a 12-byte header with the frame size and timestamp.
mp4
Package mp4 reads and writes MP4 files around hwmediacodec, which itself only handles elementary streams.
Package mp4 reads and writes MP4 files around hwmediacodec, which itself only handles elementary streams.
webm
Package webm reads and writes WebM files around hwmediacodec, which itself only handles elementary streams.
Package webm reads and writes WebM files around hwmediacodec, which itself only handles elementary streams.
net
hls
Package hls serves encoder output as a live HLS stream.
Package hls serves encoder output as a live HLS stream.
webrtc module
Package playback plays a video stream against a clock: it decodes on a background goroutine through hwmediacodec, a few frames ahead, and hands out the frame that is due as the caller advances the time.
Package playback plays a video stream against a clock: it decodes on a background goroutine through hwmediacodec, a few frames ahead, and hands out the frame that is due as the caller advances the time.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL