README
¶
go-macos/videotoolbox
Hardware H.264 and HEVC decoding on macOS from pure Go — CGO_ENABLED=0, via
purego. You bring the coded frames; this
brings the decoder.
cfg, _ := reader.TrackConfig(track.ID) // go-avkit/avkit/container
codec, _ := videotoolbox.CodecFor(cfg.Codec) // "avc1" -> H264
s, _ := videotoolbox.New(videotoolbox.Config{
Codec: codec, VPS: cfg.VPS, SPS: cfg.SPS, PPS: cfg.PPS,
})
defer s.Close()
for _, sample := range samples {
frames, err := s.Decode(videotoolbox.Sample{Data: sample.Data, PTS: pts})
for _, f := range frames {
// f.Pix aliases the decoder's buffer: Stride bytes per row, BGRA
use(f)
f.Release()
}
}
Why this exists
go-macos/avfoundation already
decodes a video file end to end, demuxing included, and it is the right tool
when it works. It does not work on Matroska. AVFoundation does not demux MKV or
WebM: measured on two real files, AVURLAsset reports no video track at all,
whatever the file holds.
$ avprobe screencasts_defguard-screencast.mkv
avprobe: avfoundation: file has no video track: screencasts_defguard-screencast.mkv
A great deal of 3D and immersive material ships as MKV, so the only way through
is to demux it ourselves and hand the coded frames to the hardware decoder
directly. go-avkit/avkit/container does
the demuxing — MP4, Matroska/WebM and MPEG-TS, in pure Go, parameter sets
included. This package is the other half: VTDecompressionSession, reached
without cgo.
$ vtprobe screencasts_defguard-screencast.mkv
matroska, 54.700 s, 1 tracks (read in 3ms, demuxed in 3ms)
video track 1: avc1 1920x1080, timescale 1000
3279 samples, h264, 0 VPS / 1 SPS / 1 PPS
frame 0: 1920x1080 stride=7680 pts=34ms
wrote frame00.png
What it covers
| Codecs | H.264 (avc1, avc3) and HEVC (hvc1, hev1), set up from parameter sets |
| Input | one access unit at a time, AVCC length prefixes (1, 2 or 4 bytes) or Annex B start codes |
| Output | zero-copy Frame, BGRA, with the PTS of the sample it came from |
| Order | frames come out in decoding order; a player sorts by Frame.PTS |
| Platforms | darwin arm64 and amd64; every other platform builds and answers ErrUnsupported |
Not here: audio, seeking (a caller starts at a sync sample itself, as vtprobe -from shows), and multi-plane pixel formats.
Three measured limits
BGRA only. The planar formats the hardware natively prefers — NV12 and
friends — return NULL from CVPixelBufferGetBaseAddress, because their bytes
live in per-plane allocations reachable only through
CVPixelBufferGetBaseAddressOfPlane. A single-plane Frame cannot describe
those, so this asks for kCVPixelBufferPixelFormatType_32BGRA and refuses any
other request up front, with a reason. Frame.ToRGBA converts for the callers
that need it.
Use Stride, not Width*4. Decoders pad rows. Measured here, 1280-wide
frames come back with a 5120-byte stride and 1920-wide ones with 7680, but that
is not a promise. Indexing by width shears the picture progressively down the
frame, which looks like a decode bug and is not one.
The bitstream form is stated, not sniffed. Sniffing was tried, and it is not sound. On a plain H.264 MP4 that decodes perfectly, sample 205 begins:
00 00 01 05 41 9a ef 34 …
Those first four bytes are the AVCC length of a 261-byte NAL unit. They are also,
byte for byte, an Annex B start code. Any test that reads the leading bytes calls
that sample Annex B, converts it, and hands the decoder rubbish —
kVTVideoDecoderBadDataErr, 204 good frames into the file. A per-track guess is
no better, only luckier: it is the same test run once. So Config.Bitstream
states it, and the default is AVCC, which is what MP4 and Matroska both hold.
Frames are not copied
A Frame holds a CVPixelBuffer locked and its pixels stay valid until
Release(). At 4K that avoids ~33 MB of copying per frame. Every frame you
receive must be released, once — holding many unreleased frames stalls the
decoder, which is waiting for its own buffers back. ReleaseAll does a batch.
What is copied is the coded frame going in: one memcpy per access unit into a
CMBlockBuffer. The alternative is to hand CoreMedia a pointer into the Go heap
and hope the sample buffer dies before the garbage collector notices, which is
not an alternative.
Measured
M4 Max, CGO_ENABLED=0, decoding through the public API of this package and of
go-avkit/avkit/container:
| file | frames | rate | |
|---|---|---|---|
| 4-minute presentation | MP4, H.264 720p | 7 740 / 7 740 | 965 fps, 32× real time |
| screencast | MKV, H.264 1080p | 3 279 / 3 279 | 537 fps, 9× real time |
| feature film, 1 h 32 | MKV, H.264 720p, 2.0 GB | 133 389 / 133 389 | 805 fps, 34× real time |
| 8-second clip | MP4, HEVC 720p | 240 / 240 | — |
| the same clip, remuxed | MKV, HEVC 720p | 240 / 240 | 1013 fps, 34× real time |
The first ten frames of the H.264 MP4, and the first five of the HEVC one, are byte-for-byte identical to the same frames decoded by AVFoundation. That is the control this was built against: a decoder that is wrong is wrong everywhere, and a Matroska path that cannot reproduce a known-good MP4 result has not proved anything.
The HEVC pair is the whole argument in two lines. Remuxed into Matroska — same
samples, same parameter sets, different container — the file still decodes here,
frame for frame, and avprobe answers file has no video track.
avfoundation reaches 2165 fps on the H.264 MP4. The difference is this
package's shape, not the decoder's: it waits for each frame before returning it,
so the caller gets its picture from the call that submitted the sample rather
than from a queue it has to manage.
cmd/vtprobe
vtprobe movie.mkv # info and the first 3 frames as PNG
vtprobe -n 10 -out /tmp movie.mp4
vtprobe -from 25m movie.mkv # start at the sync sample nearest 25 minutes
vtprobe -all movie.mkv # decode the whole track, report the rate
vtprobe -hw movie.mkv # refuse a software-decoder fallback
It reads the file whole, because the demuxer takes a byte slice. That is the tool's limit, not the package's.
Testing
The portable layer is at 100% statement coverage behind platform seams, and CI gates on that file rather than on the total: the purego bindings need a real decoder, and a total-coverage gate would either be a lie or force a video file into the repository.
The bindings are covered three ways. A real VTDecompressionSession is
built on every CI run from an eleven-byte SPS/PPS pair — format description,
callback record, decode, flush, teardown, all exercised with no media anywhere
near the runner. Failure paths (parameter sets that describe nothing, samples
that overrun their own length prefix, a session used after close) run
everywhere. And the end-to-end decode is opt-in:
VIDEOTOOLBOX_TEST_FILE=/path/to/movie.mkv go test -race ./...
Licence: BSD-3-Clause.
Documentation
¶
Overview ¶
Package videotoolbox decodes H.264 and HEVC elementary streams on macOS through VideoToolbox, with no cgo.
It exists because of a measured hole. github.com/go-macos/avfoundation decodes a file end to end — demux included — but AVFoundation does not demux Matroska: given an MKV or a WebM it reports no video track at all, whatever the file holds. Much of the 3D and immersive material worth decoding ships as MKV, so the only way through is to demux it ourselves and hand the coded frames to the hardware decoder directly. That is what this package is: the second half of that path, with github.com/go-avkit/avkit/container the first.
The model is a push-pull: build a Session from the track's parameter sets, hand it one coded frame at a time with Session.Decode, and take back the frames the decoder emitted for it. Everything goes through github.com/ebitengine/purego, so a consumer still builds with CGO_ENABLED=0.
What a sample must look like ¶
A Sample holds ONE access unit — one coded picture with the non-picture units that belong to it. The form its NAL units are in is STATED, in Config.Bitstream, and never sniffed: AVCC length prefixes, which is what an MP4 sample and a Matroska block both hold and so what container.Reader hands back for either, or AnnexB start codes, which is what an MPEG-TS payload and a raw encoder output hold.
Sniffing was tried and it is not sound. Measured on a plain H.264 MP4, sample 205 begins 00 00 01 05 — which is the four-byte AVCC length of a 261-byte NAL unit, and is also, byte for byte, an Annex B start code. Any test that reads the leading bytes calls that sample Annex B, converts it, and hands the decoder rubbish; VideoToolbox answers kVTVideoDecoderBadDataErr 204 good frames into a file that decodes perfectly. A per-track guess is no better, only luckier: it is the same test run once. The caller knows which form its demuxer produces, so the caller states it.
The parameter sets in Config are RAW NAL units — no start code, no length prefix — which is the form container.Reader's TrackConfig states them in. A start code is stripped if one is there anyway.
Frames ¶
Frames are NOT copied. A Frame holds a CVPixelBuffer locked, and its pixels stay valid until Frame.Release — which the caller must call, once, for every frame it receives.
Frames come out in DECODING order, not display order: a stream with B-frames emits them out of presentation order, and each frame carries the PTS of the sample it came from. A player must reorder by Frame.PTS; this package will not hold frames back to do it for you, because a decoder that buffers is a decoder that adds latency without being asked.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ErrUnsupported is returned by every entry point on non-darwin platforms. ErrUnsupported = errors.New("videotoolbox: unsupported on this platform (darwin only)") // ErrClosed is returned when a Session is used after [Session.Close]. ErrClosed = errors.New("videotoolbox: session is closed") // ErrUnsupportedCodec is returned by [New] for a codec this package does // not build a format description for. ErrUnsupportedCodec = errors.New("videotoolbox: unsupported codec") // ErrUnsupportedFormat is returned by [New] for a pixel format the decoder // will not produce. See [Options.Format]. ErrUnsupportedFormat = errors.New("videotoolbox: the decoder will not produce that pixel format") // ErrParameterSets is returned by [New] when the configuration does not // carry the parameter sets its codec needs. ErrParameterSets = errors.New("videotoolbox: incomplete parameter sets") // ErrNALUnitLength is returned for a length prefix size VideoToolbox does // not accept. ErrNALUnitLength = errors.New("videotoolbox: invalid NAL unit length prefix size") // ErrSample is returned when a sample cannot be submitted as given. ErrSample = errors.New("videotoolbox: invalid sample") )
Errors reported by the package. They are stable and may be tested with errors.Is.
Functions ¶
func AnnexBToAVCC ¶
AnnexBToAVCC rewrites a start-code separated access unit as a length-prefixed one, with lengthSize bytes of big-endian length before each NAL unit — the form a VideoToolbox format description states and the decoder expects.
It allocates: the two forms are not the same size, so this cannot be done in place. A caller decoding an MP4 or a Matroska file never reaches it.
func ReleaseAll ¶
func ReleaseAll(frames []*Frame)
ReleaseAll releases every frame in a batch, which is what a caller that has finished with the result of one Session.Decode wants.
func StripStartCode ¶
StripStartCode returns a NAL unit without its Annex B start code, and unchanged when it has none. Parameter sets travel both ways in the wild: an avcC record holds them bare, an elementary stream holds them separated.
Types ¶
type Bitstream ¶
type Bitstream uint8
Bitstream is how a sample separates its NAL units.
const ( // AVCC separates NAL units with a big-endian length prefix of // [Config.NALUnitLengthSize] bytes. It is the zero value because it is // what an MP4 sample and a Matroska block hold, and so what a demuxed // track almost always is. It is also the only form VideoToolbox takes. AVCC Bitstream = iota // AnnexB separates NAL units with 00 00 01 start codes, as an MPEG-TS // payload and a raw encoder output do. Samples in this form are rewritten // as AVCC before they are submitted, which allocates. AnnexB )
type Codec ¶
type Codec uint8
Codec names the bitstream a session decodes.
const ( // H264 is ITU-T H.264 / MPEG-4 AVC, described by its SPS and PPS. H264 Codec = iota + 1 // HEVC is ITU-T H.265, described by its VPS, SPS and PPS. HEVC )
The codecs VideoToolbox will build a format description for from parameter sets alone, which is what a demuxed track gives us.
type Config ¶
type Config struct {
// Codec is the bitstream in the samples. It is taken as stated and never
// inferred: the two NAL-based codecs spell a unit's type in different
// bits, so reading one as the other turns a picture into a parameter set.
Codec Codec
// Bitstream is how a sample separates its NAL units. The zero value is
// [AVCC], which is what an MP4 and a Matroska file both hold. It is taken
// as stated and never sniffed; the package documentation says why, with
// the measured counter-example.
Bitstream Bitstream
// VPS, SPS and PPS are the track's parameter sets as raw NAL units. HEVC
// needs all three; H.264 needs SPS and PPS and ignores VPS.
VPS, SPS, PPS [][]byte
// NALUnitLengthSize is how many bytes of big-endian length precede each
// NAL unit in a sample. Zero means 4, which is what an MP4 and a Matroska
// block both use; VideoToolbox accepts 1, 2 and 4 and nothing else.
NALUnitLengthSize int
// Width and Height are the coded frame size, carried for the caller's
// benefit: the decoder reads its own from the SPS.
Width, Height int
}
Config describes the track a session decodes, the way a demuxer states it.
container.Reader's TrackConfig hands back everything here but the codec: SPS, PPS and VPS are its fields of the same name, raw NAL units without a start code or a length prefix.
type Frame ¶
type Frame struct {
// Width and Height are this frame's dimensions in pixels.
Width, Height int
// Stride is the number of bytes per row, which is USUALLY more than
// Width*4: the decoder pads rows for alignment. Measured here, a
// 1280-wide frame comes back with a 5120-byte stride — which happens to
// be Width*4 — and a 1920-wide one does not. Index by Stride.
Stride int
// Format is the pixel layout, always [BGRA] today.
Format PixelFormat
// PTS is the presentation timestamp of the sample this frame was decoded
// from. Frames arrive in decoding order, so a stream with B-frames hands
// these back out of order and a player must sort by them.
PTS time.Duration
// Pix is the frame's bytes, Stride*Height of them.
Pix []byte
// contains filtered or unexported fields
}
Frame is one decoded frame. Its pixels alias a CVPixelBuffer and are valid until Frame.Release — which the caller must call, once, for every frame it receives. Holding many unreleased frames stalls the decoder, which is waiting for its own buffers back.
func (*Frame) Release ¶
func (f *Frame) Release()
Release hands the buffer back to the decoder. It is safe to call more than once, and a released Frame's Pix must not be read.
func (*Frame) ToRGBA ¶
ToRGBA copies the frame into an *image.RGBA, converting from BGRA if needed.
dst is reused when it is exactly the right size, so a render loop can hold one image and not allocate per frame; pass nil, or an image of the wrong size, to get a fresh one. It returns nil for a released frame.
type Options ¶
type Options struct {
// Format is the pixel format to decode into. Zero means [BGRA], which is
// also the only accepted value.
Format PixelFormat
// RequireHardware refuses a session that would fall back to the software
// decoder, rather than decoding slowly and saying nothing.
RequireHardware bool
}
Options parametrise New. The zero value asks for BGRA from whichever decoder VideoToolbox picks, which on Apple silicon is the hardware one.
type PixelFormat ¶
type PixelFormat uint32
PixelFormat is a CoreVideo pixel format, which is a four-character code.
const ( // BGRA is 32-bit BGRA, 8 bits per channel, one plane. It is the only // format this package asks for, and the reason is measured rather than // chosen: the planar formats the hardware natively prefers (NV12 and // friends) return NULL from CVPixelBufferGetBaseAddress, because their // bytes live in per-plane allocations reachable only through // CVPixelBufferGetBaseAddressOfPlane. A single-plane [Frame] cannot // describe those, so asking for one would hand back an empty picture. BGRA PixelFormat = 0x42475241 // 'BGRA' // RGBA is 32-bit RGBA. It DESCRIBES a frame but is not accepted as a // decode request: measured on macOS, a decompression session asked for it // fails to produce usable buffers. Use [Frame.ToRGBA] instead. RGBA PixelFormat = 0x52474241 // 'RGBA' )
The formats this package can ask the decoder for.
func (PixelFormat) String ¶
func (f PixelFormat) String() string
String renders the format as its four-character code.
type Sample ¶
type Sample struct {
// Data is the access unit, in the form [Config.Bitstream] states:
// length-prefixed by default, start-code separated when the session was
// built for [AnnexB].
Data []byte
// PTS is when this picture should be shown, relative to the start of the
// track. It travels through the decoder and comes back on the [Frame].
PTS time.Duration
// Duration is how long the picture lasts; zero if the demuxer does not
// say. Nothing here depends on it, but VideoToolbox is told.
Duration time.Duration
}
Sample is one coded access unit handed to the decoder.
type Session ¶
type Session struct {
// contains filtered or unexported fields
}
Session is a VideoToolbox decompression session: one track's decoder, fed a coded frame at a time.
It is NOT safe for concurrent use.
func (*Session) Close ¶
Close tears the session down. Frames already handed out stay valid until they are individually released.
func (*Session) Config ¶
Config returns the configuration the session was built from, with the defaults resolved.
func (*Session) Decode ¶
Decode submits one coded frame and returns the frames the decoder emitted for it — usually one, none while the decoder is filling its reference buffers, and occasionally more.
The caller must Frame.Release every frame it receives; ReleaseAll does a batch.
func (*Session) Flush ¶
Flush waits for every frame still inside the decoder and returns them. A caller that has submitted its last sample must call it, or lose whatever the decoder was still holding.
func (*Session) Format ¶
func (s *Session) Format() PixelFormat
Format returns the pixel format frames are decoded into.
type StatusError ¶
type StatusError struct {
// Op is the C function that failed.
Op string
// Status is the OSStatus it returned.
Status int32
}
StatusError carries an OSStatus from VideoToolbox or CoreMedia, naming the call that returned it. VideoToolbox says everything through these codes, and an unexplained -12909 is a long walk from the mistake that caused it.
func (*StatusError) Error ¶
func (e *StatusError) Error() string
Source Files
Directories