avfoundation

package module
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 2, 2026 License: BSD-3-Clause Imports: 5 Imported by: 0

README

go-macos/avfoundation

Hardware-accelerated video decoding and playback on macOS from pure Go — CGO_ENABLED=0, via purego.

There are two ways in, and they are not interchangeable.

Reader — decode as fast as the hardware allows. No clock, no sound.

r, err := avfoundation.Open("movie.mp4")
defer r.Close()

for {
        f, err := r.NextFrame()
        if errors.Is(err, io.EOF) { break }
        // f.Pix aliases the decoder's buffer: Stride bytes per row, BGRA
        use(f)
        f.Release()
}

Player — real-time playback. Sound, pause, seek, speed and volume.

p, err := avfoundation.OpenPlayer("movie.mp4")
defer p.Close()

p.SetVolume(0.8)
p.Seek(90 * time.Second)
p.Play()

for drawing {
        p.Pump(4 * time.Millisecond) // only where nothing else runs the run loop
        f, err := p.TryFrame()
        if f == nil { continue }     // nothing new: draw the last picture again
        use(f)
        f.Release()
}
Which one
Reader Player
Audio no yes
Pause, seek, rate, volume no yes
Owns the clock you do AVPlayer does
Frames every one, in order the one belonging to now, dropped to stay in sync
Speed 2165 fps measured (72× real time) real time, by definition
Ends with io.EOF never — it is a clock, ask it where it is

Use Reader to transcode, analyse, or render on your own schedule. Use Player when a person is watching and listening.

Why this exists

There is no realistic pure-Go alternative. A software H.264 or HEVC decoder written in Go will not keep up with 4K60; the hardware decoder in every Mac does it without warming up. Measured here on an M4 Max, 720p H.264:

decoded 7740 frames in 3.575s  (2165 fps, 72x real time)

The constraint the fleet actually cares about is no cgo, not "no operating system". This is a system framework reached without a C toolchain.

Design notes

Frames are not copied. A Frame holds the decoder's buffer locked and its pixels stay valid until Release(). At 4K that avoids ~33 MB of copying per frame. Every frame you receive must be released, once — holding many unreleased frames stalls the decoder, which is waiting for its own buffers back.

Use Stride, not Width*4. Decoders pad rows: a 1280-wide frame comes back with a 5120-byte stride here, but that is not guaranteed. Indexing by width shears the picture progressively down the frame, which looks like a decode bug and is not one.

With Reader, the caller owns the clock. It is a pull API: frames come out as fast as they decode, each carrying its presentation timestamp. That is the right shape for a renderer with its own frame loop — an immersive viewer must draw when the display is ready, not when a player decides. Player is the other bargain: AVPlayer keeps the clock, and gives you audio for it.

Two measured limits

BGRA only, for both paths. Asking for RGBA does not convert — it makes the reader fail (AVAssetReaderStatus 3), and the failure surfaced as an opaque status code from deep inside AVFoundation. Open therefore refuses a non-decodable format up front with a reason, and Frame.ToRGBA converts. The planar YUV formats the hardware natively prefers (NV12 and friends) need a multi-plane Frame this package does not have yet; that is the path to a zero-conversion Metal pipeline.

No Matroska. AVFoundation does not demux MKV or WebM: Open and OpenPlayer both report ErrNoVideoTrack for them. MP4, MOV and M4V work. That path exists elsewhere: demux with go-avkit/avkit/container, which does read EBML, and decode with go-macos/videotoolbox, which feeds the elementary stream to VideoToolbox directly and hands back the same zero-copy BGRA Frame this package does. Its cmd/vtprobe decodes a 1 h 32 MKV end to end, and its first ten frames of a control MP4 are byte-for-byte what avprobe produces here.

Player notes

TryFrame, not NextFrame. AVPlayerItemVideoOutput answers about a moment in time, not about a position in a stream: it vends at most one buffer per item time, and after a seek or a rate change the moment can go backwards. TryFrame returns (nil, nil) when there is nothing new, which is the common case — a display loop polls faster than the video's frame rate. Draw the last picture again.

Seeks are exact. Seek goes to AVFoundation with zero tolerance, so it lands on the time asked for rather than on the nearest keyframe: measured, Seek(60s) lands at 60.000000s, where a plain seekToTime: lands at 58.333s. That costs decoding forward from the previous keyframe, which is the trade a scrubber wants.

Open it on the main thread — this one is not negotiable. AVFoundation loads a file through the main dispatch queue, and only the main thread's run loop drains that queue. Measured: with the main thread parked, an AVPlayerItem sits at status 0 forever, however hard another thread's run loop is pumped, and on a cold file reports a duration of 0 with it. Start the main run loop and the same item loads in ~100 ms.

The nasty part is that a player in that state still answers: seek it to 90s and it reports 90s — the value you just handed it. That echo is how a binding that has opened nothing looks exactly like one that works. OpenPlayer therefore runs the run loop itself while loading and returns an error rather than a half-loaded Player, and every live test here asserts something only a loaded asset could know. TestLiveLoadNeedsTheMainRunLoop is the control: it switches the main run loop off, shows the echo, switches it back on and shows the item load.

So: runtime.LockOSThread() on the main goroutine, open the player there, drive it from there. A GUI application already does this — the main run loop is the event loop, and nothing extra is needed. A program with no event loop must run one; that is what Pump is for.

After loading, measured again, the clock advances and frames come out with nothing but time.Sleep between the calls. Pump is what a headless loop should wait with rather than sleeping; it is not the engine. A Player is not safe for concurrent use.

cmd/avprobe and cmd/avplay

go run ./cmd/avprobe movie.mp4              # info + first 3 frames as PNG
go run ./cmd/avprobe -n 10 movie.mp4
go run ./cmd/avprobe -all movie.mp4         # decode everything, report the rate

go run ./cmd/avplay -for 5s movie.mp4                    # play it, with sound
go run ./cmd/avplay -for 3s -seek 2m -rate 2 movie.mp4   # from 2m, double speed
go run ./cmd/avplay -for 2s -rate -1 -png f.png movie.mp4 # backwards, one frame out

Measured on an M4 Max, 720p H.264:

  playing at rate 1, volume 0.2 for 4s
  118 frames vended over 4.004s wall (29.5 fps), 759 polls had nothing new
  clock moved 3.922s (0.98x wall), frame PTS 0s -> 3.902s

  playing at rate 2, volume 0 for 3s
  176 frames vended over 3.002s wall (58.6 fps), 487 polls had nothing new
  clock moved 5.842s (1.95x wall), frame PTS 2m0s -> 2m5.841s

Testing

The portable layer is at 100% statement coverage behind platform seams. The purego bindings need real media, which a CI runner has no business shipping, so they are covered two ways: error paths (absent, junk and empty files, refused formats) run everywhere, and a live decode test runs when AVFOUNDATION_TEST_FILE names a video file:

AVFOUNDATION_TEST_FILE=/path/to/movie.mp4 go test -race ./...

That lane asserts only things checkable from outside the package — the dimensions and frame rate the file itself reports, that presentation timestamps advance by about one frame period, that the player's clock advances by itself and that a seek lands where it was sent. It also cross-checks the two paths against each other: Reader and Player must agree about the file's duration and size.

The trap it is built around is the one described above. Every live assertion is something only a loaded asset can answer — a status, a duration, a clock that moves — never merely "the value I passed came back", which an item that opened nothing answers just as readily.

Licence: BSD-3-Clause.

Documentation

Overview

Package avfoundation decodes video files on macOS through AVFoundation, with no cgo.

It exists because there is no realistic pure-Go alternative for the job: a software H.264 or HEVC decoder in Go will not keep up with 4K at 60 frames a second, while the hardware decoder in every Mac will do it without warming up. AVFoundation is the system's own front door to that decoder, and it brings demuxing and format support along with it. Everything here goes through github.com/ebitengine/purego, so a consumer still builds with CGO_ENABLED=0 — the constraint the fleet actually cares about is no cgo, not no operating system.

There are two ways in, and they are not interchangeable. Open gives a Reader: decode as fast as the hardware allows, in order, with no clock and no sound — for transcoding, analysis, or a renderer that owns its own timing. OpenPlayer gives a Player: AVPlayer's real-time playback, which brings audio, pause, seeking, speed and volume, and which owns the clock itself.

The Reader model is a pull: Open a file and call Reader.NextFrame until it reports io.EOF. Frames come out as fast as they decode, carrying their presentation timestamps, and the caller owns the clock. That is the right shape for a renderer that has its own frame loop — an immersive viewer must draw when the display is ready, not when a player decides.

Frames are NOT copied. A Frame holds the decoder's own buffer locked, and its pixels stay valid until Frame.Release. At 4K that saves about 33 MB of copying per frame, which is the difference between comfortable and not.

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrUnsupported is returned by every entry point on non-darwin platforms.
	ErrUnsupported = errors.New("avfoundation: unsupported on this platform (darwin only)")
	// ErrNoVideoTrack is returned by [Open] for a file with no video in it.
	ErrNoVideoTrack = errors.New("avfoundation: file has no video track")
	// ErrClosed is returned when a Reader or Player is used after its Close.
	ErrClosed = errors.New("avfoundation: reader or player is closed")
	// ErrReleased is returned by [Frame] accessors after [Frame.Release].
	ErrReleased = errors.New("avfoundation: frame has been released")
	// ErrUnsupportedFormat is returned by [Open] for a pixel format the decoder
	// will not produce. See [Options.Format].
	ErrUnsupportedFormat = errors.New("avfoundation: the decoder will not produce that pixel format")
)

Errors reported by the package. They are stable and may be tested with errors.Is.

Functions

This section is empty.

Types

type Frame 

type Frame struct {
	// Width and Height are this frame's dimensions in pixels.
	Width, Height int
	// Stride is the number of bytes per row, which is USUALLY more than
	// Width*4: the decoder pads rows for alignment. Indexing by Width*4 instead
	// of Stride produces a picture that shears progressively down the frame,
	// which looks like a decode bug and is not one.
	Stride int
	// Format is the pixel layout, as requested when the reader was opened.
	Format PixelFormat
	// PTS is the presentation timestamp: when this frame should be shown,
	// relative to the start of the file.
	PTS time.Duration
	// Pix is the frame's bytes, Stride*Height of them.
	Pix []byte
	// contains filtered or unexported fields
}

Frame is one decoded frame. Its pixels alias the decoder's buffer and are valid until Frame.Release — which the caller must call, once, for every frame it receives. Holding many unreleased frames will stall the decoder, because it is waiting for its own buffers back.

func (*Frame) Release 

func (f *Frame) Release()

Release hands the buffer back to the decoder. It is safe to call more than once, and a released Frame's Pix must not be read.

func (*Frame) Released 

func (f *Frame) Released() bool

Released reports whether the frame's buffer has been handed back.

func (*Frame) ToRGBA 

func (f *Frame) ToRGBA(dst *image.RGBA) *image.RGBA

ToRGBA copies the frame into an *image.RGBA, converting from BGRA if needed.

dst is reused when it is exactly the right size, so a render loop can hold one image and not allocate per frame; pass nil, or an image of the wrong size, to get a fresh one. It returns nil for a released frame.

type Info 

type Info struct {
	// Width and Height are the coded dimensions in pixels.
	Width, Height int
	// FrameRate is the track's nominal rate in frames per second. It is nominal:
	// variable-frame-rate material reports an average, so time anything that
	// matters by a frame's own PTS rather than by counting frames.
	FrameRate float64
	// Duration is the track's duration.
	Duration time.Duration
}

Info describes a file's video track, read when it is opened.

type OpenError 

type OpenError struct {
	Path   string
	Stage  string
	Detail string
}

OpenError describes a failure to open or start reading a file, carrying whatever AVFoundation said about it.

func (*OpenError) Error 

func (e *OpenError) Error() string

type Options 

type Options struct {
	// Format is the pixel format to decode into. Zero means [BGRA].
	Format PixelFormat
	// AudioDeviceUID names the audio device to play the sound on, as
	// coreaudio reports it. Empty -- the default -- lets the system choose,
	// which means the default output.
	//
	// It matters wherever the picture is not on the machine's own screen. A
	// film played on a pair of XR glasses draws there and, with nothing named
	// here, plays out of the Mac's speakers: no error, no warning, just sound
	// coming from the wrong place. The unique id is the stable one -- it
	// survives a reboot and a re-plug, and the numeric device ids do not.
	//
	// A device that does not exist is not an error either: AVFoundation falls
	// back to the default output, so a caller that must KNOW where the sound
	// went should look the device up first and say so.
	AudioDeviceUID string
	// ReadyTimeout bounds how long [OpenPlayer] waits for a file to become
	// playable before giving up. Zero means ten seconds. [Open] ignores it: an
	// AVAssetReader is ready or it is not.
	ReadyTimeout time.Duration
}

Options parametrise Open and OpenPlayer. The zero value asks for BGRA, which is what the decoder produces natively.

type PixelFormat 

type PixelFormat uint32

PixelFormat is a CoreVideo pixel format, which is a four-character code. 

const (
	// BGRA is 32-bit BGRA, 8 bits per channel. It is what the display pipeline
	// and Metal both prefer, so it is the default: asking for RGBA instead would
	// make something, somewhere, swap two bytes per pixel for nothing.
	BGRA PixelFormat = 0x42475241 // 'BGRA'
	// RGBA is 32-bit RGBA. It DESCRIBES a frame's layout but is NOT accepted as a
	// decode request: measured on macOS, an AVAssetReaderTrackOutput asked for
	// RGBA fails outright (reader status 3) rather than converting. Use
	// [Frame.ToRGBA] to convert a decoded BGRA frame instead.
	RGBA PixelFormat = 0x52474241 // 'RGBA'
)

The formats this package can ask the decoder for.

func (PixelFormat) String 

func (f PixelFormat) String() string

String renders the format as its four-character code.

type Player added in v0.2.0 

type Player struct {
	// contains filtered or unexported fields
}

Player plays a file in real time: with sound, and with a clock that can be paused, moved and run at a different speed.

It is the second of this package's two paths, and the two are not interchangeable. Reader decodes as fast as the hardware will go and hands every frame over in order — the right shape for transcoding, analysis, or a renderer that owns its own clock. Player wraps AVPlayer, which owns the clock itself: it renders audio through the system's output, drops video frames to stay in sync, and answers questions about where in the file it is. Nothing in Reader can do any of that, and nothing here decodes at 2000 frames a second.

Video comes out through Player.TryFrame, which is a POLL, not a pull: AVPlayerItemVideoOutput vends at most one buffer per item time, and asking again for the same instant answers nothing. That is the API AVFoundation offers and it is the right one for a display loop, which draws when the screen is ready and simply wants whatever picture belongs to now.

The main thread, and the run loop

A Player must be opened and driven from the process's MAIN thread, with that thread's run loop running. This is not a style preference; it is measured.

AVFoundation loads a file through the main dispatch queue, and only the main thread's run loop drains that queue. An AVPlayerItem in a program whose main thread is parked never becomes ready: it sits at status 0 for as long as you like, however hard some other thread's run loop is pumped, and on a cold file it reports a duration of 0 with it. Worse, a player in that state still ANSWERS: seek it to ninety seconds and it reports ninety seconds — the value it was just handed. That echo is exactly how a binding that has opened nothing can be mistaken for one that works, and it is what the tests here are built to catch. Sleeping is not waiting.

So: runtime.LockOSThread on the main goroutine, open the player there, and drive it from there. An application already does this — the main thread's run loop IS the window system's event loop, and a Player used from a go-widgets or AppKit program needs nothing extra. A program with no event loop of its own must run one, which is what Player.Pump is for; OpenPlayer runs it for the duration of the load and returns an error rather than a half-loaded Player.

Once loaded, the player keeps going on AVFoundation's own queues. Measured: the clock advances, seeks complete and frames come out with nothing but time.Sleep between the calls. Pump is still what a headless loop should wait with — it costs no more than sleeping and it is where run loop work gets done — but it is not the engine.

A Player is NOT safe for concurrent use.

func OpenPlayer added in v0.2.0 

func OpenPlayer(path string, opts ...Options) (*Player, error)

OpenPlayer opens path for real-time playback and waits for it to become playable, which needs the main thread's run loop and therefore happens here rather than leaving a half-loaded Player in the caller's hands. Call it from the main thread; see Player for why.

The player starts PAUSED at the beginning of the file, at full volume. Call Player.Play to start it.

func (*Player) Close added in v0.2.0 

func (p *Player) Close() error

Close stops playback and releases the player. Frames already handed out stay valid until they are individually released.

func (*Player) CurrentTime added in v0.2.0 

func (p *Player) CurrentTime() time.Duration

CurrentTime returns where the player is in the file.

It is the player's clock, not a frame's timestamp: while playing it advances continuously, and the frame Player.TryFrame hands back is the one that belongs to it. A closed player reports 0.

func (*Player) Duration added in v0.2.0 

func (p *Player) Duration() time.Duration

Duration returns the file's duration, which is Info's.

func (*Player) Format added in v0.2.0 

func (p *Player) Format() PixelFormat

Format returns the pixel format frames are decoded into.

func (*Player) Info added in v0.2.0 

func (p *Player) Info() Info

Info returns the video track's description.

func (*Player) Pause added in v0.2.0 

func (p *Player) Pause()

Pause stops playback where it is, keeping the position. It is Player.Play's inverse and leaves the rate at 0.

func (*Player) Play added in v0.2.0 

func (p *Player) Play()

Play starts playback at normal speed, with sound.

Like AVPlayer's own play, it sets the rate to 1 — a rate set earlier with Player.SetRate is NOT restored. Use SetRate to resume at another speed.

func (*Player) Playing added in v0.2.0 

func (p *Player) Playing() bool

Playing reports whether the clock is running, which is exactly whether the rate is not zero.

func (*Player) Pump added in v0.2.0 

func (p *Player) Pump(d time.Duration)

Pump runs the calling thread's run loop for about d.

It is what a program with no event loop of its own — a command-line tool, a test — should wait with instead of time.Sleep, and it must be called from the MAIN thread: a run loop can only be run by the thread that owns it, and the main one is the one AVFoundation needs (see above).

An application whose window system already runs the main run loop must NOT call it: running a run loop that is already being run from underneath itself invites reentrancy.

A d of zero or less runs one pass and returns, which is the non-blocking form.

func (*Player) Rate added in v0.2.0 

func (p *Player) Rate() float64

Rate returns the current playback rate: 0 paused, 1 normal, 2 twice as fast, negative for backwards.

func (*Player) Seek added in v0.2.0 

func (p *Player) Seek(at time.Duration) error

Seek moves playback to at, accurately: the request goes to AVFoundation with zero tolerance, so it lands on the time asked for rather than on the nearest keyframe. That costs decoding from the previous keyframe forward, which is the trade a viewer wants — measured landing within a microsecond of the request.

The time is clamped to the file: before the start becomes the start, past the end becomes the end. Seeking is asynchronous; the run loop must run (see Player.Pump) before Player.CurrentTime reports the new position.

func (*Player) SetRate added in v0.2.0 

func (p *Player) SetRate(rate float64)

SetRate sets the playback speed. A rate of 0 pauses; 1 is normal speed; 2 is twice as fast; a negative rate plays backwards, which not every file supports — measured working on H.264 in MP4.

A NaN or infinite rate is ignored rather than handed to AVFoundation, which would take it and produce a clock that cannot be reasoned about.

func (*Player) SetVolume added in v0.2.0 

func (p *Player) SetVolume(v float64)

SetVolume sets the audio volume, clamped to the 0..1 AVPlayer accepts. NaN is ignored. This is the player's own volume, not the system's.

func (*Player) TryFrame added in v0.2.0 

func (p *Player) TryFrame() (*Frame, error)

TryFrame returns the frame for the player's current time, or (nil, nil) when there is no NEW one to give.

The nil-nil answer is not an error and is the common case: a display loop runs faster than the video's frame rate, and AVPlayerItemVideoOutput vends a buffer only when the picture has changed. A caller draws the last frame again, or nothing.

It is TryFrame rather than NextFrame because there is no "next": the output answers about a moment in time, not about a position in a stream, and after a seek or a rate change the moment can go backwards. A blocking NextFrame would have to either spin or run the run loop behind the caller's back, and both are worse than telling the truth.

Every frame returned must be released, once — see Frame.

func (*Player) Volume added in v0.2.0 

func (p *Player) Volume() float64

Volume returns the audio volume, from 0 to 1.

type Reader 

type Reader struct {
	// contains filtered or unexported fields
}

Reader decodes a file's first video track, in order, from the beginning.

It is NOT safe for concurrent use: one goroutine pulls frames. Seeking is not implemented yet — a reader plays through once.

func Open 

func Open(path string, opts ...Options) (*Reader, error)

Open opens path and prepares its first video track for decoding.

func (*Reader) Close 

func (r *Reader) Close() error

Close releases the decoder. Frames already handed out stay valid until they are individually released.

func (*Reader) Format 

func (r *Reader) Format() PixelFormat

Format returns the pixel format frames are decoded into.

func (*Reader) Info 

func (r *Reader) Info() Info

Info returns the video track's description.

func (*Reader) NextFrame 

func (r *Reader) NextFrame() (*Frame, error)

NextFrame decodes and returns the next frame, or io.EOF when the track is exhausted. The caller must Frame.Release every frame it receives.

Source Files

View all Source files

Directories

Path Synopsis
cmd
avplay command
Command avplay plays a video file in real time -- with sound -- and reports what actually happened: how many frames the video output vended, how far the clock moved, and how the two compare to the wall clock.
 Command avplay plays a video file in real time -- with sound -- and reports what actually happened: how many frames the video output vended, how far the clock moved, and how the two compare to the wall clock.
avprobe command
Command avprobe decodes the first frames of a video file and writes them as PNGs.
 Command avprobe decodes the first frames of a video file and writes them as PNGs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.