decode

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 9, 2026 License: BSD-3-Clause Imports: 10 Imported by: 0

README

decode

ci

Video decoders for avkit, pure Go, no cgo. The first codec is H.264 (ITU-T H.264 | ISO/IEC 14496-10), the second HEVC / H.265 (ITU-T H.265 | ISO/IEC 23008-2): Main and Main 10, 4:2:0. Waves D1 and D2 of go-avkit/avkit#89.

r, _ := container.NewReader(file)                   // MP4, Matroska, ...
frames, err := decode.DecodeTrack(r, trackID, decode.Config{})
for _, f := range frames {                          // display order
    use(f.Y, f.Cb, f.Cr, f.YStride, f.CStride, f.Width, f.Height, f.PTS)
}

One picture, for a poster or a thumbnail, without decoding the track:

id, _ := decode.FirstVideoTrack(r)
f, err := decode.PosterFrame(r, id, 12.5, decode.PosterConfig{}) // the picture shown at 12.5 s

PosterFrame returns the picture with the greatest presentation time at or before t (decode time plus composition offset, as DecodeTrack labels them), and the very bytes DecodeTrack returns for it. A t past the end gives the last picture, one before the first presentation time gives the first, a negative or NaN t is ErrTime. Decoding starts at the last sync sample at or before the picture's sample (the first sample if none is flagged) and stops as soon as the picture is out. H.264 (avc1, avc3) and HEVC (hvc1, hev1); another codec is ErrUnsupported.

Streaming: decode.NewTrack(trackConfig, cfg) or decode.NewAVCC(avcC, cfg), then DecodeSample(sample.Data, pts) per sample and Flush() at the end. Raw .264 files: decode.New(cfg, 4) and DecodeAnnexB.

Frame carries the planes, strides, size, Chroma format, BitDepth and PTS. Its field names match go-videodsp/yuv.Picture for 8-bit 4:2:0, so converting to RGBA is a field copy; yuv.Picture has no bit depth or time, which is why it is not used as is.

HEVC takes the same calls: DecodeTrack and NewTrack choose HEVC for a track whose sample entry is hvc1 or hev1; decode.NewHVCC(hvcC, cfg) takes an hvcC record (ParseHVCC reads one), decode.NewHEVC(cfg, 4) takes in-band parameter sets and DecodeAnnexB a raw .265 stream. Main 10 frames hold little-endian 16-bit samples (ffmpeg's yuv420p10le). Each picture's PTS is that of the sample holding its first slice, so it is exact whatever the reordering. Frame.Corrupt marks a picture the decoder lost a slice of. Config.MaxFrameMBs counts 256 luma samples for HEVC; EnforceLevel applies to H.264 only.

HEVC

The decoding is github.com/gen2brain/h265 v0.2.4 (MIT, pure Go, assembly kernels with a noasm build tag), the hevc package. It decodes more than is claimed here; only what was measured is accepted:

Profile State
Main, Main Still Picture, 4:2:0 8-bit bit-exact against ffmpeg
Main 10, 4:2:0 10-bit bit-exact against ffmpeg
Other chroma formats, bit depths other than 8 and 10, unequal luma and chroma depths refused with ErrUnsupported

Measured, every picture's MD5 against ffmpeg -f framemd5 (ffmpeg 9.0.2, one thread, native pixel format):

  • JCT-VC conformance suite HEVC_v1 (ITU-T H.265.1): 147 bitstreams. 143 match ffmpeg on every picture (testdata/hevc-conformance), and wherever the package ships the MD5 of the whole output it equals ours too. Three are decided by the suite's own MD5, because ffmpeg disagrees with it and we agree: SAODBLK_A_MainConcept_4, SAODBLK_B_MainConcept_4 (ffmpeg's output differs from picture 0, with or without its assembly) and VPSSPSPPS_A_MainConcept_1 (ffmpeg reports missing parameter sets and outputs 2 pictures of the 6). One, TSUNEQBD_A_MAIN10_Technicolor_2 (luma and chroma bit depths differ), is refused here and by ffmpeg. The bitstreams are not in the repository (their redistribution terms are not stated); TestHEVCConformance runs when DECODE_HEVC_DIR points at them (go run ./tools/jvt --codec hevc fetch --all).
  • 5 streams in testdata/hevc/streams (made here with x265 from ffmpeg's testsrc2; intra, B pyramid, slices, wavefronts, weighted prediction, cropping, Main and Main 10) run in CI against ffmpeg's MD5s, plus a 4:4:4 one that must be refused.
  • 7 real files (MP4 and Matroska, 540x960 to 3840x1606, Main and Main 10), 200 pictures each: testdata/hevc-real-files.txt, hashes only.

Speed against ffmpeg is not measured: the machine's 1-minute load was 27 to 38 during this work. Tracked for decode#2.

Bibliography: ITU-T H.265 (V11, 01/2026), opened as the PDF; read and checked: Annex B (B.2, B.3: byte stream and its NAL unit extraction) and C.5.2.2 to C.5.2.4 (output order DPB, bumping, cropping). ISO/IEC 14496-15 (hvcC) is paywalled and not opened: the record layout is that of mp4ff v0.62.0 hevc/hevcdecoderconfigurationrecord.go (opened); it is fuzzed here. The JCT-VC suite listing was opened.

What is decoded (H.264)

The decoding is github.com/oops1/go.264 (BSD-2-Clause, pure Go with assembly kernels), pinned and forced onto its software path. Where it is wrong it is fixed upstream by pull request.

Profile State
Constrained Baseline, Baseline without FMO/ASO bit-exact
Main, progressive (CAVLC and CABAC, B pictures, weighted prediction, multiple references, long-term references, MMCO) bit-exact
High, 4:2:0 8-bit, progressive (8x8 transform, scaling matrices) bit-exact on the streams below; no JVT FRExt stream measured yet
Interlaced (PAFF, MBAFF), FMO/ASO, Extended (SP/SI, data partitioning) refused with ErrUnsupported
High 10, 4:2:2, 4:4:4, lossless refused with ErrUnsupported

Frame.BitDepth and Chroma exist so the later profiles need no API change.

Measured

Every picture's MD5 against ffmpeg -f framemd5 -pix_fmt yuv420p, single thread, and against the reference yuv shipped in the suite where there is one.

  • ITU-T H.264.1 conformance suite, AVCv1: 81 bitstreams tried, 73 match ffmpeg (and the suite's yuv where it ships one): 69 with go.264 v1.12.0 as pinned here, and 4 more (MR2..MR5_TANDBERG) only with oops1/go.264#2 "decoder: handle MMCO 5 and gaps in frame_num"; they are listed in pendingUpstream until it is released. The other 8 are not claimed: 5 interlaced, 3 use FMO/ASO (ffmpeg cannot decode two of them either). ffmpeg's MD5s are in testdata/jvt; the bitstreams are not in the repository (their redistribution terms are not stated), so TestJVTConformance runs when DECODE_JVT_DIR points at them (go run ./tools/jvt fetch downloads them).
  • 29 streams in testdata/streams (from go.264, BSD-2-Clause; Baseline, Main, High) run in CI against ffmpeg's MD5s in the .md5 files.
  • 5 real files (MP4 and Matroska, 720x300 to 1920x1080, x264), 900 pictures each: testdata/real-files.txt, hashes only.

Bounds

A stream is untrusted. Config.MaxFrameMBs (default 36864, a 4096x2304 picture) and MaxSampleBytes (default 64 MiB) are checked before anything sized by the stream is allocated; go.264 also refuses pictures past its level ceiling and bounds its reference frames by max_num_ref_frames. TestBombRefusedUnderMemoryLimit runs three bomb SPSs under a 256 MiB memory limit and asserts a refusal and under 8 MiB allocated. FuzzParseAVCC, FuzzDecodeSample and FuzzDecodeAnnexB have a committed corpus (testdata/fuzz, from go run ./tools/jvt seeds) and run in CI; so do FuzzParseHVCC, FuzzHEVCSample and FuzzHEVCAnnexB. For HEVC the picture size is checked by the decoder at the first slice, from the SPS, before any plane is allocated, and what an Annex B stream may hold waiting for its next start code is bounded by MaxSampleBytes; TestHEVCSizeBomb declares pictures up to 1048576 square under a 256 MiB memory limit and asserts a refusal and under 32 MiB allocated. A picture over the limit is refused with ErrUnsupported, not ErrOverLimit, because gen2brain/h265 reports it with the error it uses for an unimplemented tool.

PosterFrame adds two bounds of its own, in PosterConfig: MaxSamples (default 600) caps the samples decoded for one picture, counted from the keyframe it starts at to the sample that outputs it (the decoder outputs a picture at least one sample late); past it the error is ErrPosterLimit, raised before a decoder is built when the picture's sample is already too far from its keyframe. MaxTrackSamples (default 1<<20) refuses a longer sample list, checked once container.Reader.Samples has listed it (a reader on a file left on disk fetches every sample there, which this package cannot avoid). FuzzPosterFrame reads an MP4 or Matroska file from bytes with any t, has a committed corpus (testdata/fuzz/FuzzPosterFrame, rewritten with DECODE_WRITE_SEEDS=1 go test -run TestWritePosterSeeds), and runs in CI under GOMEMLIMIT and ulimit -v.

Bibliography

  • ITU-T H.264 (V16, 06/2026, in force), opened as the PDF. Read and checked: 7.3.2.1 (SPS syntax entry), 7.4.3 (frame_num semantics: a picture with memory_management_control_operation 5 is inferred to have had frame_num 0; Table 7-9), 8.2.1 (tempPicOrderCnt after operation 5; 8.2.1.1 to 8.2.1.3 prevPicOrderCntMsb/Lsb and prevFrameNumOffset after it), 8.2.5.1 to 8.2.5.4 (marking, gaps in frame_num, sliding window) and C.4.4 (output order and operation 5). The fix upstream (oops1/go.264#2) follows those clauses, including the rebased prevPicOrderCntLsb. ffmpeg (libavcodec h264_refs.c, h264_picture.c, h264_slice.c, h264_parse.c, opened) sets frame_num to 0 on MMCO_RESET and takes the frame order count as the Min of the two field counts; I found no rebase of prev_poc_lsb in those files. The spec text is the reference.
  • ITU-T H.264.1, conformance specification and the bitstream list at https://www.itu.int/wftp3/av-arch/jvt-site/draft_conformance/AVCv1/ (listing and each stream's readme opened).
  • ISO/IEC 14496-15 (avcC): paywalled, not opened. The record layout is that of mp4ff v0.62.0 avc/avcdecoderconfigurationrecord.go (opened), which cites 14496-15 itself; it is fuzzed here.

Performance

Not yet measured against ffmpeg: the machine's 1-minute load was 36 to 50 during this work, above the limit for a measurement (8). Tracked in the issues.

Development

go test -race ./...                       # library, 100% of statements
cd tools && go run ./covergate ../cover.out   # the gate (reads the profile)
cd tools && go test ./...                 # oracle harness

BSD-3-Clause. The streams in testdata/streams are from go.264 (BSD-2-Clause, testdata/streams/LICENSE.go264).

Documentation

Overview

Package decode turns the coded video samples avkit's container package demuxes into pictures.

The first codec is H.264 (ITU-T H.264 | ISO/IEC 14496-10), through github.com/oops1/go.264, the second HEVC (ITU-T H.265), through github.com/gen2brain/h265: pure Go decoders that are checked here, picture for picture, against ffmpeg. See the README for the profiles that claim holds for and the conformance bitstreams it was measured on.

Input is what a container hands over: an AVCDecoderConfigurationRecord or an HEVCDecoderConfigurationRecord (ISO/IEC 14496-15) or the parameter sets of a container.TrackConfig, then one length-prefixed access unit per sample. Output is every picture in display order as planar YUV, with its chroma format, bit depth and presentation time.

Index

Constants

View Source
const (
	// DefaultMaxFrameMBs is 36864 macroblocks, a 4096x2304 picture: level 5.1
	// of H.264 Annex A.
	DefaultMaxFrameMBs = 36864
	// DefaultMaxSampleBytes is 64 MiB, far above any single access unit a
	// level 6.2 stream can carry at its maximum bit rate and frame rate.
	DefaultMaxSampleBytes = 64 << 20
)

Default limits. A stream is untrusted input: the picture size in its sequence parameter set sizes every buffer the decoder allocates, so it is bounded before the first of them is.

View Source
const (
	// DefaultMaxPosterSamples is 600 samples, ten seconds of 60 frames per
	// second video decoded for a single picture.
	DefaultMaxPosterSamples = 600
	// DefaultMaxTrackSamples is 1<<20 samples, about 4.8 hours at 60 frames per
	// second.
	DefaultMaxTrackSamples = 1 << 20
)

Default PosterConfig bounds.

Variables

View Source
var (
	// ErrConfig means the codec configuration (an avcC record or a track's
	// parameter sets) is malformed.
	ErrConfig = errors.New("decode: invalid codec configuration")
	// ErrSample means a sample is not a well formed run of length-prefixed
	// NAL units.
	ErrSample = errors.New("decode: invalid sample")
	// ErrOverLimit means the stream asks for more than Config allows: a
	// picture larger than MaxFrameMBs, or a sample larger than MaxSampleBytes.
	// It is raised before anything sized by the stream is allocated.
	ErrOverLimit = errors.New("decode: stream exceeds the configured limits")
	// ErrUnsupported means the stream uses a coding tool this decoder does
	// not implement (see the README): for H.264 interlaced coding, a chroma
	// format other than 4:2:0, a bit depth above 8, lossless transform bypass;
	// for HEVC a chroma format other than 4:2:0, a bit depth other than 8 or
	// 10, and, because the HEVC decoder reports both the same way, a picture
	// larger than Config.MaxFrameMBs.
	ErrUnsupported = errors.New("decode: unsupported coding tool")
	// ErrClosed means the decoder was used after Close.
	ErrClosed = errors.New("decode: use of a closed decoder")
)

Errors a decoder can refuse with. Each is matched with errors.Is.

View Source
var (
	// ErrPosterLimit means the picture asked for is further from the keyframe it
	// is decoded from than PosterConfig.MaxSamples allows, or the track lists
	// more samples than PosterConfig.MaxTrackSamples. It is raised before the
	// decoder is built, so a stream with a distant keyframe costs nothing.
	ErrPosterLimit = errors.New("decode: poster frame exceeds the configured work limit")
	// ErrTime means the time asked for is negative or not a number.
	ErrTime = errors.New("decode: invalid poster time")
	// ErrNoVideo means the file has no video track.
	ErrNoVideo = errors.New("decode: no video track")
)

Errors PosterFrame refuses with, besides the ones above.

View Source
var ErrBitstream = errors.New("decode: malformed bitstream")

ErrBitstream means the HEVC bitstream itself is malformed: a parameter set, slice header or slice data the decoder cannot read. A bitstream the decoder can read but does not implement is ErrUnsupported instead.

Functions

func FirstVideoTrack added in v0.2.0

func FirstVideoTrack(r *container.Reader) (uint32, error)

FirstVideoTrack returns the ID of the first video track of a demuxed file.

Types

type AVCConfig

type AVCConfig struct {
	// Profile, Compatibility and Level are AVCProfileIndication,
	// profile_compatibility and AVCLevelIndication, the first three bytes of
	// the SPS after the NAL header.
	Profile, Compatibility, Level uint8
	// LengthSize is the size in bytes of each NAL unit's length prefix:
	// lengthSizeMinusOne + 1: 1, 2 or 4 (3 is reserved).
	LengthSize int
	// SPS and PPS are the parameter sets, as NAL units without a prefix.
	SPS, PPS [][]byte
}

AVCConfig is an AVCDecoderConfigurationRecord, the content of the avcC box (ISO/IEC 14496-15). The standard's text is paywalled and was not opened; the layout is that of mp4ff v0.62.0, avc/avcdecoderconfigurationrecord.go, which was.

func ParseAVCC

func ParseAVCC(b []byte) (AVCConfig, error)

ParseAVCC parses an AVCDecoderConfigurationRecord. It reads the record only as far as the decoder needs: the extension fields a High profile record appends (chroma format, bit depths, SPS extensions) are bounds-checked but not kept, since the sequence parameter set itself is authoritative.

type Chroma

type Chroma uint8

Chroma is the chroma sampling of a Frame.

const (
	Chroma400 Chroma = iota
	Chroma420
	Chroma422
	Chroma444
)

The chroma formats of H.264 (chroma_format_idc). Only Chroma420 is produced today; the others are named so that a Frame can describe the profiles that are not decoded yet without an API change.

type Config

type Config struct {
	// MaxFrameMBs is the largest picture accepted, in 16x16 macroblocks.
	MaxFrameMBs int
	// MaxSampleBytes is the largest sample (access unit) accepted.
	MaxSampleBytes int
	// EnforceLevel also refuses a stream that exceeds its own declared
	// level's limits (Annex A), where MaxFrameMBs alone would admit it.
	EnforceLevel bool
}

Config bounds what a stream may make the decoder allocate. A zero field takes its default.

type Decoder

type Decoder struct {
	// contains filtered or unexported fields
}

Decoder decodes one H.264 or HEVC elementary stream. It is not safe for concurrent use.

func New

func New(cfg Config, lengthSize int) (*Decoder, error)

New returns a decoder for length-prefixed samples with lengthSize-byte lengths (1, 2 or 4 as avcC allows; MP4 and Matroska use 4). The parameter sets must then arrive in-band, or through Config via NewAVCC or NewTrack.

func NewAVCC

func NewAVCC(avcC []byte, cfg Config) (*Decoder, error)

NewAVCC returns a decoder configured by an avcC record, as MP4 stores it.

func NewHEVC added in v0.3.0

func NewHEVC(cfg Config, lengthSize int) (*Decoder, error)

NewHEVC returns an HEVC (H.265) decoder for length-prefixed samples with lengthSize-byte lengths (1, 2 or 4). The parameter sets must then arrive in-band, or through NewHVCC or NewTrack. For HEVC, Config.MaxFrameMBs counts 256 luma samples (a 16x16 block, as for H.264) and Config.EnforceLevel has no effect.

func NewHVCC added in v0.3.0

func NewHVCC(hvcC []byte, cfg Config) (*Decoder, error)

NewHVCC returns an HEVC decoder configured by an hvcC record, as MP4 stores it. Samples are length-prefixed with the record's length size.

func NewTrack

func NewTrack(tc container.TrackConfig, cfg Config) (*Decoder, error)

NewTrack returns a decoder for an avkit container track: an HEVC decoder for a track whose sample entry is hvc1 or hev1, an H.264 decoder otherwise. Samples are length-prefixed with four-byte lengths, which is what container.Reader hands over for MP4 and Matroska. The track's parameter sets are decoded first.

func (*Decoder) Close

func (d *Decoder) Close() error

Close releases the decoder.

func (*Decoder) DecodeAnnexB

func (d *Decoder) DecodeAnnexB(data []byte) ([]Frame, error)

DecodeAnnexB decodes part of a raw Annex B byte stream (start code delimited, as in a .264 file). The pictures' PTS is their index in display order. The chunk need not end on a NAL unit boundary.

func (*Decoder) DecodeSample

func (d *Decoder) DecodeSample(data []byte, pts int64) ([]Frame, error)

DecodeSample decodes one access unit in length-prefixed form, which is container.Sample.Data, and returns the pictures that became ready, in display order. pts is the sample's presentation time; it is attached to the pictures in increasing order as they come out, so it need not belong to the picture the sample itself contains. A picture can come out several samples late.

func (*Decoder) Flush

func (d *Decoder) Flush() ([]Frame, error)

Flush returns the pictures still held for reordering. Call it at the end of the stream.

type Frame

type Frame struct {
	// Y, Cb and Cr are the planes. A sample is one byte when BitDepth is 8
	// and a little-endian 16-bit word above that. A Chroma400 frame has no
	// Cb or Cr.
	Y, Cb, Cr []byte
	// YStride and CStride are the plane row lengths in bytes.
	YStride, CStride int
	// Width and Height are the luma size in samples.
	Width, Height int
	// Chroma is the chroma sampling.
	Chroma Chroma
	// BitDepth is the luma bit depth.
	BitDepth int
	// PTS is the presentation time in the track timescale: the decode time
	// of the sample plus its composition offset. For a raw Annex B stream it
	// is the picture's index in display order.
	PTS int64
	// Corrupt reports that the HEVC decoder lost a slice of the picture, failed
	// partway through it, or predicted it from a reference the stream never
	// delivered. The picture is still returned, as far as it was decoded. It is
	// never set for H.264.
	Corrupt bool
}

Frame is one decoded picture, cropped to the stream's display window.

The field names and meaning match go-videodsp/yuv.Picture for 8-bit 4:2:0 so that converting is a field-by-field copy. A Frame owns its planes.

func DecodeTrack

func DecodeTrack(r *container.Reader, trackID uint32, cfg Config) ([]Frame, error)

DecodeTrack decodes every sample of one video track of a demuxed file and returns the pictures in display order with their presentation times in the track's timescale. It is the whole-file form of NewTrack and DecodeSample.

func PosterFrame added in v0.2.0

func PosterFrame(r *container.Reader, trackID uint32, t float64, cfg PosterConfig) (Frame, error)

PosterFrame decodes the one picture a track displays at t seconds: the one with the greatest presentation time at or before t, where a sample's presentation time is its decode time plus its composition offset, as for DecodeTrack, whose picture and PTS it returns byte for byte.

A t past the end of the track gives the last picture, and a t before the first presentation time gives the first. A negative or NaN t is ErrTime. Decoding starts at the last sample marked Sync at or before the picture's sample (the first sample when none is) and stops as soon as the picture is out, so the work is bounded by cfg.MaxSamples, not by the length of the track (ErrPosterLimit). Only H.264 and HEVC are decoded; another codec is ErrUnsupported.

The track's samples are listed by container.Reader.Samples, which a reader on a file that stays on disk (container.NewFileReader) fetches whole: the limit on their number is checked once they are listed.

type HVCConfig added in v0.3.0

type HVCConfig struct {
	// ProfileSpace, Tier, Profile and Level are general_profile_space,
	// general_tier_flag, general_profile_idc and general_level_idc, as the
	// sequence parameter set states them.
	ProfileSpace uint8
	Tier         bool
	Profile      uint8
	Level        uint8
	// ChromaFormat is chroma_format_idc; BitDepthLuma and BitDepthChroma are the
	// sample bit depths (the record stores them minus eight).
	ChromaFormat                 uint8
	BitDepthLuma, BitDepthChroma uint8
	// LengthSize is the size in bytes of each NAL unit's length prefix:
	// lengthSizeMinusOne + 1: 1, 2 or 4 (3 is reserved).
	LengthSize int
	// VPS, SPS and PPS are the parameter sets of the record's arrays, as NAL
	// units without a prefix. Arrays of any other NAL unit type (SEI) are
	// skipped.
	VPS, SPS, PPS [][]byte
}

HVCConfig is an HEVCDecoderConfigurationRecord, the content of the hvcC box (ISO/IEC 14496-15, clause 8.3.3). The standard's text is paywalled and was not opened; the layout is that of mp4ff v0.62.0, hevc/hevcdecoderconfigurationrecord.go (DecodeHEVCDecConfRec and the DecConfRec type), which was.

func ParseHVCC added in v0.3.0

func ParseHVCC(b []byte) (HVCConfig, error)

ParseHVCC parses an HEVCDecoderConfigurationRecord. Every length it reads is checked against what is left of the record before anything is kept, so a record that announces more than it holds is refused, never trusted.

type PosterConfig added in v0.2.0

type PosterConfig struct {
	Config
	// MaxSamples is the most samples decoded for the picture: the run from the
	// keyframe it starts at, up to the sample that outputs it, which can come
	// several samples after the sample holding it when the stream reorders.
	MaxSamples int
	// MaxTrackSamples is the longest sample list accepted.
	MaxTrackSamples int
}

PosterConfig bounds the work PosterFrame does for one picture, on top of the decoder's own Config. A zero field takes its default.

Directories

Path Synopsis
tools module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL