Documentation
¶
Overview ¶
Package decode turns the coded video samples avkit's container package demuxes into pictures.
The first codec is H.264 (ITU-T H.264 | ISO/IEC 14496-10), through github.com/go-avkit/h264/decoder (a decoder imported from oops1/go.264), the second HEVC (ITU-T H.265), through github.com/go-avkit/h265 (written from the specification): pure Go decoders that are checked here, picture for picture, against ffmpeg. See the README for the profiles that claim holds for and the conformance bitstreams it was measured on.
Input is what a container hands over: an AVCDecoderConfigurationRecord or an HEVCDecoderConfigurationRecord (ISO/IEC 14496-15) or the parameter sets of a container.TrackConfig, then one length-prefixed access unit per sample. Output is every picture in display order as planar YUV, with its chroma format, bit depth and presentation time.
Index ¶
- Constants
- Variables
- func FirstAudioTrack(r *container.Reader) (uint32, error)
- func FirstVideoTrack(r *container.Reader) (uint32, error)
- type AVCConfig
- type Audio
- type AudioConfig
- type AudioDecoder
- type ChannelLayout
- type Chroma
- type CodecError
- type Config
- type Decoder
- type Frame
- type HVCConfig
- type PosterConfig
- type SampleFormat
Constants ¶
const ( LayoutMono = FrontCenter LayoutStereo = FrontLeft | FrontRight Layout3 = LayoutStereo | FrontCenter LayoutQuad = LayoutStereo | BackLeft | BackRight Layout5 = Layout3 | SideLeft | SideRight Layout51 = Layout5 | LowFrequency Layout61 = Layout3 | LowFrequency | BackCenter | SideLeft | SideRight Layout71 = Layout51 | BackLeft | BackRight // The back variants are what Vorbis names for five and six channels. Layout5Back = Layout3 | BackLeft | BackRight Layout51Back = Layout5Back | LowFrequency )
Common layouts.
const ( // DefaultMaxSampleBytes is shared with video: the largest coded sample. // DefaultMaxAudioChannels is eight, the most FLAC and MPEG audio can have; // Vorbis and Opus allow more and are bounded by it. DefaultMaxAudioChannels = 8 // DefaultMaxBlockSamples is 65535, the largest block a FLAC frame states, per // channel: far above any other codec here (Opus 5760, Vorbis 8192, MP3 1152). DefaultMaxBlockSamples = 65535 // DefaultMaxAudioOutput is 128 MiB, the most PCM one call to DecodeAudioTrack // will return. DefaultMaxAudioOutput = 128 << 20 )
Default audio limits. A stream is untrusted input: what it states about its block size and channel count sizes the buffers a decoder allocates.
const ( // DefaultMaxFrameMBs is 36864 macroblocks, a 4096x2304 picture: level 5.1 // of H.264 Annex A. DefaultMaxFrameMBs = 36864 // DefaultMaxSampleBytes is 64 MiB, far above any single access unit a // level 6.2 stream can carry at its maximum bit rate and frame rate. DefaultMaxSampleBytes = 64 << 20 )
Default limits. A stream is untrusted input: the picture size in its sequence parameter set sizes every buffer the decoder allocates, so it is bounded before the first of them is.
const ( // DefaultMaxPosterSamples is 600 samples, ten seconds of 60 frames per // second video decoded for a single picture. DefaultMaxPosterSamples = 600 // DefaultMaxTrackSamples is 1<<20 samples, about 4.8 hours at 60 frames per // second. DefaultMaxTrackSamples = 1 << 20 )
Default PosterConfig bounds.
Variables ¶
var ( // ErrConfig means the codec configuration (an avcC record or a track's // parameter sets) is malformed. ErrConfig = errors.New("decode: invalid codec configuration") // ErrSample means a sample is not a well formed run of length-prefixed // NAL units. ErrSample = errors.New("decode: invalid sample") // ErrOverLimit means the stream asks for more than Config allows: a // picture larger than MaxFrameMBs, or a sample larger than MaxSampleBytes. // It is raised before anything sized by the stream is allocated. ErrOverLimit = errors.New("decode: stream exceeds the configured limits") // ErrUnsupported means the stream uses a coding tool this decoder does // not implement (see the README): for H.264 interlaced coding, a chroma // format other than 4:2:0, a bit depth above 8, lossless transform bypass; // for HEVC a chroma format other than 4:2:0 or a bit depth other than 8 or // 10 (the decoder reads them, a Frame cannot carry them: the refusal names the // format). A picture larger than // Config.MaxFrameMBs matches ErrUnsupported too, besides ErrOverLimit. ErrUnsupported = errors.New("decode: unsupported coding tool") // ErrClosed means the decoder was used after Close. ErrClosed = errors.New("decode: use of a closed decoder") )
Errors a decoder can refuse with. Each is matched with errors.Is.
var ( // ErrPosterLimit means the picture asked for is further from the keyframe it // is decoded from than PosterConfig.MaxSamples allows, or the track lists // more samples than PosterConfig.MaxTrackSamples. It is raised before the // decoder is built, so a stream with a distant keyframe costs nothing. ErrPosterLimit = errors.New("decode: poster frame exceeds the configured work limit") // ErrTime means the time asked for is negative or not a number. ErrTime = errors.New("decode: invalid poster time") // ErrNoVideo means the file has no video track. ErrNoVideo = errors.New("decode: no video track") )
Errors PosterFrame refuses with, besides the ones above.
var ErrAudioConfig = errors.New("decode: invalid audio configuration")
ErrAudioConfig means the codec configuration of an audio track (a FLAC STREAMINFO, a Vorbis or Opus header) is malformed or disagrees with the track.
var ErrAudioSample = errors.New("decode: invalid audio sample")
ErrAudioSample means an audio sample is not a well formed packet of its codec.
var ErrAudioUnsupported = errors.New("decode: unsupported audio codec")
ErrAudioUnsupported means the track's codec is one this package does not decode, or the stream uses a feature of it that the decoder behind it does not.
var ErrBitstream = errors.New("decode: malformed bitstream")
ErrBitstream means the HEVC bitstream itself is malformed: a parameter set, slice header or slice data the decoder cannot read. A bitstream the decoder can read but does not implement is ErrUnsupported instead.
var ErrEditList = errors.New("decode: edit list not supported")
ErrEditList means the track's edit list says something this package does not apply, and applying part of it would present other pictures than the file states: a media rate other than 1, an edit that lasts to the end of the media (duration 0), an empty edit after the first edit that shows media, or a time that does not fit. The message names the edit. Set Config.IgnoreEditList to decode the track without its edit list.
var ErrNoAudio = errors.New("decode: no audio track")
ErrNoAudio means a file has no audio track.
Functions ¶
func FirstAudioTrack ¶ added in v0.4.0
FirstAudioTrack returns the ID of the first audio track of a demuxed file.
Types ¶
type AVCConfig ¶
type AVCConfig struct {
// Profile, Compatibility and Level are AVCProfileIndication,
// profile_compatibility and AVCLevelIndication, the first three bytes of
// the SPS after the NAL header.
Profile, Compatibility, Level uint8
// LengthSize is the size in bytes of each NAL unit's length prefix:
// lengthSizeMinusOne + 1: 1, 2 or 4 (3 is reserved).
LengthSize int
// SPS and PPS are the parameter sets, as NAL units without a prefix.
SPS, PPS [][]byte
}
AVCConfig is an AVCDecoderConfigurationRecord, the content of the avcC box (ISO/IEC 14496-15). The standard's text is paywalled and was not opened; the layout is that of mp4ff v0.62.0, avc/avcdecoderconfigurationrecord.go, which was.
func ParseAVCC ¶
ParseAVCC parses an AVCDecoderConfigurationRecord. It reads the record only as far as the decoder needs: the extension fields a High profile record appends (chroma format, bit depths, SPS extensions) are bounds-checked but not kept, since the sequence parameter set itself is authoritative.
type Audio ¶ added in v0.4.0
type Audio struct {
// Data holds the samples, interleaved: sample 0 of every channel, then sample
// 1 of every channel, and so on. A buffer owns its bytes.
Data []byte
// Format is the sample format of Data.
Format SampleFormat
// SampleRate is in Hz: the rate the decoder outputs, which for Opus is 48000
// whatever rate the track was recorded at.
SampleRate int
// Channels is the number of interleaved channels.
Channels int
// Layout names the speakers when the codec does; it is zero for a stream that
// numbers its channels without naming them.
Layout ChannelLayout
// PTS is the presentation time of the first sample, in samples of
// SampleRate: a buffer starting at the first presented sample has PTS 0. The
// samples an encoder's priming put ahead of it are not delivered, and the
// padding at the end of the track is not either.
PTS int64
}
Audio is one buffer of decoded PCM.
func DecodeAudioTrack ¶ added in v0.4.0
DecodeAudioTrack decodes every sample of one audio track of a demuxed file and returns the PCM in presentation order, with the track's start and end trimmed as its container states them (edit list, Opus pre-skip, DiscardPadding). The sample count is meant to be what those edit list and gapless metadata define, measured in the output rate, and is not yet in one case, a defect of this package: Matroska priming and trimming are applied in milliseconds (go-avkit/decode#54).
An MP4's iTunSMPB (container.Metadata.Gapless) is applied to an AAC track the way ffmpeg 9.0.2 does (libavformat/mov.c mov_read_custom, libavformat/demux.c), to the first audio track of the file, the one the item describes.
- Start: an encoder delay in (0, 16384) ticks replaces the start the edit list states; the packets the edit list drops whole still count against it, so the track starts at the larger of the delay and those packets. With no edit list it starts at the delay.
- End: when the padding is not 0 and the track's duration (the media header's, or the edit list's total if that is less) is longer, in samples, than both the padding and the original sample count, the track ends at the duration less the padding, in place of the end the last packet's duration gives. Otherwise, as when an edit list already cuts the track to the original count, that end stands.
- The three numbers are ticks of the track's timescale and become samples of the output rate as an edit list's start does, so an HE-AAC track whose timescale is the core rate has them doubled.
NewAudioTrack does not apply it, nor to anything but AAC in an MP4.
It equals the count ffmpeg delivers except where ffmpeg is known to differ, which are deliberate divergences:
- go-avkit/decode#57: an HE-AAC MP4 track whose timescale is the core rate, with an edit list that starts after 0 and no iTunSMPB delay to replace it. The start is in core-rate ticks and is converted to the doubled output rate (1000 ticks at 22050 Hz are 2000 samples at 44100 Hz); ffmpeg 9.0.2 drops 1000 of an implicitly signalled one, so it delivers 1000 more, and of an explicitly signalled one drops the packets before the start whole and none of the one that holds it.
- go-avkit/decode#20: a fragmented MP4, whose edit list ffmpeg does not apply.
- iTunSMPB on the wrong stream: ffmpeg attaches the item to the last stream parsed when it reads it, so in a file whose audio track comes before another track it lands on that one and ffmpeg trims no audio. Here the item trims the first audio track, as it says.
- iTunSMPB before any track: ffmpeg reads the item with no stream yet and ignores it (a moov whose udta precedes the trak); here it applies wherever it is.
The delay's range, 0 to 16384 ticks, and the padding's conditions are ffmpeg's sanity checks on the values and are kept.
type AudioConfig ¶ added in v0.4.0
type AudioConfig struct {
// MaxChannels is the largest channel count accepted.
MaxChannels int
// MaxBlockSamples is the largest block a codec may decode at once, in
// samples per channel. It is checked before the block is allocated.
MaxBlockSamples int
// MaxSampleBytes is the largest coded sample accepted.
MaxSampleBytes int
// MaxOutputBytes bounds the PCM DecodeAudioTrack returns.
MaxOutputBytes int
}
AudioConfig bounds what a stream may make an audio decoder allocate. A zero field takes its default.
type AudioDecoder ¶ added in v0.4.0
type AudioDecoder struct {
// contains filtered or unexported fields
}
AudioDecoder decodes one audio track, sample by sample. It is not safe for concurrent use.
func NewAudioTrack ¶ added in v0.4.0
func NewAudioTrack(tc container.TrackConfig, cfg AudioConfig) (*AudioDecoder, error)
NewAudioTrack returns a decoder for an audio track of a demuxed file. The codec is named by tc.Codec ("fLaC" for FLAC); its configuration comes from tc.CodecConfig. The start of the track is trimmed as the track states it: its edit list (MP4, or the one avkit states for an MP3's LAME tag) or the pre-skip of Opus, which is what ffmpeg does. StartDelay alone drops nothing: ffmpeg treats a Matroska CodecDelay as a timestamp shift.
func (*AudioDecoder) Close ¶ added in v0.4.0
func (d *AudioDecoder) Close() error
Close releases the decoder.
func (*AudioDecoder) DecodeSample ¶ added in v0.4.0
func (d *AudioDecoder) DecodeSample(s container.Sample) ([]Audio, error)
DecodeSample decodes one container sample and returns the PCM it yields, after the start of the track has been trimmed.
func (*AudioDecoder) Flush ¶ added in v0.4.0
func (d *AudioDecoder) Flush() ([]Audio, error)
Flush returns what the decoder still holds. Call it at the end of the stream.
type ChannelLayout ¶ added in v0.4.0
type ChannelLayout uint32
ChannelLayout is a set of speaker positions, one bit per position in the order of the Microsoft WAVEFORMATEXTENSIBLE channel mask (and ffmpeg's AV_CH_*). The channels of an Audio buffer are interleaved in the order of the bits, lowest first, which is also the order FLAC and WAVE use. Vorbis, Opus and MPEG audio state a different order for some layouts; the decoders here reorder to this one.
const ( FrontLeft ChannelLayout = 1 << iota FrontRight FrontCenter LowFrequency BackLeft BackRight FrontLeftOfCenter FrontRightOfCenter BackCenter SideLeft SideRight )
Speaker positions.
func (ChannelLayout) Channels ¶ added in v0.4.0
func (l ChannelLayout) Channels() int
Channels returns how many speaker positions the layout names.
type CodecError ¶ added in v0.9.0
type CodecError struct {
// Codec is the sample entry's four characters.
Codec string
}
CodecError is returned when a track's sample entry names a video codec this package does not decode (VP8, VP9, AV1, MPEG-4 Part 2, ...). It names the codec, as the container states it, so that a sound file is not reported as a corrupt one: the samples are never handed to a parser of another codec. It matches ErrUnsupported with errors.Is; errors.As reaches the fourcc.
func (*CodecError) Error ¶ added in v0.9.0
func (e *CodecError) Error() string
func (*CodecError) Is ¶ added in v0.9.0
func (e *CodecError) Is(target error) bool
Is reports a CodecError as ErrUnsupported.
type Config ¶
type Config struct {
// MaxFrameMBs is the largest picture accepted, in 16x16 macroblocks.
MaxFrameMBs int
// MaxSampleBytes is the largest sample (access unit) accepted.
MaxSampleBytes int
// EnforceLevel also refuses a stream that exceeds its own declared
// level's limits (Annex A), where MaxFrameMBs alone would admit it.
EnforceLevel bool
// IgnoreEditList makes DecodeTrack and PosterFrame label pictures with track
// time, the decoding time of a sample plus its composition offset, and return
// every picture, as ffmpeg's -ignore_editlist 1 does. The default applies the
// track's edit list: pictures it leaves out are decoded but not returned, and
// presentation times are those of the presentation, as ffmpeg does by default.
// The streaming Decoder does not read it.
IgnoreEditList bool
}
Config bounds what a stream may make the decoder allocate. A zero field takes its default.
type Decoder ¶
type Decoder struct {
// contains filtered or unexported fields
}
Decoder decodes one H.264 or HEVC elementary stream. It is not safe for concurrent use.
func New ¶
New returns a decoder for length-prefixed samples with lengthSize-byte lengths (1, 2 or 4 as avcC allows; MP4 and Matroska use 4). The parameter sets must then arrive in-band, or through Config via NewAVCC or NewTrack.
func NewHEVC ¶ added in v0.3.0
NewHEVC returns an HEVC (H.265) decoder for length-prefixed samples with lengthSize-byte lengths (1, 2 or 4). The parameter sets must then arrive in-band, or through NewHVCC or NewTrack. For HEVC, Config.MaxFrameMBs counts 256 luma samples (a 16x16 block, as for H.264) and Config.EnforceLevel has no effect.
func NewHVCC ¶ added in v0.3.0
NewHVCC returns an HEVC decoder configured by an hvcC record, as MP4 stores it. Samples are length-prefixed with the record's length size.
func NewTrack ¶
func NewTrack(tc container.TrackConfig, cfg Config) (*Decoder, error)
NewTrack returns a decoder for an avkit container track: an HEVC decoder for a track whose sample entry is hvc1 or hev1, an H.264 decoder for avc1 to avc4 (or no codec stated). Any other sample entry is refused by name with a *CodecError. Samples are length-prefixed with four-byte lengths, which is what container.Reader hands over for MP4 and Matroska. The track's parameter sets are decoded first.
func (*Decoder) DecodeAnnexB ¶
DecodeAnnexB decodes part of a raw Annex B byte stream (start code delimited, as in a .264 file). The pictures' PTS is their index in display order. The chunk need not end on a NAL unit boundary.
func (*Decoder) DecodeSample ¶
DecodeSample decodes one access unit in length-prefixed form, which is container.Sample.Data, and returns the pictures that became ready, in display order. pts is the sample's presentation time; it is attached to the pictures in increasing order as they come out, so it need not belong to the picture the sample itself contains. A picture can come out several samples late.
type Frame ¶
type Frame struct {
// Y, Cb and Cr are the planes. A sample is one byte when BitDepth is 8
// and a little-endian 16-bit word above that. A Chroma400 frame has no
// Cb or Cr.
Y, Cb, Cr []byte
// YStride and CStride are the plane row lengths in bytes.
YStride, CStride int
// Width and Height are the luma size in samples.
Width, Height int
// Chroma is the chroma sampling.
Chroma Chroma
// BitDepth is the luma bit depth.
BitDepth int
// PTS is the presentation time in the track timescale: the decode time of
// the sample plus its composition offset, as DecodeSample and, with
// Config.IgnoreEditList, DecodeTrack and PosterFrame give it; DecodeTrack and
// PosterFrame by default give that time as the track's edit list presents it. For a raw Annex B stream it
// is the picture's index in display order.
PTS int64
// Corrupt reports that the HEVC decoder lost a slice of the picture, failed
// partway through it, or predicted it from a reference the stream never
// delivered. The picture is still returned, as far as it was decoded. It is
// never set for H.264.
Corrupt bool
}
Frame is one decoded picture, cropped to the stream's display window.
The field names and meaning match go-videodsp/yuv.Picture for 8-bit 4:2:0 so that converting is a field-by-field copy. A Frame owns its planes.
func DecodeTrack ¶
DecodeTrack decodes every sample of one video track of a demuxed file and returns the pictures in display order with their presentation times in the track's timescale. It is the whole-file form of NewTrack and DecodeSample.
The track's edit list is applied, as ffmpeg applies it by default (mov.c, mov_fix_index): a picture presented before the media time of the edit or at its end or after is decoded, for it may be a reference, and not returned, and the presentation times are shifted so that the pictures of the edit start at the delay the leading empty edits state, or at 0. Several edits are laid end to end. An edit list this package does not apply is refused with ErrEditList. Config.IgnoreEditList returns every picture at its track time instead.
func PosterFrame ¶ added in v0.2.0
PosterFrame decodes the one picture a track displays at t seconds: the one with the greatest presentation time at or before t, where the presentation time is the one DecodeTrack gives: the sample's decode time plus its composition offset, as the track's edit list presents it (Config.IgnoreEditList leaves the edit list out, and gives the time before it). It returns DecodeTrack's picture and PTS byte for byte. A track whose edit list presents no picture, or states what ErrEditList describes, is refused with ErrEditList.
A t past the end of the track gives the last picture, and a t before the first presentation time gives the first. A negative or NaN t is ErrTime. Decoding starts at the last sample marked Sync at or before the picture's sample (the first sample when none is) and stops as soon as the picture is out, so the work is bounded by cfg.MaxSamples, not by the length of the track (ErrPosterLimit). Only H.264 and HEVC are decoded; another codec is ErrUnsupported.
The track's samples are listed by container.Reader.Samples, which a reader on a file that stays on disk (container.NewFileReader) fetches whole: the limit on their number is checked once they are listed.
type HVCConfig ¶ added in v0.3.0
type HVCConfig struct {
// ProfileSpace, Tier, Profile and Level are general_profile_space,
// general_tier_flag, general_profile_idc and general_level_idc, as the
// sequence parameter set states them.
ProfileSpace uint8
Tier bool
Profile uint8
Level uint8
// ChromaFormat is chroma_format_idc; BitDepthLuma and BitDepthChroma are the
// sample bit depths (the record stores them minus eight).
ChromaFormat uint8
BitDepthLuma, BitDepthChroma uint8
// LengthSize is the size in bytes of each NAL unit's length prefix:
// lengthSizeMinusOne + 1: 1, 2 or 4 (3 is reserved).
LengthSize int
// VPS, SPS and PPS are the parameter sets of the record's arrays, as NAL
// units without a prefix. Arrays of any other NAL unit type (SEI) are
// skipped.
VPS, SPS, PPS [][]byte
}
HVCConfig is an HEVCDecoderConfigurationRecord, the content of the hvcC box (ISO/IEC 14496-15, clause 8.3.3). The standard's text is paywalled and was not opened; the layout is that of mp4ff v0.62.0, hevc/hevcdecoderconfigurationrecord.go (DecodeHEVCDecConfRec and the DecConfRec type), which was.
type PosterConfig ¶ added in v0.2.0
type PosterConfig struct {
Config
// MaxSamples is the most samples decoded for the picture: the run from the
// keyframe it starts at, up to the sample that outputs it, which can come
// several samples after the sample holding it when the stream reorders.
MaxSamples int
// MaxTrackSamples is the longest sample list accepted.
MaxTrackSamples int
}
PosterConfig bounds the work PosterFrame does for one picture, on top of the decoder's own Config. A zero field takes its default.
type SampleFormat ¶ added in v0.4.0
type SampleFormat uint8
SampleFormat is the format of the samples of an Audio buffer. All are little-endian and interleaved, which is ffmpeg's s16le, s32le and f32le.
const ( // SampleS16 is signed 16-bit integer samples. SampleS16 SampleFormat = iota + 1 // SampleS32 is signed 32-bit integer samples. A source of fewer bits is // left-justified: a 24-bit FLAC sample is shifted up by eight, as ffmpeg does. SampleS32 // SampleF32 is IEEE 754 single precision samples, nominally in [-1, 1]. SampleF32 )
The sample formats a decoder produces.
func (SampleFormat) Size ¶ added in v0.4.0
func (f SampleFormat) Size() int
Size returns the bytes one sample takes, 0 for a format that is not one of the constants above.
func (SampleFormat) String ¶ added in v0.4.0
func (f SampleFormat) String() string
String names the format as ffmpeg does.
Source Files
¶
- aac.go
- aac_bits.go
- aac_config.go
- aac_dequant.go
- aac_huff.go
- aac_mdct.go
- aac_ps.go
- aac_ps_hybrid.go
- aac_ps_mix.go
- aac_ps_params.go
- aac_ps_remap.go
- aac_ps_tables.go
- aac_sbr.go
- aac_sbr_adj.go
- aac_sbr_freq.go
- aac_sbr_hf.go
- aac_sbr_noise_table.go
- aac_sbr_qmf.go
- aac_sbr_qmf_tables.go
- aac_sbr_tables.go
- aac_syntax.go
- aac_synth.go
- aac_tables.go
- audio.go
- audio_mp4a.go
- avcc.go
- decode.go
- editlist.go
- flac.go
- hevc.go
- hvcc.go
- mp3.go
- opus.go
- pcm.go
- poster.go