h265

README

h265

Pure-Go (CGO=0) reader for the H.265/HEVC bitstream: the NAL layer, the parameter sets, the slice segment header, and the boundaries between coded pictures.

units, err := h265.SplitAnnexB(data)
sps, err := h265.ParseSPS(spsUnit)
pps, err := h265.ParsePPS(ppsUnit)
pics, err := h265.SplitPictures(units, pps)   // without decoding any

What it is not

It does not decode pictures. There is no CABAC, no residual, no transform, no prediction and no deblocking — nothing here turns a bitstream into pixels. It reads the syntax a caller needs in order to describe a track, cut a stream on picture boundaries, or hand slices to something that does decode.

What is in it

SplitAnnexB, SplitLengthPrefixed, Unit, UnitType	the NAL layer, both framings
ParseSPS, ParsePPS	the parameter sets
ParseSliceSegmentHeader, SliceSegmentHeader, SliceType	the slice segment header
SplitPictures, Picture	coded pictures, from first_slice_segment_in_pic_flag
POCCounter, POC	picture order counts, clause 8.3.1
ShortTermRPS, RefPic, LongTermRefPic	the reference picture sets a sequence carries
ShortHeaderError	a unit that ended inside the syntax, said as such

⛔ HEVC's slice types are not H.264's. Here B is 0, P is 1 and I is 2 — the reverse of H.264, where P is 0 and B is 1. A reader that shares a constant between the two formats reads every B slice as a P slice and every P slice as a B one, and the stream still parses.

The order a viewer sees

A stream sends only the LOW bits of each picture's order count, so the count has to be carried across pictures rather than read from one. POCCounter does that.

⛔ The picture the high bits come from is not simply the previous one. It is the previous picture of sub-layer zero that is neither a leading picture nor a sub-layer non-reference — which is exactly the set a thinned stream may have dropped. A counter that carried from every picture would make the order of what remains depend on what was thrown away.

An IDR states no count at all and is zero by definition; a BLA restarts the high bits, because a broken link is a splice.

The derivation is held against a second implementation: FFmpeg's ff_hevc_compute_poc2, transcribed into the test, over 263 168 carry-and-low -bit pairs at two widths. The one place the two part company is a negative carry — FFmpeg reconstructs the previous low bits as a remainder, which is negative there, and 84 of 1024 negative cases disagree. 8.3.1 keeps the low and high parts apart and the low part is what the slice header stated, so that is what this keeps.

The pictures a picture may still need

A sequence parameter set carries the short-term reference picture sets its slices may name. SPS.ShortTermRefPicSets holds them in order, each as the pictures already seen (Before) and those still to come (After), nearest first.

⛔ A set may be written as a DIFFERENCE from an earlier one, and the difference cannot even have its bits counted without the set it is a difference from — so the sets are read in order and every one is kept. The smallest difference the syntax can state is one: abs_delta_rps_minus1 has no way to say "the same set", so a predicted set always moves every entry, and an entry that crosses zero changes which half it belongs to.

⛔ Everything between the order-count width and the sets is conditional and of variable length — sub-layer buffering, block and transform geometry, the scaling lists, the PCM fields. A reader that walked any of them wrongly would read the sets from the wrong bits, and nothing in a set would show it. Each shape has a test that reads a known set back out of it.

The scaling lists are skipped with the reader the picture parameter set already had: a second copy of that walk would be a second thing to drift.

A slice header may also state a set of its own, with its own way of saying which earlier set it differs from. That is not here yet.

A picture may be carried by several slice segments, and only the one with first_slice_segment_in_pic_flag set begins a new picture. That is what SplitPictures counts, which is how a stream's pictures are counted without decoding any of them.

Sibling

go-avkit/h264 is the same layer for H.264/AVC, and goes further: it also carries the derivations of clause 8.2 (picture order counts, the reference set, the reference lists, the prediction weights). HEVC's equivalents are not here yet.

Windows, macOS, Linux; six 64-bit architectures. 100% statement coverage, gated in CI.

Licence

BSD-3-Clause.

Documentation

Overview

Package h265 reads an H.265/HEVC bitstream in pure Go, with no libavcodec linkage and no external binaries.

The framing and the bit reading are go-avkit/bitstream's, shared with H.264: the two formats separate their units identically and differ in how the header bytes of a unit are read -- H.264 spends one and this spends two -- and in what the parameter sets hold. This package is that difference.

Index

• Variables
• type LongTermRefPic
• type POC
• type POCCounter
  • func (c *POCCounter) Next(u Unit, sps SPS, pps PPS) (POC, error)
  • func (c *POCCounter) Reset()
• type PPS
  • func ParsePPS(u Unit) (PPS, error)
• type Picture
  • func SplitPictures(units []Unit, pps PPS) ([]Picture, error)
• type RefPic
• type SPS
  • func ParseSPS(u Unit) (SPS, error)
• type ShortHeaderError
  • func (e *ShortHeaderError) Error() string
• type ShortTermRPS
  • func (s ShortTermRPS) Count() int
• type SliceSegmentHeader
  • func ParseSliceSegmentHeader(u Unit, pps PPS) (SliceSegmentHeader, error)
• type SliceType
  • func (t SliceType) String() string
• type Unit
  • func SplitAnnexB(data []byte) ([]Unit, error)
  • func SplitLengthPrefixed(data []byte, lengthSize int) ([]Unit, error)
  • func (u Unit) Unescape() []byte
• type UnitType
  • func (t UnitType) IsBLA() bool
  • func (t UnitType) IsIDR() bool
  • func (t UnitType) IsLeading() bool
  • func (t UnitType) IsRandomAccess() bool
  • func (t UnitType) IsSliceSegment() bool
  • func (t UnitType) IsSubLayerNonReference() bool

Variables

var (
	// ErrNotPPS means the unit handed over is not a picture parameter set.
	ErrNotPPS = errors.New("h265: unit is not a picture parameter set")
	// ErrUnsupportedPPS means the set does not end where a set must end, so
	// something in it was read wrongly.
	ErrUnsupportedPPS = errors.New("h265: picture parameter set was not consumed exactly")
)

Errors a picture parameter set can be refused with.

var (
	// ErrNotSlice means the unit handed over does not carry a slice segment.
	ErrNotSlice = errors.New("h265: unit does not carry a slice segment")
	// ErrSliceHeader means a segment states something this reader does not follow.
	ErrSliceHeader = errors.New("h265: unsupported slice segment header")
)

Errors a slice segment header can be refused with.

var (
	// ErrNotSPS means the unit handed over is not a sequence parameter set.
	ErrNotSPS = errors.New("h265: unit is not a sequence parameter set")
	// ErrUnsupportedSPS means the set states something this reader does not
	// follow.
	ErrUnsupportedSPS = errors.New("h265: unsupported sequence parameter set")
)

Errors a sequence parameter set can be refused with.

var ErrNoOrder = errors.New("h265: the unit carries no picture order count")

ErrNoOrder means the unit states no order of its own.

Types

type LongTermRefPic

type LongTermRefPic struct {
	POCLSB uint32
	Used   bool
}

LongTermRefPic is one long-term picture a sequence offers to its slices.

type POC

type POC struct {
	// Value is the count itself.
	Value int32
	// LSB is what the slice header stated, before the high bits were carried in.
	LSB uint32
	// Carried says this picture is the one later pictures count from. Only a
	// picture of sub-layer zero that is neither leading nor sub-layer
	// non-reference does that, and a counter that carried from every picture
	// would place everything after the first dropped one wrongly.
	Carried bool
}

POC is a picture's order count: where it belongs in the order a viewer sees, which is not the order it was coded in.

type POCCounter

type POCCounter struct {
	// contains filtered or unexported fields
}

POCCounter derives picture order counts across a coded video sequence.

⛔ It has to be a counter rather than a function because the format sends only the LOW bits of the count. The high bits are carried forward from an earlier picture, and the one they come from is not simply the previous picture: it is the previous one of sub-layer zero that is neither a leading picture nor a sub-layer non-reference. Those are exactly the pictures a thinned stream may have dropped, so counting from them would make the order depend on what was thrown away.

Feed it every coded picture of the sequence, in decode order, one slice per picture: the first slice segment of each. A picture carried by several segments is one picture here.

func (*POCCounter) Next

func (c *POCCounter) Next(u Unit, sps SPS, pps PPS) (POC, error)

Next gives the order count of one coded picture and remembers what the pictures after it will need.

The unit is wanted whole, not just its payload: the count depends on the unit TYPE, which says whether the high bits restart, and on the temporal identifier, which says whether this picture is one later ones count from.

func (*POCCounter) Reset

func (c *POCCounter) Reset()

Reset forgets every picture counted so far. It is what a seek is: the pictures before the new position are not the ones the counts should follow from.

type PPS

type PPS struct {
	ID                     uint32
	SPSID                  uint32
	DependentSlicesEnabled bool
	OutputFlagPresent      bool
	ExtraSliceHeaderBits   uint8
	SignDataHiding         bool
	CABACInitPresent       bool
	NumRefIdxL0            uint32
	NumRefIdxL1            uint32
	InitQP                 int32 // already un-biased
	ConstrainedIntraPred   bool
	TransformSkip          bool
	CUQPDeltaEnabled       bool
	CbQPOffset             int32
	CrQPOffset             int32
	SliceChromaQPOffsets   bool
	WeightedPred           bool
	WeightedBipred         bool
	TransquantBypass       bool
	TilesEnabled           bool
	EntropySyncEnabled     bool
	TileColumns            uint32
	TileRows               uint32
	UniformTileSpacing     bool
	DeblockingOverride     bool
	DeblockingDisabled     bool
	BetaOffsetDiv2         int32
	TCOffsetDiv2           int32
	ListsModification      bool
	SliceHeaderExtension   bool
}

PPS is what a picture parameter set says about the slices that refer to it.

Two of these fields are what a slice segment header cannot be read without: whether a segment may depend on the one before it, and how many bits of the header the stream reserves before the slice type. A reader without them is misaligned from the first field that matters.

func ParsePPS

func ParsePPS(u Unit) (PPS, error)

ParsePPS reads a picture parameter set.

⛔ Unlike H.264's, this needs no other set to be consumed exactly: HEVC states the number of scaling lists by the size class rather than by the chroma format, so the variable parts here depend only on fields the set itself carries.

type Picture

type Picture struct {
	Units []Unit
	Type  SliceType
	// TypeKnown is false when no segment of this picture stated a type, which
	// happens when a stream is joined part way through a picture.
	TypeKnown bool
	Sync      bool
	Bytes     int
}

Picture is one coded picture: the units that carry it, and what it is.

Units are in the order they arrived, parameter sets and all, so a caller writing a sample table has the bytes of a sample and nothing else to gather.

func SplitPictures

func SplitPictures(units []Unit, pps PPS) ([]Picture, error)

SplitPictures groups a stream's units into the pictures they carry.

⛔ Where H.264 needs several fields compared between neighbouring slices to find a boundary, HEVC states it outright: one flag per segment says whether it begins a picture. That is the whole rule here, and it is worth saying that the simpler format is simpler rather than carrying the other one's machinery over.

Parameter sets and delimiters that arrive before a picture's first segment belong to that picture: a decoder must hold them before the segment that uses them.

type RefPic

type RefPic struct {
	// DeltaPOC is the entry's order count minus the current picture's, so it is
	// negative for a picture already seen and positive for one still to come.
	DeltaPOC int32
	Used     bool
}

RefPic is one entry of a short-term reference picture set: where the picture sits relative to the current one, and whether the current one may use it.

A set lists pictures the DECODER must keep, which is not the same as the pictures this one predicts from. An entry that is not Used is kept because a LATER picture will want it, and dropping it would break that one instead.

type SPS

type SPS struct {
	VPSID          uint8
	MaxSubLayers   uint8
	ProfileSpace   uint8
	Tier           uint8
	ProfileIDC     uint8
	LevelIDC       uint8
	ID             uint32
	ChromaFormat   uint8
	SeparatePlanes bool
	CodedWidth     uint32
	CodedHeight    uint32
	WinLeft        uint32
	WinRight       uint32
	WinTop         uint32
	WinBottom      uint32
	Width          uint32
	Height         uint32
	BitDepthLuma   uint8
	BitDepthChroma uint8
	Log2MaxPOCLSB  uint8
	// ShortTermRefPicSets are the sets a slice may name instead of stating one
	// of its own, in the order the sequence lists them. A set may be written as
	// a difference from an earlier one, so the order is load-bearing.
	ShortTermRefPicSets []ShortTermRPS
	// LongTermRefPics are the long-term pictures the sequence offers, named by
	// the low bits of their order count.
	LongTermRefPics []LongTermRefPic
}

SPS is what a sequence parameter set says about the pictures that follow.

Width and Height are what a player shows and what a container states: the coded size less the conformance window. Unlike H.264 the coded size is stated in luma samples directly rather than in macroblocks, so there is no multiply -- but the window still has to be taken off, and its offsets are in CHROMA units.

func ParseSPS

func ParseSPS(u Unit) (SPS, error)

ParseSPS reads a sequence parameter set.

type ShortHeaderError

type ShortHeaderError struct {
	Unit  int
	Bytes int
}

ShortHeaderError means a unit is too short to hold the two bytes of header this format spends.

func (*ShortHeaderError) Error

func (e *ShortHeaderError) Error() string

type ShortTermRPS

type ShortTermRPS struct {
	Before []RefPic // negative deltas, nearest first: -1, -2, ...
	After  []RefPic // positive deltas, nearest first: 1, 2, ...
}

ShortTermRPS is a short-term reference picture set: the pictures a coded picture may still need, stated relative to its own order count.

The two halves are kept apart because the format states them apart and because they mean different things: Before holds pictures already decoded, After holds pictures that follow in order but precede this one in the stream. Both are ordered by distance, nearest first.

func (ShortTermRPS) Count

func (s ShortTermRPS) Count() int

Count is how many pictures the set names, which is the NumDeltaPocs the next set may predict from.

type SliceSegmentHeader

type SliceSegmentHeader struct {
	First        bool // first_slice_segment_in_pic_flag
	Dependent    bool // only meaningful when First is false
	PPSID        uint32
	Type         SliceType
	TypeRead     bool
	RandomAccess bool // the unit type says this picture can stand alone
	// NoOutputOfPriorPics is stated only by a random access picture.
	NoOutputOfPriorPics bool
}

SliceSegmentHeader is the beginning of a slice segment: which picture it belongs to, whether it starts one, and -- when it starts one -- how that picture is coded.

⛔ Type is read only for a segment that BEGINS a picture, and TypeRead says so. Reaching the type of a later segment means first reading its address, whose width comes from the coding tree block size in the sequence set -- a field this package does not read yet. Reporting a type it had not read would be worse than saying it did not: a picture's type would then come from whichever segment was looked at.

func ParseSliceSegmentHeader

func ParseSliceSegmentHeader(u Unit, pps PPS) (SliceSegmentHeader, error)

ParseSliceSegmentHeader reads the beginning of a slice segment.

It needs the picture parameter set, which states whether a segment may depend on the one before it and how many bits the stream reserves before the slice type. Both change where the type sits, so a reader without them is misaligned from the first field that matters.

type SliceType

type SliceType uint8

SliceType is how a slice is coded. HEVC numbers them the other way round from H.264, and naming them here rather than reusing a number is the point.

const (
	SliceB SliceType = 0
	SliceP SliceType = 1
	SliceI SliceType = 2
)

The slice types.

func (SliceType) String

func (t SliceType) String() string

String names a slice type the way tools report picture types.

type Unit

type Unit struct {
	Type UnitType
	// LayerID is which layer of a scalable stream this belongs to. A plain
	// stream states zero throughout, and a decoder that ignored it would mix the
	// layers of one that does not.
	LayerID uint8
	// TemporalID is the sub-layer, one more than the field on the wire. A stream
	// can be thinned by dropping the higher ones, which is what it is for.
	TemporalID uint8
	Payload    []byte
}

Unit is one NAL unit: its two header bytes read, and its payload still escaped.

Payload is a window into the caller's bytes and is not copied.

func SplitAnnexB

func SplitAnnexB(data []byte) ([]Unit, error)

SplitAnnexB finds the NAL units of a byte stream and reads each one's header.

func SplitLengthPrefixed

func SplitLengthPrefixed(data []byte, lengthSize int) ([]Unit, error)

SplitLengthPrefixed finds the units of the form an MP4 sample carries, where each is preceded by its length in lengthSize bytes, and reads each one's header.

lengthSize comes from the hvcC record.

func (Unit) Unescape

func (u Unit) Unescape() []byte

Unescape undoes the escaping inside this unit's payload.

type UnitType

type UnitType uint8

UnitType is what a NAL unit holds. Only the ones this package acts on are named; the rest travel as their number.

const (
	UnitTrailN    UnitType = 0
	UnitTrailR    UnitType = 1
	UnitTSAN      UnitType = 2
	UnitTSAR      UnitType = 3
	UnitSTSAN     UnitType = 4
	UnitSTSAR     UnitType = 5
	UnitRADLN     UnitType = 6
	UnitRADLR     UnitType = 7
	UnitRASLN     UnitType = 8
	UnitRASLR     UnitType = 9
	UnitBLAWLP    UnitType = 16
	UnitBLAWRADL  UnitType = 17
	UnitBLANLP    UnitType = 18
	UnitIDRWRADL  UnitType = 19
	UnitIDRNLP    UnitType = 20
	UnitCRA       UnitType = 21
	UnitVPS       UnitType = 32
	UnitSPS       UnitType = 33
	UnitPPS       UnitType = 34
	UnitAUD       UnitType = 35
	UnitEOS       UnitType = 36
	UnitEOB       UnitType = 37
	UnitFD        UnitType = 38
	UnitPrefixSEI UnitType = 39
	UnitSuffixSEI UnitType = 40
)

The unit types a reader has to tell apart.

⛔ A picture's type is not one flag here as it is in H.264. HEVC names sixteen kinds of slice segment, and which of them can be decoded without anything before it is what a sample table needs: the ones from BLA_W_LP to CRA are exactly that set, and IsRandomAccess is where the boundary is written down rather than scattered.

func (UnitType) IsBLA

func (t UnitType) IsBLA() bool

IsBLA says the stream was spliced here. A broken link picture is decoded as a random access point whose leading pictures may refer to material that is not there, and its order count restarts: the high bits are taken as zero however far the count had got.

func (UnitType) IsIDR

func (t UnitType) IsIDR() bool

IsIDR says the picture starts a new coded video sequence and states no order of its own: an IDR carries no slice_pic_order_cnt_lsb at all, and its count is zero by definition.

func (UnitType) IsLeading

func (t UnitType) IsLeading() bool

IsLeading says the picture precedes its random access point in output order while following it in decode order -- RADL if it can be decoded, RASL if it refers to material before the access point and so may not be.

func (UnitType) IsRandomAccess

func (t UnitType) IsRandomAccess() bool

IsRandomAccess says whether a picture of this type can be decoded with nothing before it, which is what a sample table calls a sync sample.

func (UnitType) IsSliceSegment

func (t UnitType) IsSliceSegment() bool

IsSliceSegment says whether a unit carries coded picture data.

func (UnitType) IsSubLayerNonReference

func (t UnitType) IsSubLayerNonReference() bool

IsSubLayerNonReference says no picture of the same sub-layer uses this one for prediction, which is what makes a stream thinnable: these can be dropped without disturbing anything left. The even types below 15 are the ones.