ext

Documentation

Index

• Constants
• Variables
• type RtpExtFrameInfo
  • func (ext *RtpExtFrameInfo) Marshal() ([]byte, error)
  • func (ext *RtpExtFrameInfo) String() string
  • func (ext *RtpExtFrameInfo) Unmarshal(data []byte) error
• type RtpExtResolution
  • func (ext *RtpExtResolution) Marshal() ([]byte, error)
  • func (ext *RtpExtResolution) String() string
  • func (ext *RtpExtResolution) Unmarshal(data []byte) error
• type RtpMetadata
  • func DecodeAudioMetadata(rtpPacket *rtp.Packet) (*RtpMetadata, error)
  • func DecodeScreenShareMetadata(rtpPacket *rtp.Packet) (*RtpMetadata, error)
  • func DecodeVideoMetadata(rtpPacket *rtp.Packet) (*RtpMetadata, error)

Constants

const (
	RTP_EXTENSION_ID_UUID                   = 7
	RTP_EXTENSION_ID_AUDIO_IV               = 6 // warning: extension id 6 is re-used
	RTP_EXTENSION_ID_SCREENSHARE_RESOLUTION = 6 // warning: extension id 6 is re-used
	RTP_EXTENSION_ID_SCREENSHARE_FRAME_INFO = 4 // Screen share
	RTP_EXTENSION_ID_VIDEO_UNKNOWN_1        = 1 // always 4ff770 or 400000
	RTP_EXTENSION_ID_VIDEO_FRAME_INFO       = 3 // Some counters that increase
	RTP_EXTENSION_ID_VIDEO_UNKNOWN_5        = 5 // always 00
	RTP_EXTENSION_ID_VIDEO_UNKNOWN_7        = 7 // always 00
	RTP_EXTENSION_UNKNOWN                   = 1
)

Variables

var (
	ErrNoData        = errors.New("no data was passed")
	ErrInvalidLength = errors.New("invalid length")
)

Types

type RtpExtFrameInfo

type RtpExtFrameInfo struct {
	Version       uint8
	Start         bool
	End           bool
	Independent   bool
	Required      bool
	Base          bool
	TemporalID    uint8
	CurrentFrame  uint16
	PreviousFrame uint16
	BaseFrame     uint16
}

H264 SVC layer / or frame information

Zoom uses a mangled form of "Frame Marking RTP Header Extension" https://www.ietf.org/id/draft-ietf-avtext-framemarking-13.html

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ID=? | L=1 |S|E|I|D|B| TID | LID | (TL0PICIDX omitted) +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The most obvious distinction between the spec and Zoom's implementation is that theirs contains additional data that contains 3 frame counters. While the exact names and uses for these frame counters is not known, they do behave very consistently which allows us to make a guess at what the might be doing.

I took the liberty to name these frame counters, giving a bit of meaning: 1. Current Frame: increases on every frame (fragmented packets contain the same counter) 2. Previous Frame: seems to always indicate the previous frame, but this may indicate the temporal layer 3. Base Frame: seems to only change when serious changes occur such as a lot of movement or screen size changes

Eitherway, the exact meaning of these frame counters is currently unknown.

Q: Why is there a need to include this fragmentation info in an extension header? A?: Maybe in an e2ee chat we can't distinguish keyframes from other frames so

they are marked in order that the SFU can figure which frames to send

[2 192 0 114 0 113 0 1] => [ 2 = version?

192 	= pkt info?
0 114 	= current frame/layer counter
0 113 	= previous frame/layer counter
0 1		= base reference frame counter

]

Fragmented:

10  11 1000 184 -> start IDR frame
00  11 1000 56 -> continuation IDR bits (probably last bits)
01  11 1000 120 -> end IDR frame

// This can't only be for pred frames, also gets used for IDR
10  10 0100 164 -> start PRED frame
00	10 0100 36 -> continuation PRED frame
01	10 0100 100 -> end PRED frame

// Med intensity changes
10 00 0000  128 -> start
00 00 0000	0 	-> continuation
01 00 0000  64  -> end

// High intensity changes
10	10 0000 160 -> start
00	10 0000 32 ->  continue single (probably predicted frames)
01	10 0000 96 ->   end

Unfragmented:

11  00 0000 192 -> single (probably predicted frames)

pkt_info bytes [S, E, I, R, B, T, ??, ??]
S = start
E = end
I = independent frame
R = required (seems to be inverted)
B = Base Layer Sync = 1 if only depends on base layer (resolution changes SPS PPS, I frames)
T = Temporal Layer Sync = 1 if P frame

baseFrame only changes if there have been spatial changes (screen resolution)

rtp id=[3] version=2 pktInfo=192 curFrame=2568 prevFrame=2567 baseFrame=229 width=1920 height=924
rtp id=[3] version=2 pktInfo=184 curFrame=2569 prevFrame=2569 baseFrame=2569 width=1150 height=914

func (*RtpExtFrameInfo) Marshal

func (ext *RtpExtFrameInfo) Marshal() ([]byte, error)

func (*RtpExtFrameInfo) String

func (ext *RtpExtFrameInfo) String() string

func (*RtpExtFrameInfo) Unmarshal

func (ext *RtpExtFrameInfo) Unmarshal(data []byte) error

type RtpExtResolution

type RtpExtResolution struct {
	Width  uint16
	Height uint16
}

func (*RtpExtResolution) Marshal

func (ext *RtpExtResolution) Marshal() ([]byte, error)

func (*RtpExtResolution) String

func (ext *RtpExtResolution) String() string

func (*RtpExtResolution) Unmarshal

func (ext *RtpExtResolution) Unmarshal(data []byte) error

type RtpMetadata

type RtpMetadata struct {
	StreamId                    []byte
	ScreenShareResolution       *RtpExtResolution
	ScreenShareFrameInfo        *RtpExtFrameInfo
	AudioHeaderEncryptedPayload []byte
}

func DecodeAudioMetadata

func DecodeAudioMetadata(rtpPacket *rtp.Packet) (*RtpMetadata, error)

func DecodeScreenShareMetadata

func DecodeScreenShareMetadata(rtpPacket *rtp.Packet) (*RtpMetadata, error)

func DecodeVideoMetadata

func DecodeVideoMetadata(rtpPacket *rtp.Packet) (*RtpMetadata, error)