[BUG] Invalid line in SDP answer when attempting to send multiple video tracks

[atyshka]

Describe the bug
I am attempting to add two different video tracks to the same streaming session, which I assume would be a valid use case, i.e. for sharing both a user's camera and screen share. When doing so, the browser fails to connect on the client because of an invalid SDP answer. Inspecting the generated SDP answer, I notice that when using a single video track or a single video and audio track, the SDP is valid, but when attempting to use two video tracks the SDP contains an invalid line which is blank.

Here is the SDP answer generated for a single video stream, no audio:

"v=0
o=- 1090333328 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0
a=msid-semantic: WMS myKvsVideoStream
m=video 9 UDP/TLS/RTP/SAVPF 125
c=IN IP4 127.0.0.1
a=candidate:1 1 udp 1694498815 192.168.1.29 55309 typ srflx raddr 0.0.0.0 rport 0 generation 0 network-cost 999
a=candidate:0 1 udp 2130706431 192.168.1.29 55370 typ host raddr 0.0.0.0 rport 0 generation 0 network-cost 999
a=msid:myKvsVideoStream /vehicle/sensors/camera/flir/front_cam/encoded
a=ssrc:2102704195 cname:qOWoEx7L1mTgS1zS
a=ssrc:2102704195 msid:myKvsVideoStream /vehicle/sensors/camera/flir/front_cam/encoded
a=ssrc:2102704195 mslabel:myKvsVideoStream
a=ssrc:2102704195 label:/vehicle/sensors/camera/flir/front_cam/encoded
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:+sRL
a=ice-pwd:76e7fiaJ0x5a7kQaW/k3I/j5
a=ice-options:trickle
a=fingerprint:sha-256 5E:53:77:77:C8:C1:59:4A:2D:B1:75:07:A2:16:E3:51:DC:A5:A2:E8:62:81:24:13:AE:BF:36:F6:4E:03:9F:B9
a=setup:active
a=mid:0
a=sendonly
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:125 H264/90000
a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtcp-fb:125 nack
a=rtcp-fb:125 transport-cc
"

Here is the answer generated for two video streams:

"v=0
o=- 1578931153 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1
a=msid-semantic: WMS myKvsVideoStream
m=video 9 UDP/TLS/RTP/SAVPF 125
c=IN IP4 127.0.0.1
a=candidate:1 1 udp 1694498815 192.168.1.29 46641 typ srflx raddr 0.0.0.0 rport 0 generation 0 network-cost 999
a=candidate:0 1 udp 2130706431 192.168.1.29 51889 typ host raddr 0.0.0.0 rport 0 generation 0 network-cost 999
a=msid:myKvsVideoStream /vehicle/sensors/camera/flir/front_cam/encoded
a=ssrc:1393935670 cname:NrITmrKOkoBmqqgu
a=ssrc:1393935670 msid:myKvsVideoStream /vehicle/sensors/camera/flir/front_cam/encoded
a=ssrc:1393935670 mslabel:myKvsVideoStream
a=ssrc:1393935670 label:/vehicle/sensors/camera/flir/front_cam/encoded
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:6wZt
a=ice-pwd:evUp22ew3I+p03sugvEpGW4l
a=ice-options:trickle
a=fingerprint:sha-256 54:89:77:FC:8E:8E:00:98:1D:B3:B2:42:89:76:91:18:06:CF:BE:E9:1B:47:6F:31:F1:B0:70:ED:5B:7F:BB:9C
a=setup:active
a=mid:0
a=sendonly
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:125 H264/90000
a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtcp-fb:125 nack
a=rtcp-fb:125 transport-cc
m=video 9 UDP/TLS/RTP/SAVPF 125
c=IN IP4 127.0.0.1
a=candidate:1 1 udp 1694498815 192.168.1.29 46641 typ srflx raddr 0.0.0.0 rport 0 generation 0 network-cost 999
a=candidate:0 1 udp 2130706431 192.168.1.29 51889 typ host raddr 0.0.0.0 rport 0 generation 0 network-cost 999
a=msid:myKvsVideoStream /vehicle/sensors/camera/flir/right_cam/encoded
a=ssrc:16767695 cname:NrITmrKOkoBmqqgu
a=ssrc:16767695 msid:myKvsVideoStream /vehicle/sensors/camera/flir/right_cam/encoded
a=ssrc:16767695 mslabel:myKvsVideoStream
a=ssrc:16767695 label:/vehicle/sensors/camera/flir/right_cam/encoded
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:6wZt
a=ice-pwd:evUp22ew3I+p03sugvEpGW4l
a=ice-options:trickle
a=fingerprint:sha-256 54:89:77:FC:8E:8E:00:98:1D:B3:B2:42:89:76:91:18:06:CF:BE:E9:1B:47:6F:31:F1:B0:70:ED:5B:7F:BB:9C
a=setup:active
a=mid:1
a=
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:125 H264/90000
a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtcp-fb:125 nack
a=rtcp-fb:125 transport-cc
"

Notice the error on line 46, after a=mid1, there is a blank a=, which the browser cannot parse. I am not sure what this is supposed to be, possibly a=sendonly?

SDK version number
commit b472c1a

To Reproduce
Steps to reproduce the behavior:

1. Start with kvs master example
2. Change audio track to be a second video track
3. Remove code for sending audio frames
4. Call sendFrame twice for each track when sending an h264 frame

Expected behavior
Valid SDP answer sent

Desktop (please complete the following information):

• OS: Linux
• Browser Chrome
• Version 88

[zhuker]

the library doesnt support multiple video streams out of the box
but i patched it like so to enable it

diff --git src/source/Sdp/Sdp.h src/source/Sdp/Sdp.h
index e3a3a11f..50cdedb1 100644
--- src/source/Sdp/Sdp.h
+++ src/source/Sdp/Sdp.h
@@ -62,16 +62,16 @@ extern "C" {

 #define MAX_SDP_TOKEN_LENGTH 128

-#define MAX_SDP_SESSION_BANDWIDTH_COUNT        2
-#define MAX_SDP_SESSION_TIME_DESCRIPTION_COUNT 2
-#define MAX_SDP_SESSION_TIMEZONE_COUNT         2
+#define MAX_SDP_SESSION_BANDWIDTH_COUNT        16
+#define MAX_SDP_SESSION_TIME_DESCRIPTION_COUNT 16
+#define MAX_SDP_SESSION_TIMEZONE_COUNT         16
 /**
  * https://tools.ietf.org/html/rfc4566#section-5.14
  *
  * reserving enough for audio, video, text, application and message for now
  */
-#define MAX_SDP_SESSION_MEDIA_COUNT   5
-#define MAX_SDP_MEDIA_BANDWIDTH_COUNT 2
+#define MAX_SDP_SESSION_MEDIA_COUNT   16
+#define MAX_SDP_MEDIA_BANDWIDTH_COUNT 16

 #define MAX_SDP_ATTRIBUTES_COUNT 128

[zhuker]

note MAX_SDP_SESSION_MEDIA_COUNT

[atyshka]

Thank you @zhuker ! Is there a technical reason this limit is so low by default?

[zhuker]

there's rarely a need to carry multiple video streams over webrtc 🤷

[hassanctech]

@atyshka Is your issue resolved now? For certain embedded devices with very limited space the SDP structures actually use too much memory in fact some of them lower MAX_SDP_SESSION_MEDIA_COUNT to 2, we've met devices that want to have 3 simultaneous viewers but keep the total memory usage under 2-3MB. Since this is an embedded SDK typically running on an embedded device we opt for a low default.

[atyshka]

Yeah I'll say it's resolved. Off topic but @zhuker I see you've been working on this on your branch, have you encountered any difficulties with browsers and multi-video-track? Googling around it seems like Chrome/Firefox have had issues but a lot of those threads are several years old.

[zhuker]

@atyshka yeah we are consistently streaming multiple cams, no issues with chrome (havent tested in firefox but should be ok)

[atyshka]

Hmm, I tried changing those limits in the Sdp header but still getting the same issue with the blank Sdp line... any other changes necessary?

[atyshka]

It seems to be related to having two tracks with the same codec and not a result of track limits. I am modifying the sample, and because I'm removing the audio track and replacing with a video track, I wouldn't expect the preset limits in Sdp.h to be the problem, as I'm still only at two total tracks.

[hassanctech]

@atyshka Yes I also did not understand how changing those limits would in any way impact you but also wasn't sure about your setup. That information is very helpful so you are taking the stock master and replacing the audio track with the video track. Now it seems like when printing out the "direction" isn't getting set so it prints out nothing. From what I can tell it looks like your assertion is correct that the line should be a=sendonly. Can you please also share the offer SDP?

[atyshka]

Here is the offer, I'm not quite sure how to interpret it.

v=0
o=- 3861851306496664494 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0
a=msid-semantic: WMS
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 122 102 121 127 120 125 107 108 109 124 119 123
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:Cc4J
a=ice-pwd:PIHms+AVTXIdTMefmLqXc8pP
a=ice-options:trickle
a=fingerprint:sha-256 CA:03:9D:0A:BD:78:9C:78:FB:C5:4D:FC:DA:75:18:29:EE:F7:9A:D0:50:36:4C:2E:CF:FA:E7:9B:B8:16:A7:20
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 urn:3gpp:video-orientation
a=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=recvonly
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 profile-id=0
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:100 VP9/90000
a=rtcp-fb:100 goog-remb
a=rtcp-fb:100 transport-cc
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 nack pli
a=fmtp:100 profile-id=2
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=100
a=rtpmap:122 VP9/90000
a=rtcp-fb:122 goog-remb
a=rtcp-fb:122 transport-cc
a=rtcp-fb:122 ccm fir
a=rtcp-fb:122 nack
a=rtcp-fb:122 nack pli
a=fmtp:122 profile-id=1
a=rtpmap:102 H264/90000
a=rtcp-fb:102 goog-remb
a=rtcp-fb:102 transport-cc
a=rtcp-fb:102 ccm fir
a=rtcp-fb:102 nack
a=rtcp-fb:102 nack pli
a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
a=rtpmap:121 rtx/90000
a=fmtp:121 apt=102
a=rtpmap:127 H264/90000
a=rtcp-fb:127 goog-remb
a=rtcp-fb:127 transport-cc
a=rtcp-fb:127 ccm fir
a=rtcp-fb:127 nack
a=rtcp-fb:127 nack pli
a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f
a=rtpmap:120 rtx/90000
a=fmtp:120 apt=127
a=rtpmap:125 H264/90000
a=rtcp-fb:125 goog-remb
a=rtcp-fb:125 transport-cc
a=rtcp-fb:125 ccm fir
a=rtcp-fb:125 nack
a=rtcp-fb:125 nack pli
a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtpmap:107 rtx/90000
a=fmtp:107 apt=125
a=rtpmap:108 H264/90000
a=rtcp-fb:108 goog-remb
a=rtcp-fb:108 transport-cc
a=rtcp-fb:108 ccm fir
a=rtcp-fb:108 nack
a=rtcp-fb:108 nack pli
a=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f
a=rtpmap:109 rtx/90000
a=fmtp:109 apt=108
a=rtpmap:124 red/90000
a=rtpmap:119 rtx/90000
a=fmtp:119 apt=124
a=rtpmap:123 ulpfec/90000

I'm creating the offer in the browser like so:

peerConnection.createOffer({
            offerToReceiveVideo: true,
        });

This method is found in the examples for the JS sdk, but apparently it's deprecated, if the problem is with the SDP offer this could be the reason. Mozilla docs say: Note: You shouldn't use this legacy property. Instead, use RTCRtpTransceiver to control whether or not to accept incoming video. I'll look into that approach if it is the SDP offer that is causing the answer to be invalid.

[hassanctech]

Is this the complete offer? There is only 1 m-line here, i.e. only 1 video track. When you send this offer if your response answer has 2 m-lines (even if you created 2 transceivers) this is a bug I will look into that. But on your end what you need to do is on the JS side which you're using to create the "offer" you need to modify that to have 2 video tracks as well -- WebRTC requires the offer/answer to match matching m-lines, so if 1 m-line is sent and 2 m-lines are received it will be rejected. The solution for this for you is the also have 2 video tranceivers/tracks on the JS side and you can set them both to recvonly. Does that make sense?

Yes we are aware of the offerToReceiveVideo being deprecated and have a work item to address that, I cannot promise any timelines for that though if it's the cause of a breakage we will prioritize it.