Skip to content

After testing, the audio can be obtained, but the video frames cannot be obtained #169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
liusuyi2021 opened this issue May 8, 2025 · 9 comments
Open

After testing, the audio can be obtained, but the video frames cannot be obtained #169

liusuyi2021 opened this issue May 8, 2025 · 9 comments
Assignees
@devopvoid

Comments

@liusuyi2021
Copy link

liusuyi2021 commented May 8, 2025 •
edited
Loading

After testing, the audio can be obtained, but the video frames cannot be obtained

            public void onTrack(RTCRtpTransceiver transceiver) {
                MediaStreamTrack track = transceiver.getReceiver().getTrack();
                if (track instanceof VideoTrack) {
                    // 设置到你的渲染器
                    VideoTrack videoTrack = (VideoTrack) track;
                    videoTrack.addSink(videoFrame -> {
                        log.warn("Received video frame: {}X{}",
                                videoFrame.buffer.getWidth(),
                                videoFrame.buffer.getHeight());
                    });
                    // videoTrack.setEnabled(true);
                }
                if (track instanceof AudioTrack) {
                    AudioTrack audioTrack = (AudioTrack) track;
                    //存储音频自定义打开这个注释
                    //audioTrack.addSink(recorder);
                    audioTrack.addSink((data, bitsPerSample, sampleRate, channels, frames) -> {
                        log.warn("Received audio data: {} sampleRate: {} channels: {} frames: {}", data.length,
                                sampleRate, channels, frames);
                    });
                }
            }
@devopvoid
Copy link
Owner

Please make sure you get into the if-condition for the video track and a video stream has actually been negotiated by the peer connection to receive video.

@liusuyi2021
Copy link
Author

I am certain that there is a peer-to-peer connection, and if there is no audio connection, I will not receive data. I am the same as problem # 132

@devopvoid
Copy link
Owner

Your code seems to be this:

RTCRtpTransceiverInit init = new RTCRtpTransceiverInit();
init.direction = RTCRtpTransceiverDirection.RECV_ONLY;

VideoDeviceSource videoSource = new VideoDeviceSource();
VideoTrack videoTrack = peerConnectionFactory.createVideoTrack("video", videoSource);

// 添加transceiver
peerConnection.addTransceiver(videoTrack, init);

Please try doing it this way:

VideoDeviceSource videoSource = new VideoDeviceSource();
VideoTrack videoTrack = peerConnectionFactory.createVideoTrack("video", videoSource);

peerConnection.addTrack(videoTrack, List.of("stream"));

for (RTCRtpTransceiver transceiver : peerConnection.getTransceivers()) {
  	MediaStreamTrack track = transceiver.getSender().getTrack();

	if (nonNull(track) && track.getId().equals("video")) {
		transceiver.setDirection(RTCRtpTransceiverDirection.RECV_ONLY);
		break;
	}
}

For audio you can also use peerConnection.addTrack(audioTrack, List.of("stream"));

@devopvoid
Copy link
Owner

Duplicate of #168

@devopvoid devopvoid marked this as a duplicate of #168 May 8, 2025
@liusuyi2021
Copy link
Author

您的代码似乎是这样的：

RTCRtpTransceiverInit init = new RTCRtpTransceiverInit();
init.direction = RTCRtpTransceiverDirection.RECV_ONLY;

VideoDeviceSource videoSource = new VideoDeviceSource();
VideoTrack videoTrack = peerConnectionFactory.createVideoTrack("video", videoSource);

// 添加transceiver
peerConnection.addTransceiver(videoTrack, init);

请尝试这样做：

VideoDeviceSource videoSource = new VideoDeviceSource();
VideoTrack videoTrack = peerConnectionFactory.createVideoTrack("video", videoSource);

peerConnection.addTrack(videoTrack, List.of("stream"));

for (RTCRtpTransceiver transceiver : peerConnection.getTransceivers()) {
  	MediaStreamTrack track = transceiver.getSender().getTrack();

	if (nonNull(track) && track.getId().equals("video")) {
		transceiver.setDirection(RTCRtpTransceiverDirection.RECV_ONLY);
		break;
	}
}

对于音频，您还可以使用peerConnection.addTrack(audioTrack, List.of("stream"));
Despite using the code you provided, we still cannot receive any video frames

@devopvoid
Copy link
Owner

Since I cannot test your code and setup, there is nothing I can help you with right now. We use this library in production and video is working.

You can try debugging the sent and received session descriptions.

@liusuyi2021
Copy link
Author

liusuyi2021 commented May 10, 2025 •
edited
Loading 

Host: 192.168.1.2:8889
Connection: Keep-Alive
Accept-Encoding: gzip
User-Agent: okhttp/3.14.9


HTTP/1.1 204 No Content
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Authorization, Content-Type, If-Match
Access-Control-Allow-Methods: OPTIONS, GET, POST, PATCH, DELETE
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Link
Server: mediamtx
Date: Sat, 10 May 2025 01:29:02 GMT


POST /62218849d69b4146b29437e16f310231_1/whep HTTP/1.1
Content-Type: application/sdp; charset=utf-8
Content-Length: 4090
Host: 192.168.1.2:8889
Connection: Keep-Alive
Accept-Encoding: gzip
User-Agent: okhttp/3.14.9

v=0
o=- 6409497023764066778 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0
a=extmap-allow-mixed
a=msid-semantic: WMS
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 127 120 125 119 124 107 108 109 123 118 122 117 35 36 114 115 116
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:+PBZ
a=ice-pwd:jzby3KXJ/etHWhO2mDUuPTsd
a=ice-options:trickle
a=fingerprint:sha-256 0D:33:92:63:39:CD:9C:48:9C:67:06:0A:C8:29:EC:68:FE:AC:A3:0C:28:4C:4C:7D:9C:59:E7:C4:71:17:AB:7F
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 urn:3gpp:video-orientation
a=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendrecv
a=msid:- videoTrack
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 profile-id=0
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:100 VP9/90000
a=rtcp-fb:100 goog-remb
a=rtcp-fb:100 transport-cc
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 nack pli
a=fmtp:100 profile-id=2
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=100
a=rtpmap:127 H264/90000
a=rtcp-fb:127 goog-remb
a=rtcp-fb:127 transport-cc
a=rtcp-fb:127 ccm fir
a=rtcp-fb:127 nack
a=rtcp-fb:127 nack pli
a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
a=rtpmap:120 rtx/90000
a=fmtp:120 apt=127
a=rtpmap:125 H264/90000
a=rtcp-fb:125 goog-remb
a=rtcp-fb:125 transport-cc
a=rtcp-fb:125 ccm fir
a=rtcp-fb:125 nack
a=rtcp-fb:125 nack pli
a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f
a=rtpmap:119 rtx/90000
a=fmtp:119 apt=125
a=rtpmap:124 H264/90000
a=rtcp-fb:124 goog-remb
a=rtcp-fb:124 transport-cc
a=rtcp-fb:124 ccm fir
a=rtcp-fb:124 nack
a=rtcp-fb:124 nack pli
a=fmtp:124 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtpmap:107 rtx/90000
a=fmtp:107 apt=124
a=rtpmap:108 H264/90000
a=rtcp-fb:108 goog-remb
a=rtcp-fb:108 transport-cc
a=rtcp-fb:108 ccm fir
a=rtcp-fb:108 nack
a=rtcp-fb:108 nack pli
a=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f
a=rtpmap:109 rtx/90000
a=fmtp:109 apt=108
a=rtpmap:123 H264/90000
a=rtcp-fb:123 goog-remb
a=rtcp-fb:123 transport-cc
a=rtcp-fb:123 ccm fir
a=rtcp-fb:123 nack
a=rtcp-fb:123 nack pli
a=fmtp:123 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d001f
a=rtpmap:118 rtx/90000
a=fmtp:118 apt=123
a=rtpmap:122 H264/90000
a=rtcp-fb:122 goog-remb
a=rtcp-fb:122 transport-cc
a=rtcp-fb:122 ccm fir
a=rtcp-fb:122 nack
a=rtcp-fb:122 nack pli
a=fmtp:122 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=4d001f
a=rtpmap:117 rtx/90000
a=fmtp:117 apt=122
a=rtpmap:35 AV1/90000
a=rtcp-fb:35 goog-remb
a=rtcp-fb:35 transport-cc
a=rtcp-fb:35 ccm fir
a=rtcp-fb:35 nack
a=rtcp-fb:35 nack pli
a=rtpmap:36 rtx/90000
a=fmtp:36 apt=35
a=rtpmap:114 red/90000
a=rtpmap:115 rtx/90000
a=fmtp:115 apt=114
a=rtpmap:116 ulpfec/90000
a=ssrc-group:FID 1678362880 366126444
a=ssrc:1678362880 cname:C8TGLz7UzhKqVgLx
a=ssrc:1678362880 msid:- videoTrack
a=ssrc:1678362880 mslabel:-
a=ssrc:1678362880 label:videoTrack
a=ssrc:366126444 cname:C8TGLz7UzhKqVgLx
a=ssrc:366126444 msid:- videoTrack
a=ssrc:366126444 mslabel:-
a=ssrc:366126444 label:videoTrack

HTTP/1.1 201 Created
Accept-Patch: application/trickle-ice-sdpfrag
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: ETag, ID, Accept-Patch, Link, Location
Content-Type: application/sdp
Etag: *
Id: e8e9641d-96a3-4f21-bd91-b17f69898165
Location: /62218849d69b4146b29437e16f310231_1/whep/5fbb65c3-953a-4e31-8461-14098a6fe9a9
Server: mediamtx
Date: Sat, 10 May 2025 01:29:02 GMT
Content-Length: 1072

v=0
o=- 4806493387780213692 1746840542 IN IP4 0.0.0.0
s=-
t=0 0
a=msid-semantic:WMS*
a=fingerprint:sha-256 36:FE:41:3C:CC:D2:6A:3F:F9:FD:6D:DE:D9:B9:31:4F:72:53:DB:86:D0:42:A9:56:E4:4D:54:4D:E4:F4:21:D7
a=extmap-allow-mixed
a=group:BUNDLE 0
m=video 9 UDP/TLS/RTP/SAVPF 124
c=IN IP4 0.0.0.0
a=setup:active
a=mid:0
a=ice-ufrag:nqjLbfeeSRnOYtvn
a=ice-pwd:IYTqFrViTgRROIOofPNCbyTbwKUiCGGW
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:124 H264/90000
a=fmtp:124 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtcp-fb:124 goog-remb 
a=rtcp-fb:124 transport-cc 
a=rtcp-fb:124 ccm fir
a=rtcp-fb:124 nack 
a=rtcp-fb:124 nack pli
a=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=ssrc:3947837704 cname:mediamtx
a=ssrc:3947837704 msid:mediamtx video
a=ssrc:3947837704 mslabel:mediamtx
a=ssrc:3947837704 label:video
a=msid:mediamtx video
a=sendrecv
a=candidate:1944728210 1 udp 2130706431 192.168.1.2 8189 typ host
a=candidate:1944728210 2 udp 2130706431 192.168.1.2 8189 typ host
a=end-of-candidates

PATCH /62218849d69b4146b29437e16f310231_1/whep/5fbb65c3-953a-4e31-8461-14098a6fe9a9 HTTP/1.1
If-Match: *
Content-Type: application/trickle-ice-sdpfrag; charset=utf-8
Content-Length: 389
Host: 192.168.1.2:8889
Connection: Keep-Alive
Accept-Encoding: gzip
User-Agent: okhttp/3.14.9

a=ice-ufrag:+PBZ
a=ice-pwd:jzby3KXJ/etHWhO2mDUuPTsd
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 127 120 125 119 124 107 108 109 123 118 122 117 35 36 114 115 116
a=mid:0
a=candidate:696122497 1 udp 2122260223 192.168.1.27 65103 typ host generation 0 ufrag +PBZ network-id 1
a=candidate:4021771414 1 udp 2122194687 172.21.176.1 65104 typ host generation 0 ufrag +PBZ network-id 4

HTTP/1.1 204 No Content
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Server: mediamtx
Date: Sat, 10 May 2025 01:29:02 GMT

This is my interactive packet capture content. Through chatgpt inquiry, I learned that the other party's streaming media only supports [H264 profile level id=42e01f] video stream encoding. I suspect that it is because we do not support it that [onTrack] cannot obtain video frames

@devopvoid
Copy link
Owner

This makes sense, since there is currently no h264 codec compiled in the snapshot.

Version 0.10.0 must have this codec.

@liusuyi2021
Copy link
Author

I am using version 0.10.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@devopvoid devopvoid

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@devopvoid @liusuyi2021