Video with WindowCapturer: framerate max out at 30

[arosov]

Describe the bug
Using VideoDesktopSource.setFrameRate to 30 or above only produces a frame every 32ms (ie. 30fps).
Setting it to a lower value than 30 does work as expected.
I'm not even going over the network at this stage, just using a LocalSink.

To Reproduce
Steps to reproduce the behavior:

1. Use WindowCapturer to list windows / desktopsources.
2. Select a window
3. Setup a VideoDesktopSource with that window id and setFrameRate(60 or 120)
4. factory.createVideoTrack(videoDesktopSource)
5. videoTrack.addSink(localSink)
6. Measure time between frames in localSink

Expected behavior
Since I'm testing on Windows and DXGI should be used, the requested framerate should be respected and it's currently not the case.

Desktop (please complete the following information):

• OS: Windows 10

Additional context
The framerate I observe is very regular so it seems highly unlikely that this issue is the result of resource starvation.

Didn't see any relevant information even with webrtc Logging enabled.

It could be a limitation of using a localSink but I didn't see anything pointing that way. Also, since the framerate can be set to less than 30 and the localsink honors it, it doesn't seem to be the problem. Plus, using a localsink seems to be a "normal" testing thing for webrtc, from what I gathered.

At first glance, webrtc-java only uses the framerate to sleep the appropriate amout of time between frames.
So it looks like the issue is internal to webrtc or a limitation from windows capture API.

I looked at https://chromium.googlesource.com/external/webrtc/+/branch-heads/4844/modules/desktop_capture/

Didn't find anything pointing toward a webrtc limitation but I could have missed it.

But on Windows, DXGI should be used and from what I understand it should be fast enough for way more than 30fps.
I wasn't able to ascertain what was the capture API used.

I'm kinda hoping it's using a fallback (GDI ? ) instead of DXGI and that's the issue but this is a wild guess.

[devopvoid]

Hi, I'm currently working on the next release with a more recent version of WebRTC. They have this method for desktop capture:

  // Sets max frame rate for the capturer. This is best effort and may not be
  // supported by all capturers. This will only affect the frequency at which
  // new frames are available, not the frequency at which you are allowed to
  // capture the frames.
  virtual void SetMaxFrameRate(uint32_t /* max_frame_rate */) {}

https://github.com/maitrungduc1410/webrtc/blob/86a736a4a1647532fe1b4c780c2e82baeb373d39/modules/desktop_capture/desktop_capturer.h#L112

The Java DesktopCapturer now has the respective method to set the max framerate.

webrtc-java/webrtc/src/main/java/dev/onvoid/webrtc/media/video/desktop/DesktopCapturer.java

Line 110 in 0d44451

public native void setMaxFrameRate(int maxFrameRate);

But I'm not sure whether this will change anything. Its worth a try, I guess.
You can test this without setting up a peer connection. Simply using the DesktopCapturer class.

[devopvoid]

Just tested this with the DesktopCapturer. The set frame rate seems to be respected. I get every ~17ms (60 fps) a frame. Still have to test with the VideoTrack.

[devopvoid]

<dependency>
	<groupId>dev.onvoid.webrtc</groupId>
	<artifactId>webrtc-java</artifactId>
	<version>0.11.0-SNAPSHOT</version>
</dependency>

[arosov]

Hi, thanks for having a look at this.

Great news that you're working on updating the webrtc base branch \o/

Unfortunately, I don't actually have a build env set up on windows 😩
I may get around to that soon but right now I don't have it, I only have a Linux build env working and I'm doing video things on Windows only (for now at least).

Any chance you can publish the 0.11.0-SNAPSHOT (at least the Windows "part") ? Or is it my mistake and there's actually a specific maven repo to hit to get it ?

Also, just for confirmation, when you tested that, was it under Windows ?

[arosov]

Yet another side question but, the Visual Studio link in your README points to the master branch of webrtc.
That file states it wants Visual Studio 2022.

But your workflows (besides the one for the newer WebRTC) are using Gituhb Action Runner Windows 2019.

Am I correct in assuming that it means Visual Studio 2019 before 0.11.0 and Visual Studio 2022 from now on ?

[devopvoid]

Any chance you can publish the 0.11.0-SNAPSHOT (at least the Windows "part") ?

Below the Maven repository config to get the latest snapshot for all supported systems:

<repository>
	<name>Central Portal Snapshots</name>
	<id>central-portal-snapshots</id>
	<url>https://central.sonatype.com/repository/maven-snapshots/</url>
	<releases>
		<enabled>false</enabled>
	</releases>
	<snapshots>
		<enabled>true</enabled>
	</snapshots>
</repository>

Also, just for confirmation, when you tested that, was it under Windows ?

I tested that on Windows.

[devopvoid]

Am I correct in assuming that it means Visual Studio 2019 before 0.11.0 and Visual Studio 2022 from now on ?

Correct!

I forgot that the README points to the master branch of WebRTC,

[arosov]

Thanks for publishing the snapshot :)

For anyone else reading this, the gradle translation of the Maven bit above:

 repositories {
        maven {
            name = "Central Portal Snapshots"
            url = uri("https://central.sonatype.com/repository/maven-snapshots/")
            metadataSources {
                mavenPom()
                artifact()
            }
            mavenContent {
                snapshotsOnly()
            }
        }
    }

[arosov]

I can't seem to get 60 fps as it is 😖
No behavior change on my end, I do capture and render frames in my app but even if I look at timestamp before rendering I'm still at 30 fps.

But:

• I now have a hint in webrtc's logging that wasn't there before 0.11.0, I double checked.
  • (window_capture_utils.cc:326): Fail to create instance of VirtualDesktopManager
    Single occurence of that log when capture starts.
• We're not doing exactly the same thing, I assumed it was close enough but maybe not.
  • You're using DesktopCapturer and I'm using WindowCapturer + VideoTrack.
    I'll give a shot at a simpler test with just DesktopCapturer later but I actually want/need the WindowCapturer.
• I'm assuming you're checking the timestamp of the frames from the DesktopCapturer.start() callback ?
  My WindowCapturer doesn't seem interested in triggering that callback at all.
  In fact, it turns out that I actually don't even need to call WindowCapturer.start
  It seems that VideoDesktopSource.start is triggering the capture on my end.
  I'm checking timestamps from VideoTrackSink callback since it's the earliest place I get frames.
• It only occured to me now that maybe I am actually capturing at 60fps but encoding at 30 fps ? I naively assumed there wouldn't be encoding involved in a LocalSink but that could have been a dumb assumption.
  • Would you have any idea on how to check if that's actually the case ? Or how to fiddle with the encoder in general ?

[arosov]

I now use both WindowCapturer.setMaxFrameRate(60) and VideoDesktopSource.setFrameRate(60).

Also, just pasting webrtc's logs, just in case.

(audio_device_buffer.cc:66): AudioDeviceBuffer::ctor
(audio_device_impl.cc:142): current platform is Win32
(audio_device_impl.cc:170): webrtc::AudioDeviceModuleImpl::CreatePlatformSpecificObjects
(audio_device_impl.cc:909): webrtc::AudioDeviceModuleImpl::PlatformAudioLayer
(audio_device_impl.cc:263): Dummy Audio APIs will be utilized.
(audio_device_impl.cc:276): webrtc::AudioDeviceModuleImpl::AttachAudioBuffer
(audio_device_impl.cc:296): virtual webrtc::AudioDeviceModuleImpl::Init
(input_volume_stats_reporter.cc:98): Will not log any `WebRTC.Audio.Apm.AppliedInputVolume.*` histogram stats.
(input_volume_stats_reporter.cc:98): Will not log any `WebRTC.Audio.Apm.RecommendedInputVolume.*` histogram stats.
(audio_processing_impl.cc:463): Injected APM submodules:
Echo control factory: 0
Echo detector: 0
Capture analyzer: 0
Capture post processor: 0
Render pre processor: 0
(audio_processing_impl.cc:476): AudioProcessing: AudioProcessing::Config{ pipeline: { maximum_internal_processing_rate: 48000, multi_channel_render: 0, multi_channel_capture: 0 }, pre_amplifier: { enabled: 0, fixed_gain_factor: 1 },capture_level_adjustment: { enabled: 0, pre_gain_factor: 1, post_gain_factor: 1, analog_mic_gain_emulation: { enabled: 0, initial_level: 255 }}, high_pass_filter: { enabled: 0 }, echo_canceller: { enabled: 0, mobile_mode: 0, enforce_high_pass_filtering: 1 }, noise_suppression: { enabled: 0, level: Moderate }, transient_suppression: { enabled: 0 }, gain_controller1: { enabled: 0, mode: AdaptiveAnalog, target_level_dbfs: 3, compression_gain_db: 9, enable_limiter: 1, analog_gain_controller { enabled: 1, startup_min_volume: 0, clipped_level_min: 70, enable_digital_adaptive: 1, clipped_level_step: 15, clipped_ratio_threshold: 0.1, clipped_wait_frames: 300, clipping_predictor:  { enabled: 0, mode: 0, window_length: 5, reference_window_length: 5, reference_window_delay: 5, clipping_threshold: -1, crest_factor_margin: 3, use_predicted_step: 1 }}}, gain_controller2: { enabled: 0, fixed_digital: { gain_db: 0 }, adaptive_digital: { enabled: 0, headroom_db: 5, max_gain_db: 50, initial_gain_db: 15, max_gain_change_db_per_second: 6, max_output_noise_level_dbfs: -50 }, input_volume_control : { enabled 0}}
(webrtc_voice_engine.cc:476): WebRtcVoiceEngine::WebRtcVoiceEngine
(webrtc_voice_engine.cc:498): WebRtcVoiceEngine::Init
(webrtc_voice_engine.cc:506): Supported send codecs in order of preference:
(webrtc_voice_engine.cc:511): opus/48000/2 { minptime=10 useinbandfec=1 } (111)
(webrtc_voice_engine.cc:511): red/48000/2 { =111/111 } (63)
(webrtc_voice_engine.cc:511): G722/8000/1 (9)
(webrtc_voice_engine.cc:511): PCMU/8000/1 (0)
(webrtc_voice_engine.cc:511): PCMA/8000/1 (8)
(webrtc_voice_engine.cc:511): CN/8000/1 (13)
(webrtc_voice_engine.cc:511): telephone-event/48000/1 (110)
(webrtc_voice_engine.cc:511): telephone-event/8000/1 (126)
(webrtc_voice_engine.cc:514): Supported recv codecs in order of preference:
(webrtc_voice_engine.cc:519): opus/48000/2 { minptime=10 useinbandfec=1 } (111)
(webrtc_voice_engine.cc:519): red/48000/2 { =111/111 } (63)
(webrtc_voice_engine.cc:519): G722/8000/1 (9)
(webrtc_voice_engine.cc:519): PCMU/8000/1 (0)
(webrtc_voice_engine.cc:519): PCMA/8000/1 (8)
(webrtc_voice_engine.cc:519): CN/8000/1 (13)
(webrtc_voice_engine.cc:519): telephone-event/48000/1 (110)
(webrtc_voice_engine.cc:519): telephone-event/8000/1 (126)
(audio_device_impl.cc:296): virtual webrtc::AudioDeviceModuleImpl::Init
(audio_device_impl.cc:646): virtual webrtc::AudioDeviceModuleImpl::SetPlayoutDevice
(adm_helpers.cc:44): Unable to set playout device.
(audio_device_impl.cc:815): virtual webrtc::AudioDeviceModuleImpl::RegisterAudioCallback
(webrtc_voice_engine.cc:604): WebRtcVoiceEngine::ApplyOptions: AudioOptions {aec: 1, agc: 1, ns: 1, hf: 1, swap: 0, audio_jitter_buffer_max_packets: 200, audio_jitter_buffer_fast_accelerate: 0, audio_jitter_buffer_min_delay_ms: 0, }
(audio_device_impl.cc:831): virtual webrtc::AudioDeviceModuleImpl::BuiltInAECIsAvailable
(audio_device_generic.cc:18): virtual webrtc::AudioDeviceGeneric::BuiltInAECIsAvailable: Not supported on this platform
(audio_device_impl.cc:834): output: 0
(audio_device_impl.cc:847): virtual webrtc::AudioDeviceModuleImpl::BuiltInAGCIsAvailable
(audio_device_generic.cc:28): virtual webrtc::AudioDeviceGeneric::BuiltInAGCIsAvailable: Not supported on this platform
(audio_device_impl.cc:850): output: 0
(audio_device_impl.cc:863): virtual webrtc::AudioDeviceModuleImpl::BuiltInNSIsAvailable
(audio_device_generic.cc:38): virtual webrtc::AudioDeviceGeneric::BuiltInNSIsAvailable: Not supported on this platform
(audio_device_impl.cc:866): output: 0
(audio_processing_impl.cc:675): AudioProcessing::ApplyConfig: AudioProcessing::Config{ pipeline: { maximum_internal_processing_rate: 48000, multi_channel_render: 0, multi_channel_capture: 0 }, pre_amplifier: { enabled: 0, fixed_gain_factor: 1 },capture_level_adjustment: { enabled: 0, pre_gain_factor: 1, post_gain_factor: 1, analog_mic_gain_emulation: { enabled: 0, initial_level: 255 }}, high_pass_filter: { enabled: 1 }, echo_canceller: { enabled: 1, mobile_mode: 0, enforce_high_pass_filtering: 1 }, noise_suppression: { enabled: 1, level: High }, transient_suppression: { enabled: 0 }, gain_controller1: { enabled: 1, mode: AdaptiveAnalog, target_level_dbfs: 3, compression_gain_db: 9, enable_limiter: 1, analog_gain_controller { enabled: 1, startup_min_volume: 0, clipped_level_min: 70, enable_digital_adaptive: 1, clipped_level_step: 15, clipped_ratio_threshold: 0.1, clipped_wait_frames: 300, clipping_predictor:  { enabled: 0, mode: 0, window_length: 5, reference_window_length: 5, reference_window_delay: 5, clipping_threshold: -1, crest_factor_margin: 3, use_predicted_step: 1 }}}, gain_controller2: { enabled: 0, fixed_digital: { gain_db: 0 }, adaptive_digital: { enabled: 0, headroom_db: 5, max_gain_db: 50, initial_gain_db: 15, max_gain_change_db_per_second: 6, max_output_noise_level_dbfs: -50 }, input_volume_control : { enabled 0}}
(render_delay_buffer.cc:364): Applying total delay of 5 blocks.
(matched_filter.cc:789): Filter 0: start: 0 ms, end: 128 ms.
(matched_filter.cc:789): Filter 1: start: 96 ms, end: 224 ms.
(matched_filter.cc:789): Filter 2: start: 192 ms, end: 320 ms.
(matched_filter.cc:789): Filter 3: start: 288 ms, end: 416 ms.
(matched_filter.cc:789): Filter 4: start: 384 ms, end: 512 ms.
(transparent_mode.cc:243): AEC3 Transparent Mode: Legacy
(echo_canceller3.cc:806): AEC3 created with sample rate: 16000 Hz, num render channels: 1, num capture channels: 1
(clipping_predictor.cc:358): [AGC2] Clipping prediction disabled.
(agc_manager_direct.cc:483): [agc] analog controller enabled: yes
(agc_manager_direct.cc:486): [agc] Min mic level: 12 (overridden: no)
(peer_connection_factory.cc:353): Using default network controller factory
(bitrate_prober.cc:54): Bandwidth probing enabled, set to inactive
(pacing_controller.cc:200): bwe:pacer_updated pacing_kbps=300 padding_budget_kbps=0
(cpu_info.cc:53): Available number of cores: 16
(transport_sequence_number_feedback_generator.cc:52): Maximum interval between transport feedback RTCP messages: 250 ms
(aimd_rate_control.cc:87): Using aimd rate control with back off factor 0.85
(remote_bitrate_estimator_single_stream.cc:53): RemoteBitrateEstimatorSingleStream: Instantiating.
(webrtc_session_description_factory.cc:173): DTLS-SRTP enabled; sending DTLS identity request (key type: 1).
(openssl_key_pair.cc:40): Making key pair
(openssl_key_pair.cc:93): Returning key pair
(boringssl_certificate.cc:189): Making certificate for WebRTC
(boringssl_certificate.cc:245): Returning certificate
(webrtc_session_description_factory.cc:447): Setting new certificate.
(window_capture_utils.cc:326): Fail to create instance of VirtualDesktopManager

[arosov]

Also, a quick google-fu gave me this : https://chromium.googlesource.com/external/webrtc/+/HEAD/modules/desktop_capture/win/window_capture_utils.cc#318