Improve speed for initial sync with virtual files

[wonx]

How to use GitHub

• Please use the 👍 reaction to show that you want to have the same feature implemented.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

Feature description

When using virtual files, the first log in after a new installation will start a syncing process that can take a very long time depending on the number of files to synchronize.

In my case, i'm syncing around ~700000 files, my computer has been already up for 29 hours without a restart and the sync process has now reached the 50% mark. I can see that the virtual files are created one by one, but it can be as slow as 2 per second. Two or more days until Nextcloud can be usable is too much in my opinion.

It would be cool if there was any way to speed up the initial sync.

PS: This is related to #4421

[marcotrevisan]

I'm experiencing a similar problem on Mac OS (around 200k files). In my humble opinion, syncrhonizing the full hierarchy is the key problem here. The typical end user doesn't need to have the full folder hierarchy saved and synchronized. A lazier approach (i.e. trigger on open and/or scan the opened subfolders only, and not the whole depth trough but perhaps 1 or 2 levels below) would grant more scalability and decrease the load on the NC server.

[johannes-luebke]

I'd like to add, that restarting the sync, client or PC will result in a complete restart of the process. Also the sync doesn't seem to start immediately, but it first counts all files it will sync and then starts syncing. The counting alone takes two days for me and the sync isn't done after more than 10 days. At least the sync should pick up where it left of.

[CWempe]

I have the same issue.

The most annoying part is that I do not need the folders with all the little files available on my desktop.

So it would be enough if I could say "do not sync this folder unless it is accessed by the user".

I think the suggestion from @marcotrevisan (see #4464) also sounds promising.

[PhilippSchlesinger]

See #4918 (comment) for a description of a problem with the tray window related to speed issues for inital sync

[CWempe]

I can confirm this issue.

I started syncing virtual files (~1 million) on a new notebook.
I knew it would take a while.
The next day I checked and saw about 30 % finished.
The day after that only 10 % more (= 40%).
In the application window I could see that roughly one file was processed per second.

Then I read about restarting the client software here.
And the syncing (files per second) increased dramatically.

Now I took some data to verify this behavior:

So the best workaround would be a script that restarts the Nextcloud client every 30 minutes or so. 😜

Bu it would be great if this could be fixed.

Server: 24.0.7 (docker)
Client: 3.6.2 (Windows)

[PhilippSchlesinger]

With latest Nextcloud Client 3.7.3 an inital sync on ~150k files took <1 hour where it was a whole night and endless errors in the past.
Maybe you guys could also check again and see if it improved with the latest version.

[PhilippSchlesinger]

With latest Nextcloud Client 3.7.3 an inital sync on ~150k files took <1 hour where it was a whole night and endless errors in the past.
Maybe you guys could also check again and see if it improved with the latest version.

@CWempe
Since you described the issue in detail and with numbers previously, could you maybe check again with 3.7.3 or later and report if anything changed?

[tomdereub]

Like I said here : #3120 (comment), I'm still having the problem with Nextcloud 25 and desktop client 3.8.2. In 24 hours it had not yet finished to count files to synchronize, then it lost connexion, and restarted from scrath... About 2 000 000 files.