Call for testers - release candidate v0.11-rc3

[Zygo]

Continued from #296. Thanks to all who tested -rc1 and -rc2!

New since -rc2:

• Support for openat2 on kernels that have it. This uses weak symbols for the first time. It may cause building or running issues for anyone building or running with non-mainstream libc or toolchain.
• Scheduler improvements to update progress more consistently and shorten queue times.
• Lots of doc updates. The minimum supported kernel has been moved to v5.4, which means the docs don't need to discuss ancient kernel bugs.

These changes might cause a regression relative to -rc2. If there are any, I'll back out the problematic changes, and v0.11 will be based on -rc2 or -rc3 with the offending commits reverted.

Todo:

• Fix for cancelling with CTRL+C need to be followed by umount /run/bees/mnt/$UUID #281
• Finalize the beescrawl.dat format for extent scan? (Might be too large of a change for v0.11, and we might have to support what -rc1 did for the foreseeable future anyway)

[manxorist]

Building v0.11-rc3 fails for me with

[...]
g++ -Wall -Wextra -Werror -O3 -I../include -D_FILE_OFFSET_BITS=64 -std=c++11 -Wold-style-cast -Wno-missing-field-initializers  -M -MF openat2.dep -MT openat2.o openat2.cc
In file included from openat2.cc:1:
../include/crucible/openat2.h:4:10: fatal error: linux/openat2.h: No such file or directory
    4 | #include <linux/openat2.h>
      |          ^~~~~~~~~~~~~~~~~
compilation terminated.
make[1]: *** [Makefile:36: openat2.dep] Error 1
[...]

on Ubuntu 20.04.6 LTS.

As far as I understand it, Ubuntu 20.04 only provides Linux kernel userspace headers for kernel 5.4 (package linux-libc-dev version 5.4.0-204.224), even when running HWE kernel 5.15 (package linux-image-5.15.0-129-lowlatency version 5.15.0-129.139~20.04.1).

bees v0.11-rc2 builds and works fine.

I do not have strong feelings about Ubuntu 20.04 support, so if you decide to not support it any more, that would be fine by me.

[XZVB12]

Hello, thank you for your work.
Checked on the kernel 6.12.10 for me works fine, but I did not check it on the main system, only on external drives, because I don’t want to be stuck with this problem: #249
I think for convenience it would be a good idea to write the remaining time statistics on the console, for example every 5 minutes / % and complete the process when everything is done or at least write about it, now the application just stays open.
There was a suggestion to create a convenient tool for status viewing #193, but it seems the idea was thrown.

I read a lot of comments about simultaneous compression and dedup and seem confused, can please you clarify the following :
it says here, #70 (comment) that any defragmentation cancels the bees work. will it affect the autodefrag flag mounting ? or only on manual start of full defragmentation: btrfs filesystem defragment -r -v -czstd ?
The same question applies to compression. Does compression compress=zstd:10 cancel deduplication or deduplication cancel compression?
As I understand from this message: #296 (comment) It is better to completely abandon compression , but this message #185 (comment) says that using compression and deduplication at the same time is a good idea.

Sorry to ask here, some themes have not been updated for a long time and I didn’t want to create a new theme that duplicated the old one.

[Zygo]

There was a suggestion to create a convenient tool for status viewing #193, but it seems the idea was thrown.

There's something like that on roadmap, but it doesn't make bees run any faster, so it's a low priority (or an opportunity for external contribution--patches welcome).

any defragmentation cancels the bees work

btrfs fi defrag and autodefrag both work by copying the data to a new physical location. The coalescing of small extents into larger ones is a side-effect of the copying process. All dedupe is the opposite of that process, replacing separate copies with references to a single physical location. Dedupe requires fragment boundaries to align exactly to boundaries between duplicate and unique data, while defrag ignores such boundaries, requiring correction from bees when the duplicate copies are deduped again.

Any form of defragmentation looks like new data to bees, which will attempt to dedupe the new data, possibly reinstating some of its earlier fragmentation. In v0.11, bees will now only introduce fragments when most of the space used by the fragmented extent can be freed as a result of the fragmentation. This produces a healthier balance between defrag and dedupe.

The same question applies to compression. #296 (comment)

Compression and dedupe are independent features, and they can be combined to get the good effects of both.

Note that in the graph, the compressed and uncompressed data sets are both 45% smaller after dedupe, but they were different sizes at the beginning. The compressed data set went from 100 GiB to 55 GiB, while the uncompressed data set went from 160 GiB to 90 GiB. The effect of dedupe was the same, but the compressed data was smaller due to compression, so fewer bytes were saved by dedupe because they were already saved by compression.

Combining compression and dedupe, the data went from 160 GiB (uncompressed without dedupe) to 55 GiB (compressed with dedupe), a total savings of 65%.

bees won't recompress data for you. You'll have to do that yourself, with btrfs fi defrag -vrczstd or similar.

[Zygo]

[...Ubuntu 20, kernel 5.15, kernel headers 5.4...]

It turned out to be not too painful to add compatibility code for older headers, similar to what bees already does for btrfs, so I pushed a fix to master.

If anyone wants to test it on their DEC Alpha...let me know if it works. 😉

[KeinNiemand]

In #296 you said
"Only one thread reads from disk at any time".
Would it make sense to use 2 threads to read from disk on raid 1 and raid10 filesystems (3 for raid1c3, 4 for raid1c4)? I don't know exactly how btrfs handles reading from raid1 but I recall reading somewhere that it splits different reads between the disks so maybe using 2 threads (1 per copy of the data in the raid) would be faster.

[manxorist]

[...Ubuntu 20, kernel 5.15, kernel headers 5.4...]

It turned out to be not too painful to add compatibility code for older headers, similar to what bees already does for btrfs, so I pushed a fix to master.

not quite yet ...

[...]
g++ -Wall -Wextra -Werror -O3 -I../include -D_FILE_OFFSET_BITS=64 -std=c++11 -Wold-style-cast -Wno-missing-field-initializers  -M -MF openat2.dep -MT openat2.o openat2.cc
In file included from openat2.cc:1:
../include/crucible/openat2.h:22:2: error: ‘__u64’ does not name a type
   22 |  __u64 flags;
      |  ^~~~~
../include/crucible/openat2.h:23:2: error: ‘__u64’ does not name a type
   23 |  __u64 mode;
      |  ^~~~~
../include/crucible/openat2.h:24:2: error: ‘__u64’ does not name a type
   24 |  __u64 resolve;
      |  ^~~~~
make[1]: *** [Makefile:41: openat2.o] Error 1
[...]

[kakra]

bees won't recompress data for you

Doesn't bees create compressed temporary files to place dedupe extents?

[kakra]

Would it make sense to use 2 threads to read from disk

Not necessarily... btrfs stripes reads by PID modulo. So if you have two disk candidates to read from, btrfs does PID modulo 2, and the result selects stripe 0 or 1 for read.

So if bees happens to read only from "even" PIDs, two threads would read the same stripe - always - because bees threads are relatively long living.

I wonder if we could get some stripe selector into the kernel at some time which decides reading based on IO latency. But this is difficult to do, because switching the stripe too often will reduce throughput. And the gains seem not to be very high: There has been a patchset in the past trying to do something like that, and it didn't improve a lot.

So if bees could read from two stripes in parallel, it would probably not gain a lot.

[KeinNiemand]

Would it make sense to use 2 threads to read from disk

Not necessarily... btrfs stripes reads by PID modulo. So if you have two disk candidates to read from, btrfs does PID modulo 2, and the result selects stripe 0 or 1 for read.

So if bees happens to read only from "even" PIDs, two threads would read the same stripe - always - because bees threads are relatively long living.

Could bees account for the PID modulo 2 thing? Like just use 2 threads for reading but make sure that one has an odd PID and one an even. If PID can't be controlled when running threads, getting and even and an odd still shouldn't be to hard, just create threads until you have at least one even and one odd PID then get rid of the extra threads you made.
I assume the same logic apply with module 3 for raid1c3 and module 4 for raid1c4.

I wonder if we could get some stripe selector into the kernel at some time which decides reading based on IO latency. But this is difficult to do, because switching the stripe too often will reduce throughput. And the gains seem not to be very high: There has been a patchset in the past trying to do something like that, and it didn't improve a lot.

If switching is bad how about just assigning each thread a stripe to read from but instead of making it based on PID base in on IO latency or throughput or utilization of each stripe or whatever trying to balance reads between stripes, switching could be done very infrequently maybe once per minute or less.

So if bees could read from two stripes in parallel, it would probably not gain a lot.

If you account for the PID thing you'd have 2 independent threads doing reads without switching, why would it not gain a lot? In theory reading from 2 stripes should be an up to 100% improvement in read speed.

[kakra]

If you account for the PID thing you'd have 2 independent threads doing reads without switching, why would it not gain a lot? In theory reading from 2 stripes should be an up to 100% improvement in read speed.

Because there is still meta data which may not follow the same logic, and because btrfs is not really good at parallelizing reads and writes from the same consumer.

Most of the time in bees seems to be spend on very short extents which involves a lot of metadata IO. So while reads of large extents could benefit from such a logic, it would be only a very small gain in the overall extent processing. I'd rather leave that extra "free" IO to other processes.

But I'm pretty sure bees did multiple read threads in the past... @Zygo? So this feature must have been dropped for a reason.

[KeinNiemand]

If you account for the PID thing you'd have 2 independent threads doing reads without switching, why would it not gain a lot? In theory reading from 2 stripes should be an up to 100% improvement in read speed.

Because there is still meta data which may not follow the same logic, and because btrfs is not really good at parallelizing reads and writes from the same consumer.

How is two bees thread doing reads any different then two completely independent consumers? As long as the two threads work on scanning different data (don't try to read the same file/extent/whatever bees works with 2 threads) they should not really interfere with each other. How is bees any different then two compactly separate applications reading 2 different things from the 2 copies?

Most of the time in bees seems to be spend on very short extents which involves a lot of metadata IO. So while reads of large extents could benefit from such a logic, it would be only a very small gain in the overall extent processing. I'd rather leave that extra "free" IO to other processes.

Is there a reason metadata reads wouldn't follow the same logic assuming it's also raid1 or higher (I guess if data and metadata are at different raid levels they would use different module values) for me data is raid10 and metadata is raid1c3.

But I'm pretty sure bees did multiple read threads in the past... @Zygo? So this feature must have been dropped for a reason.

My guess (I have no idea what bees actually did in previous and I'm just going of one part of last issue saying it now only uses one thread for reading.) is that previously bees just didn't have a limit on how many threads could read at the same time, which probably degraded performance/gave no performance gains since you had multiple threads reading from the same disks.
Trying to use on thread per copy on a raid (accounting for the PID thing) might improve performance.

[Zygo]

But I'm pretty sure bees did multiple read threads in the past... @Zygo? So this feature must have been dropped for a reason.

It wasn't a feature. Old versions simply hurled IO requests at the kernel, and the kernel generally scheduled them in the least efficient way possible. There were readahead requests to try to streamline that by prefetching data, but it turns out that's not what readahead is designed to handle. Kernel IO schedulers promised a lot, before and after the mq transition, but failed to deliver anything useful to bees.

As a temporary fix for v0.11, all readahead/prefetching was moved to userspace, and a mutex was introduced to ensure that only one thread at a time does prefetch (the downstream processing is still parallel, but it's all working from page cache after prefetch is done). Extent scan ensures that IO patterns generated by bees reused cached pages more often, which gains back most of the performance that subvol scans were losing by design.

It's a temporary fix because the more permanent fix will be to remove as much block reading as possible. There's no need to make that code run faster or in parallel, if we can avoid running it in the first place. Getting rid of reads is next on the todo list, but we want to get the non-trivial gains we have so far into a stable release so distros will pick it up, and people can stop running v0.10 and earlier.

It doesn't make a lot of sense to micro-optimize reads when:

1. Reads are mostly going away. The goal is to eventually read less than 1% of the unique uncompressed data by reading the csum tree instead of data blocks to find duplicates. Once that's done, even if read overhead was magically reduced all the way to zero by micro-optimization, it would only result in a 1% performance improvement in the scanning phase.
2. Reads don't dominate the IO time any more. The temporary fix was good enough that now the IO overhead is a roughly even mix of reads, LOGICAL_INO, and dedupe operations. If we get rid of most reads, we don't need most of the LOGICAL_INO calls either--in most cases, we're doing LOGICAL_INO only so we can read data blocks from files.
3. If the kernel ever gets around to merging any of the existing read policy patches (some of which are more than 5 years old now), the btrfs raid1 mirror optimization won't work any more, and will make reads slower when attempted. In that event, any prior effort spent on optimizing bees for that case would be wasted--users could upgrade their kernels and delete a few lines of bees code, and get the optimization for free.
4. The reads have to go somewhere. On SSDs, where reads could be issued in parallel, the improvement is limited by all the CPU bees uses after reading the data. Improvement on SSD would require optimizing other parts of bees to be able to consume the data at speeds over a few hundred MiB/s. Most of those parts will be replaced as part of the transition to csum scan, because they're slow and limiting performance even more than single-threaded reads do.

I made some prototypes where bees figures out where we can send reads in parallel based on the chunk tree, and schedule tasks to avoid reading from the same devices at the same times, using the same framework bees currently uses to avoid inode and extent lock contention. They suffer from two problems:

1. Only a subset of btrfs configurations can use the optimization. Any single-device filesystem, or any filesystem using raid0, raid10, raid5, or raid6, can't usefully do parallel reads from userspace as they are already parallelized in the kernel. On raid1, raid1c3, or raid1c4, the raid1 optimization relies on not having a non-default read policy enabled. That leaves only multi-disk single profile filesystems, and raid1 filesystems with 4 or more devices, raid1c3 with 6 or more devices, or raid1c4 with 8 or more devices, where the micro-optimization can be enabled. Those configurations seem unusual[citation needed].

2. When it's enabled, it almost never works without restructuring other parts of bees or making work queues insanely large. e.g. to get two dedupe reads to run in parallel, we need to find 2 duplicate data extents that are stored on 4 separate drives. The current extent scanner pulls all of one side of the dedupe from the same block group, which means we almost never get more than 3 drives involved concurrently in 2 parallel dedupes. Even if that is fixed, duplicates are found where duplicates are found, so the optimizer is at the mercy of the user's data to find opportunities to work.

Some of these problems could be solved, but the csum tree scanner simply doesn't have the problems to begin with, and it's hundreds of times faster on all storage configurations. After that's done, we can look at where bees spends its time and focus effort there.

Another possible micro-optimization is to use the CLONE_IO flag for bees threads, so they appear to be separate processes to the kernel. I say "possible" because it would involve reimplementing pthread_clone just to attempt it, and it's not clear there's any benefit compared to having the kernel treat bees as multiple processes for IO vs. having bees organize the read workload itself as a single process. On the todo list, CLONE_IO experiments are last, in a bucket with "using io_uring" and other tweaks that are also likely to be irrelevant in the long term.

[Zygo]

bees won't recompress data for you

Doesn't bees create compressed temporary files to place dedupe extents?

Only for partially duplicate extents where it can't find a duplicate copy of some of the data, and only when those fragments are compressible.

In the 160 GiB uncompressed test, only 293 MiB was compressed by bees (and about 7 GiB was zeros, so those are now holes that don't count as space usage).

Processed 151 files, 55179 regular extents (310127 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL       99%       90G          91G         153G
none       100%       90G          90G         151G
zstd        30%      293M         948M         2.2G

[KeinNiemand]

1. Only a subset of btrfs configurations can use the optimization. Any single-device filesystem, or any filesystem using raid0, raid10, raid5, or raid6, can't usefully do parallel reads from userspace as they are already parallelized in the kernel. On raid1, raid1c3, or raid1c4, the raid1 optimization relies on not having a non-default read policy enabled. That leaves only multi-disk single profile filesystems, and raid1 filesystems with 4 or more devices, raid1c3 with 6 or more devices, or raid1c4 with 8 or more devices, where the micro-optimization can be enabled. Those configurations seem unusual[citation needed].

Why wouldn't raid10 benefit? Yes it already does parallel reads from 2 devices because of the 2 stripes but there's also another copy like in raid1. Wouldn't the raid1 optimization also work on raid10 (going from reading from 2 devices to all 4 in a 4 drive raid 10 setup)?
I might be confusing raid10 with raid0 +1 here I don't know in what order btrfs does raid10 in.