Preferred metadata design

[josefbacik]

We need to agree on how this system will work before we start writing code again. The main goal is to provide users the ability to specify a disk (or set of disks) to dedicate for metadata. There's a few policy questions that need to be asked here. Please discuss with new comments, and as we come to a consensus I will update this main entry with the conclusion of our discussions.

Policy questions

• ENOSPC
  • Do we fail when we run out of chunks on the metadata disks?
  • Do we allow the use of non-metadata disks once the metadata disk is full?
  • Do we allow the user to specify the behavior in this case?
• Device Replace/Removal
  • Does the preferred flag follow the device that's being replaced?
  • What do you do if you remove the only preferred device in the case that we hard ENOSPC if there are no metadata disks with free space?
• On the fly changing of existing metadata disks.
  • If you unset the flag on one disk, is the metadata moved?
  • If you unset the flag on the only metadata disk, what happens? (Similar to device removal?)

Userspace implementation

• sysfs interface for viewing current status of all elements.
• sysfs interface at least for setting any policy related settings.
• A btrfs command for setting a preferred disk.
  • The ability to set this at mkfs time without the file system mounted.

Kernel implementation

• The device settings need to be persistent (ie which device is preferred for the metadata).
• The policy settings must also be persistent.
  • The question of how to store this is an open question. I have suggested using the btrfs property xattrs and storing them in the tree-root, so they wouldn't be visible to anything except btrfs. We need to decide on some sort of way to deal with system wide settings, and we'll use this feature to figure out what we think the canonical way forward is for that.

[josefbacik]

For me

• ENOSPC
  • I'm fine with a hard/soft policy for this. Hard being we run out of allocatable chunks on the preferred metadata devices then you simply ENOSPC. This is tricky to convey with df however, and would need to have some reflection in btrfs filesystem usage so the user knows they only have X allocatable space for metadata in the case of the hard policy.
• Device replace
  • If we're replacing a metadata preferred device, then it sticks with the device. The way device replace works currently is the device item is left untouched mostly, we just update the id to point at the new device path. Mechanically this would work out fine, just don't touched the metadata preferred property. If the user wants to change it they can do so later.
• On the fly changing.
  • If we're unsetting the preferred metadata flag I think we should trigger a balance for any metadata chunks that are on that device.
  • If we have the "hard" policy then you should simply be disallowed from unsetting the flag on the last disk. For "soft" it should be allowed.
• Where to store the preferred policy.
  • We are going to want to have more policies in the future that apply to the entire file system. I, for example, want to be able to specify allocator behaviors persistently. I think the btrfs property thing is a nice fit for this, because these are simple selections. "hard" or "soft" for the preferred metadata policy. We simply add new strings to the property as we think of new horrible ways to abuse it, and we don't have to worry about tooling for a new item flag or whatever. We store the property xattr in the tree_root so that it's not visible via lsattr and doesn't get pulled down via rsync or anything like that. However I could be convinced of alternative approaches, but generally I want to have something generic we all agree on so that I (and I believe @morbidrsa) can use it for our own purposes in the future.

[Zygo]

Policy questions

• ENOSPC

  • I'm opposed to the global "hard" vs "soft" vs "metadata" idea. It's easier to label all the drives by preferred chunk type, and whether they allow fallback to the other type or not. It's not possible to specify good behavior for more than 2 speeds of drive with a single global option.

  • There are use cases for allowing and disallowing metadata on data disks and vice versa. The policy should be separate for each disk, i.e. each disk should specify whether it accepts data or metadata separately, and each disk should specify whether it prefers metadata or data separately. That makes 4 preference levels for each disk:

    • metadata only - first in line for metadata chunks, data is never allocated here
    • metadata preferred - second in line for metadata chunks, third in line for data chunks
    • data preferred - third in line for metadata chunks, second in line for data chunks
    • data only - metadata is never allocated here, first in line for data chunks

  • Note the sort order for data is the opposite of metadata.

  • The behavior in the "never allocate here" cases should be exactly the same as if all space on the disk in question was fully allocated, and you need a new chunk of the respective type.

  • df should not count metadata-only devices as free space (bavail).

  • Each level includes the previous level. For a multi-disk profile, we try to use the minimum number of preference levels that satisfy the minimum device requirement. i.e. first we try to get min_devs for a data chunk using only data-only disks, then we try data-only and data-preferred, then we try data-only, data-preferred, and metadata-preferred, then we give up and ENOSPC because policy does not allow us to use any other disks. This affects the stripe widths you get with the striping RAID profiles.

  • There needs to be a default policy for new drives when they are added to the filesystem. I think it should either be "data preferred", or the user must specify the default policy when they set up any preference at all (this is easiest if the config is a single string).

  • I don't believe we can have fewer than 4 states for each disk, as fewer than 4 will make some performance use cases impossible. I had an "extra credit" version of this on the mailing list which allows specifying arbitrary sort orders across all devices, but arbitrary device fill order isn't a feature I desperately need.

    • e.g. more than 2 preference levels: you could have 4 disks of different speeds (1 is fastest, 4 is slowest), and want metadata to overflow from disk 1 to disk 2, but not disk 3 or 4, because metadata performance on disks 3 and 4 would be crippling. Data would be the opposite, overflow from 4 to 3 would be allowed, but not 2 or 1, because disks 1 and 2 are the only place we can put metadata and we can't afford to waste that space on data.
    • e.g. when performance is more important than space: a more typical configuration is 2 NVME's and 6 or more spinning disks. In that case we dedicate the NVME to metadata-only and everything else is data-only. No metadata allowed on the spinning disks as suddenly everything gets 95% slower when that happens. No data on the NVME as it steals metadata space, and on some NVME firmware we want to keep them only half full to avoid write multiplication.
    • e.g. when performance is not as important as space: you prefer to have metadata on disk 1 and data on disk 2, but the SSD is too big to waste on just metadata, and you don't want to ENOSPC with room still available on the slow drive. In that case you'd only use "metadata preferred" and "data preferred" to get most of the chunks in the right place, but still use all of the available space.

  • If the configured preferences say we can't use a disk for (meta)data, then we can't use that disk for (meta)data (same as if the disk was full). The user must choose between prioritizing performance or space, they are mutually opposite goals.

• Device Replace/Removal

  • Device preferences follow the device that's being replaced. The replaced drive will have exactly the same chunks on it that it had before, so it's reasonable to assume it has the same preferences too. If the device preferences change (e.g. replacing SSD with HDD) then the admin can change the preference attributes and start a balance to rearrange the data.
  • There should be no requirement for a device to be online to change its preference. So if you are replacing a SSD with a HDD, you can change the preference on the device before or after the replace.
  • When we remove a drive, we check to see if there are enough devices for all the currently existing RAID profiles. This would be adjusted to count or not count devices depending on preference, e.g. if you have RAID1 metadata, you can't remove a device from the filesystem if it means you are left with fewer than 2 data-preferred, metadata-preferred, or metadata-only drives. The admin can change the preferences first and then the removal would succeed, same way they can convert the metadata to single and then remove succeeds. You also have to have sufficient space for the relocated data on the remaining devices after preference filtering, but I don't know off the top of my head whether we bother to check that in dev remove.
  • The preferences only take effect at chunk allocation time, so you can change preferences at any time (provided that the new preferences have enough of each kind of disk for RAID profiles). The preference affects only new chunks, so existing chunks are left alone.
  • If you want the data moved to match the updated preferences, run a balance at some convenient time. We definitely do not want this balance auto-triggered. Metadata balances take weeks to months to run on the kinds of servers I want to use this feature on--they have to be scheduled on a calendar, and paused outside of maintenance windows.
  • Extra credit: add a new balance filter for matching "chunk that is on a disk with non-matching data/metadata preferences." So if you flip some disks from metadata-preferred to metadata-only, you run btrfs balance start -dpreferred /... and any data chunks on those disks are relocated onto other disks (or you run out of space and ENOSPC).

Userspace implementation

• Yes to all of these:
  • sysfs interface for viewing current status of all elements.
  • sysfs interface at least for setting any policy related settings.
  • A btrfs command for setting a preferred disk.
• Extra credit:
  • The ability to set this at mkfs time without the file system mounted.

Kernel implementation

• Yes to this:

• The device settings need to be persistent (ie which device is preferred for the metadata).

• The policy settings must also be persistent.

• Store it as a string. (Just one?)

• Update string from btrfs tools. Kernel verifies and compiles the string before allowing changes, including minimum device count checks. If the new string is rejected, the old string (if any) is kept. btrfs userspace doesn't vet the string, so we don't need to change it for every kernel (though userspace might analyze the string to provide helpful or localized error messages).

  • I'd also be OK with the tools compiling it to a binary blob and passing that to the kernel, e.g. like we do for balance args.

• If the kernel can't understand the on-disk string (old kernel on filesystem touched by new kernel) then...just ignore it, since that's what all existing kernels will do. Drop a warning in dmesg in new kernels.

  • We could set a read-only incompatible bit, but that seems like overkill at the moment? Technically the allocator from old kernels will still correctly read and write data, just in undesirable places. Maybe only set the incompat bit when an "-only" preference is set, as violations of the "-only" requirement are very expensive to recover from.

• Store the config (whatever form it takes) as a persistent item in the dev tree, where we currently put device stats and replace progress (but not balance progress?). Plenty of persistent item objectids left, and the offset could be a syntax revision field.

[josefbacik]

Ok so you envision something more generic than simply "allocate metadata on metadata preferred devices", and more classifying devices on a spectrum, and using that spectrum to infer policy.

I actually like that idea better, because it removes the need to have a separate flag to indicate what the policy is, we simply tag devices with their policy and carry on. From the point of view of the user it would map out roughly like this

# mkfs
mkfs.btrfs --data-preferred /dev/sdb,/dev/sdc --metadata-preferred /dev/sdd,/dev/sde --metadata-only /dev/nvme0n1

# on the fly
echo 'metadata-only' > /sys/fs/btrfs/<uuid>/allocation-preferences/sdc/preference

cat /sys/fs/btrfs/<uuid>/allocation-preferences/sdc/preferences
none [metadata-only] metadata-preferred data-only data-preferred

# unmounted
btrfs device preference --set 'metadata-only' /dev/nvme0n1

# mounted
btrfs device preference --set 'metadata-only' /dev/nvme0n1 /path/to/fs

For this the on disk implementation would simply be a new item on the device tree per device. Implementation wise I would do something like

• Lookup this device item for each device of the file system, default to no preference if the item doesn't exist.
• No preference means the device goes on fs_devices->alloc_list. Add a respective list to fs_devices for each of the different allocation preferences.
• Change __btrfs_alloc_chunk() to abstract out searching the fs_devices->alloc_list and instead search through the respective list. The ENOSPC handling will do the correct thing from there, all you need to do is populate it based on the list you need to be searching through.
• Unfortunately we sort of need something like Qu's work to fix ->free_chunk_space, as now this becomes an even more important number based on our more complicated setup. If this is wrong we'll make really bad decisions early on in the ENOSPC machinery.

I think that's it code wise.

[Zygo]

Sounds good. One question: What does "no preference" mean? metadata-only, metadata-preferred, data-preferred, and data-only imply an ordering, so what is "none" and where does it fit in the order?

Or is "none" an alias to one of the others and the default value for new devices, e.g.

# default filesystem with no preferences
cat /sys/fs/btrfs/<uuid>/allocation-preferences/default/preference
metadata-only metadata-preferred data-only [data-preferred]

# set our fastest devices to metadata-only, all other devices to data-only
echo metadata-only | tee /sys/fs/btrfs/<uuid>/allocation-preferences/nvme*/preference
echo data-only | tee /sys/fs/btrfs/<uuid>/allocation-preferences/sd*/preference

# change what 'none' means, so metadata doesn't leak onto new devices with default preferences:
echo data-only > /sys/fs/btrfs/<uuid>/allocation-preferences/default/preference

and if so, wouldn't it be better to call it "default" instead of "none"?

[kreijack]

Policy questions

ENOSPC

Do we fail when we run out of chunks on the metadata disks?

IMHO no, a lower performance is far better than a metadata -ENOSPC.
Anyway we should create a logging system to warn the user about this situation. (like the one that warns the user if there are different profiles in the same filesystem).

Do we allow the use of non-metadata disks once the metadata disk is full?

As above, yes. However there are some corner cases that have to be addressed.
Supposing to have:

• sda -> metadata
• sdb -> metadata
• sdc -> data
• sde -> data
• sdd -> data
• data profile: raid5
• metadata profile: raid1

Normal case: data spans sd[cde], metadata spans sd[ab]. What should happen if sdd is full:

1. return -ENOSPC
2. should data span sd[abce] or span only 3 disks ? (the data could have even more space )

Do we allow the user to specify the behavior in this case?

I think no. However I am open to change my idea if there is a specific user case.

Device Replace/Removal

Does the preferred flag follow the device that's being replaced?

In the general case no, I don't see any obious behavior; we can replace a faster disk with a slower one. However btrfs disk replace/remove should warn the user about the possibles risks

What do you do if you remove the only preferred device in the case that we hard ENOSPC if there are no metadata disks with free space?

I think that the "preferred" is just an hint. I prefer a better btrfs-progs that warn the user about this situations (preferred disks full)

Userspace implementation

sysfs interface for viewing current status of all elements.

Definitely yes

sysfs interface at least for setting any policy related settings.

I agree, the only limits is that is difficult to implement an "atomic change of multiple values". I don't know if this
is the case however.

A btrfs command for setting a preferred disk. The ability to set this at mkfs time without the file system mounted.

It does make sense. Anyway my suggestion is still to allow a flag to the mount command to set a "standard" behavior in an emergence situation. May be that the mount options are "transitory"; instead the sysfs are "permanentely"

Kernel implementation

The device settings need to be persistent (ie which device is preferred for the metadata).

Agree

The policy settings must also be persistent.

Agree

The question of how to store this is an open question.

The xattr was a fascinating idea. Unfortunately it suffers of two problems:

• a simple cp or a rsync commands between two filesystem may create unexpected results. This may be mitigated preventing flistxattr() to return these setting. The biggest limits is that we never know which xattr are set. We could only try. (what happens if an xattr with an incorrect name is set ? it will last forever )
• it requires that the "root subvol" (id=5) is mounted. To mitigated this, we could force btrfs to access the root subvolume xattr when the prefix of the xattr name starts with "btrfs.fs." (instead of "btrfs.sub." for subvolume or "btrfs.file." for file)

The other way is the one which was used to store the default subvolume. The only differences is to use an extendible (and versioned) structure to hold several parameters (even not related). A "dirty" flag marks the structure to be committed in the next transaction (see btrfs_feature_attr_store() ).

[Zygo]

This is one area where Goffredo and I disagree. I have use cases where there absolutely must not be data or metadata on a non-preferred device type. In my examples above "metadata-only" and "data-only" get used more often than "metadata-preferred" or "data-preferred."

In my test cases I never use metadata-preferred or data-preferred at all. I could live with just metadata-only and data-only, but I know others can't, so I included metadata-preferred and data-preferred in all variations of my proposal.

On a server with 250GB of NVME and 50TB of spinning disk, It's a 95% performance hit to put metadata on a non-preferred device, and a 0.5% space gain to use preferred devices for data. That tradeoff is insane, we never want that to happen and we'd rather have it just not be possible. We're adults, we can buy NVME devices big enough for our metadata requirements.

[josefbacik]

I'm with Zygo here, having first hand had teams tell me they'd rather have X fall over hard than slowly degrade. I think its valuable to have the metadata-preferred/data-preferred model for people who want the perf boost+safety, but it's equally valuable for Zygo and other people who would rather it be fast and plan appropriately than to have it slowly break.

[kreijack]

Ok, let me to summarize the algorithm.
We have 5 disk classes:

• METADATA_ONLY (the disk can host only metadata)
• PREFERRED_METADATA (the disk is preferred for metadata, in low space condition it can host data)
• UNSPECIFIED (a disk without any hints)
• PREFERRED_DATA (...)
• DATA_ONLY (...)

The above ordering is for metadata. For data, the ordering is reversed.
The disks are ordered first by level and then by available space. After the sorting the last "*_ONLY" disks are excluded from the consideration by the allocator.

The allocator takes the disks from the first group.
If these are not enough (ndevs < devs_min), it extends the list of disks using the second group.
If these are not enough, it extends the list of available disks up to the 3rd group.
If these are not enough, it extends the list of available disks up to the 4rd group.

To simplify I would like to know your opinion about grouping the different levels:

• group, in the ordering, the devices tagged by METADATA_ONLY and PREFERRED_METADATA
• group, in the ordering, the devices tagged by UNSPECIFIED and PREFERRED_DATA

[Zygo]

The "disk with no hints" case is a problem. If we have split the filesystem into disks that only have data or metadata, but we add a new disk, btrfs might put the wrong type of chunk on it.

There's a few solutions:

1. Require that of the following must be true:

• all devices must be unspecified, metadata-preferred, or data-preferred. "unspecified" is an alias for "data-preferred".
• if any device has an "only" preference, devices that are "unspecified" get no chunks at all.
  A side-effect of this is that we can't go directly from "all devices unspecified" to "all devices data-only and metadata-only", we have to transition through data-preferred and metadata-preferred until the numbers are high enough for RAID profiles, then flip them to data-only and metadata-only in a second pass.

2. Make "unspecified" an alias for one of the other 4 types, that can be set at run time (this is my earlier comment Preferred metadata design #19 (comment)). For multi-device data-only and metadata-only setups, we follow these steps:

• set "unspecified" as an alias for data-preferred, so every device is initially data-preferred.
• set data-only and metadata-only preferences on each device.
• set "unspecified" as an alias for data-only to prevent metadata from leaking onto slow devices.

3. Get rid of "unspecified", make the initial state of every device "data-preferred". This can lead to the wrong chunk type on some devices, but data chunks are far faster to relocate with balance, and it makes the user manual much shorter.

My preference is option #3. Even though it can put data on a SSD sometimes, it is much simpler for users.

[kreijack]

Get rid of "unspecified", make the initial state of every device "data-preferred". This can lead to the wrong chunk type on some devices, but data chunks are far faster to relocate with balance, and it makes the user manual much shorter.

I think that this is the "least surprise" option; I like this

[Zygo]

To simplify I would like to know your opinion about grouping the different levels:

group, in the ordering, the devices tagged by METADATA_ONLY and PREFERRED_METADATA

I can't think of cases where you'd have both -preferred and -only preferences set on devices in a filesystem without also wanting the ordering between them. Conversely, if you don't want the ordering, you also don't need two of the preference levels.

• e.g. 1, in the NVME/SSD/HDD case, where there are devices of different speeds, you want metadata on NVME first, SSD last, so metadata-only and metadata-preferred are two distinct groups that should be ordered separately.
• e.g. 2 if you had 5 disks of equal speed, you wouldn't bother making 2 of them data-preferred and 3 data-only, you'd just make them all data-preferred or all data-only. Only one of data-preferred or data-only exists in the filesystem, making the other group irrelevant for sorting.

We could dig out my chunk_order variant proposal from the mailing list. That allows specifying device allocation order explicitly, and would allow user control over grouping. For that matter, there's no reason why we couldn't do allocation ordering as a separate feature proposal. In that case, we would use the m-o, m-p, d-p, d-o ordering by default, but if the user gives a different explicit ordering, we use that order instead, then the device preferences only affect what chunk types are allowed on which devices.

[MarkRose]

Is there any effort underway to implement this? I may be interested in contributing

[hauuau]

Are there any news about this feature? The patch series from the last year doesn't apply cleanly to recent kernels anymore.

@MarkRose, There is a patch series by @kreijack on the linux-btrfs mailing list. Try searching its archives for "allocation_hint".

[kakra]

Here's a patchset I'm trying to keep updated for LTS versions:
kakra/linux#26