Remove KF_SLEEPABLE from arena kfuncs #10310
Open
+582
−59
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Author
|
Upstream branch: 4722981 |
kernel-patches-daemon-bpf
bot
force-pushed
the
bpf-next_base
branch
from
4a6b8b7 to
1efb39d
Compare
Author
|
Upstream branch: 7dc211c |
kernel-patches-daemon-bpf
bot
force-pushed
the
series/1024361=>bpf-next
branch
from
bbbedb8 to
91d2dfb
Compare
kernel-patches-daemon-bpf
bot
force-pushed
the
bpf-next_base
branch
from
1efb39d to
5b97b4a
Compare
Author
|
Upstream branch: ec12ab2 |
kernel-patches-daemon-bpf
bot
force-pushed
the
series/1024361=>bpf-next
branch
from
91d2dfb to
8617009
Compare
kernel-patches-daemon-bpf
bot
force-pushed
the
bpf-next_base
branch
from
5b97b4a to
7b6b51d
Compare
Author
|
Upstream branch: d6ec090 |
kernel-patches-daemon-bpf
bot
force-pushed
the
series/1024361=>bpf-next
branch
from
8617009 to
11d7e64
Compare
kernel-patches-daemon-bpf
bot
force-pushed
the
bpf-next_base
branch
from
7b6b51d to
2412df8
Compare
Author
|
Upstream branch: d6ec090 |
vm_area_map_pages() may allocate memory while inserting pages into bpf arena's vm_area. In order to make bpf_arena_alloc_pages() kfunc non-sleepable change bpf arena to populate pages without allocating memory: - at arena creation time populate all page table levels except the last level - when new pages need to be inserted call apply_to_page_range() again with apply_range_set_cb() which will only set_pte_at() those pages and will not allocate memory. - when freeing pages call apply_to_existing_page_range with apply_range_clear_cb() to clear the pte for the page to be removed. This doesn't free intermediate page table levels. Signed-off-by: Puranjay Mohan <[email protected]>
To make arena_alloc_pages() safe to be called from any context, replace kvcalloc() with kmalloc_nolock() so as it doesn't sleep or take any locks. kmalloc_nolock() returns NULL for allocations larger than KMALLOC_MAX_CACHE_SIZE, which is (PAGE_SIZE * 2) = 8KB on systems with 4KB pages. So, round down the allocation done by kmalloc_nolock to 1024 * 8 and reuse the array in a loop. Signed-off-by: Puranjay Mohan <[email protected]>
Make arena related kfuncs any context safe by the following changes: bpf_arena_alloc_pages() and bpf_arena_reserve_pages(): Replace the usage of the mutex with a rqspinlock for range tree and use kmalloc_nolock() wherever needed. Use free_pages_nolock() to free pages from any context. apply_range_set/clear_cb() with apply_to_page_range() has already made populating the vm_area in bpf_arena_alloc_pages() any context safe. bpf_arena_free_pages(): defer the main logic to a workqueue if it is called from a non-sleepable context. specialize_kfunc() is used to replace the sleepable arena_free_pages() with bpf_arena_free_pages_non_sleepable() when the verifier detects the call is from a non-sleepable context. In the non-sleepable case, arena_free_pages() queues the address and the page count to be freed to a lock-less list of struct arena_free_spans and raises an irq_work. The irq_work handler calls schedules_work() as it is safe to be called from irq context. arena_free_worker() (the work queue handler) iterates these spans and clears ptes, flushes tlb, zaps pages, and calls __free_page(). Signed-off-by: Puranjay Mohan <[email protected]>
As arena kfuncs can now be called from non-sleepable contexts, test this by adding non-sleepable copies of tests in verifier_arena, this is done by using a socket program instead of syscall. Add a new test case in verifier_arena_large to check that the bpf_arena_alloc_pages() works for more than 1024 pages. 1024 * sizeof(struct page *) is the upper limit of kmalloc_nolock() but bpf_arena_alloc_pages() should still succeed because it re-uses this array in a loop. Augment the arena_list selftest to also run in non-sleepable context by taking rcu_read_lock. Signed-off-by: Puranjay Mohan <[email protected]>
kernel-patches-daemon-bpf
bot
force-pushed
the
series/1024361=>bpf-next
branch
from
11d7e64 to
d573c74
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull request for series with
subject: Remove KF_SLEEPABLE from arena kfuncs
version: 3
url: https://patchwork.kernel.org/project/netdevbpf/list/?series=1024361