Windows: classic queues run into eacces filesystem errors in rabbit_classic_queue_store_v2:recover_segments/5

[ophirzk]

Describe the bug

After discovering issue: #14094 I upgraded to RabbitMQ 4.1.2 but the issue seems to reproduce in 4.1.2.

Erlang version used
27.3.x

Operating system (distribution) used
Windows Server 2019

How is RabbitMQ deployed?
Windows installer

{{{badmatch,{error,eacces}}, [{rabbit_classic_queue_index_v2, recover_segments,5, [{file, "src/rabbit_classic_queue_index_v2.erl"}, {line,312}]}, {rabbit_classic_queue_index_v2, recover_segments,8, [{file, "src/rabbit_classic_queue_index_v2.erl"}, {line,285}]}, {rabbit_classic_queue_index_v2, recover,7, [{file, "src/rabbit_classic_queue_index_v2.erl"}, {line,255}]}, {rabbit_variable_queue,init,5, [{file, "src/rabbit_variable_queue.erl"}, {line,457}]}, {rabbit_priority_queue,init,3, [{file, "src/rabbit_priority_queue.erl"}, {line,150}]}, {rabbit_amqqueue_process,init_it2, 3, [{file, "src/rabbit_amqqueue_process.erl"}, {line,216}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"}, {line,1056}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"}, {line,329}]}]}, {gen_server2,call, [<0.11162.0>,info,infinity]}}.

supervisor: {<0.1569.0>,rabbit_amqqueue_sup}, errorContext: child_terminated, reason: {{badmatch,{error,eacces}}, [{rabbit_classic_queue_index_v2,recover_segments,5, [{file,"src/rabbit_classic_queue_index_v2.erl"},{line,312}]}, {rabbit_classic_queue_index_v2,recover_segments,8, [{file,"src/rabbit_classic_queue_index_v2.erl"},{line,285}]}, {rabbit_classic_queue_index_v2,recover,7, [{file,"src/rabbit_classic_queue_index_v2.erl"},{line,255}]}, {rabbit_variable_queue,init,5, [{file,"src/rabbit_variable_queue.erl"},{line,457}]}, {rabbit_priority_queue,init,3, [{file,"src/rabbit_priority_queue.erl"},{line,150}]}, {rabbit_amqqueue_process,init_it2,3, [{file,"src/rabbit_amqqueue_process.erl"},{line,216}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"},{line,1056}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,329}]}]}, offender: [{pid,<0.11164.0>}, {id,rabbit_amqqueue}, {mfargs, {rabbit_amqqueue_process,start_link, [{amqqueue, {resource,<<"/">>,queue, <<"vsb.retention_topic=events_flow_output">>}, true,false,none,[],<0.12155.0>,[],[],[],undefined, undefined,[],undefined,live,0,[],<<"/">>, #{user => <<"RabbitMQ">>}, rabbit_classic_queue,#{}}, <0.1568.0>]}}, {restart_type,transient}, {significant,true}, {shutdown,600000}, {child_type,worker}],


{<0.1569.0>,rabbit_amqqueue_sup}, errorContext: child_terminated, reason: {{badmatch,{error,eacces}}, [{rabbit_classic_queue_index_v2,recover_segments,5, [{file,"src/rabbit_classic_queue_index_v2.erl"},{line,312}]}, {rabbit_classic_queue_index_v2,recover_segments,8, [{file,"src/rabbit_classic_queue_index_v2.erl"},{line,285}]}, {rabbit_classic_queue_index_v2,recover,7, [{file,"src/rabbit_classic_queue_index_v2.erl"},{line,255}]}, {rabbit_variable_queue,init,5, [{file,"src/rabbit_variable_queue.erl"},{line,457}]}, {rabbit_priority_queue,init,3, [{file,"src/rabbit_priority_queue.erl"},{line,150}]}, {rabbit_amqqueue_process,init_it2,3, [{file,"src/rabbit_amqqueue_process.erl"},{line,216}]}, {gen_server2,handle_msg,2, [{file,"gen_server2.erl"},{line,1056}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,329}]}]}, offender: [{pid,<0.11160.0>}, {id,rabbit_amqqueue}, {mfargs, {rabbit_amqqueue_process,start_link, [{amqqueue, {resource,<<"/">>,queue, <<"vsb.retention_topic=events_flow_output">>}, true,false,none,[],<0.12155.0>,[],[],[],undefined, undefined,[],undefined,live,0,[],<<"/">>, #{user => <<"RabbitMQ">>}, rabbit_classic_queue,#{}}, <0.1568.0>]}}, {restart_type,transient}, {significant,true}, {shutdown,600000}, {child_type,worker}],

These eaccess errors seems like they have causes some queue to crash and not recover:
Channel error on connection <0.229992.0> ([::1]:60991 -> [::1]:5672, vhost: '/', user: 'RabbitMQ'), channel 4:, operation queue.declare caused a channel exception not_found: queue 'vsb.topic=short_id=5;role=Tasks;task_type=short_task;' in vhost '/' has crashed and failed to restart

Reproduction steps

1. Install RabbitMQ 4.1.2 using Windows Installer
2. Run 20+ publishers
3. Publisher to header exchanges bound to durable queues with different headers
4. After several hours we get eaccess and queue start to crash
   ...

Expected behavior

• No eaccess
• Queues don't crash

Additional context

No response

[michaelklishin]

@ophirzk "run a bunch of publishers for N hours" is not particularly helpful, most real world RabbitMQ installations do exactly that and most do not run into eacces exceptions, including most Windows users.

The author of #14094 did provide enough details and then twice given us feedback on the effectiveness of the changes after running a 4.1.2 previous for several days.

[ophirzk]

@michaelklishin
The issue reproduces under stress. It will be hard to give a step by step repro scenario.
In addition, I don't see that in #14094 there were more details on the repro steps:
"Steps to reproduce the behavior in question
I am unsure what causes this behavior. It happens intermittantly. Other servers with the exact same RabbitMQ/Erlang combination do not have this problem."

[michaelklishin]

We will see where else in the message store or CQ code the same "retries mechanism" from #14108 #14178 can/have to be applied.

I do not have an ETA. The workarounds are well known:

1. Do not use Windows (the eacces error when a file handle still in use is closed by the same process is a Windows-specific filesystem API design choice)
2. Switch to QQs and/or streams for the affected parts. In theory the issue can affect anything that uses files but somehow we haven't see it with QQs or streams

[michaelklishin]

@lhoguin perhaps #14108 and #14178 need to be extended to a few more functions in rabbit_classic_queue_store_v2, the same way #14178 was an extension of #14108 in a different place.

[tomerrt]

@michaelklishin , @lhoguin, @ophirzk

Thank you for looking into that and trying to workaround it.

Looking more into that, it appears that Microsoft changed the default delete file semantic versions of windows 10 > 18363.657 to be posix compliant (FILE_DISPOSITION_POSIX_SEMANTICS). Meaning it is deleted right away (but still allow access by existing handles to the data stream). If I understand, this would allow you to then open the file again and get a new link. See more here _FILE_DISPOSITION_INFORMATION_EX (ntddk.h) - Windows drivers | Microsoft Learn., and here DeleteFileA function (fileapi.h) - Win32 apps | Microsoft Learn

Based on this: Offer Posix-style file and directory deletion APIs · Issue #32737 · dotnet/runtime
"Starting with Windows 10 version 1909, (at least) build 18363.657 the DeleteFile Windows API function now exhibits synchronous behavior at the Windows API level"

If that is correct, it means that the error is dependent on the OS version. From our current data, we see the issue only on windows server 2019, and not on windows server 2022. @lhoguin , it worth checking if you get the same issue after upgrading to WS2022.

While in WS2022 the default delete semantic was changed to FILE_DISPOSITION_POSIX_SEMANTICS, it is still achievable in previous versions by calling ZwSetInformationFile and setting FILE_DISPOSITION_INFORMATION_EX::FILE_DISPOSITION_POSIX_SEMANTICS before deleting the file.

I wonder if that would be considered a more bulletproof solution to this race condition.

[michaelklishin]

I wonder if that would be considered a more bulletproof solution to this race condition.

Perhaps. Since RabbitMQ is based on an operating system-neutral runtime, we will have to investigate if said setting is exposed to the file I/O API we use, or its lower-level version (prim_file).

We will report our findings here, of course.

[ophirzk]

@michaelklishin
I think we can either implement a NIF and call Windows API with POSIX semantics instead of using Erlang's file module. Or changing the implementation of Erlang's file module so by default it will pass the POSIX flags to the Win APIs. As you said, RabbitMQ should be OS neutral, and making sure file operations have the same semantics is part of it. It can be done by enforcing POSIX semantics in Windows.

Retrying 3 times and waiting for 10ms is just a shot in the air and not a fix. I believe enforcing POSIX semantics is the right solution. Until then, I believe that a warning should be generated in RabbitMQ 4.1.x during classic queue initialization in case the OS is older than Windows 2022.

[michaelklishin]

@ophirzk an Erlang NIF (native code extension) plugin is also an option for those willing to develop and deploy it.

[ophirzk]

@michaelklishin Where would be the right place to put the native code implementation within RabbitMQ server folder structure? Do we need a seperate build process to compile the native code as part of RabbitMQ server build process?

[lhoguin]

The fix is a best effort to unblock some users. It mirrors another fix that was made for QQs but is obviously far from ideal.

A proper fix would be to upgrade to a more recent Windows version, or to make Erlang/OTP include the POSIX semantics flag in its file:delete operations (and then upgrade to the Erlang/OTP version that includes this fix). There's already a function that accepts options, but what we need isn't there and it's not likely that OTP team wants to add a Windows-only option to that module.

I do not think that a NIF is a good solution for this as you would have to replace all file:delete calls with the NIF, and that can't easily be done as a RabbitMQ plugin. You would have to use Meck or similar and that's not something I would recommend for production.