corrupted darshan log file when using openMPI

[wkliao]

Attached zip file contains a Darshan log file generated by GitHub CI, ran with
the following commands (MPI is OpenMPI).

darshan-config
  --log-hints       -> cb_nodes=4

CMD=/usr/bin/mpiexec --oversubscribe -n 4 ./tst_mpi_io ./testfile.dat

ls -lt /home/runner/work/darshan/darshan/darshan_logs/2025/8/11
total 4
-r-------- 1 runner docker 3181 Aug 11 15:32 runner_tst_mpi_io_id14869-14869_8-11-55935-8583482323186166128_1.darshan

DARSGAN_PARSER=/home/runner/work/darshan/darshan/darshan_install/bin/darshan-parser

Error: Z_DATA_ERROR
Error: unable to inflate darshan log data.
Error: failed to read name hash from darshan log file.

logs.zip

[carns]

I suspect #1000 is related.

[carns]

I tried to reproduce this on my laptop and also was not able to do it. I had a theory that maybe it had something to do with the fact that the github ci runner probably doesn't have many (or maybe just one) virtual core available. I tried to simulate that configuration on my laptop by running the test like this:

~/working/install-openmpi-5.0.8/bin/mpiexec --cpu-set 0 -n 4 --oversubscribe ./tst_mpi_io testfile.dat nbytes=

The resulting log file was fine though, no corruption. This was with origin/main of Darshan and OpenMPI 5.0.8 compiled from source with default options.

[carns]

I also tried it with openMPI 5.0.7 from Ubuntu and couldn't trigger it there either.

[carns]

From strace I can see that OpenMPI's MPI-IO implementation is doing some sort of read/modify/write (data sieving?) on my laptop. There are significant pread calls to the log in addition to the pwrite calls on the output file descriptor, at any rate.

There are also accesses to control file in the same directory as the darshan log file as well as accesses to control files in /dev/shm, and several calls to fnctl SET_LCK options.

It may be worth checking the mount options in the github ci runner to make sure there aren't any unusual mount options for the file system we are writing to or /dev/shm that could alter behavior.

[carns]

From strace I can see that OpenMPI's MPI-IO implementation is doing some sort of read/modify/write (data sieving?) on my laptop. There are significant pread calls to the log in addition to the pwrite calls on the output file descriptor, at any rate.

There are also accesses to control file in the same directory as the darshan log file as well as accesses to control files in /dev/shm, and several calls to fnctl SET_LCK options.

It may be worth checking the mount options in the github ci runner to make sure there aren't any unusual mount options for the file system we are writing to or /dev/shm that could alter behavior.

There are also accesses to /tmp as well while the darshan log is being written.

[carns]

Another option might be to try something like act to run the docker images and actions locally so that its easier to inspect what's going on. I'm not sure how hard it is to set up though: https://github.com/nektos/act .

[wkliao]

After more testings using OpenMPI 4.1.4, 5.0.2, 5.0.3, ..., 5.0.8, I finally
find that the data sieving problem was actually fixed in 5.0.6. It turned out
my claim of using 5.0.8 also producing an error was incorrect. My bad.

PR #1070 was created to demonstrate my findings.

The next question for us is how Darshan should react when 5.0.5 and earlier was used...

[carns]

Fantastic!

Are we doing something that (inadvertently) triggers data sieving? It may also be worth looking to see what the heuristic is, because its not intuitive to me why that would be done with 4 processes writing a few KiB to ext4.

[carns]

I looked in the OpenMPI code out of curiosity. Once the code translates down to individual POSIX operations (regardless of how they originated at the MPI API level), it checks to see if it is being asked to write a vector and then does some heuristics to determine if it should activate data sieving or not for that vector:

https://github.com/open-mpi/ompi/blob/6ec63e2b69d4ef5b0e085d38abfab35d24008318/ompi/mca/fbtl/posix/fbtl_posix_pwritev.c#L58

The parameters that it is using for the heuristic are at https://github.com/open-mpi/ompi/blob/6ec63e2b69d4ef5b0e085d38abfab35d24008318/ompi/mca/fbtl/posix/fbtl_posix_component.c#L78

This is odd, because I don't believe that Darshan is intentionally writing anything with gaps in the file at any given stage of writing the log file (so I wouldn't anticipate any vectors at all; much less vectors with gaps between segments). The header is uncompressed and a fixed size, and then the remaining data (I think?) is all compressed into dense buffers. Either a mistake on our part (putting something at an unusual offset) or a mistake in the OpenMPI heuristic?

[wkliao]

I hack into Darshan to print the offset and length of each MPI write request made by each process.
Their patterns appear to me is only one-contiguous file space appending, no "vector".
Using these offset-length pairs, I wrote a small test program to recreate Darshan's write
operation. It failed when using OpenMPI 5.0.5, but not 5.0.6.

Maybe you are reading the OpenMPI main branch, not 5.0.5.

My guess is OpenMPI 5.0.5 triggers data sieving even for such appending pattern.
I would like to get some feedbacks from OpenMPI folks on how to disable data
sieving when using 5.0.5 and earlier. Even better if I know how to set MPI-IO hints
in general in OpenMPI. Their user guide is not clear to me.

In addition to cb_nodes, Darshan also sets romio_no_indep_rw, I do not think
OpenMPI implements romio_no_indep_rw.

[carns]

The reason I'm wondering about why data sieving is triggered is because I see data sieving happening on my laptop (there are fnctl locks and preadv calls mixed in with the pwritev calls) even in newer versions of OpenMPI. The log file appears to e written correctly, but data sieving looks entirely unnecessary.

I suspect that the bug that was fixed didn't alter whether data sieving was used or not, but just fixed a bug in it when it is activated. I think there may be an additional bug that is causing it to be activated in the first place, if the Darshan data is all contiguous.