Duplicate mmap log file at the beginning of darshan_core_shutdown #1053

wkliao · 2025-07-15T21:48:07Z

This is in case an interrupt occurs while processing the log file, resulting the log file corrupted.
See issue #1052.

carns

This looks good, just a few minor suggestions

darshan-runtime/lib/darshan-core.c

carns · 2025-07-18T15:21:10Z

darshan-runtime/lib/darshan-core.c

+                     final_core->mmap_log_name);
+            mmap_fd = open(dup_log_fame, O_CREAT | O_WRONLY, 0644);
+            if (mmap_fd != -1) {
+                write(mmap_fd, fileSize, buf);


we probably need to check the return code of the write here as well. If it fails we just do a best effort cleanup (unlink etc.).

I on purpose ignore all the error codes in this code block, as we
do not want any error to affect the log processing after this code block.
The duplicated file will be unlinked at the end, line 809, any way.

If this is OK, I can add some comments into the codes.

Yeah that strategy makes sense. It may be good to add a comment explaining.

Comments have been added in commit b3ce870.

darshan-runtime/lib/darshan-core.c

wkliao · 2025-07-18T20:57:01Z

I also notice that no CI is testing the configure option --enable-mmap-logs.
Maybe we should add one.

wkliao · 2025-07-18T23:34:20Z

The solution proposed in this PR only works when --enable-mmap-logs is set at configure time.
When this is not the case, we may need another solution.

carns · 2025-07-20T19:16:12Z

I also notice that no CI is testing the configure option --enable-mmap-logs. Maybe we should add one.

Yeah we really should.

carns

This looks great. Let's see if Mikaila can test it. If it works for her then this can be merged.

This is in case an interrupt occurs while processing the log file, resulting the log file corrupted.

…s used

…ntime

carns · 2025-08-21T19:45:07Z

darshan-runtime/lib/darshan-core.c

+    if (mmap_fd != -1) {
+        int dup_mmap_fd;
+        void *buf = (void*) malloc(final_core->mmap_size);
+        read(mmap_fd, buf, final_core->mmap_size);


FYI, this block of code produces the following warning with gcc 14.2:

../../../darshan-runtime/lib/darshan-core.c: In function ‘darshan_core_shutdown’: ../../../darshan-runtime/lib/darshan-core.c:494:9: warning: ignoring return value of ‘read’ declared with attribute ‘warn_unused_result’ [-Wunused-result] 494 | read(mmap_fd, buf, final_core->mmap_size); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ../../../darshan-runtime/lib/darshan-core.c:500:13: warning: ignoring return value of ‘write’ declared with attribute ‘warn_unused_result’ [-Wunused-result] 500 | write(dup_mmap_fd, buf, final_core->mmap_size); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I ignored them on purpose, as these codes are for file duplication, wouldn't
want any error stop the duplication.

Anyway, I added checks in 14a755c.
Strangely, I also run gcc 14.2.1, but did not get those warnings.
Please see if they are silenced.

carns · 2025-08-21T20:04:29Z

darshan-runtime/lib/darshan-core.c

+        void *buf = (void*) malloc(final_core->mmap_size);
+        read(mmap_fd, buf, final_core->mmap_size);
+        close(mmap_fd);
+        snprintf(dup_log_fame, strlen(final_core->mmap_log_name), "%s.dup",


This snprintf truncates the files name to something like /tmp/carns_python3_id1445957_mmap-log-3779006568372861245-0.darsha because the 2nd argument to is calling strlen() on the original mmap_log_name. Maybe this size limit on the snprintf should be based on the size of the dup_log_fname instead?

This should be fixed in 6a5c60f.

Actually I have one other request here. Sorry this didn't occur to me sooner. I like that the duplicate file has "dup" in the file name, but it may be better if it weren't used as the file extension. I find myself doing things like ls /tmp/*.darshan to look for intermediate files, when with this patch it would really need to be ls /tmp/*.darshan /tmp/*.darshan.dup to find them all.

Can we change the naming convention so that even the duplicate file names end with .darshan?

Another way is to run "ls /tmp/*.darshan*".

The log file name is constructed in:

darshan/darshan-runtime/lib/darshan-core.c

Lines 830 to 832 in b7e4e89

snprintf(core->mmap_log_name, __DARSHAN_PATH_MAX,

"/%s/%s_%s_id%d_mmap-log-%" PRIu64 "-%d.darshan",

core->config.mmap_log_path, cuser, __progname, jobid, logmod, my_rank);

Where do you suggest to insert substring "dup"?

There can be other things in /tmp with "darshan" in the name that aren't log files, though (I have a bunch for sure, though I'm an outlier because I've been experimenting with a lot of Darshan things :))

I'm not picky about where to put dup in the name (other than not at the end). Maybe just making the extension .dup.darshan?

Updated. Please give it a try.

wkliao · 2025-08-22T15:29:22Z

I have a doubt this issue may be related to the data sieving bug in OpenMPI in #1064.
Is it possible to get @mikail-g to confirm whether or not she is using OpenMPI, etc.?

wkliao added the Work in Progress label Jul 15, 2025

wkliao requested a review from carns July 15, 2025 21:48

carns requested changes Jul 18, 2025

View reviewed changes

wkliao force-pushed the copy_mmap_file branch from 6ac8902 to fa44852 Compare July 23, 2025 17:12

carns mentioned this pull request Jul 28, 2025

Processes being abruptly killed in the window of time between the start of shutdown and the shutdown completing #1052

Open

carns previously approved these changes Jul 28, 2025

View reviewed changes

wkliao mentioned this pull request Jul 30, 2025

Change mmap I/O default mode to enable #1055

Merged

wkliao added 4 commits July 30, 2025 16:07

Duplicate mmap log file at the beginning of darshan_core_shutdown

dad0e9b

This is in case an interrupt occurs while processing the log file, resulting the log file corrupted.

Move calls to unlink up, to right after darshan_log_finalize()

e081fe5

printf on screen whether or not configure option --enable-mmap-logs i…

99f568b

…s used

reuse mmap file size, mmap_size, added as a member of darshan_core_ru…

7af3cb4

…ntime

wkliao force-pushed the copy_mmap_file branch from fa44852 to 7af3cb4 Compare July 30, 2025 21:08

carns reviewed Aug 21, 2025

View reviewed changes

wkliao added 2 commits August 21, 2025 15:40

silence gcc 14.2 warning by checking return values

14a755c

fix snprintf length argument

6a5c60f

wkliao dismissed carns’s stale review via 6a5c60f August 21, 2025 20:44

Use ".dup.darshan" suffix for duplicated log file name

56f0c82

wkliao force-pushed the copy_mmap_file branch from 6a47d49 to 56f0c82 Compare August 25, 2025 16:47

	snprintf(core->mmap_log_name, __DARSHAN_PATH_MAX,
	"/%s/%s_%s_id%d_mmap-log-%" PRIu64 "-%d.darshan",
	core->config.mmap_log_path, cuser, __progname, jobid, logmod, my_rank);

Duplicate mmap log file at the beginning of darshan_core_shutdown #1053

Are you sure you want to change the base?

Duplicate mmap log file at the beginning of darshan_core_shutdown #1053

Uh oh!

Conversation

wkliao commented Jul 15, 2025

Uh oh!

carns left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

wkliao commented Jul 18, 2025

Uh oh!

wkliao commented Jul 18, 2025

Uh oh!

carns commented Jul 20, 2025

Uh oh!

carns left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wkliao commented Aug 22, 2025

Uh oh!

Uh oh!