Not really finalising even with MPI_Finalize() when `LD_PRELOAD`ed

[kingshuk00]

With the following minimal example, I can confirm that with preloaded the libmpitrace.so, the PMPI_Finalize() is not called with MPI_Finalize() in code.

#include"mpi.h"
#include<stdio.h>

int main(int argc, char *argv[])
{
  int ret= MPI_Init(&argc, &argv);
  int rank= -1;
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);

  ret= MPI_Finalize();
  {
    int initialised= -1, finalised= -1;
    MPI_Initialized(&initialised);
    MPI_Finalized(&finalised);
    printf("%d: initialised= %d, finalised= %d\n", rank, initialised, finalised);
    fflush(stdout);
  }
  return ret;
}

$ mpicc -O3 simple.c
$ mpirun -n 2 ./a.out
0: initialised= 1, finalised= 1
1: initialised= 1, finalised= 1
$ mpirun -n 2 env EXTRAE_CONFIG_FILE=./extrae.xml LD_PRELOAD=${EXTRAE_ROOT}/lib/libmpitrace.so ./a.out
1: initialised= 1, finalised= 0
0: initialised= 1, finalised= 0

This is because of the check at src/tracer/wrappers/MPI/mpi_wrapper.c:2287 which will never be fulfilled when preloading.
This causes unexpected SIGTERM when exiting larger jobs.
Is this intended?

[gllort]

Yes, this is intended behavior. When preloaded, the tracing library initializes in its constructor (before MPI_Init) and finalizes via an atexit handler at program termination. This design lets it capture events even after MPI_Finalize (e.g., system calls) and launch the merger step in parallel while the MPI environment is still active.

To have the tool finalize at MPI_Finalize instead, set the environment variable EXTRAE_SKIP_AUTO_LIBRARY_INITIALIZE=1. This disables the constructor so that initialization occurs at MPI_Init and finalization at MPI_Finalize.

[kingshuk00]

Thank you for your reply. This is helpful.
In this scenario, would it makes sense to include a check and finalize it afterall in one of the atexits or destructors?
I'll close this issue either way.

[kingshuk00]

I'm closing this. Feel free to comment whether putting a check and finalize in one of the cleanup functions is a good idea.

[gllort]

I'm reopening this issue. It's a good idea to include a final call to MPI_Finalize during the cleanup phase. Even in successful runs, we've observed various errors that appear to result from improper MPI finalization, likely due to MPI_Finalize not being called when EXTRAE_SKIP_AUTO_LIBRARY_INITIALIZE is not set.

[kingshuk00]

There is a possible fix for this sitting in PR #134