mdtest tasks stuck when trying to run on 6 or more hosts

[klin938]

I am trying to do some benchmarking on our cluster. My used the command:
mpirun -np 12 --hostfile ./machinefile ./mdtest -b 4 -z 3 -I 10 -d /PATH/TO/NFS_STORAGE

I started with two hosts, then adding one node each time. Eventually the mdtest stuck when I was trying to use six nodes. Typically the task output stuck as this one forever:

$ mpirun -np 12 --hostfile ./machinefile ./mdtest -b 4 -z 3 -I 10 -d /PATH/TO/NFS_STORAGE
-- started at 10/14/2016 11:40:35 --

mdtest-1.9.3 was launched with 12 total task(s) on 6 node(s)
Command line used: ./mdtest -b 4 -z 3 -I 10 -d /PATH/TO/NFS_STORAGE
Path: /share/ScratchGeneral/luffy
FS: 90.9 TiB Used FS: 74.0% Inodes: 181.9 Mi Used Inodes: 74.0%
^Cmpirun: killing job...

I am using mdtest-1.9.3 on CentOS 6.8. Any idea will be appreciated.

Cheers

[klin938]

I did another simple MPI test on my cluster, the same machinefile with six nodes went well without an issue:

$ mpirun -np 12 --hostfile ./machinefile ./hello_world
Hello world from processor delta-4-6.local, rank 7 out of 12 processors
Hello world from processor delta-4-6.local, rank 1 out of 12 processors
Hello world from processor delta-4-5.local, rank 0 out of 12 processors
Hello world from processor delta-4-9.local, rank 3 out of 12 processors
Hello world from processor delta-4-5.local, rank 6 out of 12 processors
Hello world from processor delta-4-9.local, rank 9 out of 12 processors
Hello world from processor delta-5-0.local, rank 5 out of 12 processors
Hello world from processor delta-4-7.local, rank 2 out of 12 processors
Hello world from processor delta-5-2.local, rank 4 out of 12 processors
Hello world from processor delta-5-2.local, rank 10 out of 12 processors
Hello world from processor delta-5-0.local, rank 11 out of 12 processors
Hello world from processor delta-4-7.local, rank 8 out of 12 processors

So it seems not to be caused by MPI itself....

[brettkettering]

Frankly, the branching (-b), depth (-z), and items per node (-I) are arguments that we never use in our work. This may be the reason the program does not terminate.

Let's isolate those by not using them. Try running:

$ mpirun -np 12 --hostfile ./machinefile ./mdtest -n 10 -C -F -d /PATH/TO/NFS_STORAGE

That will create 10 files per process in the directory specified by -d. If this works, we can start to isolate which phase is hanging on you. It's been a while since I used mdtest, but as I recall the default behavior is to create (-C), stat (-T), read (-E), and delete (-r) in that order.

Generally we don't read the files in our experiments, rather we use IOR for that purpose. So, you could try each phase on its own:

$ mpirun -np 12 --hostfile ./machinefile ./mdtest -C -b 4 -z 3 -I 10 -d /PATH/TO/NFS_STORAGE

then,

$ mpirun -np 12 --hostfile ./machinefile ./mdtest -T -b 4 -z 3 -I 10 -d /PATH/TO/NFS_STORAGE

then,

$ mpirun -np 12 --hostfile ./machinefile ./mdtest -E -b 4 -z 3 -I 10 -d /PATH/TO/NFS_STORAGE

then,

$ mpirun -np 12 --hostfile ./machinefile ./mdtest -r -b 4 -z 3 -I 10 -d /PATH/TO/NFS_STORAGE

See if you can get through any individual phase and terminate normally using the branching, depth, and items per node arguments.

Brett

[atorrez]

Thanks for the suggestions Brett.

One more suggestion would be to reduce the -b and -z arguments. He is targeting NFS and that may be a limitation although I do not think it should be for this size of job.

I just wanted to add that I ran the same command as the user specified on mdtest 1.9.3 and it worked, targeting a parallel file system.

When I have some time, I will look into this further.

Alfred

From: Brett Kettering <notifications@github.commailto:notifications@github.com>
Reply-To: MDTEST-LANL/mdtest <reply@reply.github.commailto:reply@reply.github.com>
Date: Friday, October 14, 2016 at 8:01 AM
To: MDTEST-LANL/mdtest <mdtest@noreply.github.commailto:mdtest@noreply.github.com>
Subject: Re: [MDTEST-LANL/mdtest] mdtest tasks stuck when trying to run on 6 or more hosts (#9)

Frankly, the branching (-b), depth (-z), and items per node (-I) are arguments that we never use in our work. This may be the reason the program does not terminate.

Let's isolate those by not using them. Try running:

$ mpirun -np 12 --hostfile ./machinefile ./mdtest -n 10 -C -F -d /PATH/TO/NFS_STORAGE

That will create 10 files per process in the directory specified by -d. If this works, we can start to isolate which phase is hanging on you. It's been a while since I used mdtest, but as I recall the default behavior is to create (-C), stat (-T), read (-E), and delete (-r) in that order.

Generally we don't read the files in our experiments, rather we use IOR for that purpose. So, you could try each phase on its own:

$ mpirun -np 12 --hostfile ./machinefile ./mdtest -C -b 4 -z 3 -I 10 -d /PATH/TO/NFS_STORAGE

then,

$ mpirun -np 12 --hostfile ./machinefile ./mdtest -T -b 4 -z 3 -I 10 -d /PATH/TO/NFS_STORAGE

then,

$ mpirun -np 12 --hostfile ./machinefile ./mdtest -E -b 4 -z 3 -I 10 -d /PATH/TO/NFS_STORAGE

then,

$ mpirun -np 12 --hostfile ./machinefile ./mdtest -r -b 4 -z 3 -I 10 -d /PATH/TO/NFS_STORAGE

See if you can get through any individual phase and terminate normally using the branching, depth, and items per node arguments.

Brett

You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/9#issuecomment-253808079, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AB5L1i7fp0oCIDobzCSsAcSjvqVg1v7mks5qz4tRgaJpZM4KWozK.

[klin938]

Thanks guys, I will give it a try