Run Regent on a distributed system

[crl123]

Good afternoon,
I am running Regent on my cluster of 9 node with the following parameters:
mpirun -np 9 -ppn 1 ./TaskBench/task-bench/regent/main.shard14 -steps 10 -type fft -kernel compute_bound -iter 1000000
And it is giving me the following problem:
main.shard14: core.cc:588: void TaskGraph::execute_point(long int, long int, char*, size_t, const char**, const size_t*, size_t, char*, size_t) const: Assertion `input[i].second == dep' failed.

= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= EXIT CODE: 6
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES

APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)
And sometimes the following problem:
main.shard14: core.cc:565: void TaskGraph::execute_point(long int, long int, char*, size_t, const char**, const size_t*, size_t, char*, size_t) const: Assertion `offset <= point && point < offset+width' failed.

= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= EXIT CODE: 6
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES

APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)
I have the same problem when I use the tree type, but when I use the stencil_1d type I don't have the problem.
I compile regent as follows:
DEFAULT_FEATURES=0 USE_REGENT=1 ./get_deps.sh
export CXX=mpicxx
export CC=mpicc
./build_all.sh
Thank you in advance for your help,

[elliottslaughter]

Hi @crl123,

This means that Task Bench is computing the wrong result. I'm a little confused, I thought the Regent implementation was fully debugged.

I'm not expecting this to make a difference, but can you confirm what Task Bench branch/tag you're on?

I'll try to confirm on my end as well.

[crl123]

I'm on the 'origin/master' branch.
I updated the repository in my local machine on this Sunday.

[elliottslaughter]

Ok, I'm a bit swamped with things going on this week, but I'll try to find time to verify the Regent implementation on my own machine.

[elliottslaughter]

Sorry for taking so long to get back to this.

Looking back at your configuration here, I don't see any settings for the network. Typically you'd use something like:

export USE_GASNET=1
export CONDUIT=aries

Otherwise what you're doing is running N copies of the single-node program. Which is probably why this is misbehaving.

[ysfess22]

Hi @elliottslaughter. I have a further question about multi-node benchmarks.
Using gasnet the way you explained for a cluster with two nodes (udp conduit) creates double the number of tasks in the graph; half of the tasks is ran by node 1 and the other half by node 2. Is that the expected behaviour? Or is there a way to have the tasks be split between nodes? E.g., Given a 10x10 stencil graph, the 100 tasks would be split between two nodes.

[elliottslaughter]

@ysfess22 Please submit this as a new issue unless it's specifically related to the original posting.

The answer will depend on how you have configured your system, and I will require more information, which will clog this thread if it's not specifically related.