Better suppor for interactive apps in Multi-Node

[danielfrg]

When wrapping cuPyNumeric (and legate code in general) in interactive Apps (streamlit) or APIs (FastAPI) cuPyNumeric code will get executed in different order, when clicking in the app or making a request.

This works fine in Single-Node but for Multi-Node the execution of the code will diverge and creates issues.

We should support this scenario. with either a client-server architecture or also safe single node execution that allows one rank to trigger compute in others?

[lightsighter]

or also safe single node execution that allows one rank to trigger compute in others?

Legion already supports this, but there will be limits to its scalability depending on task granularity. You will not be able to program 1000s (and probably not even 100s) of nodes like this.

[manopapad]

Opened nv-legate/legate#1008 for one possible solution to this problem. As was mentioned already, providing a single-controller execution mode would be another (orthogonal) way to solve the same problem, but that's a longer-term item.