Runtime performance: reduce the number of threads and the synchronization

[dzufferey]

Right now we Netty has its own thread pool and we use an additional thread per instance. This is too much. We need to move back to a version where the threads are decoupled from the instances and pooled. There was an earlier version doing that (which was faster than the current version) but the code was insanely complex.

The tasks are:

• create an instance
• destroy an instance
• receiving a message
• react to a timeout

Question: Do we need a finer granularity ?

Each task needs to own the instance since the semantics of an instance is sequential.

Question: What does a thread do when its task works on a instanced already owned by another thread?

• Do we block? (the easiest to implement but not so efficient)
• Do we push the additional task to the thread which own the instance? This might be some form or reverse work-stealing. (The synchronization protocol between the threads might be tricky. When pushing a task, be careful about weak memory issues)

In the current version, each thread owns an instance. The receive tasks are pushed through the message buffer. The thread is responsible for the timeout, and the destruction comes through interruption.

[dzufferey]

In some example, we have blocking operation within the update methods, e.g., synchronization across multiple instances in parallel. For this use case, using separate threads for the different instances make sense.

Ideally, we should be able to provide both a runtime with thread pooling and using one thread per instance.