Async switch

[asmodehn]

Since pyzmp role is mostly to transfer messages from Node A to Node B, and our current usecase is to transfer messages between Web Servers and Ros Nodes, It makes sense to use an async framework.
Check : http://klen.github.io/py-frameworks-bench/

Since we also have to support python 2.7 we have a number of candidates :

• trollius -> deprecated
• asyncio -> python3 only
• twisted -> looks good. Might move to support asyncio. Autobahn supports twisted and asyncio.
• tornado -> primary usecase is web. our usecase is more generic.

So twisted seems the way to go.

@dhirajdhule I have thought about that for a while, and I think this is the way to go for pyzmp. To start integrating twisted to replace our main loop in there, and have it managed by a long time proven and maintained python package.

To be clear, the communication between two processes is another topic, but inside one process, we should start implementing asynchronous behaviors. Later on we will be able to improve pyros and rostful with this as well.

[asmodehn]

Another interesting candidate : gevent
Because it is based on greenlet which supports both usecases :

• concurrent.futures API use case : threads returning future that handles result or exception (but does not allow to proactively interrupt long running threads -> it s up to the user )
• direct threading API use case : proactively interrupting long running thread (but does not handle result or exception -> it s up to the user)

From http://www.gevent.org/intro.html :

    join – waits until the greenlet exits;
    kill – interrupts greenlet’s execution;
    get – returns the value returned by greenlet or re-raises the exception that killed it.

From a first look, gevent based implementation would be complementary to twisted...
From what I understand so far, Twisted is full async (launching and managing multiple process on a machine) whereas gevent is async but only inside the same interpreter (will use only one CPU), unless we extend it with https://gehrcke.de/gipc/ ...
It might also be easier to migrate from the current code.

Just for memory, there are also a few benchmarks out there with other possible options:
http://nichol.as/asynchronous-servers-in-python
https://magic.io/blog/uvloop-blazing-fast-python-networking/
http://ptspts.blogspot.kr/2010/05/feature-comparison-of-python-non.html

[asmodehn]

my first try with gevent went simpler than I expected : https://gist.github.com/asmodehn/1c8ded26f543ce72b0686ca0bea76d27

[asmodehn]

Few problem with gevent though :

• actively killing the greenlet from outside prevent cleanup (probably the reason why it s not directly doable with futures API), and exception will not be thrown in parent thread (unless it happens to be blocking on get())
• start() method returns even before the child process has declared itself "ready" -> we need to wait a bit outside (not clean... but so far all async implementation I have seen have the same issue)

[asmodehn]

This will likely depend on how colos implements "collaborative processes" ( as a set of co(llaborative)routines ) and should be done after that.

There area few other roadbloacks : python3, ros distro compatibility, packaging, etc.
All in all, far from simple.