Unix Socket Transport plugin causes a crash while under load

[timothyfroehlich]

20-03-04 20:19:38.992 ERR arch_node_tasks.tasks.broadcast_forwarder.forwarder Unexpected main loop error:
Exception

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/arch_node_tasks/tasks/broadcast_forwarder/forwarder.py", line 238, in main_loop
    self._scan(hw.scan())
  File "</usr/lib/python3.7/site-packages/decorator.py:decorator-gen-31>", line 2, in scan
  File "/usr/lib/python3.7/site-packages/typedargs/utils.py", line 32, in _check_and_execute
    retval = func(*convargs, **convkw)
  File "/usr/lib/python3.7/site-packages/iotile/core/hw/hwmanager.py", line 764, in scan
    devices = self.stream.scan(wait=wait)
  File "/usr/lib/python3.7/site-packages/iotile/core/hw/transport/adapterstream.py", line 137, in scan
    self._loop.run_coroutine(self.adapter.probe())
  File "/usr/lib/python3.7/site-packages/iotile/core/utilities/async_tools/event_loop.py", line 536, in run_coroutine
    return future.result()
  File "/usr/lib/python3.7/concurrent/futures/_base.py", line 435, in result
    return self.__get_result()
  File "/usr/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.7/site-packages/iotile_transport_socket_lib/generic/device_adapter.py", line 78, in probe
    await self._send_command(OPERATIONS.PROBE, None, COMMANDS.ProbeResponse)
  File "/usr/lib/python3.7/site-packages/iotile_transport_socket_lib/generic/device_adapter.py", line 238, in _send_command
    return await self.client.send_command(name, args, verifier, timeout=timeout)
  File "/usr/lib/python3.7/site-packages/iotile_transport_socket_lib/generic/socket_client.py", line 137, in send_command
    await self._implementation.send(packed)
  File "/usr/lib/python3.7/site-packages/iotile_transport_socket_lib/unix_socket/unixsocket_implementation.py", line 117, in send
    await self.con.send(encoded)
  File "/usr/lib/python3.7/site-packages/iotile_transport_socket_lib/generic/abstract_socket_implementation.py", line 48, in send
    raise ConnectionError from Exception
ConnectionError
20-03-04 20:19:39.028 DEB iotile.core.utilities.async_tools.event_loop Stopping task socket_client
20-03-04 20:19:39.030 ERR iotile.core.utilities.async_tools.event_loop object NoneType can't be used in 'await' expression
20-03-04 20:19:39.039 DEB transitions.core Initiating transition from state connected to state disconnected...
20-03-04 20:19:39.039 DEB transitions.core Exiting state connected. Processing callbacks...
20-03-04 20:19:39.040 INF transitions.core Exited state connected
20-03-04 20:19:39.040 DEB transitions.core Entering state disconnected. Processing callbacks...
20-03-04 20:19:39.041 INF transitions.core Entered state disconnected
20-03-04 20:19:41.048 DEB iotile.core.utilities.async_tools.event_loop Stopping task <Task finished coro=<AsyncSupervisorClient.register_service() done, defined at /usr/lib/python3.7/site-packages/iotilegateway/supervisor/client.py:377> result=None>
20-03-04 20:19:41.049 DEB iotile.core.utilities.async_tools.event_loop Stopping task <Task finished coro=<AsyncSupervisorClient.update_state() done, defined at /usr/lib/python3.7/site-packages/iotilegateway/supervisor/client.py:265> result=None>
20-03-04 20:19:41.051 DEB iotile.core.utilities.async_tools.event_loop Stopping task <Task finished coro=<AsyncSupervisorClient.register_agent() done, defined at /usr/lib/python3.7/site-packages/iotilegateway/supervisor/client.py:397> result=None>
20-03-04 20:19:41.052 DEB iotile.core.utilities.async_tools.event_loop Stopping task supervisor_client
20-03-04 20:19:41.053 DEB iotile.core.utilities.async_tools.event_loop Stopping task <Task pending coro=<MqttClient._subscription_loop() running at /usr/lib/python3.7/site-packages/arch_node_tasks/utilities/mqtt_client.py:145> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x74b45130>()]>>
20-03-04 20:19:41.055 DEB iotile.core.utilities.async_tools.event_loop Stopping task <Task finished coro=<MqttClient.subscribe() done, defined at /usr/lib/python3.7/site-packages/arch_node_tasks/utilities/mqtt_client.py:121> result=None>
20-03-04 20:19:41.057 INF iotilegateway.supervisor.client Closing connection to server due to stop()
20-03-04 20:19:41.068 DEB iotile.core.utilities.async_tools.event_loop A task wasn't properly registered/cancelled as expected: <Task cancelling coro=<sleep() running at /usr/lib/python3.7/asyncio/tasks.py:595> wait_for=<Future cancelled> cb=[gather.<locals>._done_callback() at /usr/lib/python3.7/asyncio/tasks.py:691]>
20-03-04 20:19:41.069 DEB iotile.core.utilities.async_tools.event_loop A task wasn't properly registered/cancelled as expected: <Task pending coro=<_repeat() running at /usr/lib/python3.7/site-packages/iotile/core/utilities/async_tools/event_loop.py:753> wait_for=<_GatheringFuture pending cb=[<TaskWakeupMethWrapper object at 0x72deb230>()]> cb=[_chain_future.<locals>._call_set_state() at /usr/lib/python3.7/asyncio/futures.py:355]>
20-03-04 20:19:41.070 DEB iotile.core.utilities.async_tools.event_loop A task wasn't properly registered/cancelled as expected: <Task pending coro=<ProtocolHandler.mqtt_deliver_next_message() running at /usr/lib/python3.7/site-packages/hbmqtt/mqtt/protocol/handler.py:461> wait_for=<Future cancelled>>
20-03-04 20:19:41.074 ERR iotile.core.utilities.async_tools.event_loop Error stopping task <Task cancelled coro=<sleep() done, defined at /usr/lib/python3.7/asyncio/tasks.py:582>>: CancelledError()
20-03-04 20:19:41.075 DEB iotile.core.utilities.async_tools.event_loop Finished loop in background thread
20-03-04 20:19:41.191 ERR asyncio Future exception was never retrieved
future: <Future finished exception=CancelledError()>
concurrent.futures._base.CancelledError
sys:1: RuntimeWarning: coroutine 'wait_for' was never awaited
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
root@arch-ap-002a:~# 
[1] 0:bash*Z