multi-threading for nuttx

[JanStaschulat]

Hi,

@jamoralp @pablogs9 @ralph-lange

I am running into problems for an application with multiple threads for NuttX and Olimex board. You know, that I try to get the multi-threaded executor for NuttX running.
What I have done so far:

• I tested the multi-threaded exectutor under Linux (Ubuntu 18.04, local Foxy installation), which works.
• I tested single-threaded executor with a ping-pong example (communication of Agent (linux, Ubuntu 18.04, Foxy) and Olimex/NuttX , branch Foxy works
• Now I am trying to get the multi-threading executor running with micro-ros on the Olimex board. The high level concept of multithreadd executor (with single-threaded access to RCL):
  • There is one executor thread, which accesses RCL layer (rcl_wait, rcl_take)
  • The executor thread spawns one thread for every subscription
  • If a new message arrives (tested with rcl_wait) , then it takes the data (with rcl_take), copies it and signals the worker thread to start, which will then call the callback of the subscription (using condition variables and locks)

When I try to run it on Olimex board, the threads start, but there is no progress in the main thread (aka executor).
The application uses two subscriptions and two publishers and one executor. So I don't need to change anything in the micro-ros configuration, right?

nuttx_apps/examples/uros_rbs/main_rbs.c

Line 216 in 0746a00

RCCHECK(rclc_executor_start_multi_threading_for_nuttx(&high_executor));

The processing just stops after calling rcl_wait
https://github.com/micro-ROS/rclc/blob/7a5d0d254f4dbf744b04f46a14fd05de061bbeb3/rclc/src/rclc/executor.c#L1559

However, rcl_wait might not be the problem - maybe something goes wrong with the other threads and then everything stops.
I configured also the priorities:

• main_thread (aka executor loop): 130
• worker thread 1: 120
• worker thread 2: 100
  So even if the worker threads are executing, the NuttX OS should be able to execute the executor loop.

I also noticed that sometimes the green light on the Olimex board starts blinking. after that not output is seen in the nsh shell (via screen terminal). What does that mean? Something really went wrong?

I wrote a simple program a main thread and two worker threads, which seems to works fine (without micro-ros functions used).

Is there anything regarding STACKSIZE I have to consider? Currently it is set to 65000 in the Makefile

When spawning threads (pthread_create), then no stack size is configured. What is the default stack size. Is it maybe too small, too large?

rclc executor:
https://github.com/micro-ROS/rclc/tree/feature/rbs-nuttx

application on olimex:
https://github.com/micro-ROS/nuttx_apps/tree/feature/foxy_rbs_executor_demo/examples/uros_rbs

[pablogs9]

Regarding the stack size, usually calling to micro-ROS functions use "big" stack:

• Crazyflie: 2250 Words = 9 kB
• Zephyr: 25 kB
• Mbed: 20 kB

These values are completely experimental and usually depend on the functionality used. The point is that every time I see an inexplicable crash in an embedded application, it usually is the stack.

This stack is only the main thread? The other threads share this amount?

[JanStaschulat]

I guess. How do I know? Do you think that 65000 (aka 65kB) is sufficient for all - or should I define an extra STACKSIZE for these threads?

If I configure a new thread like this:

nuttx_apps/examples/uros_rbs/main_rbs.c

Line 138 in 0746a00

// (void)pthread_attr_setstacksize(&attr, CONFIG_UROS_RBS_EXAMPLE_STACKSIZE);

the stack-size is exclusive for the thread?

[JanStaschulat]

The worker thread does not have any local data. However the thread gets as parameter this struct (with 4 pointers)
https://github.com/micro-ROS/rclc/blob/7a5d0d254f4dbf744b04f46a14fd05de061bbeb3/rclc/include/rclc/executor.h#L56

The worker thread
https://github.com/micro-ROS/rclc/blob/7a5d0d254f4dbf744b04f46a14fd05de061bbeb3/rclc/src/rclc/executor.c#L1442
only calls the callback of the subscription (which is defined by the user) - thats all:
https://github.com/micro-ROS/rclc/blob/7a5d0d254f4dbf744b04f46a14fd05de061bbeb3/rclc/src/rclc/executor.c#L1466

I wonder, what stacksize for such a thread should be - what would you suggest?

[JanStaschulat]

What is the maximum total STACKSIZE for all threads? How much memory is available for that at the Olimex board?

[pablogs9]

I'm not sure about how stack size is set for threads in Nuttx, AFAIK it should follow strictly the POSIX Thread API.

These samples that I provide are for threads that execute the whole micro-ROS application.

The handling of the available memory depends on the RTOS, for example, FreeRTOS let you create static memory blocks for thread stacks, and Zephyr (by default) use heap for allocating dynamically the thread stack. So I guess that the available memory depends on the heap/bss/data sections defined in the linker script because I guess that the stack section (if exists) will be used to startup code and RTOS initialization.

Are you sure that only one thread is accessing the middleware? In some tests that we did time ago, multithreaded access to the XRCE middleware breaks it really easy because of buffer corruption.

[JanStaschulat]

"Are you sure that only one thread is accessing the middleware? In some tests that we did time ago, multithreaded access to the XRCE middleware breaks it really easy because of buffer corruption."

Yes. I designed the multi-threaded executor in such a way, that only one thread accesses all calls to XRCE middleware.
However, a guard condition is signalled from the worker thread - but that should be okay, or not?

[pablogs9]

Please explain how the worker thread uses the guard condition.

[JanStaschulat]

The reason for the guard condition is to respond as fast as possible to incoming messages:

• First of all, the wait-set contains only the subscriptions for which its worker thread is ready. This is necessary because if the rcl_wait checks for new data and its corresponding worker thread is busy, then the executor thread will grab CPU 100% - as rcl_wait always come back immediately and the message will not be consumed with rcl_take.
• Therefore, if a worker thread is busy, then its corresponding subscription will not be of the wait set. In case, a new message fo r this corresponding subscription is available, then the executor will not notice and the rcl_wait call will block until the timeout. (one might argue that is still okay - but i want to process the next message - as soon as the worker thread is ready again)
• I wanted to have a way to wake up rcl_wait if the worker thread is ready again. For this reason, the worker thread has a guard_condition, which it will signal when the worker thread is ready again, aka after processing the subscription callback.
• This wakes up rcl_wait and in the next iteration of the executor loop, the wait-set is created with the corresponding subscription:

In the main-thread

• at initialization:

  • a guard_condition is created
  • guard condition is added to executor (no access to XRCE-micro-ros)
  • worker_threads are created and started

• in executor loop

  • wait_set is constructed with this guard_condition which calls rcl_wait_set_add_guard_condition
  • rcl_wait is called

In worker thread:

• after the callback of the subscription has been called, the worker thread is ready again and the guard_condition is signalled, calling rcl_trigger_guard_condition
• the call to rcl_trigger_guard_condition is guarded with a mutex, so multiple worker threads cannot call this funtion at the same time

[pablogs9]

Ok, I'm trying to understand this...

Some thoughts:

1. rcl_trigger_guard_condition at middleware level is not thread-safe: link. But I guess that if the worker is the only one in charge of triggering it, it is ok.
2. Ok I was reading the points one by one, you just solved this with the mutex....
3. The RMW of XRCE will handle the wait call by letting the XRCE session runs for a certain amount of time, check here. This means that the the rmw_wait() and the rcl_wait() call will only return before timeout if an XRCE data message arrives (Subscription, Request or Reply). Once the session has been ran, we check the guard conditions, check here. That means that if you are running a rcl_wait() for N ms, it will wait N ms despite a guard condition is triggered in between.

Let me know what do you think about this approach or if this interferes too much with your implementation. Maybe we can add some kind of concept of "guard condition" to the XRCE middleware in order to abort the session wait...

Last thing regarding this

I tested the multi-threaded exectutor under Linux (Ubuntu 18.04, local Foxy installation), which works.

Is possible for you to test the executor under Linux but using the XRCE-DDS middleware? Like we do with micro-ROS demos. This way we should be able to determine if this is a middleware problem and debug and fix it in such that case.

[JanStaschulat]

• increased STACKSIZE of application from 65000 to 68000 (69000 => software crashed)
• introduced mutex around all micro-ros functions: subscription callback calls rcl_publish which is called from worker thread. Limitation: micro-ros is single threaded. Now all functions: rcl_wait, rcl_take, rcl_publish are all guarded with the lock. Drawback: trade-off needs to be found between timeout for rcl_wait and throughput of worker_threads (which might publish data)
  • small timeout
    • frequent re-build of wait_set, greater overhead
    • higher troughput of user callback functions
  • large timeout
    • less frequent re-build of wait_set, minimal overhead
    • low throughput of user callback function (as they all have to wait for timeout or rcl_wait - if no new messages arrive)
• removed guard_condition, because as Pablo says, it does not wake up the rcl_wait earlier than the timeout

then I could run the ping-pong example successfully.

[pablogs9]

Jan, which one of these points makes the application works? I would like to know it because if it is a stack limitation it would be "ok".

But if we are having concurrency issues I would like to investigate a bit about making the library multithreaded because in this use-case it should be working theoretically with the current approach.

[JanStaschulat]

Definitly, this stacksize adjustment make it run. The configured stacksize is the total stacksize of the entire application (with two pthreads). The stacksizes of the threads are not configured - so I assume that they are using the stacksize of their spawning application.

The lock in the worker-thread around the the execution of the user-callback, which might call rcl_publish, and in the executor-thread (for rcl_wait and rcl_take) are necessary because micro-ros is single-threaded. For a multi-threaded micro-ros implementation, this lock would not be necessary. The potential waiting time when publishing messages in the user-callback or calling rcl_wait would be removed.

To summarize, this version of a multi-threaded rclc-executor works with the single-threaded micro-ros library.

It was designed to demonstrate budget-based scheduling with NuttX, but it can be used to assign priorities on Linux/FreeRTOS/Zephyr as well - assuming creation of pthreads, and assignment of priorities (sched_param) are supported.

[iluetkeb]

I've followed this on the side since Jan asked me about it earlier. The fact that we needed 65k stack, and now even 69k stack, has puzzled me for a long time and I would love to learn more about what this is for. It doesn't seem to match the memory requirement benchmarks for micro-xrce-dds, but I also don't see another big consumer of memory in this app.

[pablogs9]

@iluetkeb we have profiled the memory consumption of micro-ROS in FreeRTOS where you have complete control over dynamic memory allocations and stack of each task, as you can see here results are different from what we can see in Nuttx.

[JanStaschulat]

@pablogs9 The memory consumption measurements show publisher and subscriber in isolation. But what would be the memory for an application with two publishers and two subscribers? Can I just add these values:

• 2 publishers: 43,5kb (according to the diagramm reliable stream, BE stream is similar)
• 2 subscribers: 60kb (according to the diagramm reliable stream, BE stream is similar)
• total: 103,5kb???

Really?

The Olimex board STM32-E407 has only 196kB RAM.