Record service requests

[kunaltyagi]

Description

In ROS1, recording services was not implemented. However, actions were implemented on top of messages.

This made it possible to debug/enhance/test a system by recording just messages and replaying them. In ROS2, server client handshake is done over services. Would it be possible to implement a rosbag record of the sent goals?

Related Issues

ROS1: ros/ros_comm#250

Completion Criteria

Successful trigger of action by following this sequence:

1. ros2 run action_tutorials_py fibonacci_action_server
2. ros2 bag record -a --include-hidden # or any other flag
3. ros2 run action_tutorials_py fibonacci_action_client
4. Close all nodes
5. ros2 run action_tutorials_py fibonacci_action_server
6. ros2 bag play rosbag2_
7. Wait to see if server responds

This sequence with ROS1 works if the goal messages are recorded. If this also works with ROS2, we can consider this feature as "ready"

Implementation Notes / Suggestions

On the interface, we can have 3 different behaviors, all of which make sense from a certain perspective:

1. ros2 bag record --include-actions or ros2 bag record --include-services to allow replaying of service requests (most sensible, keeps concerns separate)
2. Make it default behavior to record action goal/cancellations but not services (same behavior from ROS1)
3. Extend the existing ros2 bag record --include-hidden-topics (better if not done this way)

Since the services and topics are implemented similarly at a RMW level, it should be possible to record the data sent via services.

On the other hand, just because all current RMW implementations use the same mechanism for topic and service communication, this might not hold true in future.

Testing Notes / Suggestions

The integrated testing for this would be quite simple:

1. start service/action server
2. start recording
3. call service/action
4. stop recording and play the bag
5. expect a call for the service/action

I don't know how it'd be unit tested

[emersonknapp]

Given that this is a feature request - I would appreciate if you could reformat using the feature request template instead of the bug report template - it will be easier to understand.

[Karsten1987]

@kunaltyagi while it's a valid feature request, it actually goes beyond the scope of rosbag2. For this to happen, we actually need support in ROS2 for serialized service (and action) calls. Unfortunately, up to today, we only have serialized messages.

[emersonknapp]

Are ROS 2 Services implemented under the hood using Topics? If so, these don't appear in the ros2 topic list --include-hidden-topics output, so they're "very hidden"

[clalancette]

Are ROS 2 Services implemented under the hood using Topics?

Correct, assuming a DDS RMW. All three of our in-tree RMWs use a pair of topics (prefixed with rq and rr, respectively) for the call and response side of the service.

If so, these don't appear in the ros2 topic list --include-hidden-topics output, so they're "very hidden"

Yes, ros2 topic doesn't even consider them as part of the message network because they don't start with the correct prefix (rt/).

That said, I don't think that any of this is a requirement imposed by ROS 2, more of a convention. The DDS RMWs all do it since the DDS spec does not have a definition for an RPC-like call (as far as I understand). You can imagine a different RMW that does have support for true RPC doing something different.

[kunaltyagi]

@emersonknapp Updated the format of issue.

For this to happen, we actually need support in ROS2 for serialized service (and action) calls

Aren't services and actions a superset of the message format? If a message can be (de)serialized, why is doing the same with service request/response and actions an issue (provided they are visible)?

Is this a constraint?

@clalancette Are the prefixes exposed by some mechanism? Or should it be requested at a rclcpp/rclpy level?

[emersonknapp]

Aren't services and actions a superset of the message format?

Yes - but the definition format and the network mechanism / format are not necessarily the same thing.

The more important point, I think is, per @clalancette comment - Services are not necessarily a Topic under the hood by definition of the ROS 2 specification. Though, in the case of all DDS RMW implementations, they are implemented using DDS topics, and could theoretically then be exposed in a way that DDS could record. This might be worth figuring out - given that we understand that we couldn't guarantee the mechanism would work for all possible RMW implementations.

[Karsten1987]

well, DDS topics yes or no but I think this doesn't really solve the task at hand here. The functionality has to be exposed through an RMW interface and at this point it's RMW agnostic. We can't just have a big "if DDS, then record services by hacking topics" within rosbag2 :D

What @clalancette said is certainly correct, but I believe there's more to it. On the ROS2 core stack we prefix all services with rq/rr, but even the actual DDS implementation is free to do whatever with the topic. In the case of RTI connext, I believe they added a magic number (gid) to the response topic to uniquely identify a service server and the respective client.

[Barry-Xu-2018]

I am considering the requirement of recorded data and the feasibility of realization.
From the perspective of use case,

Service/Client

• Record data for debugging service side
  I think this feature request can cover this use case.

• Record data for debugging client side
  Impossible to implement (from my thought)

Action

• Record data for debugging action service side
  Record sent goal.
  I think the modification for this feature can be updated to support this.
  But I don't sure whether anyone wants to get this data in practice?
  Any comments ? @emersonknapp @clalancette @Karsten1987 @kunaltyagi

• Record data for debugging action client side
  Impossible to implement (from my thought)

[fujitatomoya]

being off possibility or feasibility, we would want to support the following use cases? (at least i would do as application engineer.)

• service server development with recorded request. probably it would be nice to load the recorded request bag file to ros2 action and then we can see the status, feedback and response. (same use case can be applied to service server.)
• all recorded data of service/action client/server will be still useful to view the transmission activities later. i believe this is really useful to debug distributed system to see what they are actually transmitting when the problem observed.

[kunaltyagi]

If this feature isn't supported out-of-the-box, the community would have to create message based services and actions. Wouldn't that kind of defeat the purpose of the design process of ROS2? The community based solutions might not have the same level of guarantees that the core ros2 libraries come with.

This is a very important feature for systems actually in deployment, and I can't stress this enough: debugging, replay, simulation, etc. use this very heavily. In fact, some people and entities I know have stopped using services and only use messages to ensure the presence of record and replay features

There is neither an alternative nor steps taken to mitigate this and in absence of these, it doesn't motivate the effort required tor migrating to ros2 (assuming creating custom srv and action interface is more complicated than staying on ros1)

[Barry-Xu-2018]

I think we need to consider how to record service/action beyond the limitations of the current implementation of rosbag2.
e.g.
rosbag2 will record all communication message between service/action service and client/action client.
And then user can use tool like wireshark to check the content of messages at each record time point , which may come from the client or from the server.

This needs to further consider.