Side Mirrors

[ThomasNaderBMW]

Hello everyone,

a colleague had a question what isn't clear in the cooperation with other companies:

In BaseMoving (https://opensimulationinterface.github.io/open-simulation-interface/structosi3_1_1BaseMoving.html) there is a picture that includes the side mirrors.

It is included in the moving object, but in the moving object itself (https://opensimulationinterface.github.io/open-simulation-interface/structosi3_1_1MovingObject.html) there is the note, that the side mirrors are not included:

What is countig? Did I miss something?

Thank you!

[HendrikAmelunxen]

In addition to this, what about the base_polygon? Since it is described as the two dimensional (flat) contour of the object, I would assume that side mirrors are included within the polygon. But then the polygon would not be consistent with the bounding box, if side mirrors are excluded there.
Therefore, should the comment, that the bounding box does NOT include mirrors, be deleted? Maybe a good way to make it consistent.

[kmeids]

Hello,

Even though the picture should be pointing out to the doted bounding-box, it is a bit miss-leading since the side-mirrors are still present and we can't see if they are included in the doted 3D bounding box or not.

For the GroundTruth/SensorView message, simulation environments usually provide the width of the vehicle without the side-mirrors, and since the Note: The bounding box does NOT includes mirrors for vehicles is for the entire base message than I would assume that it also applies to the base_polygon. This definitally have its limitations since most probably some simulation environments can also provide the width including the side-mirrors.

For the SensorData message, the note (Note: The bounding box does NOT includes mirrors for vehicles
) may not be valid in several use cases, e.g. a NN would most probably include the side-mirrors as well (acting on raw-image data or on a point cloud).

To stay flexible and support all cases, my suggestion is to add another Boolean message under BaseMoving to indicate whether the side-mirrors are included or not, e.g. includes_side_mirror. By setting its default value to False this would also ensure backward compatibility.

@ThomasNaderBMW. @HendrikAmelunxen what do you think?

[HendrikAmelunxen]

We have discussed this topic in detail within the harmonization working group today.

@kmeids In our opinion it would be good if OSI specifies the bounding box; therefore a Boolean would be disadvantageous since it allows both ways instead of setting a way.

We would prefer the following approach:

• Since the polyline and the bounding-box is derived from the 3d model (which typically includes the side mirrors) the polyline and the bounding-box should also include them, to be consistent.
• In addition to this, we would need additional base_moving attributes for the mirror lengths (e.g. length_mirror_left and length_mirror_right), to be able to deduce the “core”-bounding-box without mirrors.
  If needed, this approach could be further extended to add-on components at other surfaces, e.g. to define antennas on the roof. Of course; the side mirrors are the main usecase.
• Furthermore, we need a modification of the documentation, which clearly states how the polyline and the bounding-box are defined.

This would be a consistent and full approach for the side-mirrors.

@pmai We were wondering why it was explicitly stated that side mirrors are excluded for the bounding box. Do you know for which use-case this statement was introduced? Are we missing a point?

[kmeids]

@HendrikAmelunxen

Since the polyline and the bounding-box is derived from the 3d model (which typically includes the side mirrors) the polyline and the bounding-box should also include them, to be consistent.

That is true, the 3D model includes the side mirrors, however simulation environments don't get the bounding box from the 3D model rather from vehicle dynamics (most probably chassis data) and there you would rarely find that the rare mirrors are included in the width. So defining the 3D bounding box to include the side mirrors will also be disadvantageous, since this data is being provided by the simulation environment.

An alternative suggestion is to keep the 3D bounding box as it is (without the side mirrors) and add the field "length_mirror_x"as you have suggested.
So basically to get the final size of the 3D bounding box, you go through an additive process and not a subtractive one.
The advantage here is that this will still work even if the simulation environment does not have/provide the "length_mirror_x".

[pmai]

@pmai We were wondering why it was explicitly stated that side mirrors are excluded for the bounding box. Do you know for which use-case this statement was introduced? Are we missing a point?

The major motivation would be from the PoV of the sensor modeling: For most sensors, the mass of the object is contained in the current definition of a bounding box and this will have a major impact on e.g. visibility calculations, etc. While objects that stick out will in some situations also have a major impact, those situations tend to depend on the orientation and exact location of the thing sticking out, and so would tend to use the exact geometry anyway. Not saying this was the exact reasoning at the time, but that would be one of the reasons to prefer bounding boxes that hug the core object more closely.

I think the way forward would rather be to add further bounding boxes, since those can then move independently: The major problem right now is not the mirrors, I'd venture, but rather the fact that we cannot represent an open passenger door or an open trunk door on a 5 door vehicle very well, and those things move somewhat independently.

Of course the bounding box approach quickly reaches its limits, i.e. I would not want to go much further than the major doors/trunk/front, the wheel carriers we already do, and maybe the side mirrors. Antennae etc. should probably be dealt directly at 3d level.

[HendrikAmelunxen]

@kmeids

however simulation environments don't get the bounding box from the 3D model rather from vehicle dynamics (most probably chassis data) and there you would rarely find that the rare mirrors are included in the width.

This is maybe a use case for an OEM where the parameters for the ego vehicle have been derived from a vehicle dynamics model. In this case, the side mirror information might not be available. However, in many other use cases this information will be available for the ego vehicle.
Besides the ego vehicle, all the other moving objects have a 3d model, from which the bounding box and the contour line are derived. In my opinion, in this case the side mirrors definitely must be included in the bounding box and in the base_polygon. Take the following example for a typical compact car:

• The base polygon (blue line) is derived from a 3d-model
• The red line is the bounding box including the side mirrors
• The magenta line is one bounding box including the wheel boxes
• The magenta line is one bounding box along the side doors

In this case only the red bounding box is unambiguous and consistent with the 3d-model and the base_polygon. Excluding the side mirrors does not specify the bounding box clearly, e.g. should the wheel boxes be included like shown in the picture? A definition like this (bounding box does not include side mirrors) would be a serious problem for simulation environments since it is not clear what the bounding box should be for all the moving objects within the scenario.

Since the ego vehicle is just one of the moving objects, I would definitely prefer to handle the bounding box and base_polygon definitions exactly the same among all moving objects.

The bounding box concept has to extended, e.g. to be capable to simulate situations with opening car doors, or bicyclists indicating that they want to make a turn, etc. In this case I would assume that the bounding box and the base polygon changes accordingly and consistently. The base polygon can change dynamically during simulation but the bounding box needs additional features to handle these use cases correctly. This should be a further enhancement of OSI.

Until this I think we should definitely clarify how to handle side mirrors. From the harmonization working group we think it is necessary to correct the statement for the bounding box, by stating that the side mirrors must be included in the bounding box.

[thempen]

Just as a suggestion. I tried to define a draft for a BoundingBox message to solve the problem. For the requirement of defining different bounding boxes, e.g. with or without mirrors, there is an enum type. It defines, which elements are included. Therefore there has to be a separation on which elements are within the selection. For example: Is cargo part of the bounding box? Is the bicyclist with his or her arms included?...
Are there only "static elements" to consider? - not directly. Doors, hitches, roof rails, etc. can be moved or dismounted. Therefore there is the definition made of "from oem factory side".
Within that range, there are various possibilities to define a BoundingBox (see code).

To consinder movable parts, like opening doors, cargo or arms and legs for motorcycles and bicycles, there is the offset. The offset is calculated through adding a value to the corresponding side of the bounding box and can be positive or negative.

Recapitulated, there is the reference bounding box defined through the Reference type and the possibility to change the bounding box on runtime through offset values in 5 directions.

syntax = "proto2";

option optimize_for = SPEED;

package osi3;

message BoundingBox
{
    // The dimension of the BoundingBox. 
    // The dimension calculation is defined by the reference value.
    //
    optional Dimension3d dimension = 1;
	
    // The reference type of the BoundingBox
    //
    optional Reference reference = 2;

    // The reference has to be defined through vehicle elements from oem factory 
    // So to say, roof boxes, cargo, etc. must not be included
    // For vehicles with closed doors and wheel angle 0.
    //
    enum Reference
    {
	// Reference of the object is unknown (must not be used in ground truth).
        //
	REFERENCE_UNKNOWN = 0;
		
	// BoundingBox is defined through the whole object geometry. Everything of the geometry is included, e.g. side mirrors, antennas, etc.
        //	
	REFERENCE_OUTER_GEOMETRY = 1;
		
	// BoundingBox is defined through the object geometry, except protruding elements, e. g. side mirrors, antennas, etc. Here we need a clear definition!
        //
	REFERENCE_BASE_GEOMETRY = 2; 
		
	// BoundingBox is defined in respect of the vehicle dynamics. Here we need a clear definition!
	//
	REFERENCE_PHYSICAL = 3;
		
	// BoundingBox calculation based in the Sensor-Detection. It is not directly defined, whether side mirrors, antennas, etc. are included.
        // The including/excluding depend on the resolution, detection algorithm, etc.
        //
        REFERENCE_SENSOR_DETECION = 4;
				
	// Further definitions are thinkable
    }
	
	// For the usage with dynamic elements, e.g. opening doors, wheels in certain angles, crane booms at trucks 
	// or elements not included from oem factory side, e.g. some cargo or body parts for bicyclists,
	// offsets can be defined dynamically. An encapsulation of the offsets into an internal message e.g. "message BoundingBoxDynamics { }" can be thinkable.
	
	// The offset front defines a value, to calculate the actual BoundingBox to use.
	// The direction front is the direction in positive x-axis
	//
	optional double offset_front = 3;
	
	// The offset back defines a value, to calculate the actual BoundingBox to use.
	// The direction back is the direction in negative x-axis
	//
	optional double offset_back = 4;
	
	// The offset left defines a value, to calculate the actual BoundingBox to use.
	// The direction left is the direction in positive y-axis
	//
	optional double offset_left = 5;
	
	// The offset right defines a value, to calculate the actual BoundingBox to use.
	// The direction right is the direction in negative y-axis
	//
	optional double offset_right = 6;
	
	// The offset top defines a value, to calculate the actual BoundingBox to use.
	// The direction top is the direction in positive z-axis
	//
	optional double offset_top = 7;
	
}

A question to @pmai and @kmeids: How do you understand BoundingBoxes and and base polygons? For me, the base polygon is projection of the contours of an object to the ground, ignoring roll, yaw and pitch. In contrary the BoundingBox does have the possibility to roll, yaw and pitch together with the geometry of an object. This can be a very big difference in using the values for e.g. hill situations or motorcycles.

[kmeids]

@thempen and @HendrikAmelunxen

There is no doubt that the current polygon and bounding boxes definitions should be adapted and enhanced.
Regardless of the solution, the possibility to communicate the vehicle's dimensions (width in this case) with and without the side mirrors must not be deprecated.

I suggest that detailed discussions on whether to do this on the base_polygon, dimension or even the VehicleAttributes message should at this stage migrate to the WG meetings.

[thempen]

Discussion results from Harmonisation group at 10.12.2020:
Bounding Box:
A quick change for 3.3, bringing in the side mirrors, will result in losing the backward compatibility. We further see, there is a bigger change needed for future use cases. These changes will probably break the backwards compatibility again for release 4.0.
To reduce those breaks, we want to have one bigger change for 4.0. The side mirrors will stay excluded from the bounding box for now.

For further development, we need support from the SensorModeling group, to have more use cases and requirements.

Base Polygon:
The description of the base polygon is irritating together with the bouding box description:

    // Usage as ground truth:
    // The two dimensional (flat) contour of the object. This is an extension of
    // the concept of a bounding box as defined by \c Dimension3d. The contour
    // is the projection of the object's outline onto the z-plane in the object
    // frame (independent of its current position and orientation). The height
    // is the same as the height of the bounding box.

The projection of the object's outline would include the side mirrors. To make this point clear, we simply would suggest, cancelling out This is an extension of the concept of a bounding box as defined by \c Dimension3d.. Resulting in an independent view on these data types.

For the work to the major release 4.0, we can bring these concepts back together. Thinkable is a more direct connection through including the base polygon directliy inside a bounding box message as an extension.

syntax = "proto2";

option optimize_for = SPEED;

package osi3;

message BoundingBox
{
    // ... additional code see comment above

    // Base polygon represents the outer contour line of the object as a topdown projection on the BoundingBox ground plane
    message BasePolygon
    {
        // ...
    }
}

@kmeids we think, bringing the discussion inside the architecture group is a bit to early. We want to make sure to understand the requirements of the users. Getting input from the SensorModeling group will help us developing a draft for future improvements and discussions.

[kmeids]

Thanks for the update, I agree.
I will bring this topic into the SensorModeling WG meeting taking place tomorrow at 9:00 CET.
@thempen, @HendrikAmelunxen please feel free to join!