Missing 3d Bouding box?

[xixinzhang]

I have some questions about the labels of 3d BBox:

1. I found the start frame(2897) of boxes in sequences 4 does not match the start frame(0) of the LiDAR scan. Is this something expected?
2. I've also seen a similar issue Some dynamic objects' 3D bbox labels are missing #40, which is related to data of sequence 0. There seem to be a lot of missing bboxes, I haven't checked all frames of all sequences, but here are some examples below. I don't know if I missed some data or if they were just not given, but the case I give for sequence 4 is out of the range of the first 250 frames.

• sequence 2 (frame 4433 front view):
  

• sequence 4 (frame 2907 front view)
  
  (frame 3368 front view, no label at all)
  
  (frame 3368 back view, no label at all)
  
  Only three static cars are being labeled for the frame window [3356, 3586], which is wrong from the above visualization

  <object1100>
    <index>8576</index>
    <label>car</label>
    <semanticId_orig>7</semanticId_orig>
    <semanticId>13</semanticId>
    <instanceId>129</instanceId>
    <category>instance</category>
    <timestamp>-1</timestamp>
    <dynamic>0</dynamic>
    <transform type_id="opencv-matrix">
      <rows>4</rows>
      <cols>4</cols>
      <dt>f</dt>
      <data>
        1.14138556e+00 2.14470959e+00 -6.63366588e-03 3.82886426e+03
        -5.34604931e+00 4.57898617e-01 -1.12612871e-03 -1.66201196e+03
        9.46977234e-04 9.00010951e-03 1.63808441e+00 2.39120895e+02 0. 0.
        0. 1.</data></transform>
    <vertices type_id="opencv-matrix">
      <rows>8</rows>
      <cols>3</cols>
      <dt>f</dt>
      <data>
        5.00000000e-01 5.00000000e-01 5.00000000e-01 5.00000000e-01
        5.00000000e-01 -5.00000000e-01 5.00000000e-01 -5.00000000e-01
        5.00000000e-01 5.00000000e-01 -5.00000000e-01 -5.00000000e-01
        -5.00000000e-01 5.00000000e-01 -5.00000000e-01 -5.00000000e-01
        5.00000000e-01 5.00000000e-01 -5.00000000e-01 -5.00000000e-01
        -5.00000000e-01 -5.00000000e-01 -5.00000000e-01 5.00000000e-01</data></vertices>
    <faces type_id="opencv-matrix">
      <rows>12</rows>
      <cols>3</cols>
      <dt>u</dt>
      <data>
        0 2 1 2 3 1 4 6 5 6 7 5 4 5 1 5 0 1 7 6 2 6 3 2 5 7 0 7 2 0 1 3 4
        3 6 4</data></faces>
    <start_frame>3356</start_frame>
    <end_frame>3586</end_frame></object1100>
  <object1101>
    <index>8577</index>
    <label>car</label>
    <semanticId_orig>7</semanticId_orig>
    <semanticId>13</semanticId>
    <instanceId>130</instanceId>
    <category>instance</category>
    <timestamp>-1</timestamp>
    <dynamic>0</dynamic>
    <transform type_id="opencv-matrix">
      <rows>4</rows>
      <cols>4</cols>
      <dt>f</dt>
      <data>
        4.78651315e-01 1.92826843e+00 7.02159759e-03 3.82994263e+03
        -4.01186228e+00 2.29451627e-01 -1.06386833e-01 -1.66836865e+03
        -3.10723543e-01 7.85248913e-03 1.38441491e+00 2.38654770e+02 0. 0.
        0. 1.</data></transform>
    <vertices type_id="opencv-matrix">
      <rows>8</rows>
      <cols>3</cols>
      <dt>f</dt>
      <data>
        5.00000000e-01 5.00000000e-01 5.00000000e-01 5.00000000e-01
        5.00000000e-01 -5.00000000e-01 5.00000000e-01 -5.00000000e-01
        5.00000000e-01 5.00000000e-01 -5.00000000e-01 -5.00000000e-01
        -5.00000000e-01 5.00000000e-01 -5.00000000e-01 -5.00000000e-01
        5.00000000e-01 5.00000000e-01 -5.00000000e-01 -5.00000000e-01
        -5.00000000e-01 -5.00000000e-01 -5.00000000e-01 5.00000000e-01</data></vertices>
    <faces type_id="opencv-matrix">
      <rows>12</rows>
      <cols>3</cols>
      <dt>u</dt>
      <data>
        0 2 1 2 3 1 4 6 5 6 7 5 4 5 1 5 0 1 7 6 2 6 3 2 5 7 0 7 2 0 1 3 4
        3 6 4</data></faces>
    <start_frame>3356</start_frame>
    <end_frame>3586</end_frame></object1101>
  <object1103>
    <index>8579</index>
    <label>car</label>
    <semanticId_orig>7</semanticId_orig>
    <semanticId>13</semanticId>
    <instanceId>131</instanceId>
    <category>instance</category>
    <timestamp>-1</timestamp>
    <dynamic>0</dynamic>
    <transform type_id="opencv-matrix">
      <rows>4</rows>
      <cols>4</cols>
      <dt>f</dt>
      <data>
        -1.57342708e+00 -1.90071630e+00 -6.44791964e-03 3.83258594e+03
        4.33695078e+00 -6.89578295e-01 -1.15723989e-03 -1.65412012e+03
        -3.29058268e-03 -8.37895181e-03 1.55791259e+00 2.39629547e+02 0.
        0. 0. 1.</data></transform>
    <vertices type_id="opencv-matrix">
      <rows>8</rows>
      <cols>3</cols>
      <dt>f</dt>
      <data>
        5.00000000e-01 5.00000000e-01 5.00000000e-01 5.00000000e-01
        5.00000000e-01 -5.00000000e-01 5.00000000e-01 -5.00000000e-01
        5.00000000e-01 5.00000000e-01 -5.00000000e-01 -5.00000000e-01
        -5.00000000e-01 5.00000000e-01 -5.00000000e-01 -5.00000000e-01
        5.00000000e-01 5.00000000e-01 -5.00000000e-01 -5.00000000e-01
        -5.00000000e-01 -5.00000000e-01 -5.00000000e-01 5.00000000e-01</data></vertices>
    <faces type_id="opencv-matrix">
      <rows>12</rows>
      <cols>3</cols>
      <dt>u</dt>
      <data>
        0 2 1 2 3 1 4 6 5 6 7 5 4 5 1 5 0 1 7 6 2 6 3 2 5 7 0 7 2 0 1 3 4
        3 6 4</data></faces>
    <start_frame>3356</start_frame>
    <end_frame>3586</end_frame></object1103>

[yiyiliao]

Thank you for reporting the issues!

I found the start frame(2897) of boxes in sequences 4 does not match the start frame(0) of the LiDAR scan. Is this something expected?

Yes, it is expected. We have withheld some RGB images from sequence 4 because they are used as test sequences for the NVS benchmark.

I've also seen a similar issue #40, which is related to data of sequence 0. There seem to be a lot of missing bboxes, I haven't checked all frames of all sequences, but here are some examples below. I don't know if I missed some data or if they were just not given, but the case I give for sequence 4 is out of the range of the first 250 frames.

I investigated this issue and found that the missing bounding boxes are likely due to retrieving them using the annotation window. Some regions have multiple annotations because the ego-car passed the same areas repeatedly or because of overlaps between adjacent windows. To address this, we implemented an automatic check to filter out duplicate annotations, which might explain why some boxes appear from a different window. Below are screenshots showing the same sequence (seq 4, 0000003356_0000003586.ply) overlaid with instance bounding boxes, comparing retrieval using the annotation window versus retrieval based on distance."

1. Annotation window-based retrieval

2. Distance-based retrieval

[yiyiliao]

I have updated kitti360Viewer3D.py to visualize the bounding boxes based on distance instead of annotation window:

kitti360Scripts/kitti360scripts/viewer/kitti360Viewer3D.py

Lines 319 to 345 in d4d4885

	if not len(v.bboxes):
	raise RuntimeError('No bounding boxes found! Please set KITTI360_DATASET in your environment path')
	pcdFileList = v.annotation3DPly.pcdFileList
	# group the bboxes by windows
	windows_unique = np.unique(np.array(v.bboxes_window), axis=0)
	for idx,window in enumerate(windows_unique):
	# load point cloud for visualization
	if not '%010d_%010d' % (window[0], window[1]) in pcdFileList[idx]:
	raise RuntimeError("Window %s does not match with point cloud name %s! Please make sure that you downloaded the accumulated point clouds correctly." % (window, pcdFileList[idx]) )
	pcd = v.loadWindow(pcdFileList[idx], args.mode)
	pcdCenter = np.median(np.asarray(pcd.points),axis=0)
	# filter out outliers in pcd
	validPoints = np.where(np.linalg.norm(np.asarray(pcd.points)-pcdCenter,axis=1)<1e+3)[0]
	pcd = pcd.select_by_index(validPoints)
	pcdCenter = np.median(np.asarray(pcd.points),axis=0)
	# load instance bounding boxes not too far away from the point cloud center
	bboxes = [v.bboxes[i] for i in range(len(v.bboxes)) if (np.linalg.norm(np.mean(np.asarray(v.bboxes[i].vertices), axis=0)-pcdCenter)<120)]
	print('Visualizing %06d_%06d with %d objects' % (window[0], window[1], len(bboxes)))
	if len(bboxes)>args.max_bbox:
	print('Randomly sample %d/%d bboxes for rendering efficiency' % (args.max_bbox, len(bboxes)))
	random_list = np.random.permutation(len(bboxes))[:args.max_bbox]
	bboxes = [bboxes[i] for i in random_list]
	open3d.visualization.draw_geometries(bboxes + [pcd])