[Validate] Drop invalid polygons for IOU matching (#287)

* Drop invalid polygons for IOU matching

* Change log for invalid itemsm

* Add CHANGELOG entry

Author: gatli

CHANGELOG.md

@@ -5,6 +5,11 @@ All notable changes to the [Nucleus Python Client](https://github.com/scaleapi/n
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.10.5](https://github.com/scaleapi/nucleus-python-client/releases/tag/v0.10.5) - 2022-04-26
+
+### Fixed
+- Invalid polygons are dropped form PolygonMetric iou matching
+
 ## [0.10.4](https://github.com/scaleapi/nucleus-python-client/releases/tag/v0.10.4)) - 2022-05-02
 
 ### Added
@@ -15,14 +20,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [0.10.3](https://github.com/scaleapi/nucleus-python-client/releases/tag/v0.10.3) - 2022-04-22
 
 ### Fixed
-
 - Polygon and bounding box matching uses Shapely again providing faster evaluations
 - Evaluation function passing fixed for Polygon and Boundingbox configurations
 
 ## [0.10.1](https://github.com/scaleapi/nucleus-python-client/releases/tag/v0.10.1) - 2022-04-21
 
 ### Added
-
 - Added check for payload size
 
 ## [0.10.0](https://github.com/scaleapi/nucleus-python-client/releases/tag/v0.10.0)) - 2022-04-21

nucleus/metrics/polygon_utils.py

@@ -1,3 +1,4 @@
+import logging
 import sys
 from functools import wraps
 from typing import Dict, List, Tuple
@@ -76,6 +77,40 @@ def _iou_assignments_for_same_reference_id(
     polygon_annotations = list(map(polygon_annotation_to_shape, annotations))
     polygon_predictions = list(map(polygon_annotation_to_shape, predictions))
 
+    invalid_anns = [
+        ann
+        for ann, poly in zip(annotations, polygon_annotations)
+        if not poly.is_valid
+    ]
+    invalid_preds = [
+        pred
+        for pred, poly in zip(predictions, polygon_predictions)
+        if not poly.is_valid
+    ]
+
+    if invalid_anns or invalid_preds:
+        # Filter out invalid polys
+        polygon_annotations = [
+            poly
+            for ann, poly in zip(annotations, polygon_annotations)
+            if poly.is_valid
+        ]
+        polygon_predictions = [
+            poly
+            for pred, poly in zip(predictions, polygon_predictions)
+            if poly.is_valid
+        ]
+        invalid_dataset_ids = set(
+            ann.reference_id for ann in invalid_anns
+        ).union(set(pred.reference_id for pred in invalid_preds))
+        # TODO(gunnar): change to .id once field is surfaced)
+        logging.warning(
+            "Invalid polygons for dataset items: %s Annotations:%s, predictions: %s",
+            invalid_dataset_ids,
+            [a.annotation_id for a in invalid_anns],
+            [p.annotation_id for p in invalid_preds],
+        )
+
     # Compute IoU matrix and set IoU values below the threshold to 0.
     iou_matrix = _iou_matrix(polygon_annotations, polygon_predictions)
     iou_matrix[iou_matrix < iou_threshold] = 0

pyproject.toml

@@ -21,7 +21,7 @@ exclude = '''
 
 [tool.poetry]
 name = "scale-nucleus"
-version = "0.10.4"
+version = "0.10.5"
 description = "The official Python client library for Nucleus, the Data Platform for AI"
 license =  "MIT"
 authors = ["Scale AI Nucleus Team <nucleusapi@scaleapi.com>"]