Simulate Geometric Inputs/Prompts for Training (Custom Dataset)

[biggeR-data]

Hi team,

thank you for the great project!

I'm currently working on fine-tuning a model on my own custom dataset. I've been wondering how the model can be trained for prompts / bounding boxes which would represent user's input faithfully.

In the repository I found: sam3/train/transforms/point_sampling.py with this class:

class RandomGeometricInputsAPI:
    """
    For geometric queries, replaces the input box or points with a random
    one sampled from the GT mask. Segments must be provided for objects
    that are targets of geometric queries, and must be binary masks. Existing
    point and box queries in the datapoint will be ignored and completely replaced.
    Will sample points and boxes in XYXY format in absolute pixel space.

    Geometry queries are currently determined by taking any query whose
    query text is a set value.

    Args:
      num_points (int or (int, int)): how many points to sample. If a tuple,
        sample a random number of points uniformly over the inclusive range.
      box_chance (float): fraction of time a box is sampled. A box will replace
        one sampled point.
      box_noise_std (float): if greater than 0, add noise to the sampled boxes
        with this std. Noise is relative to the length of the box side.
      box_noise_max (int): if not none, truncate any box noise larger than this
        in terms of absolute pixels.
      resample_box_from_mask (bool): if True, any sampled box will be determined
        by finding the extrema of the provided mask. If False, the bbox provided
        in the target object will be used.
      point_sample_mode (str): In ["centered", "random_mask", "random_box"],
        controlling how points are sampled:
          "centered": points are sampled farthest from the mask edges and each other
          "random_mask": points are sampled uniformly from the mask
          "random_box": points are sampled uniformly from the annotation's box
        Note that "centered" may be too slow for on-line generation.
      geometric_query_str (str): what string in query_text indicates a
        geometry query.
      minimum_box_area (float): sampled boxes with area this size or smaller after
        noising will use the original box instead. It is the input's responsibility
        to avoid original boxes that violate necessary area bounds.
      concat_points (bool): if True, any sampled points will be added to existing
        ones instead of replacing them.

    """

Unfortunately I was unable to find any examples or even a code block which was using this class. By the looks of it, it seems to be exactly what I am looking for, but I am not sure how it could be incorporated into the current training loop.

Has anyone already figured out a way how to train on simulated clicks / bounding boxes?
I would be very interested in potential attempts from others.