Help with spinning runes

[f12Gamer]

Hello,

can someone help me with detecting spinning runes? I wrote my own program in Python, and the code below uses rune_nms.onnx. It works fine overall, though it doesn’t detect 100% of the runes.

Now I’d like to add a feature to detect spinning runes, similar to what this program can do. Could anyone give me a hint or suggestion on how to achieve this in my code?

MODEL_PATH = "rune_nms.onnx"
DIRECTIONS = ['UP', 'DOWN', 'LEFT', 'RIGHT']
CONFIDENCE_THRESHOLD = 0.75

# Load the ONNX model
session = ort.InferenceSession(resource_path(MODEL_PATH))
input_name = session.get_inputs()[0].name

def getFoundArrows():
    # Grab region from screen — replace these with your coordinates or interface
    image = getRuneScreenshot()

    # Preprocess
    input_tensor, scale, pad_left, pad_top, orig_w, orig_h = preprocess_image_yolo_style(image)

    # Run inference
    outputs = session.run(None, {input_name: input_tensor})
    detections = outputs[0][0]

    # Filter and remap
    filtered = [det for det in detections if det[4] > CONFIDENCE_THRESHOLD]
    remapped = remap_boxes_to_original(filtered, scale, pad_left, pad_top)

    # Sort by confidence descending, then pick top 4
    top_4 = sorted(remapped, key=lambda det: -det[4])[:4]

    # Sort those 4 by x1 (left to right)
    remapped_sorted = sorted(top_4, key=lambda det: det[0])

    # Output sorted directions
    arrows = [DIRECTIONS[int(det[5])] for det in remapped_sorted]
    confidences = [round(float(det[4]), 2) for det in remapped_sorted]

    print(f"{getTime()}{arrows}")
    print(f"{getTime()}{confidences}")

    return arrows

def preprocess_image_yolo_style(pil_image, target_size=640):
    img = pil_image.convert("RGB")
    orig_w, orig_h = img.size

    scale = min(target_size / orig_w, target_size / orig_h)
    new_w = int(round(orig_w * scale))
    new_h = int(round(orig_h * scale))

    pad_w = target_size - new_w
    pad_h = target_size - new_h
    pad_left = int(round(pad_w / 2 - 0.1))
    pad_right = int(round(pad_w / 2 + 0.1))
    pad_top = int(round(pad_h / 2 - 0.1))
    pad_bottom = int(round(pad_h / 2 + 0.1))

    resized_img = img.resize((new_w, new_h), Image.BICUBIC)
    padded_img = ImageOps.expand(resized_img, border=(pad_left, pad_top, pad_right, pad_bottom), fill=(114, 114, 114))

    img_array = np.asarray(padded_img).astype(np.float32) / 255.0
    img_array = np.transpose(img_array, (2, 0, 1))
    input_tensor = np.expand_dims(img_array, axis=0)

    return input_tensor, scale, pad_left, pad_top, orig_w, orig_h


def remap_boxes_to_original(boxes, scale, pad_left, pad_top):
    remapped = []
    for box in boxes:
        x1, y1, x2, y2, conf, class_id = box
        x1 = max((x1 - pad_left) / scale, 0)
        y1 = max((y1 - pad_top) / scale, 0)
        x2 = max((x2 - pad_left) / scale, 0)
        y2 = max((y2 - pad_top) / scale, 0)
        remapped.append([x1, y1, x2, y2, conf, class_id])
    return remapped

[f12Gamer]

No one?
@sasanquaa can you pin point me where in your code you implemented the spinning runes?

[sasanquaa]

Oh, I was checking this tab yesterday but didn't saw any new question. Today it shows up as asked 4 days ago so sorry for the late reply. It is this part https://github.com/sasanquaa/komari/blob/master/backend/src/detect.rs#L1859-L2211

[sasanquaa]

The spin rune detection is the two functions:

• calibrate_for_spin_arrows -> Checks for existence of spinning arrows
• detect_spin_arrow -> Detects actual spin arrow by checking for the "spin lag"

If you need a summary of how it works:

1. Detects spin arrow "region"s - these are basically just rectangles each containing the whole spin arrow even while it is spinning
2. Crops to each spin arrow region and extracts Saturation (S) channel from HSV color
3. Detects the spin arrow contour and finds its minEnclosingTriangle
4. The middle point of shortest edge of the triangle is the spin arrow's tip
5. Detects "spin lag" by determining previous frame and current frame spin arrow's tip is spinning in the same direction

If you want to visualize how it works, download the debug version -> Go to Debug tab -> Choose the spin rune sandbox test