Skip to content

feat: add freeform drawing gestures #186

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
aidenybai wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: add freeform drawing gestures #186

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
168 changes: 168 additions & 0 deletions packages/react-grab/e2e/freeform-drawing.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,168 @@
import { test, expect } from "./fixtures.js";
Copy link
Contributor

@cubic-dev-ai cubic-dev-ai bot Feb 20, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Missing test coverage for the draw mode toolbar button. The tests currently only cover the 'Alt' key activation path.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At packages/react-grab/e2e/freeform-drawing.spec.ts, line 6:

<comment>Missing test coverage for the draw mode toolbar button. The tests currently only cover the 'Alt' key activation path.</comment>

<file context>
@@ -0,0 +1,168 @@
+const FREEFORM_IDLE_TIMEOUT_MS = 600;
+const GESTURE_SETTLE_BUFFER_MS = 300;
+
+test.describe("Freeform Drawing", () => {
+  test("should enter comment mode via arrow gesture", async ({ reactGrab }) => {
+    await reactGrab.activate();
</file context>
Fix with Cubic


const FREEFORM_IDLE_TIMEOUT_MS = 600;
const GESTURE_SETTLE_BUFFER_MS = 300;

test.describe("Freeform Drawing", () => {
test("should enter comment mode via arrow gesture", async ({ reactGrab }) => {
await reactGrab.activate();

const listItem = reactGrab.page.locator("li").first();
const box = await listItem.boundingBox();
if (!box) throw new Error("Could not get bounding box");

const targetX = box.x + box.width / 2;
const targetY = box.y + box.height / 2;

await reactGrab.page.keyboard.down("Alt");
await reactGrab.page.mouse.move(targetX - 150, targetY - 80);
await reactGrab.page.mouse.down();
await reactGrab.page.mouse.move(targetX, targetY, { steps: 20 });
await reactGrab.page.mouse.up();
await reactGrab.page.keyboard.up("Alt");

await expect
.poll(() => reactGrab.isPromptModeActive(), {
timeout: FREEFORM_IDLE_TIMEOUT_MS + GESTURE_SETTLE_BUFFER_MS + 2000,
})
.toBe(true);
});

test("should enter comment mode via circle gesture", async ({
reactGrab,
}) => {
await reactGrab.activate();

const firstItem = reactGrab.page.locator("li").first();
const thirdItem = reactGrab.page.locator("li").nth(2);
const firstBox = await firstItem.boundingBox();
const thirdBox = await thirdItem.boundingBox();
if (!firstBox || !thirdBox)
throw new Error("Could not get bounding boxes");

const centerX = (firstBox.x + thirdBox.x + thirdBox.width) / 2;
const centerY = (firstBox.y + thirdBox.y + thirdBox.height) / 2;
const radiusX = (thirdBox.x + thirdBox.width - firstBox.x) / 2 + 30;
const radiusY = (thirdBox.y + thirdBox.height - firstBox.y) / 2 + 30;

const strokeSteps = 40;
const totalAngle = Math.PI * 1.8;

await reactGrab.page.keyboard.down("Alt");

const startX = centerX + radiusX;
const startY = centerY;
await reactGrab.page.mouse.move(startX, startY);
await reactGrab.page.mouse.down();

for (let stepIndex = 1; stepIndex <= strokeSteps; stepIndex++) {
const angle = (stepIndex / strokeSteps) * totalAngle;
await reactGrab.page.mouse.move(
centerX + radiusX * Math.cos(angle),
centerY + radiusY * Math.sin(angle),
);
}

await reactGrab.page.mouse.up();
await reactGrab.page.keyboard.up("Alt");

await expect
.poll(() => reactGrab.isPromptModeActive(), {
timeout: FREEFORM_IDLE_TIMEOUT_MS + GESTURE_SETTLE_BUFFER_MS + 2000,
})
.toBe(true);
});

test("should hide crosshair during freeform session", async ({
reactGrab,
}) => {
await reactGrab.activate();

await reactGrab.page.mouse.move(400, 300);
await reactGrab.page.waitForTimeout(100);

const crosshairBefore = await reactGrab.isCrosshairVisible();
expect(crosshairBefore).toBe(true);

await reactGrab.page.keyboard.down("Alt");
await reactGrab.page.mouse.move(200, 200);
await reactGrab.page.mouse.down();
await reactGrab.page.mouse.move(300, 300, { steps: 10 });

const crosshairDuring = await reactGrab.isCrosshairVisible();
expect(crosshairDuring).toBe(false);

await reactGrab.page.mouse.up();
await reactGrab.page.keyboard.up("Alt");
});

test("should not start freeform when overlay is inactive", async ({
reactGrab,
}) => {
await reactGrab.page.keyboard.down("Alt");
await reactGrab.page.mouse.move(300, 300);
await reactGrab.page.mouse.down();
await reactGrab.page.mouse.move(400, 400, { steps: 10 });
await reactGrab.page.mouse.up();
await reactGrab.page.keyboard.up("Alt");

await reactGrab.page.waitForTimeout(
FREEFORM_IDLE_TIMEOUT_MS + GESTURE_SETTLE_BUFFER_MS,
);

const state = await reactGrab.getState();
expect(state.isActive).toBe(false);
});

test("should keep overlay active in comment mode after freeform gesture", async ({
Copy link
Contributor

@cubic-dev-ai cubic-dev-ai bot Feb 20, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This test completely duplicates the logic from 'should enter comment mode via arrow gesture' (lines 7-29). Consider moving the isOverlayVisible assertion into the first test and removing this duplicate to reduce maintenance burden.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At packages/react-grab/e2e/freeform-drawing.spec.ts, line 117:

<comment>This test completely duplicates the logic from 'should enter comment mode via arrow gesture' (lines 7-29). Consider moving the `isOverlayVisible` assertion into the first test and removing this duplicate to reduce maintenance burden.</comment>

<file context>
@@ -0,0 +1,168 @@
+    expect(state.isActive).toBe(false);
+  });
+
+  test("should keep overlay active in comment mode after freeform gesture", async ({
+    reactGrab,
+  }) => {
</file context>
Fix with Cubic

reactGrab,
}) => {
await reactGrab.activate();

const listItem = reactGrab.page.locator("li").first();
const box = await listItem.boundingBox();
if (!box) throw new Error("Could not get bounding box");

const targetX = box.x + box.width / 2;
const targetY = box.y + box.height / 2;

await reactGrab.page.keyboard.down("Alt");
await reactGrab.page.mouse.move(targetX - 150, targetY - 80);
await reactGrab.page.mouse.down();
await reactGrab.page.mouse.move(targetX, targetY, { steps: 20 });
await reactGrab.page.mouse.up();
await reactGrab.page.keyboard.up("Alt");

await expect
.poll(() => reactGrab.isPromptModeActive(), {
timeout: FREEFORM_IDLE_TIMEOUT_MS + GESTURE_SETTLE_BUFFER_MS + 2000,
})
.toBe(true);

const isActive = await reactGrab.isOverlayVisible();
expect(isActive).toBe(true);
});

test("should discard gesture with insufficient points", async ({
reactGrab,
}) => {
await reactGrab.activate();

await reactGrab.page.keyboard.down("Alt");
await reactGrab.page.mouse.move(300, 300);
await reactGrab.page.mouse.down();
await reactGrab.page.mouse.move(305, 305);
await reactGrab.page.mouse.up();
await reactGrab.page.keyboard.up("Alt");

await reactGrab.page.waitForTimeout(
FREEFORM_IDLE_TIMEOUT_MS + GESTURE_SETTLE_BUFFER_MS,
);

const isActive = await reactGrab.isOverlayVisible();
expect(isActive).toBe(true);

const isPromptMode = await reactGrab.isPromptModeActive();
expect(isPromptMode).toBe(false);
});
});
28 changes: 28 additions & 0 deletions packages/react-grab/src/components/icons/icon-draw.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
import type { Component } from "solid-js";

interface IconDrawProps {
size?: number;
class?: string;
}

export const IconDraw: Component<IconDrawProps> = (props) => {
const size = () => props.size ?? 14;

return (
<svg
xmlns="http://www.w3.org/2000/svg"
width={size()}
height={size()}
viewBox="0 0 24 24"
fill="currentColor"
class={props.class}
>
<path
opacity="0.4"
d="M17.6963 13.5207C17.8504 13.2971 17.9275 13.1854 17.9177 13.0772C17.9078 12.9691 17.8184 12.8797 17.6397 12.701L11.3001 6.36084C11.1214 6.18208 11.032 6.0927 10.9239 6.08284C10.8158 6.07298 10.704 6.15004 10.4804 6.30416C9.60263 6.90931 8.74758 7.15064 7.84873 7.23856L7.81049 7.2423C7.12164 7.30966 6.55058 7.3655 6.10482 7.45299C5.65134 7.54199 5.20715 7.68375 4.8563 8.00532C4.5207 8.31291 4.34173 8.69556 4.21709 9.10487C4.09986 9.48987 4.01003 9.97373 3.9053 10.5379L2.5018 18.0963C2.37765 18.7646 2.26942 19.3472 2.25247 19.8126C2.23443 20.3076 2.31049 20.8455 2.73272 21.2678C3.15495 21.69 3.69286 21.7661 4.18791 21.7481C4.65322 21.7311 5.23583 21.6229 5.9041 21.4987L13.4622 20.0952C14.0263 19.9904 14.5101 19.9006 14.895 19.7834C15.3043 19.6588 15.6869 19.4798 15.9945 19.1442C16.316 18.7934 16.4578 18.3492 16.5469 17.8958C16.6344 17.45 16.6904 16.8788 16.7578 16.1899L16.7616 16.1517C16.8496 15.2529 17.0911 14.3983 17.6963 13.5207Z"
/>
<path d="M18.9948 4.74718C18.2226 3.97492 17.5836 3.33591 17.0124 2.91293C16.4108 2.46741 15.7737 2.17625 15.0174 2.26633C14.0173 2.38542 13.2605 3.04927 12.7141 3.68764C12.5171 3.91777 12.3302 4.16657 12.1565 4.41288C12.0232 4.60202 11.9565 4.6966 11.9659 4.80565C11.9753 4.91471 12.0591 4.99856 12.2268 5.16626L18.833 11.773C19.0007 11.9408 19.0845 12.0246 19.1936 12.034C19.3027 12.0434 19.3972 11.9767 19.5864 11.8433C19.8328 11.6697 20.0818 11.4826 20.312 11.2855C20.9504 10.7392 21.6142 9.98232 21.7333 8.98227C21.8234 8.2259 21.5322 7.58879 21.0867 6.98721C20.6637 6.41607 20.0247 5.77711 19.2525 5.00491L18.9948 4.74718Z" />
<path d="M3.68674 21.7255L9.62581 15.7864C10.005 15.9936 10.4154 16.1292 10.8376 16.1977C11.3827 16.2862 11.8964 15.916 11.9849 15.3709C12.0734 14.8257 11.7032 14.3121 11.1581 14.2236C10.7852 14.163 10.4714 14.0069 10.2312 13.7667C9.99109 13.5265 9.83491 13.2127 9.77438 12.8399C9.68587 12.2947 9.1722 11.9245 8.62705 12.013C8.0819 12.1015 7.71172 12.6152 7.80022 13.1604C7.86876 13.5825 8.00437 13.993 8.21159 14.3721L2.2735 20.3103C2.32138 20.6495 2.44887 20.9844 2.73185 21.2674C3.01422 21.5498 3.34832 21.6774 3.68674 21.7255Z" />
</svg>
);
};
103 changes: 102 additions & 1 deletion packages/react-grab/src/components/overlay-canvas.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import type {
OverlayBounds,
SelectionLabelInstance,
AgentSession,
StrokePoint,
} from "../types.js";
import { lerp } from "../utils/lerp.js";
import {
Expand All @@ -19,6 +20,9 @@ import {
OVERLAY_FILL_COLOR_DRAG,
OVERLAY_BORDER_COLOR_DEFAULT,
OVERLAY_FILL_COLOR_DEFAULT,
FREEFORM_STROKE_COLOR,
FREEFORM_STROKE_MIN_WIDTH_PX,
FREEFORM_STROKE_MAX_WIDTH_PX,
} from "../constants.js";

const LAYER_STYLES = {
Expand All @@ -44,7 +48,13 @@ const LAYER_STYLES = {
},
} as const;

type LayerName = "crosshair" | "drag" | "selection" | "grabbed" | "processing";
type LayerName =
| "crosshair"
| "drag"
| "selection"
| "grabbed"
| "processing"
| "freeform";

interface OffscreenLayer {
canvas: OffscreenCanvas | null;
Expand Down Expand Up @@ -90,6 +100,10 @@ export interface OverlayCanvasProps {
agentSessions?: Map<string, AgentSession>;

labelInstances?: SelectionLabelInstance[];

freeformStrokePoints?: StrokePoint[][];
freeformStrokeVisible?: boolean;
freeformStrokeCompletedAt?: number | null;
}

export const OverlayCanvas: Component<OverlayCanvasProps> = (props) => {
Expand All @@ -106,6 +120,7 @@ export const OverlayCanvas: Component<OverlayCanvasProps> = (props) => {
selection: { canvas: null, context: null },
grabbed: { canvas: null, context: null },
processing: { canvas: null, context: null },
freeform: { canvas: null, context: null },
};

const crosshairCurrentPosition: Position = { x: 0, y: 0 };
Expand All @@ -114,6 +129,8 @@ export const OverlayCanvas: Component<OverlayCanvasProps> = (props) => {
let dragAnimation: AnimatedBounds | null = null;
let grabbedAnimations: AnimatedBounds[] = [];
let processingAnimations: AnimatedBounds[] = [];
let freeformFadeOpacity = 1;
let freeformCompletedAtTimestamp: number | null = null;

const createOffscreenLayer = (
layerWidth: number,
Expand Down Expand Up @@ -366,6 +383,46 @@ export const OverlayCanvas: Component<OverlayCanvasProps> = (props) => {
}
};

const renderFreeformLayer = () => {
const layer = layers.freeform;
if (!layer.context) return;

const context = layer.context;
context.clearRect(0, 0, canvasWidth, canvasHeight);

const strokes = props.freeformStrokePoints;
Copy link
Contributor

@cubic-dev-ai cubic-dev-ai bot Feb 20, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Add a check for props.freeformStrokeVisible at the beginning of renderFreeformLayer to ensure the stroke is hidden when the prop is false.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At packages/react-grab/src/components/overlay-canvas.tsx, line 393:

<comment>Add a check for `props.freeformStrokeVisible` at the beginning of `renderFreeformLayer` to ensure the stroke is hidden when the prop is false.</comment>

<file context>
@@ -366,6 +383,46 @@ export const OverlayCanvas: Component<OverlayCanvasProps> = (props) => {
+    const context = layer.context;
+    context.clearRect(0, 0, canvasWidth, canvasHeight);
+
+    const strokes = props.freeformStrokePoints;
+    if (!strokes || strokes.length === 0) return;
+    if (freeformFadeOpacity <= 0) return;
</file context>
Suggested change
const strokes = props.freeformStrokePoints;
if (!props.freeformStrokeVisible) return;
const strokes = props.freeformStrokePoints;
Fix with Cubic

if (!strokes || strokes.length === 0) return;
if (freeformFadeOpacity <= 0) return;

context.lineCap = "round";
context.lineJoin = "round";
context.globalAlpha = freeformFadeOpacity;
context.strokeStyle = FREEFORM_STROKE_COLOR;

for (const stroke of strokes) {
if (stroke.length < 2) continue;

let pressureSum = 0;
for (const point of stroke) {
pressureSum += point.pressure;
}
const averagePressure = pressureSum / stroke.length;
context.lineWidth =
FREEFORM_STROKE_MIN_WIDTH_PX +
averagePressure *
(FREEFORM_STROKE_MAX_WIDTH_PX - FREEFORM_STROKE_MIN_WIDTH_PX);

context.beginPath();
context.moveTo(stroke[0].x, stroke[0].y);
for (let pointIndex = 1; pointIndex < stroke.length; pointIndex++) {
context.lineTo(stroke[pointIndex].x, stroke[pointIndex].y);
}
context.stroke();
}

context.globalAlpha = 1;
};

const compositeAllLayers = () => {
if (!mainContext || !canvasRef) return;

Expand All @@ -375,13 +432,15 @@ export const OverlayCanvas: Component<OverlayCanvasProps> = (props) => {

renderCrosshairLayer();
renderDragLayer();
renderFreeformLayer();
renderSelectionLayer();
renderGrabbedLayer();
renderProcessingLayer();

const layerRenderOrder: LayerName[] = [
"crosshair",
"drag",
"freeform",
"selection",
"grabbed",
"processing",
Expand Down Expand Up @@ -517,6 +576,28 @@ export const OverlayCanvas: Component<OverlayCanvasProps> = (props) => {
}
}

if (freeformCompletedAtTimestamp !== null) {
const freeformElapsed = Date.now() - freeformCompletedAtTimestamp;
const freeformFadeDeadline = FEEDBACK_DURATION_MS + FADE_OUT_BUFFER_MS;

if (freeformElapsed >= freeformFadeDeadline) {
freeformFadeOpacity = 0;
} else if (freeformElapsed > FEEDBACK_DURATION_MS) {
freeformFadeOpacity =
1 - (freeformElapsed - FEEDBACK_DURATION_MS) / FADE_OUT_BUFFER_MS;
shouldContinueAnimating = true;
} else {
freeformFadeOpacity = 1;
shouldContinueAnimating = true;
}
} else if (
Copy link
Contributor

@cubic-dev-ai cubic-dev-ai bot Feb 20, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: Remove the else if block that continuously sets shouldContinueAnimating = true when drawing to prevent an infinite requestAnimationFrame loop and high CPU usage.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At packages/react-grab/src/components/overlay-canvas.tsx, line 593:

<comment>Remove the `else if` block that continuously sets `shouldContinueAnimating = true` when drawing to prevent an infinite `requestAnimationFrame` loop and high CPU usage.</comment>

<file context>
@@ -517,6 +576,28 @@ export const OverlayCanvas: Component<OverlayCanvasProps> = (props) => {
+        freeformFadeOpacity = 1;
+        shouldContinueAnimating = true;
+      }
+    } else if (
+      props.freeformStrokeVisible &&
+      props.freeformStrokePoints &&
</file context>
Fix with Cubic

props.freeformStrokeVisible &&
props.freeformStrokePoints &&
props.freeformStrokePoints.length > 0
) {
shouldContinueAnimating = true;
}

compositeAllLayers();

if (shouldContinueAnimating) {
Expand Down Expand Up @@ -755,6 +836,26 @@ export const OverlayCanvas: Component<OverlayCanvasProps> = (props) => {
),
);

createEffect(
on(
() =>
[
props.freeformStrokePoints,
props.freeformStrokeVisible,
props.freeformStrokeCompletedAt,
] as const,
([, , completedAt]) => {
if (completedAt != null) {
freeformCompletedAtTimestamp = completedAt;
} else {
freeformCompletedAtTimestamp = null;
freeformFadeOpacity = 1;
}
scheduleAnimationFrame();
},
),
);

onMount(() => {
initializeCanvas();
scheduleAnimationFrame();
Expand Down
5 changes: 5 additions & 0 deletions packages/react-grab/src/components/renderer.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,9 @@ export const ReactGrabRenderer: Component<ReactGrabRendererProps> = (props) => {
grabbedBoxes={props.grabbedBoxes}
agentSessions={props.agentSessions}
labelInstances={props.labelInstances}
freeformStrokePoints={props.freeformStrokePoints}
freeformStrokeVisible={props.freeformStrokeVisible}
freeformStrokeCompletedAt={props.freeformStrokeCompletedAt}
/>

<div
Expand Down Expand Up @@ -209,6 +212,8 @@ export const ReactGrabRenderer: Component<ReactGrabRendererProps> = (props) => {
toolbarActions={props.toolbarActions}
onToggleMenu={props.onToggleMenu}
isMenuOpen={Boolean(props.toolbarMenuPosition)}
isDrawMode={props.isDrawMode}
onToggleDraw={props.onToggleDraw}
/>
</Show>

Expand Down
Loading
Loading