Skip to content
This repository was archived by the owner on Apr 18, 2024. It is now read-only.

Commit 18fc2e5

Browse files
BradNeubergBrad Neubergfarioashlomziknikitabelonogov
authored
feat: DEV-4081: Implements Magic Wand tool for image segmentation (#1119)
* Rename floodfill tools to magic wand in preparation of work. * Implements image segmentation Magic Wand. This PR implements a Magic Wand, making it possible to click in a region of an image a user is doing segmentation labeling on, drag the mouse to dynamically change flood filling tolerance, then release the mouse button to get a new labeled area. It is particularly effective at labeling broad, diffuse, complex edged objects, such as clouds, cloud shadows, snow, etc. in earth observation applications or organic shapes in biomedical applications. In-depth technical overview on how it works at the top of src/tools/MagicWand.js. * Collapse multiple Magic Wand additions into the same class if appropriate. * Remove console statement. * Stack classes from successive Magic Wand thresholds when we are able to. * Be able to collapse wand regions even if panning has happened. * Remove spurious comments. * Remove spurious console statement. * Fix doc issue. * Fix code line that got hidden by comment somehow. * Double to single quotes to match new code style. * Change protocol and storage bucket for Magic Wand example. The old bucket was configured under the wrong project, and using https instead of http causes mixed content issues for CORS. Local development is done with http so using that. * Whitespace fix. * Capture undo/redoes and invalidate offscreen cache. This means undo/redo will now work correctly, but will end up creating a new class layer if more magic wanding is done on the current instance. * Put Magic Wand behind a feature flag (fflag_feat_front_dev_4081_magic_wand_tool). * Fix path for feature flag file. * Make the crossOrigin attribute more granular and flexible for different deployment scenarios. * Remove listing the magic wand feature flag here. * Magic Wand will be on in front end by default, but the server config will have it off by default. Note that this is needed so that Magic Wand e2e integration tests will work. * Whitespace fix. * e2e integration tests for the Magic Wand. * Lint fix -- remove unused import. * Doc fix. * Bring Brush tag in when Magicwand tag used, to deal with Eraser tool getting duplicated. * Make sure the eraser shows up when just magicwand tag present. To do this, we have to ensure that the eraser doesn't get added twice if the brush tag is also present. * Make sure magicwand drafts are properly saved for autosave. * Enable eraser integration tests now that this bug is fixed. * Put duplicate tools panel logic behind magicwand feature flag as well. * Fix incorrect import path. * Make Magic Wand feature flag more dynamic, and tie into it for e2e teste. * Code review feedback. * Follow standard `node scripts/create-docs.js` flow. Note that we have not been able to make sure the docs are generated correctly until the https://api.github.com/repos/heartexlabs/label-studio/contents/docs/source/tags API endpoint registers the magicwand tag. * Fix docs so they are autogenerated correctly. * Make code review feedback changes. * Have the Brush be shown in the Magic Wand example as well. * Lint and compilation fixes. * Make Magic Wand thresholding more robust with better UX. Basically, we now calculate the Magic Wand thresholding from the entire screen, not just relative to the image. We also correctly allow you to drag in any direction to get positive or negative thresholding. * Ensure currently selected region still works when changing between some tools. The Magic Wand, the Brush, and the Eraser should all retain selection when moving between them, as well as when moving between those tools and the Zoom or Pan tools. This fix allows removing some hacky specific code I added for the Magic Wand for this behavior. Fixes HumanSignal/label-studio#3510. * Removed unused imports. * Change path to Heartex AWS buckets. * Make changes based on code review feedback. * Add feature flags around unselect behavior. * Fix region unselection for Selection, Erase & others Erase previously had false, so it should not unselect region in any case. Tools like Selection, Zoom and others should not unselect it as well. So set default to false and added true to DrawingTool mixin. * Temporarily disable gallery in e2e because of CORS Currently tests are failing when image in gallery is switched. These CORS errors can be fixed in separate PR, so had to disable it. --------- Co-authored-by: Brad Neuberg <bradneuberg@planet.com> Co-authored-by: Sergey Zhuk <sergey.zhuk@heartex.com> Co-authored-by: hlomzik <hlomzik@gmail.com> Co-authored-by: Nikita Belonogov <nikita.belonogov@heartex.com>
1 parent f62c94d commit 18fc2e5

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

47 files changed

+36525
-187
lines changed

‎e2e/fragments/AtImageView.js‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,7 @@ module.exports = {
9595
},
9696

9797
/**
98-
* Get pixel color at point
98+
* Get pixel color at point
9999
* @param {number} x
100100
* @param {number} y
101101
* @param {number[]} rgbArray

‎e2e/tests/helpers.js‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -444,6 +444,14 @@ const areEqualRGB = (a, b, tolerance) => {
444444
return true;
445445
};
446446

447+
const setKonvaLayersOpacity = ([opacity]) => {
448+
const stage = window.Konva.stages[0];
449+
450+
for (const layer of stage.getLayers()) {
451+
layer.canvas._canvas.style.opacity = opacity;
452+
}
453+
};
454+
447455
const getKonvaPixelColorFromPoint = ([x, y]) => {
448456
const stage = window.Konva.stages[0];
449457
const colors = [];
@@ -731,6 +739,7 @@ module.exports = {
731739
getCanvasSize,
732740
getImageSize,
733741
getImageFrameSize,
742+
setKonvaLayersOpacity,
734743
setZoom,
735744
whereIsPixel,
736745
countKonvaShapes,

‎e2e/tests/image.magic-wand.test.js‎

Lines changed: 199 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,199 @@
1+
/* global Feature, Scenario */
2+
3+
const {
4+
initLabelStudio,
5+
hasKonvaPixelColorAtPoint,
6+
setKonvaLayersOpacity,
7+
serialize,
8+
waitForImage,
9+
} = require('./helpers');
10+
const assert = require('assert');
11+
12+
Feature('Test Image Magic Wand');
13+
14+
const CLOUDSHADOW = {
15+
color: '#1EAE3B',
16+
rgbArray: [30, 174, 59],
17+
};
18+
const HAZE = {
19+
color: '#68eee5',
20+
rgbArray: [104, 238, 229],
21+
};
22+
const CLOUD = {
23+
color: '#1b32de',
24+
rgbArray: [27, 50, 222],
25+
};
26+
27+
const config = `
28+
<View>
29+
<Labels name="labels_1" toName="image_1">
30+
<Label hotkey="1" value="Snow" background="#F61E33" />
31+
<Label hotkey="2" value="Cloud Shadow" background="${CLOUDSHADOW.color}" />
32+
<Label hotkey="3" value="Haze" background="${HAZE.color}" />
33+
<Label hotkey="4" value="Cloud" background="${CLOUD.color}" />
34+
<Label hotkey="5" value="Exclude" background="#2C2C2B" />
35+
</Labels>
36+
<MagicWand name="magicwand_1" toName="image_1" />
37+
<View style="width: 100%; margin-bottom: 20%; margin-side: 10%;">
38+
<Image name="image_1" value="$image" zoomControl="true" zoom="true" crossOrigin="anonymous" />
39+
</View>
40+
</View>`;
41+
42+
const annotationEmpty = {
43+
id: '1000',
44+
result: [],
45+
};
46+
47+
// TODO: Change these URLs to heartex URLs, and ensure the heartex bucket allows CORS access
48+
// for these to work.
49+
const data = {
50+
'image': [
51+
'http://htx-pub.s3.amazonaws.com/samples/magicwand/magic_wand_scale_1_20200902_015806_26_2235_1B_AnalyticMS_00750_00750.jpg',
52+
'http://htx-pub.s3.amazonaws.com/samples/magicwand/magic_wand_scale_2_20200902_015806_26_2235_1B_AnalyticMS_00750_00750.jpg',
53+
'http://htx-pub.s3.amazonaws.com/samples/magicwand/magic_wand_scale_3_20200902_015806_26_2235_1B_AnalyticMS_00750_00750.jpg',
54+
'http://htx-pub.s3.amazonaws.com/samples/magicwand/magic_wand_false_color_20200902_015806_26_2235_1B_AnalyticMS_00750_00750.jpg',
55+
],
56+
'thumb': 'http://htx-pub.s3.amazonaws.com/samples/magicwand/magic_wand_thumbnail_20200902_015806_26_2235_1B_AnalyticMS_00750_00750.jpg',
57+
};
58+
59+
async function magicWand(I, { msg, fromX, fromY, toX, toY }) {
60+
I.usePlaywrightTo(msg, async ({ page }) => {
61+
await page.mouse.move(fromX, fromY);
62+
await page.mouse.down();
63+
await page.mouse.move(toX, toY);
64+
await page.mouse.up();
65+
});
66+
I.wait(1); // Ensure that the magic wand brush region is fully finished being created.
67+
}
68+
69+
async function assertMagicWandPixel(I, x, y, assertValue, rgbArray, msg) {
70+
const hasPixel = await I.executeScript(hasKonvaPixelColorAtPoint, [x, y, rgbArray, 1]);
71+
72+
assert.equal(hasPixel, assertValue, msg);
73+
}
74+
75+
Scenario('Make sure the magic wand works in a variety of scenarios', async function({ I, LabelStudio, AtImageView, AtSidebar }) {
76+
const params = {
77+
config,
78+
data,
79+
annotations: [annotationEmpty],
80+
};
81+
82+
I.amOnPage('/');
83+
84+
LabelStudio.setFeatureFlags({
85+
'fflag_feat_front_dev_4081_magic_wand_tool': true,
86+
});
87+
88+
I.executeScript(initLabelStudio, params);
89+
90+
AtImageView.waitForImage();
91+
await AtImageView.lookForStage();
92+
I.executeScript(waitForImage);
93+
94+
I.say('Making sure magic wand button is present');
95+
I.seeElement('.lsf-toolbar__group button[aria-label="magicwand"]');
96+
97+
I.say('Making sure Eraser button is present');
98+
I.seeElement('.lsf-toolbar__group button[aria-label="eraser"]');
99+
100+
I.say('Select magic wand & cloud class');
101+
I.pressKey('W');
102+
I.pressKey('4');
103+
104+
AtSidebar.seeRegions(0);
105+
106+
I.say('Magic wanding clouds with cloud class in upper left of image');
107+
await magicWand(I, { msg: 'Fill in clouds upper left', fromX: 258, fromY: 214, toX: 650, toY: 650 });
108+
await magicWand(I, { msg: 'Fill in clouds lower left', fromX: 337, fromY: 777, toX: 650, toY: 650 });
109+
110+
I.say('Ensuring repeated magic wands back to back with same class collapsed into single region');
111+
AtSidebar.seeRegions(1);
112+
AtSidebar.see('Cloud');
113+
114+
// Force all the magic wand regions to be a consistent color with no opacity to make testing
115+
// magic wand pixel colors more robust.
116+
I.say('Ensuring cloud magic wand pixels are correctly filled color');
117+
await I.executeScript(setKonvaLayersOpacity, [1.0]);
118+
assertMagicWandPixel(I, 0, 0, false, CLOUD.rgbArray,
119+
'Far upper left corner should not have magic wand cloud class');
120+
assertMagicWandPixel(I, 260, 50, true, CLOUD.rgbArray,
121+
'Upper left should have magic wand cloud class');
122+
assertMagicWandPixel(I, 300, 620, true, CLOUD.rgbArray,
123+
'Lower left should have magic wand cloud class');
124+
assertMagicWandPixel(I, 675, 650, false, CLOUD.rgbArray,
125+
'Far lower right corner should not have magic wand cloud class');
126+
127+
// Make sure the region made from this is correct.
128+
I.say('Ensuring magic wand brushregion was created correctly');
129+
const result = await I.executeScript(serialize);
130+
const entry = result[1];
131+
132+
assert.equal(entry['from_name'], 'labels_1');
133+
assert.equal(entry['type'], 'labels');
134+
const labels = entry['value']['labels'];
135+
136+
assert.equal(labels.length, 1);
137+
assert.equal(labels[0], 'Cloud');
138+
assert.equal(entry['value']['rle'].length > 0, true);
139+
140+
// Undo the bottom left area we just added, make sure its gone but our region list is still
141+
// 1, then redo it and ensure its back and our region list is still 1 again.
142+
I.say('Undoing last cloud magic wand and ensuring it worked correctly');
143+
I.click('button[aria-label="Undo"]');
144+
assertMagicWandPixel(I, 300, 620, false, CLOUD.rgbArray,
145+
'Undone lower left should not have magic wand cloud class anymore');
146+
assertMagicWandPixel(I, 260, 50, true, CLOUD.rgbArray,
147+
'Upper left should still have magic wand cloud class');
148+
AtSidebar.seeRegions(1);
149+
150+
I.say('Redoing last cloud magic wand and ensuring it worked correctly');
151+
I.click('button[aria-label="Redo"]');
152+
assertMagicWandPixel(I, 300, 620, true, CLOUD.rgbArray,
153+
'Redone lower left should have magic wand cloud class again');
154+
assertMagicWandPixel(I, 260, 50, true, CLOUD.rgbArray,
155+
'Upper left should still have magic wand cloud class');
156+
AtSidebar.seeRegions(1);
157+
158+
I.say('Unselecting last magic wand region');
159+
I.pressKey('Escape');
160+
161+
// @todo currently gallery doesn't work well with CORS, so this is not covered by test
162+
// Change to the false color view, zoom in, and magic wand with a new class.
163+
// I.say('Changing to false-color view');
164+
// I.click('[class*="gallery--"] img[src*="false_color"]');
165+
166+
I.say('Zooming in');
167+
I.click('button[aria-label="zoom-in"]');
168+
I.click('button[aria-label="zoom-in"]');
169+
I.click('button[aria-label="zoom-in"]');
170+
I.click('button[aria-label="zoom-in"]');
171+
I.click('button[aria-label="zoom-in"]');
172+
173+
I.say('Selecting cloud shadow class');
174+
I.pressKey('2');
175+
176+
I.say('Magic wanding cloud shadows with cloud shadow class in center of zoomed image');
177+
await magicWand(I, { msg: 'Cloud shadow in middle of image', fromX: 390, fromY: 500, toX: 500, toY: 500 });
178+
179+
I.say('Ensuring new cloud shadow magic wand region gets added to sidebar');
180+
AtSidebar.seeRegions(2);
181+
AtSidebar.see('Cloud Shadow');
182+
183+
I.say('Ensuring cloud shadow magic wand pixels are correctly filled color');
184+
await I.executeScript(setKonvaLayersOpacity, [1.0]);
185+
assertMagicWandPixel(I, 0, 0, false, CLOUDSHADOW.rgbArray,
186+
'Zoomed upper left corner should not have cloud shadow');
187+
assertMagicWandPixel(I, 350, 360, true, CLOUDSHADOW.rgbArray,
188+
'Center area should have magic wand cloud shadow class');
189+
190+
// Make sure if you have a region selected then change the class the region class changes.
191+
I.say('Changing class of existing selected region to Haze should change it to new class');
192+
I.pressKey('3');
193+
AtSidebar.seeRegions(2);
194+
AtSidebar.dontSee('Cloud Shadow');
195+
AtSidebar.see('Haze');
196+
await I.executeScript(setKonvaLayersOpacity, [1.0]);
197+
assertMagicWandPixel(I, 350, 360, true, HAZE.rgbArray,
198+
'Center area should have magic wand haze class');
199+
});

‎examples/image_magic_wand/START.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
2+
# Magic Wand for Image Segmentation
3+
4+
![Magic Wand](/images/screenshots/image_magic_wand.png "Magic Wand")
5+
6+
# Install
7+
8+
## Linux & Ubuntu guide
9+
10+
Install python and virtualenv
11+
12+
```bash
13+
# install python and virtualenv
14+
apt install python3.6
15+
pip3 install virtualenv
16+
17+
# setup python virtual environment
18+
virtualenv -p python3 env3
19+
source env3/bin/activate
20+
21+
# install requirements
22+
cd backend
23+
pip install -r requirements.txt
24+
```
25+
26+
## Cross Domain Image Access
27+
28+
Note that if you are storing images that you'd like to apply the Magic Wand to cross-domain, such as on Google Storage Buckets, you will have to [enable CORS headers for the storage buckets to enable cross-domain pixel access](https://developer.mozilla.org/en-US/docs/Web/HTML/CORS_enabled_image) so that the Magic Wand can get the raw pixel data to threshold. By default browsers block JavaScript from accessing pixel-level image data unless the right CORS headers are set.
29+
30+
As an example, if you wanted to configure a Google Storage Bucket with the right headers, you might do the following:
31+
32+
```bash
33+
gsutil cors set gcp_cors_config.json gs://BUCKET-NAME
34+
```
35+
36+
Note that in the gcp_cors_config.json example given in this directory that we have set `origin` to `*`, which means all origins can access that data, as well as set `responseHeader` to `*`, which means all HTTP response headers can be accessed. In a real scenario you probably want to think through the security ramifications of this for your own particular Label Studio setup.
37+
38+
# Start
39+
40+
Magic Wand for image segmentation:
41+
42+
```bash
43+
fflag_feat_front_dev_4081_magic_wand_tool=1 python server.py -c config.json -l ../examples/image_magic_wand/config.xml -i ../examples/image_magic_wand/tasks.json -o output
44+
```

0 commit comments

Comments
 (0)