Filling transparent area with closest average color

[zachallaun]

I need to blur the background of some product images. I'm using an API that automatically removes the background of the product, and then am just blurring the original image and stacking the images. This works fairly well, but it creates a blurred edge around the product.

To improve this, I'd like to use the image with the background removed as a mask to remove the product from the original image, and then fill in that removed area with the closest average background color -- a sort of naive generative background. My hope is that when this is blurred and the product stacked on top, it no longer creates the effect of a blurred product edge.

I'm having trouble figuring out how to replace the transparent area of an image with the closest average color. Any tips on this would be greatly appreciated!

[kipcole9]

Any chance you can share an example of an original image and then its counterpart with the background removed? I can then work up a livebook as a working experiment.

To improve this, I'd like to use the image with the background removed as a mask to remove the product from the original image

This part should be quite straight forward. The image with the background removed will have an alpha band that is itself the mask. If you invert that mask and then attach it as an alpha band to the original image you should have the original background without the product in it. Something like:

# Assuming rgba image so 0 band is r, 1 band is g etc
alpha_band = 3
mask = background_removed_image[alpha_band]

# Instead of masking the background, mask the product
inverted_mask = Image.invert!(mask)

# Attach this alpha band to the original image
image_without_product = Image.add_alpha!(orginal_image, inverted_mask)

replace the transparent area of an image with the closest average color

I'm not clear on what "closest average color" means - can you clarify?

[kipcole9]

I wonder if you blurred the background image (instead of the original image) and then composed the product image over this blurred background you would get crisper edges? This because the product itself is not being blurred?

[zachallaun]

Some images to make things more concrete.

Here's the outcome I'm trying to achieve (generated by another service):

Original image:

Segmented image:

When I say "closest average color", what I mean is to fill in the transparent section with the average color of some "window" around the pixel being filled. So say you have the background that now looks like this:

I'd want the fill on the top right to be predominantly beige to match the wall, and the section in the middle to be predominantly dark gray to match the ground, etc.

[kipcole9]

I did the following as an example to start working on. Is this heading in the right direction?

iex> original = Image.open!("/Users/kip/Desktop/original_image.png")
iex> product = Image.open!("/Users/kip/Desktop/product.png")
iex> inverted_mask = Image.invert!(product[3])
iex> background = Image.add_alpha!(original, inverted_mask)
iex> blurred_background = Image.blur!(background)
iex> composed = Image.compose!(blurred_background, product)

[kipcole9]

Hmmm, I see, still has artefacts around the edges of the product. I wonder if only blurring the background rgb bands (not the alpha band) will help. Seems to:

iex> original = Image.open!("/Users/kip/Desktop/original_image.png")
iex> product = Image.open!("/Users/kip/Desktop/product.png")
iex> inverted_mask = Image.invert!(product[3])
iex> background = Image.add_alpha!(original, inverted_mask)
iex> blurred_background = Image.blur!(background[0..2]) |> Image.add_alpha!(background[3])
iex> composed = Image.compose!(blurred_background, product)

[zachallaun]

Nice! This is looking great, and definitely Good EnoughTM for this application :) Thanks for the suggestions!

[kipcole9]

Cool, glad that gets you moving forward. I've put a simple livebook up. I couldn't do drag/drop for the images since Kino apparently only does rgb (not rgba) images.

[kipcole9]

Can you let me know what service you use to do the image segmentation? I'm a bit curious how far we might get using image with evision to do that in Elixir.

[zachallaun]

I'm using pixelcut.ai, but not for any particular reason beyond it seeming to do a decent job compared to some other services I tried out.