grid artifacts

[dabeschte]

Hi!

I did some experiments with this approach and I struggle to get rid of the grid artifacts that you can see also in your reference images like here if you zoom in really close.

Do you know where these artifacts come from and how to get rid of them? I even tried varying the convolution kernel randomly every iteration, but it does not really help. Using smaller kernels, the grid seems to get smaller as well. But that unfortunately does not solve it.

[ariel415el]

Hi,
I think grid artifact are a sign of a local minma in the optimization process (patches with grid signs are out of distribution).
Changing the kernel size is a good idea but I can recommend also changing the number of scales you work with, the size of the coarsest optimization and number of steps (especially in first levels where they are not costly).

[dabeschte]

From what I can tell, the grid only appears in the later steps with reasonably high resolution and it just gets much more pronounced with every other step. The explanation sounds intuitive, but I am still a bit surprised how regular this grid is.
I will try to do more steps in these stages and see if it helps.

And thanks for the quick response! :)

[ariel415el]

Very interesting, I haven't noticed that. It might also be useful to use more SWD projections
Please do share your conclusions.

[dabeschte]

The grid does not appear on all textures. I used a brick wall as example and the result was completely free of artifacts.
But using this wooden texture as input:

the grid is clearly visible on the output:

it does appear to become less noticable with a greater number of adam steps, but it takes a lot more to really get rid of it. with 1000 instead of 300 steps, it did not change much. but with 3000 steps for every scale, the results becomes significantly better, even if still not perfect. But then, the model essentially just creates a copy of the input. I already increase the coarsest level by 100% for this result:

If I find the solution at some point, I'll post it here!

[ariel415el]

Very good example. I tried it myself and I too get these grid effect.
I'm not quite sure if the grid size is determined by the patch size which was my first suspect

[dabeschte]

I wonder if this only happens on jpeg inputs. because at closer look, you can see some small grid artifacts in the input image as well, just not as pronounced

[dabeschte]

I played around with a different wooden texture and it does not appear here. even if I convert the texture to lower quality jpeg where you can clearly see a grid in the source image - the result gets rid of the grid in this case

It can remove them even when basically copying the source: