Visualization of global and local information in images in the paper

[jgy0]

I have a question: in Figure 3, b represents global information, so why is the house purple (low value), whereas in c, the raindrop should be local information that requires attention (also low value)? Could these two figures be reversed?

[hainuo-wang]

Hi jgy0,

Thanks for your attention and the question! The figures are correct. The high (yellow) and low (purple) values represent the magnitude of feature response to different types of information within the model. This behavior can be explained by the fact that (b) is a judgment based on the model's hidden state, while (c) is a judgment based on its current state.

Figure 3 (b): In an all-in-one setting, the primary goal of the global branch is to identify the overall degradation type for the entire image. The model first needs to understand, "Is this a rainy, snowy, or hazy image?" This judgment is based on the model's hidden state, representing a conclusion it has already formed. Therefore, Figure 3 (b) shows which regions in the image contribute more to supporting this global classification decision. The house itself is image content; its textures and structure contribute relatively little to identifying the weather type and are thus "ignored" by this branch.

Figure 3 (c): This branch focuses on the local textures that need to be preserved. Its judgment is based on the current state, directly processing the raw pixel information at hand. Therefore, the 'good' background textures receive a high value (yellow). The raindrop is an artifact to be removed, not a texture to be preserved, so it logically receives a low value (purple) to be distinguished from the valid textures.

The detailed modeling of this mechanism is further explained in Section 3 of the paper. More feature visualizations can be found in the Appendix.

Please feel free to ask if you have any other questions.

Best regards,

Hainuo Wang

[jgy0]

Hi jgy0,

Thanks for your attention and the question! The figures are correct. The high (yellow) and low (purple) values represent the magnitude of feature response to different types of information within the model. This behavior can be explained by the fact that (b) is a judgment based on the model's hidden state, while (c) is a judgment based on its current state.

Figure 3 (b): In an all-in-one setting, the primary goal of the global branch is to identify the overall degradation type for the entire image. The model first needs to understand, "Is this a rainy, snowy, or hazy image?" This judgment is based on the model's hidden state, representing a conclusion it has already formed. Therefore, Figure 3 (b) shows which regions in the image contribute more to supporting this global classification decision. The house itself is image content; its textures and structure contribute relatively little to identifying the weather type and are thus "ignored" by this branch.

Figure 3 (c): This branch focuses on the local textures that need to be preserved. Its judgment is based on the current state, directly processing the raw pixel information at hand. Therefore, the 'good' background textures receive a high value (yellow). The raindrop is an artifact to be removed, not a texture to be preserved, so it logically receives a low value (purple) to be distinguished from the valid textures.

The detailed modeling of this mechanism is further explained in Section 3 of the paper. More feature visualizations can be found in the Appendix.

Please feel free to ask if you have any other questions.

Best regards,

Hainuo Wang

Thank you so much for the author's reply; I understood immediately. I have another question about what the LPEM module in figure 5 is, because I only saw this module appear once in the manuscript (I was looking at the version on arXiv).

[hainuo-wang]

The LPEM in Figure 5 stands for Logarithmic Positional Encoding Matrix. This module represents the specific initialization and parameterization design for the matrix $A$, which we adopted from the S4D and Mamba architectures.

For the specific implementation details, please refer to:

MODEM/basicsr/models/archs/modem_arch.py

Line 224 in 78f6334

def A_log_init(d_state, d_inner, copies=1, device=None, merge=True):