About loss and results

[LinLin2004]

I am really interested in your work and i am trying to make a reproduction for your experiment. But i meet some problems ,will you give more detailed code for me?
If you are not convenient to release your code to the public , you can contact me by email. My e-mail is 1071753343@qq.com and I propose i just use it to my study research .
THANK YOU AND LOOKING TO YOUR REPLY!

[DY112]

Hello, thank you for your interest in my research.
Unfortunately, even if it is for personal research purposes, sharing the code via email could lead to legal issues.
To support your implementation, I have shared the model architecture and the Slot Attention repository in the README.
If you have any further questions, please feel free to ask — I’ll do my best to assist you.

[LinLin2004]

I want to know how can i get the srgb format (e,g .jpg) image in visual not
in the .tiff format. Would you like to share your code about tiff2jpg ?
@DY112

[DY112]

Hello, if it’s for visualization purposes, I recommend saving the image as a PNG after either white balancing the raw or applying gamma correction (such as 2.2) to the raw, rather than converting it to sRGB.

[LinLin2004]

Hello~ I want to know the detailed experiment settings. I have tried only use l1loss experiment settings about “2000epoch 32batchsize adamw lr 1e-4” resulting MAE is 2.6 for mixed-illuminant scenario using LSMI galaxy dataset. Now I use two loss functions which you mentioned in your paper setting my experiment in “5000epoch 128batchsize adamw lr 5e-4”,but the results look like not well.

I hope know your detailed settings when you use LSMI datasets!

Thank you and looking forward for your reply! @DY112

[LinLin2004]

Also, I wrote the CentroidLoss after reading your paper following like this

class CentroidLoss(nn.Module):
    def __init__(self, kmeans_res_file='./kmeans_results_slot7.json'):
        super().__init__()
        with open(kmeans_res_file, 'r') as f:
            self.centroids = torch.tensor(json.load(f), dtype=torch.float32).reshape(-1, 3)
        # self.centroids = self.centroids[:, [0, 2]] # del green channel 
    
    def forward(self, pred_chorm, gt_chorm, pred_wmap, gt_wmap):
        # calculate l1 cost
        b, n, _ = gt_chorm.shape # n: number of illumination
        _, k, _ = pred_chorm.shape # k: number of centroids (slots)

        gt_ = gt_chorm.reshape(b, n, 1, 3).expand(b, n, k, 3) # BN3 -> BNK3
        cen_ = self.centroids.reshape(1, 1, k, 3).expand(b, n, k, 3).to(gt_chorm.device) # K3 -> BNK3

        l1cost = torch.abs(gt_ - cen_).sum(dim=-1).reshape(b, n*k)

        # find the n smallest l1cost idx
        _, idx = torch.topk(l1cost, n, dim=-1, largest=False) # b, n 
        idx = idx % k

        # compute center loss
        chorm_idx = idx.reshape(b, n, 1).expand(b, n, 3)
        used_pred_chorm = torch.gather(pred_chorm, dim=1, index=chorm_idx) # BN3
        chorm_loss = F.l1_loss(used_pred_chorm, gt_chorm, reduction='mean') 

        _, _, h, w = pred_wmap.shape
        map_idx = idx.reshape(b, n, 1, 1).expand(b, n, h, w)
        used_pred_wmap = torch.gather(pred_wmap, dim=1, index=map_idx)
        wmap_loss = F.l1_loss(used_pred_wmap, gt_wmap, reduction='mean')
        return chorm_loss + wmap_loss

And I found my train loss did not decrease, test loss did decrease .
Could you tell me where I get wrong?@DY112

[DY112]

Hello, we used a batch size of 20 and the default Adam optimizer (beta1 = 0.9, beta2 = 0.999).
The learning rate was set to 1e-4, and we used the StepLR scheduler with gamma = 0.5 and step_size = 300.
Training was conducted for 658 epochs, and the best checkpoint on the validation set was used during testing.
Regarding the loss design, I currently do not have enough time to examine it in detail, so I will get back to you soon.
Thank you.

[LinLin2004]

Hello, I have used your new experiment settings and only used l1loss resulting MAE is 2.6 for mixed-illuminant scenario using LSMI galaxy dataset. But when I use 2 loss including CentroidLoss resulting worse.
Could you help me find what my problems about what I wrote? I really in hurry and I have ever tried many times to fix it, but it did not work well.
@DY112 Thank you very much!!

[ex2333]

Hello, I've also been struggling to reproduce CentroidLoss recently. Could you confirm if this reproduction is correct? I'm in urgent need of using it and look forward to your reply!
@DY112

Also, I wrote the CentroidLoss after reading your paper following like this

class CentroidLoss(nn.Module):
    def __init__(self, kmeans_res_file='./kmeans_results_slot7.json'):
        super().__init__()
        with open(kmeans_res_file, 'r') as f:
            self.centroids = torch.tensor(json.load(f), dtype=torch.float32).reshape(-1, 3)
        # self.centroids = self.centroids[:, [0, 2]] # del green channel 
    
    def forward(self, pred_chorm, gt_chorm, pred_wmap, gt_wmap):
        # calculate l1 cost
        b, n, _ = gt_chorm.shape # n: number of illumination
        _, k, _ = pred_chorm.shape # k: number of centroids (slots)

        gt_ = gt_chorm.reshape(b, n, 1, 3).expand(b, n, k, 3) # BN3 -> BNK3
        cen_ = self.centroids.reshape(1, 1, k, 3).expand(b, n, k, 3).to(gt_chorm.device) # K3 -> BNK3

        l1cost = torch.abs(gt_ - cen_).sum(dim=-1).reshape(b, n*k)

        # find the n smallest l1cost idx
        _, idx = torch.topk(l1cost, n, dim=-1, largest=False) # b, n 
        idx = idx % k

        # compute center loss
        chorm_idx = idx.reshape(b, n, 1).expand(b, n, 3)
        used_pred_chorm = torch.gather(pred_chorm, dim=1, index=chorm_idx) # BN3
        chorm_loss = F.l1_loss(used_pred_chorm, gt_chorm, reduction='mean') 

        _, _, h, w = pred_wmap.shape
        map_idx = idx.reshape(b, n, 1, 1).expand(b, n, h, w)
        used_pred_wmap = torch.gather(pred_wmap, dim=1, index=map_idx)
        wmap_loss = F.l1_loss(used_pred_wmap, gt_wmap, reduction='mean')
        return chorm_loss + wmap_loss

And I found my train loss did not decrease, test loss did decrease . Could you tell me where I get wrong?@DY112

[DY112]

Hello, @ex2333 @LinLin2004 please send your email address to my email.
I'll help your reproduction.

[LinLin2004]

I have already send you to your email dongyoung.kim@yonsei.ac.kr,
my email is 1071753343@qq.com!
Looking forward to your reply! @DY112

[LinLin2004]

Hello, I want to know how you preprocess your LSMI dataset.
I have already use you original code for LSMI dataset to preprocess these but I remove illum_augmentation. Also, I used the traditional U-Net as an encoder after reading your paper, the link is https://github.com/praann07/Multi-Scale-Attention-Enhanced-U-Net-for-Image-Deblurring-with-Composite-Loss-Functions . So do you use the same U-Net as me?
Looking forward to your reply! @DY112