rgb_only_on_original in Nerfiller_model.py

[Harr7y]

https://github.com/ethanweber/nerfiller/blob/main/nerfiller/nerf/nerfiller_model.py
if self.config.rgb_only_on_original:
loss_dict["rgb_loss"] = self.rgb_loss(gt_rgb * batch["original_mask"], pred_rgb * batch["original_mask"])

should have an "else" case?
else:
loss_dict["rgb_loss"] = self.rgb_loss(gt_rgb, pred_rgb)

[Harr7y]

I am a little confused with nerfstudio code, can you point out where makes the masked out part (by an additional mask image) not be used during the training.

Is it the "erode_mask" function in PatchPixelSampler ?

Besides, I wonder how to set breakpoint into the code to check the immediate features, such like outputs after dealing with the ray_bundle. I tried the breakpoint() or pdb.set_trace(). They both didn't work.

For example:
Adding the breakpoint to the nerfiller_pipeline.py
ray_bundle, batch = self.datamanager.next_train(step)
model_outputs = self._model(ray_bundle)
breakpoint())

[ethanweber]

Loss function
Hi @Harr7y, interesting catch! I think it should have an else case. It must not have mattered in my results (surprisingly) because I first pretrain with Nerfacto and then fine-tune with Nerfiller. This would imply that LPIPS loss can do the heavy lifting of adding the content consistently. Let me know if the else case helps your results, as I'd be very curious and eager to modify the code if so. Thanks!

Masks
Here is some masking logic in the datamanager, where we sample from the images.

Breakpoints
I typically use import pdb; pdb.set_trace();, which has worked fine for me. It doesn't work for me when I am using a rich progress bar, but otherwise it's been good for me.

[Harr7y]

Thanks for your reply~
Masks
I apologize if my words have caused any misunderstanding. What I meant to ask is which part of NerfStudio ensures that the content under the mask is not used in the training process.

Breakpoints
After adding the breakpoint to the nerfiller_pipeline.py and pip install -e ., then I run ns-train train-occluder with "grid-prior-du" config. It just didn't work. 😥 Can you provide some advice?

ray_bundle, batch = self.datamanager.next_train(step)
model_outputs = self._model(ray_bundle)
pdb.set_trace()

[ethanweber]

Masks
Oh yes, for masks it's coming from the pixel sampler for Nerfacto, but I sample everything for the custom methods. I actually set the images after inpainting in the pipeline file, which is why I can sample everything and put a loss everywhere on the patch.

Breakpoints
Oh I think you mean ns-launch train-occluder with grid-prior-du, instead of "ns-train". Make sure you don't launch with multiple processes when you run this. You'll want to run the command ns-train grid-prior-du ... in its own thread, rather than relying on the command that launches many python commands at once. So, you can run with --dry-run and then copy paste the command into its own terminal. This way I think pdb will start working, since the terminal is for just one command. (If you've already done this, I'm a bit lost for why pdb wouldn't work.)

[Harr7y]

It works, thanks a lot!!! 👏👏

[ethanweber]

Hey @Harr7y, I just wanted to follow up from a while ago. Did you ever implement the "else statement" above for the rgb loss and did it prove to be important? I'd like to update the codebase if that ended up being important for results. Thanks!

[Harr7y]

Hey @Harr7y, I just wanted to follow up from a while ago. Did you ever implement the "else statement" above for the rgb loss and did it prove to be important? I'd like to update the codebase if that ended up being important for results. Thanks!

Sorry, this experiment is a little too long ago, and I've cleaned up the test records. In my (not very clear) memory, the implement of 'else statements' has little effect on the result. It seems that LPIPS loss has played a great role.

[ethanweber]

No worries at all, and thanks for the reply! :) I thought the same, as I double checked my code and experiments before releasing the code. Thanks @Harr7y! Going to leave the issue open in case anyone has similar questions regarding this topic.