Low level of details for synthetic dragon

[arterms]

Hello!
Thank you for the great work!
Right now I'm trying to reproduce with your code reconstruction of dragon from 200 frames, rendered by mitsuba in 512x512 resolution. Unfortunately after 50000 steps of the second stage I realized that many details on the mesh are lost (if comparing to those pictures from the paper).

Could you please tell me, whether womask_iron.conf config file is the valid one for training with preserving small details? Which parameters might be tuned for better quality? Should be gamma-correction be applied for exr-files generated by mitsuba-renderer before feeding into networks? How input resolution affects shape precision?
Thanks!

[Kai-46]

Hi -- can you share some of your rendered images?

Note that we do gamma-correct our mitsuba-rendered images (also w/ proper global scaling to reasonable brightness) and save as uint8 png images. Then we will use these png images (no inverse gamma correction) in the first stage; in the second stage, we will first inverse-gamma-correct the png images, as rendering equation is linear.

[arterms]

Hello!
Without gamma-correction mitsuba-rendered images looks like under-exposed:

For the first stage I used this code for reading *.exr files, which performs gamma-correction to reasonable brightness:

IRON/models/dataset.py

Lines 69 to 78 in 5993e35

	print("Loading png images failed; try loading exr images")
	import pyexr
	self.images_lis = sorted(glob(os.path.join(self.data_dir, "image/*.exr")))
	self.n_images = len(self.images_lis)
	self.images_np = np.clip(
	np.power(np.stack([pyexr.open(im_name).get()[:, :, ::-1] for im_name in self.images_lis]), 1.0 / 2.2),
	0.0,
	1.0,
	)

Is it flag --inv_gamma_gt supposed to be set during the second stage to perform inverse gamma-correction of the input images? I'm asking, because train_scene.sh script doesn't set this flag for the second stage.

[Kai-46]

Hi -- the exr images need to be scaled (same scaling factor applied to all your training and test images) and then gamma-corrected to have well-exposed uint8 png images. I can share some example script to achieve this exr to png conversion later.

And yes, --inv_gamma_gt would be needed in the second stage for the synthetic data. (the setting in train_scene.sh was meant for real-world images; somehow I found that without this --inv_gamma_gt, the method recovers better specularity for real-world captures sometimes).

[arterms]

Hello!
Thank you for the reply!
Could you suggest any reasonable scaling factor to apply for all exr images before gamma-correction (with grade 1/2.2)?

[Kai-46]

Let's say you put all your training and test images into a tensor of shape [N, H, W, 3]; then a good heuristic might be to scale the 98-percentile (might worth trying other percentiles) of the flattened tensor to be 1; then you clip the scaled tensor to the range (0,1), and perform gamma correction, followed by saving as uint8 png.

[arterms]

Hi!
I experimented a bit with the different percentiles for normalization and gamma-correction setups for the second stage. The best result I got is with 95-percentile and disabled --gamma_pred and enabled --inv_gamma_gt flags. Unfortunately, this result is still doesn't contain small details and sharp edges like that presented in the paper.


Do you have any suggestion how to tune configs/setups to get the shape more accurate?
Thanks!