Registration and generation of composite image

[NichD]

Hi @cdgatenbee - I was exploring using VALIS to register and fuse an H&E + fluorescence image. I know it's not 'correct' to fuse an RGB image data with single channel fluorescent data. However, I think for my use case, this might work. I'm unclear on a few items related to VALIS behavior:

1. How would VALIS handle this task? Would the warp_and_merge_slides export a 4-channel image (e.g. R, G, B, Fluorescence)?
2. Are there issues to be aware of regarding combining U8 with U16 dtypes?
3. Looking at the parameter resolution_xyu, I'm inferring that VALIS assumes all slides use the same pixel size / resolution in real-world coords. In this workflow, the fluorescent data is not at the same resolution as the WSI-H&E data. Will VALIS generate an initial rigid transform that includes scaling components to address this discrepancy?
4. If I did the above workflow as an initial processing step to register each slide with its corresponding fluorescence data, would I then be able to register those outputs as though they were a stack of histology images, or will that likely blow up?

I realize I'm probably straining the tool in such a workflow. Thanks in advance!

[cdgatenbee]

Hi @NichD,
I think all of the above should work, but would probably require a bit of custom code. I haven't tested using warp_and_merge_slides to merge RGB and Fluorescence images yet, but it's all (py)vips under the hood, so combining the images should work ok. Would just have to double check that U8 gets upcast to U16. I'm less certain about the ome-xml metadata though. I'll try to do some testing to see how it goes. In theory one could alternatively have an ome-tiff with one series being the RGB and a second being the fluorescence image, but I don't think that's something pyvips (and thus valis) currently supports.

Regarding registering the merged images created in the 1st step, that should also work. But it would probably involve creating a new ImageProcessor class. In it's most simple form it could extract either the RGB or fluorescence sub-images from the registered and merged image, and then use one of the built-in preprocessing methods to process that sub-image to use for registration. Or, if you have some way you'd like to process the merged image using information in both the RGB and fluorescence image, you could do that too. The ImageProcessor. process_image method just needs to return a single channel uint8 image. The uninstantiated ImageProcessor would then passed into the if_processing_cls argument of the Valis.register method, since this 4channel U16 image would probably wouldn't be considered RGB(A) (right now valis considers everything that isn't RGB as "immunofluorescent", even though it could be another modality).

When performing the registration, valis actually ignores the resolution_xyu information, and simply resizes the images based on pixel dimensions. So I don't think it will be an issue if the RGB and fluorescence images have different resolutions, as they'll all be resized to the same maximum dimension (i.e. same width or height, in pixels). The resolution_xyu is only really used for the metadata. However, if it does become an issue it is something that should be addressable.

This certainly sounds like an interesting project you're working on. It is pushing the boundaries, but I think ensuring valis has this sort of functionality/flexibility would be very useful. If I can be of help along the way, please let me know.

Best,
-Chandler

[NichD]

@cdgatenbee - That was a very detailed response, thank you so much! Based on what you said, I have several things to try.

In the meantime, I've been trying to hand-jam the workflow through. It hasn't been successful, but has provided some insights. Some additional context. My fluorescence image is a basic TIFF file with the minimal metadata required to make it ImageJ friendly. The WSI H&E input is a standard CZI file from Zeiss AxioScan.

VALIS is able to execute the workflow using these two files. The workflow consists of rigid-registration only, followed by warp & merge (see attached script). What is generated seems to be a uint16 OME-TIFF with 16 channels. The extra 12 channels appear to be all zeros. The fluorescence image (saved as channel 4) does seem to have been up-sampled to the correct size, and the RGB uint8 data does appear to be correctly up-converted to uint16. Additionally, the moving image (i.e. the fluorescence image), doesn't seem to have been transformed at all. The 'original' and 'rigid' images in the overlaps folder are identical. I don't know if this indicates a failure to converge, or some error in the merging process, clearly the extra 12 channels are unintended. I have tried a few tricks, like up-sampling the fluorescent pixel sizes to match the H&E, adding metadata tags in the fluorescent image file, but it has not improved results.

In terms of the grander workflow, after reflection it seems like it would be easier to simply register the WSI-H&E image stack collectively, but apply the resulting transforms to the composite 4-channel H&E + Fluoro image. Given that I am designating the H&E slide as the reference slide for each H&E + fluorescent pair, this seemed like it would work. That said, I haven't gotten that far yet.

Any insight you can give on this issue would be most appreciated!!

Best-

Nich

[NichD]

After closer examination, I realize that VALIS is applying some warping / transformation, but it looks sub-optimal. I realize this particular dataset has significant image artifacts and that may be confounding results. If you have any suggestions as to a preferred feature selector or optimizer, please share!

[cdgatenbee]

Hi @NichD,

Apologies for taking so long to get back to you. Were you able to make any progress on this? If not, there are other feature detectors available (see here), but I think what might help more is creating a custom ImageProcesser for the "green image", which seems to have a lot of artifacts, some of which may be able to be cleaned up. The ImageProcesser needs to have a create_mask method to create a mask that covers the tissue, and a process_image method that returns a uint8 image that is passed to the feature detectors and non-rigid registration method. You can see an example of how one is implemented here. Sorry I can't provide more specific advice in this case.

Best,
-Chandler

[NichD]

Hi @Cgatenbee - I haven't pushed this very much. This particular aspect of the effort has been a lower priority recently. That said, I think your suggestion is very sound and I suspect a modified ImageProcessor that does some preprocessing before yielding an image for feature detection is a great idea and would likely resolve this issue. Thank you for the links and details. I'll follow up with you if/when I pick this thread up again. Much appreciated! Best, -Nich

…

On Mon, Mar 6, 2023 at 7:34 AM cdgatenbee ***@***.***> wrote: Hi @NichD <https://github.com/NichD>, Apologies for taking so long to get back to you. Were you able to make any progress on this? If not, there are other feature detectors available (see here <https://valis.readthedocs.io/en/latest/feature_detectors.html>), but I think what might help more is creating a custom ImageProcesser <https://valis.readthedocs.io/en/latest/preprocessing.html#base-imageprocesser> for the "green image", which seems to have a lot of artifacts, some of which may be able to be cleaned up. The ImageProcesser needs to have a create_mask method to create a mask that covers the tissue, and a process_image method that returns a uint8 image that is passed to the feature detectors and non-rigid registration method. You can see an example of how one is implemented here <https://valis.readthedocs.io/en/latest/_modules/valis/preprocessing.html#ColorfulStandardizer>. Sorry I can't provide more specific advice in this case. Best, -Chandler — Reply to this email directly, view it on GitHub <#37 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADDBJ4I2R5S4YCJMME6W4ZTW2XYWLANCNFSM6AAAAAAUXBJIYQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>