Multimodal deep learning sample model for in-hospital mortality prediction

[rohansuri17]

This notebook demonstrates how to build a multimodal deep learning model for in-hospital mortality prediction by combining:

• MIMIC-IV electronic health record (EHR) time-series data
• MIMIC-CXR chest X-ray images

[jhnwu3]

By any chance, can you do this with the existing PyHealth API here?

[rohansuri17]

@jhnwu3 We did think about this actually. PyHealth does not currently have built-in support for multimodal fusion with EHR + images (like chest X-rays). We used the HAIM repo to assist us in generating data pairs https://github.com/lrsoenksen/HAIM/blob/main/1_Generate%20HAIM-MIMIC-MM%20from%20downloaded%20MIMIC-IV%20and%20MIMIC-CXR.ipynb. We would have to perform all preprocessing and concatenation externally (like we already do) which feels like it defeats the purpose of using pyhealth. Also, it seems like if we did do pyhealth we can only do unimodal tasks with either EHR data or CXR data.

Would it be better to focus on just EXR/CXR datasets exclusively and create model examples around those or can we use our existing approach above? In my opinion the multimodal data scenario is super novel, but I think we would need to have Image data support, Multimodal fusion architectures, Vision encoders (ResNet, etc.), and Image-text or EHR-image pairing utilities to do so. HAIM + Medmod repo (https://github.com/[nyuad-cai/MedMod](https://github.com/nyuad-cai/MedMod)) do the heavy lifting in that regard.

Let me know your suggestions!

[jhnwu3]

Actually, the way PyHealth works right now for MIMIC4 (which includes MIMIC-CXR and MIMIC-Note) directly supports the loading of both images and EHR.

https://github.com/sunlabuiuc/PyHealth/blob/master/pyhealth/datasets/mimic4.py

You just have to specify them in the path. Once that happens, it's pretty trivial to define a new MultiModalMIMIC4Task() using all of the different patient event tables.

https://github.com/sunlabuiuc/PyHealth/blob/master/pyhealth/tasks/mortality_prediction.py

For reference, this mortality_prediction example tasks here can already pre-load images and other events to each patient. It's also a lot cleaner than using HAIM here (HAIM looks like a monstrosity here haha).

The only thing I can see preventing you guys from replicating this is the fact that I doubt you'll have the compute to properly load MIMIC4 in its entirety (we're currently working on a memory fix here).