Google Colab to play around with MedViLL for report generation in inference mode?

[PabloMessina]

Hi! First of all, congrats for the awesome work.

I wanted to play around with the pretrained model in the downstream task of report generation. Given that the pretrained weights of MedViLL are available, I thought that it should be straightforward to implement a Google Colab notebook to use a pretrained MedViLL in inference mode to generate reports from Chest X-ray images. However, I'm having a hard time figuring out how to do this. I've been reading the code inside https://github.com/SuperSupermoon/MedViLL/tree/master/downstream_task/report_generation_and_vqa but I don't see an easy to way to do it.

Basically I want to create a Google Colab notebook that does the following things:

• Downloads a single Chest X-ray image
• Downloads the pretrained weights of MedViLL
• Has the code of MedViLL model
• Creates an instance of MedViLL and loads the pretrained weights
• Uses MedViLL in inference mode to generate a report from a single Chest X-ray image
• Visualizes this image next to the generated report

Is something like that easy to implement? I would appreciate some guidance on how to achieve this.

Thanks in advance.

[xaleza]

I'm trying to do the same but i can't seem to firgure how to run the model on inference mode. Do you have an update on this?