Large-scale Benchmarking and Boosting Transfer Learning for Medical Image Analysis

A comprehensive systematic study of transfer learning of numerous models with distinct architectures pretrained on various datasets using different strategies for medical imaging applications, focusing on (i) benchmarking numerous conventional and modern convolutional neural network (ConvNet) and vision transformer architectures across various medical tasks; (ii) investigating the impact of fine-tuning data size on the performance of ConvNets compared with vision transformers in medical imaging; (iii) examining the impact of pretraining data granularity on transfer learning performance; (iv) evaluating transferability of a wide range of recent self-supervised methods with diverse training objectives to a variety of medical tasks across different modalities; and (v) delving into the efficacy of domain-adaptive pretraining on both photographic and medical datasets to develop high-performance models for medical tasks.

Publication

Large-scale benchmarking and boosting transfer learning for medical image analysis

Mohammad Reza Hosseinzadeh Taher¹, Fatemeh Haghighi¹, Michael B. Gotway², Jianming Liang^1
1 Arizona State University, ² Mayo Clinic
Medical Image Analysis (MedIA), 2025

Paper | Code

Installation

Clone the repository and install dependencies using the following command:

$ git clone https://github.com/MR-HosseinzadehTaher/BenchmarkTransferLearning.git
$ cd BenchmarkTransferLearning/Version2
$ pip install -r requirements.txt

Model Zoo

You can download the pretrained models used/developed in our paper as follows:

Category	Backbone	Platform	Training Dataset	Training Objective	model
Domain-adapted models	ResNet50	Pytorch	ImageNet → ChestX-ray14	Supervised → Supervised	download
	ResNet50	Pytorch	ImageNet → ChexPert	Supervised → Supervised	download
	ResNet50	Pytorch	ImageNet → MIMIC	Supervised → Supervised	download
	Swin-B	Pytorch	ImageNet → MIMIC	Supervised → Supervised	download
	ConvNeXt-B	Pytorch	ImageNet → MIMIC	Supervised → Supervised	download
ImageNet-1K models	Swin-T	Pytorch	ImageNet-1K	Supervised	download
	Swin-S	Pytorch	ImageNet-1K	Supervised	download
	Swin-B	Pytorch	ImageNet-1K	Supervised	download
	ConvNeXt-T	Pytorch	ImageNet-1K	Supervised	download
	ConvNeXt-S	Pytorch	ImageNet-1K	Supervised	download
	ConvNeXt-B	Pytorch	ImageNet-1K	Supervised	download
iNat2021 models	ResNet50	Pytorch	iNat2021	Supervised	download
	ResNet50	Pytorch	iNat2021 mini	Supervised
Self-supervised models	ResNet50	Pytorch	ImageNet	Barlow Twins	download
	ResNet50	Pytorch	ImageNet	SwAV	download
	ResNet50	Pytorch	ImageNet	DeepCluster-v2	download
	ResNet50	Pytorch	ImageNet	Sela-v2	download
	ResNet50	Pytorch	ImageNet	InsDis	download
	ResNet50	Pytorch	ImageNet	MoCo-v1	download
	ResNet50	Pytorch	ImageNet	MoCo-v2	download
	ResNet50	Pytorch	ImageNet	PCL-v1	download
	ResNet50	Pytorch	ImageNet	PCL-v2	download
	ResNet50	Pytorch	ImageNet	SimCLR-v1	download
	ResNet50	Pytorch	ImageNet	SimCLR-v2	download
	ResNet50	Pytorch	ImageNet	InfoMin	download
	ResNet50	Pytorch	ImageNet	PIRL	download
	ResNet50	Pytorch	ImageNet	BYOL	download

For all pretrained models, except those we domain-adapted, we employed the official publicly released weights from the respective original papers, and we acknowledge the corresponding authors.

Datasets

Datasets can be downloaded from the following links.

ChestX-ray14

ChexPert

DRIVE (pre-processed dataset can be downloaded from here)

Montgomery

Shenzhen

RSNA PE detection

SIIM-ACR Pneumothorax Segmentation

For the classificatin tasks, we have provided data splits in the dataset directory. For each of train, test, and validation sets, a text file including images and labels information is provided. For the segmentation tasks, you need to put the train, test, and validation sets into separate folders.

Fine-tuning of pre-trained models on target tasks

1. Download the desired pre-trained model from the above-mentioned links.
2. Download the desired dataset; you can simply add any other dataset that you wish.
3. For classification tasks, run the following command by the desired parameters. For example, to finetune ImageNet-1K model with ConvNeXt-base backbone on ChestX-ray14, run:

python main_classification.py --data_set ChestXray14  \
--init imagenet_1k \
--model convnext_base \
--proxy_dir path/to/pre-trained-model \
--data_dir path/to/dataset \
--train_list dataset/Xray14_train_official.txt \
--val_list dataset/Xray14_val_official.txt \
--test_list dataset/Xray14_test_official.txt \
--num_class 14

Or, to evaluate ImageNet-1K model with Swin-base backbone on ChestX-ray14, run:

python main_classification.py --data_set ChestXray14  \
--init imagenet_1k \
--data_dir path/to/dataset \
--model swin_base \
--train_list dataset/Xray14_train_official.txt \
--val_list dataset/Xray14_val_official.txt \
--test_list dataset/Xray14_test_official.txt \
--num_class 14

4. For segmentation tasks, run the following command by the desired parameters. For example, to finetune ImageNet-1K model with ConvNeXt-base backbone on clavicle segmentation, run:

python main_segmentation.py --data_set SCR-Clavicle  \
--arch convnext_upernet  \
--backbone convnext_base \
--init imagenet_1k \
--proxy_dir path/to/pre-trained-model \
--train_data_dir path/to/train/images \
--train_mask_dir path/to/train/masks \
--valid_data_dir path/to/validation/images \
--valid_mask_dir path/to/validation/masks \
--test_data_dir path/to/test/images \
--test_mask_dir path/to/test/masks \
--train_file_path path/to/train/file/split \
--valid_file_path path/to/val/file/split \
--test_file_path path/to/test/file/split

Or, to evaluate ImageNet-1K model with Swin-base backbone on clavicle segmentation, run:

python main_segmentation.py --data_set SCR-Clavicle  \
--arch swin_upernet  \
--backbone swin_base \
--init imagenet_1k \
--train_data_dir path/to/train/images \
--train_mask_dir path/to/train/masks \
--valid_data_dir path/to/validation/images \
--valid_mask_dir path/to/validation/masks \
--test_data_dir path/to/test/images \
--test_mask_dir path/to/test/masks \
--train_file_path path/to/train/file/split \
--valid_file_path path/to/val/file/split \
--test_file_path path/to/test/file/split

Citation

If you use this code or use our pre-trained weights for your research, please cite our paper:

@article{taher2025large,
  title={Large-scale benchmarking and boosting transfer learning for medical image analysis},
  author={Taher, Mohammad Reza Hosseinzadeh and Haghighi, Fatemeh and Gotway, Michael B and Liang, Jianming},
  journal={Medical image analysis},
  volume={102},
  pages={103487},
  year={2025},
  publisher={Elsevier}
}

Acknowledgement

This research has been supported in part by ASU and Mayo Clinic through a Seed Grant and an Innovation Grant, and in part by the NIH under Award Number R01HL128785. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. This work has utilized the GPUs provided in part by the ASU Research Computing and in part by the Bridges-2 at Pittsburgh Supercomputing Center through allocation BCS190015 and the Anvil at Purdue University through allocation MED220025 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296. The content of this paper is covered by patents pending. We thank Nahid Ul Islam and Ruibin Feng for evaluating 14 self-supervised ImageNet models, including InsDis, MoCo-v1, MoCo-v2, SimCLR-v1, SimCLR-v2, BYOL, PIRL, DeepCluster-v2, SeLa-v2, PCL-v1, PCL-v2, SwAV, InfoMin, and Barlow Twins, on target tasks: ECC (by Nahid), and DXC_14 and DXC_5 (by Ruibin), respectively. The content of this paper is covered by patents pending. This work utilizes implementations of ConvNeXt and Swin Transformer, adapted from their official GitHub repositories: ConvNeXt and Swin Transformer. We also build U-Net architecture for segmentation tasks by referring to the released code at segmentation_models.pytorch.

License

Released under the ASU GitHub Project License.