Unable to reproduce the performance in Table1 of re-trained model following configs

[zw-SIMM]

When I used your trained model files/clipzyme_model.ckpt, to run the analysis/results, I can got the same results in Table 1.

# Test （not remove exact match uniprot_id）
        * mean BEDROC_85:	 0.44694237653821683
        * mean BEDROC_20:	 0.6298173546254426
        * mean EF_0.05:	 14.085124553319226
        * mean EF_0.1:	 8.060006004418565
        * mean percentile:	 95.10050213381196
        * median percentile:	 99.26118813166505
        * 
# Test （remove exact match uniprot_id)
        * mean BEDROC_85:	 0.36846560413957946
        * mean BEDROC_20:	 0.5723366739501967
        * mean EF_0.05:	 13.135639054964582
        * mean EF_0.1:	 7.702268103727128
        * mean percentile:	 94.22180948902022
        * median percentile:	 98.66822956240192

But, when I tried to train from scracth following the instruction in README.md (4 gpus),

For example, to run the first row in Table 1, run:

python scripts/dispatcher.py -c configs/train/clip_egnn.json -l ./logs/

I got poor performance when evaluating (low val_clip_accuracy and high val_clip_loss)

Epoch 29: 100%|██████████| 1075/1075 [14:39<00:00,  1.22it/s, v_num=7, train_clip_accuracy=0.938, train_clip_quantile=0.966, train_clip_loss=0.122, val_clip_accuracy=0.211, val_clip_quantile=0.784, val_clip_loss=2.830, val_loss=2.830, train_loss=0.141]
Saving args to ./logs/0337abd6ab11244491b4200e9ae37d8a.args

and poor results in analysis/Results.ipynb (Just change the path of best checkpoints)

        * mean BEDROC_85:	 0.01979012780672189
        * mean BEDROC_20:	 0.06457525835087777
        * mean EF_0.05:	 1.273378323980406
        * mean EF_0.1:	 1.2462138252576678
        * mean percentile:	 59.4702234656787
        * median percentile:	 62.850935637459095

Can you help check the configs/clip_egnn.json ? Thanks

For example, I tried to add

"train_esm_with_graph": [true],

But the performance is still poor.

Can you help re-check the training process? Thanks very much.

[pgmikhael]

Hi,

Since this is about the training process rather than the trained model, this may take time (as data / code have moved and we are not actively working on this repo). I will mention however that that we train on 8 gpus, and we note in the paper that fine-tuning ESM does not help.

[zw-SIMM]

Hi,

Since this is about the training process rather than the trained model, this may take time (as data / code have moved and we are not actively working on this repo). I will mention however that that we train on 8 gpus, and we note in the paper that fine-tuning ESM does not help.

Thanks for your reply, I' ll try 8 gpus again. But I don't think 4 or 8 gpus would lead to such a big difference in the results.

[zw-SIMM]

Hi,
Since this is about the training process rather than the trained model, this may take time (as data / code have moved and we are not actively working on this repo). I will mention however that that we train on 8 gpus, and we note in the paper that fine-tuning ESM does not help.

Thanks for your reply, I' ll try 8 gpus again. But I don't think 4 or 8 gpus would lead to such a big difference in the results.

I have tried 8 gpus, still poor performance.

python scripts/dispatcher.py -c configs/train/clip_egnn.json -l ./logs/            
Launching Dispatcher with 1 jobs!

Start gpu worker 0,1,2,3,4,5,6,7

Launched exp: CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -u scripts/main.py --batch_size 8 --precision bf16 --accumulate_grad_batches 1 --dataset_name enzymemap_reaction_graph --dataset_file_path files/enzymemap.json --dataset_cache_path files/cached_enzymemap.p --remove_duplicate_reactions --max_protein_length 650 --class_bal_key ec3 --assign_splits --split_type rule_id --ec_level 1 --split_probs 0.8 0.1 0.1 --split_seed 0 --min_reaction_quality -1 --version 2 --create_sample_per_sequence --lightning_name base --model_name enzyme_reaction_clip --protein_msa_dir /home/datasets/embed_msa_transformer --train_esm_dir esm2_t33_650M_UR50D.pt --esm_dir esm2_t33_650M_UR50D.pt --chemprop_node_dim 9 --chemprop_edge_dim 3 --protein_encoder egnn_sparse_network --use_protein_graphs --protein_dim 1280 --neighbour_aggr mean --pool_type sum --message_dim 24 --norm_feats --norm_coors --update_feats --update_coors --egcl_layers 6 --use_sinusoidal --protein_graphs_dir /home/clipzyme_datasets/quickprot_caches --protein_structures_dir /home/clipzyme_datasets/download_AF2_stuctures --chemprop_num_layers 5 --chemprop_hidden_dim 1280 --chemprop_pool none --compute_metrics_every_step --callback_names lr_monitor checkpointer --max_epochs 30 --lr_decay 0.1 --lr 0.0001 --weight_decay 0.05 --loss_names clip_loss --clip_loss_use_gather --metric_names clip_classification clip_quantile --checkpoint_save_last --monitor val_clip_quantile --num_workers 24 --optimizer_name adamw --scheduler_name linear_warmup_cosine_lr --warmup_init_lr 0.0001 --warmup_min_lr 1e-05 --warmup_start_lr 1e-06 --warmup_steps 100 --train --shuffle_eval_loader --num_sanity_val_steps 0 --val_check_interval 1.0 --gpus 8 --dropout 0 --checkpoint_dir /home/clipzyme_snapshots/metabolomics --logger_name tensorboard --workspace anonymous --project_name enzymes --logger_tags clip  --results_path ./logs/82b548dcd63250711bb39da1a8eb5abb --experiment_name 82b548dcd63250711bb39da1a8eb5abb > ./logs/82b548dcd63250711bb39da1a8eb5abb.txt 2>&1

[zw-SIMM]

Hi,

Since this is about the training process rather than the trained model, this may take time (as data / code have moved and we are not actively working on this repo). I will mention however that that we train on 8 gpus, and we note in the paper that fine-tuning ESM does not help.

"Hi, do you have any plans to update this repo? Thanks!"

[Octopus-Detective]

Hi,
Since this is about the training process rather than the trained model, this may take time (as data / code have moved and we are not actively working on this repo). I will mention however that that we train on 8 gpus, and we note in the paper that fine-tuning ESM does not help.

"Hi, do you have any plans to update this repo? Thanks!"

Hi how long did you take for training？and what kind of gpus did you use？i wanna follow this project but have no idea about how many memories should be used.