computing the inference FLOPs

[twwwwx]

Hello,
Thanks for your talented work! I was trying to run your model in the test set. However, I have some questions about how did you compute FLOPs.
To have the amazing acceleration rates as you mentioned in the paper, does it need to train an end-to-end sparsified model, or running your finetuning codes for several epoches is just enough? As is mentioned in the paper, "In the training procedure, Transkimmer does not prune the hidden state tensors as it does in the inference time." So in the inference time, by which means you prune the hidden state tensors to turn it to the inference mode? Can I use torchfile directly on the trained transkimmer model to compute the inference FLOPs?
Looking forward to your reply.

[ChandlerGuan]

Hello, thanks for your comment.
You can make the evaluation batch size to 1 with --per_gpu_eval_batch_size==1 in the launching script at this branch. This will perform the hidden state tensor pruning in the inference. And finetuning is good enough to optimize the model.

[twwwwx]

Thanks for your reply! It actually helps me a lot in understanding your work. However I'm still confused about how did you compute the FLOPs. In your paper it's said that it is measured using torchprofile. But in the "run_classification.py" there's a "evaluation" function it seems that compute flops by multiplicaion.
Looking forward to your reply.

[ChandlerGuan]

You can evaluate the FLOPs with torchprofile function profile_macs in the evaluation function of "run_classification.py" similar to the following.
profile_macs(model, args=(batch['input_ids'], batch['attention_mask'], None, None, None, None, batch['labels']))

[twwwwx]

You can evaluate the FLOPs with torchprofile function profile_macs in the evaluation function of "run_classification.py" similar to the following. profile_macs(model, args=(batch['input_ids'], batch['attention_mask'], None, None, None, None, batch['labels']))

Thanks again with a star: )! Now I've got it. Besides, Is it has the same effect if I added torchprofile function profile_macs in your "run_glue_no_trainer.py" and set --per_gpu_eval_batch_size==1 to check the speedup in GLUE dataset tasks?

[ChandlerGuan]

Yes, it should be the same. Thanks for your comments.😊