Purning

[rafecchang]

Attempted with unstructured purning (converting same weights to 0.) However, doing so will not reduce the model size since the model saves 0 that still occupy space. To assess energy saving I employed EmissionTracker through codecarbon. With an optuna study, I tried different pruning_params in the 6 transformer layers (2 linear layers each.)

Finding:
(In the trail)

• The min energy used in validation is 0.000395 kWh (0.9524200164068909 accuracy) and the max energy used is 0.000409 (0.9647251845775225 accuracy.)
• The test data set has 1219 rows, which converts to 15 more errors with 0.000014 kwh energy reduction.
• With a minimal decrease in energy using and substantial

(Compare to original)

• The original model with no purning, the energy consumption is 0.000420 kWh with the accuracy of 0.9794913863822805. The highest accuracy of the optuna study consumes 0.000403 kWh with the accuracy of 0.977850697292863.
• This translates to a 4.05% reduction in energy consumption and a 0.17% decrease in accuracy compared to the original model.

[rafecchang]

Hyper parameters of the described trail (0.9779 accuracy):
lin1_layer0_pruning = 0.2017953262091607
lin2_layer0_pruning = 0.15729525440967862
lin1_layer1_pruning = 0.12969848850621088
lin2_layer1_pruning = 0.454859759610882
lin1_layer2_pruning = 0.2310579033256247
lin2_layer2_pruning = 0.1607487810536109
lin1_layer3_pruning = 0.22990726915583418
lin2_layer3_pruning = 0.29273042218157586
lin1_layer4_pruning = 0.44861219131635766
lin2_layer4_pruning = 0.23703770072386673
lin1_layer5_pruning = 0.45099619043007
lin2_layer5_pruning = 0.4751512722238028

[rafecchang]

Tested in a different notebook (Rafe_dev/purningTest), the purned model consumes 0.000445 kWh when in validation while the original model consumes 0.000433 kWh with the same reduction in accuracy. The energy reduction dropped from 4% to 2%; there seems to be instability with the measurement method. Though this still decreases energy consumption, we are not going to adopt purning in this project.