Hyperparameter Optimization Complete

[rafecchang]

Here I am trying a complete hyperparameter optimization.

Objective:

• For model accuracy, I am optimizing learning_rate, num_trainable_layers, dropout_rate, epochs, batch_size, step_size, and gamma- these are the original hyperparameters that had went through a optimization in a previous optuna study.
• For energy reduction, I am optimizing pruning_rate (new variable.)
• This attempt also uses gradient clipping and early stopping (new variables.)
• To make the optimization more generalizable, I am also switching the training and testing set in each trial. The function randomly draws 4 out of 5 batches of augmented data as training, and use the unused original data as validation. This should not cause data leakage since each trial is create a model and train from ground up (new variable.)
• In this optimization, I am also increasing the max epoch from 5 to 10.

Finding:

• After 7 hours of running, the optuna study completed 15 out of 40 trials planned.
• The best accuracy is 0.57 from trial 11. The hyperparameters of the trail is 'learning_rate': 4.412782360446137e-05, 'num_trainable_layers': 1, 'dropout_rate': 0.22861677124968877, 'epochs': 8, 'batch_size': 16, 'step_size': 1, 'gamma': 0.8669394692021379, 'pruning_rate': 0.05915694799122513.

Discussion:

• The optimization is slow and not optimal. This optimization included new variables that was not present in the previous runs, therefore it's hard to tell which contributes to the result.