Hyperparameter Optimization Isolate

[rafecchang]

Here I am trying to optimize hyperparameter individually. Since the original run of optimization (including learning_rate, num_trainable_layers, dropout_rate, batch_size, step_size, gamma, and epochs) gave a higher accuracy score, I am running new trials of hyperparameter optimization with new variables to investigate which are the variables that improve model's performance.

Objective:

• Run a few trials of original model + early stopping + higher epochs max (increase from 5 to 10)
• Run a few trials of original model + gradient clipping
• Run a few trials of original model + switching training / testing

Evaluation:

• Train/ validate with original split
• Train/ validate with a random split

Next Step:

• Determine which hyperparameters to keep
• Use the combination of these hyperparameters to test for a few trails and assess performance

[rafecchang]

Update: Since purning (included in issue#14) is a method to reduce size but not increase accuracy, the method will be tested in a separate attempt. This branch is focused on increasing accuracy.

[rafecchang]

The result of both 1. Run a few trials of original model + early stopping + higher epochs max (increase from 5 to 10,) and 2. Run a few trials of original model + gradient clipping did not match expectation after 5 trials. Therefore I decided to not consider these hyperparameters moving forward.

[rafecchang]

See the result under this branch (15-hyperparameter-optimization-isolate.) Dir: ClimateDebunk/Rafe_dev/simpleisolate.ipynb