Experiment history

[hbandukw]

Hackathon Details

Current submissions

[hbandukw]

Data: ~6000 climate-quotes

Task: Classify them into one of 8 labels

Data Imbalance

#To-Do Add plot

Current Model Features:

1. Tokenizer: distilbert/distilbert-base-uncased: Same as BERT but smaller. Trained by distillation of the pretrained BERT model, meaning it’s been trained to predict the same probabilities as the larger model.

• Padding based on max-length: 75% percentile of all quote lengths #To-Do Add histogram of quote lengths

3. Classification model: DistilBert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output)

Improve model performance

1. Data Augmentation: To resolve class imbalance
   Results: #To-Do
   Adopted: No

2. Learning rate scheduler
   Results: #To-Do

3. Hyperparameter optimization

• Batch size (val + train)
• Epochs
• Num of trainable layers
• Dropout
• Attention dropout
• Learning rate
• Gamma # lr scheduler
• Step size # lr scheduler

Results: #To-Do

Improve energy/resource utilization

1. Quantization

[rafecchang]

• Note on Data Augmentation: Implemented naw.ContextualWordEmbsAug to create a balanced dataset.

• Note on Hyperparameter optimization: Used optuna for trials of 40 on both original and augmented data. However, each trail takes around 25 minutes and not be able to see the full result yet.