Select Kernal

Model run configurations for our 4 tests

Preparing OpenWeb data (first)

$ python data/openwebtext/prepare.py

The following configurations can be ran in parallel

Baseline (Dot Product)

$ torchrun --standalone --nproc_per_node=8 train.py config/train_gpt2.py --kernel_config=0 --out_dir=out-baseline

Polynomial

$ torchrun --standalone --nproc_per_node=8 train.py config/train_gpt2.py --kernel_config=1 --out_dir=out-polynomial

Periodic

$ torchrun --standalone --nproc_per_node=8 train.py config/train_gpt2.py --kernel_config=2 --out_dir=out-periodic

Gaussian

$ torchrun --standalone --nproc_per_node=8 train.py config/train_gpt2.py --kernel_config=3 --out_dir=out-gaussian

Evaluate GPT2 model through AI2 Reasoning Challenge (ARC)

1. Download the ARC dataset and Tokenize the ARC Corpus.
   $ python data/arc/prepare.py
   (Remark: if not properly work, download the ARC dataset from this link and unzip the file at data/arc folder. Then rerun the script.)
2. Rename the folder of each ckpt.pt to "out-arc-baseline", "out-arc-polynomial", "out-arc-periodic" or "out-arc-gaussian" respectively.
3. Evaluate the models for each kernel_config:
   1. Fine-tuning the GPT2 model.
      $ python train.py config/finetune_arc.py --init_from=resume --kernel_config=///
   2. Run evaluation.
      $ python eval_arc.py --kernel_config=///
   3. Evaluate the next model.

Evaluate GPT2 model through EN-FR translation and BLEU

Evaluate the models one by one:

1. Fine-tuning the GPT2 model.
   $ python train.py config/finetune_en-fr.py --init_from=///
2. Run evaluation.
   $ python eval_BLEU.py
3. Evaluate the next model.