To-Do Overview

[alexisdrakopoulos]

This issue is meant to serve as an overview of things that need to get done, larger parts will require their own open issues.

Code

Core Model:

• Implement 3 Stage Network
• Implement Proxy Anchor Loss from https://github.com/tjddus9597/Proxy-Anchor-CVPR2020
• Implement Model Saves/Checkpoints/Loading
• Add inference code to allow loading a pretrained model -> add images to inference (without HDF5)
• Add Augmentations to core model file rather than importing augmentations (to allow for easier inference using augmentations)
• Ability to switch Trunk to other architectures
• Train ResNet-18, ResNet-50, MobileNetv2 on Food-693
• Allow Classifier to be switched out by reworking API
• Add k-NN GPU/CPU PyTorch code
• Allow caching of images in memory <- added a routine that loads jpegs/pngs without conversion to numpy to keep compressed form

Quality Improvements:

• Update to PyTorch 1.6 and allow for mixed precision training
• Add pretrained weights and allow users to download/use
• Change path handling for model saving/logging procedures
• Look into incorporating better model logging/analysis
• Loading weights without optimizers + loading individual stages
• Ability to save weights without optimizers/individual stages
• Refactor the way in which indices are currently handled
• Add the images -> HDF5 code
• Extend data class to allow combination of multiple datasets or HDF5 files rather than 1 large file
• add mixed precision parallel training functionality

Maintainability/Documentation:

• Refactor/Modularize code
• Finish adding examples/docs to model code
• Write ReadME
• Add some tests for dataloader to avoid the HDF5 corruption
• Save EfficientNet weights and add load procedure so we don't require depencency
• remove pytorch-metric-learning dependency by implementing MS miner and Triplet/MS loss
• remove efficientnet dependency

Minor Bugs/Issues:

• During model training a line prints an extra loading bar <- This is due to sampler printing after calling the tqdm
• Model is not asynchronously reading data due to pulling data from a global cached dictionary, not easy to fix

Research

Pretraining

• Add loss information of unseen classes

Embedding Space Work

• Embedding Space collision detection
• Embedding Space quality evaluation (Perhaps some clustering metrics etc...)
• Pretraining LMNN and using learned matrix as initialization

Few-Shot Fine Tuning

• Effect of better Augmentation procedures
• Hyperparameter Search on losses + augmentation + LRs/Regularization
• Incorporating kNN as classifier loss (end-to-end approach essentially with kNN rather than MLP classifier)

k-NN/LMNN

• LMNN on GPU (Jax implementation was attempted but is slower)
• Reducing variance in initial predictions in k-NN through other normalization/robustness schemes