transfer testing method #3
Description
We need core algorithm implemented, for testing purposes we don't need book dataset nor the dataloader mentioned in other issue, we could use imagenet dataset with dataloader provided from torchvision lib.
Transfer testing should work in this manner:
- Train model on sub sample of dataset (for example dogs from imagenet)
- Compute test error
- Add other class to training dataset (for example cats)
- Compute test error for new class only
- Hypothesis: test error on dataset with additional class should be similar with previous one without additional data
our goal is also to find proportion of new data to old data that is needed to fulfill our hypothesis (treat is as hyperparameter, 10%, 20% etc)