Modeling hyperparameters config file

[EzicStar]

A model_params.yaml file would let users configure hyperparameters without the need of modifying the code. It is better to have this parameters in a different file which is referenced in config.yaml so it doesn't become too long and unreadable.

[georg-wolflein]

I actually think it would be better to keep all configs in one file, so there is a single source of truth (though I am perhaps open to being convinced otherwise).
If we keep everything in one file, there could be a dedicated section for the hyperparameters so it’s still readable…
What do you think? @FWao @s1787956

[EzicStar]

Good point. My motivation to do it in a separate file is that config.yaml file already has ~270 lines. Here is the train_params.yaml file that I thought about:

# Advanced parameters for training.
# Default values work usually well. 
# Its path can be specified in the main config.yaml.
max_epochs: 64
patience: 16
batch_size: 64
# Only for tile-level training. Reducing its amount could affect
# model performance. Reduces memory consumption. Default value works
# fine for most cases.
bag_size: 512
# Optional parameters
#num_workers: 16 # Default chosen by cpu cores

# Model to use. The available models are "vit" or "mlp". 
# Defaults based on feature type.
#model_name: "vit"

model_params:
  # Tile-level training models:
  vit: # Vision Transformer
    dim_model: 512
    dim_feedforward: 512
    n_heads: 8
    n_layers: 2
    dropout: 0.25
    # Experimental feature: Use ALiBi positional embedding
    use_alibi: false

  # Patient-level training models:
  mlp: # Multilayer Perceptron
    dim_hidden: 512
    num_layers: 2
    dropout: 0.25

The user can use the default table or specify its own params file inside config.yaml. For only 30 more lines I think that is not worth it to create a new file but if we keep adding models it will get bigger. What do you think? We can also still keep it in config file and move it later if it gets bigger.