New experiment variants #22
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| #!/bin/bash | ||
| #SBATCH --job-name=rl_das_experiment | ||
| #SBATCH --output=logs/experiment_%A_%a.out | ||
| #SBATCH --error=logs/experiment_%A_%a.err | ||
| #SBATCH --ntasks=1 | ||
| #SBATCH --cpus-per-task=1 | ||
| #SBATCH --mem=32G | ||
| #SBATCH --time=48:00:00 | ||
| #SBATCH --partition=plgrid-gpu-a100 | ||
| #SBATCH --array=0-9 # 10 tasks total | ||
|
|
||
| CDB_VAL=${1:-1.5} | ||
|
|
||
| if [ "$#" -gt 0 ]; then | ||
| shift | ||
| fi | ||
|
|
||
| if [ "$#" -eq 0 ]; then | ||
| PORTFOLIO=('JDE21' 'MADDE' 'NL_SHADE_RSP') | ||
|
Collaborator
There was a problem hiding this comment. Shouldn't we use |
||
| else | ||
| PORTFOLIO=("$@") | ||
| fi | ||
| PORTFOLIO_STR=$(IFS="_"; echo "${PORTFOLIO[*]}") | ||
|
|
||
|
|
||
| # CONFIGURATION | ||
| ENV_PATH="$SCRATCH/DynamicAlgorithmSelection/.venv/bin/activate" | ||
| source "$ENV_PATH" | ||
| mkdir -p logs | ||
|
|
||
| # Array of Dimensions | ||
| DIMS=(2 3 5 10) | ||
|
|
||
| # 1. Dimension-specific CV-LOIO (Indices 0-3) | ||
| if [[ $SLURM_ARRAY_TASK_ID -ge 0 && $SLURM_ARRAY_TASK_ID -le 3 ]]; then | ||
|
Collaborator
There was a problem hiding this comment. This is a reward_study, but where is the reward actually changed? I don’t see -O or --reward-option. As it stands, this runner seems to just repeat the experiments from single_algorithm_CDB_study.slurm. |
||
| MODE="CV-LOIO" | ||
| DIM=${DIMS[$SLURM_ARRAY_TASK_ID]} | ||
| echo "Running Mode: $MODE | Dimension: $DIM" | ||
|
|
||
| python3 dynamicalgorithmselection/main.py ${PORTFOLIO_STR}_PG_${MODE}_${CDB_VAL}_${DIM} \ | ||
| -p "${PORTFOLIO[@]}" -r ELA --mode $MODE --dimensionality $DIM \ | ||
| --cdb $CDB_VAL --n_epochs 3 --agent policy-gradient | ||
|
|
||
| # 2. Dimension-specific CV-LOPO (Indices 4-7) | ||
| elif [[ $SLURM_ARRAY_TASK_ID -ge 4 && $SLURM_ARRAY_TASK_ID -le 7 ]]; then | ||
| MODE="CV-LOPO" | ||
| DIM=${DIMS[$((SLURM_ARRAY_TASK_ID - 4))]} | ||
| echo "Running Mode: $MODE | Dimension: $DIM" | ||
|
|
||
| python3 dynamicalgorithmselection/main.py ${PORTFOLIO_STR}_PG_${MODE}_${CDB_VAL}_${DIM} \ | ||
| -p "${PORTFOLIO[@]}" -r ELA --mode $MODE --dimensionality $DIM \ | ||
| --cdb $CDB_VAL --n_epochs 3 --agent policy-gradient | ||
|
|
||
| # 3. Multidimensional CV-LOIO (Index 8) | ||
| elif [[ $SLURM_ARRAY_TASK_ID -eq 8 ]]; then | ||
| MODE="CV-LOIO" | ||
| echo "Running Mode: $MODE | Multidimensional PG" | ||
| python3 dynamicalgorithmselection/main.py ${PORTFOLIO_STR}_PG_MULTIDIMENSIONAL_${MODE}_${CDB_VAL} \ | ||
| -p "${PORTFOLIO[@]}" -r ELA --mode $MODE --cdb $CDB_VAL --agent policy-gradient | ||
|
|
||
| # 4. Multidimensional CV-LOPO (Index 9) | ||
| elif [[ $SLURM_ARRAY_TASK_ID -eq 9 ]]; then | ||
| MODE="CV-LOPO" | ||
| echo "Running Mode: $MODE | Multidimensional PG" | ||
| python3 dynamicalgorithmselection/main.py ${PORTFOLIO_STR}_PG_MULTIDIMENSIONAL_${MODE}_${CDB_VAL} \ | ||
| -p "${PORTFOLIO[@]}" -r ELA --mode $MODE --cdb $CDB_VAL --agent policy-gradient | ||
| fi | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| #!/bin/bash | ||
| #SBATCH --job-name=rl_das_experiment | ||
| #SBATCH --output=logs/experiment_%A_%a.out | ||
| #SBATCH --error=logs/experiment_%A_%a.err | ||
| #SBATCH --ntasks=1 | ||
| #SBATCH --cpus-per-task=1 | ||
| #SBATCH --mem=32G | ||
| #SBATCH --time=48:00:00 | ||
| #SBATCH --partition=plgrid-gpu-a100 | ||
| #SBATCH --array=0-9 # 10 tasks total | ||
|
|
||
| REWARD_OPTION=${1:-1} | ||
|
|
||
| CDB_VAL=1.5 | ||
|
|
||
| PORTFOLIO=('MADDE') | ||
|
|
||
| PORTFOLIO_STR=$(IFS="_"; echo "${PORTFOLIO[*]}") | ||
|
|
||
|
|
||
| # CONFIGURATION | ||
| ENV_PATH="$SCRATCH/DynamicAlgorithmSelection/.venv/bin/activate" | ||
| source "$ENV_PATH" | ||
| mkdir -p logs | ||
|
|
||
| # Array of Dimensions | ||
| DIMS=(2 3 5 10) | ||
|
|
||
| # 1. Dimension-specific CV-LOIO (Indices 0-3) | ||
|
Collaborator
There was a problem hiding this comment. To be honest, for a single-algorithm experiment I wouldn’t run that many experiments. Since we are not comparing it against a baseline (e.g., RL-DAS), there may be less need to evaluate it across many different CV variants. I would probably focus on a single setting, such as Multidimensional CV-LOPO or Multidimensional CV-LOIO.
Collaborator
There was a problem hiding this comment. The key question is what exactly we want to measure for a single algorithm, and whether this will be compared against any baseline or used only to explain the motivation or introduce our own metric. |
||
| if [[ $SLURM_ARRAY_TASK_ID -ge 0 && $SLURM_ARRAY_TASK_ID -le 3 ]]; then | ||
| MODE="CV-LOIO" | ||
| DIM=${DIMS[$SLURM_ARRAY_TASK_ID]} | ||
| echo "Running Mode: $MODE | Dimension: $DIM" | ||
|
|
||
| python3 dynamicalgorithmselection/main.py ${PORTFOLIO_STR}_PG_${MODE}_DIM${DIM} \ | ||
| -p "${PORTFOLIO[@]}" -r ELA --mode $MODE --dimensionality $DIM \ | ||
|
Collaborator
There was a problem hiding this comment. Oh, that’s interesting: I thought you used custom by default. Which representation works better? |
||
| --cdb $CDB_VAL --n_epochs 3 --agent policy-gradient | ||
|
|
||
|
|
||
| # 2. Dimension-specific CV-LOPO (Indices 4-7) | ||
| elif [[ $SLURM_ARRAY_TASK_ID -ge 4 && $SLURM_ARRAY_TASK_ID -le 7 ]]; then | ||
| MODE="CV-LOPO" | ||
| DIM=${DIMS[$((SLURM_ARRAY_TASK_ID - 4))]} | ||
| echo "Running Mode: $MODE | Dimension: $DIM" | ||
|
|
||
| python3 dynamicalgorithmselection/main.py ${PORTFOLIO_STR}_PG_${MODE}_DIM${DIM} \ | ||
| -p "${PORTFOLIO[@]}" -r ELA --mode $MODE --dimensionality $DIM \ | ||
| --cdb $CDB_VAL --n_epochs 3 --agent policy-gradient | ||
|
|
||
| # 3. Multidimensional CV-LOIO (Index 8) | ||
|
Collaborator
There was a problem hiding this comment. Here we also include |
||
| elif [[ $SLURM_ARRAY_TASK_ID -eq 8 ]]; then | ||
| MODE="CV-LOIO" | ||
| echo "Running Mode: $MODE | Multidimensional PG" | ||
| python3 dynamicalgorithmselection/main.py ${PORTFOLIO_STR}_PG_MULTIDIMENSIONAL_${MODE} \ | ||
| -p "${PORTFOLIO[@]}" -r ELA --mode $MODE --cdb $CDB_VAL --agent policy-gradient | ||
|
|
||
| # 4. Multidimensional CV-LOPO (Index 9) | ||
| elif [[ $SLURM_ARRAY_TASK_ID -eq 9 ]]; then | ||
| MODE="CV-LOPO" | ||
| echo "Running Mode: $MODE | Multidimensional PG" | ||
| python3 dynamicalgorithmselection/main.py ${PORTFOLIO_STR}_PG_MULTIDIMENSIONAL_${MODE} \ | ||
| -p "${PORTFOLIO[@]}" -r ELA --mode $MODE --cdb $CDB_VAL --agent policy-gradient | ||
| fi | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just to confirm: are notebooks used for data analysis intentionally not tracked in this repository?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For consistency, it would be preferable to use the format
${PORTFOLIO_STR}_PG_${MODE}_CDB${CDB_VAL}_DIM${DIM}, or alternatively introduce a shared function that adds a common prefix to all parameters so that naming remains consistent across Slurm jobs.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But that’s (^) just nitpicking.