Conversation
Co-authored-by: Copilot <copilot@github.com>
|
Currently testing with a different pipeline, still probably needs some adjustments |
…ugin-slurm by using the official distribution method pip instead
|
Seems to be working now in https://github.com/gmanthey/vcfstats-snakemake, though the array jobs seem to sometimes have issues determining if jobs are running or not, might be that including this setting requires more babysitting the pipeline. This will probably get clear with some more testing. Also there was a version mismatch in bioconda where the slurm executor plugin for snakemake was calling for the wrong version in a dependency, causing the array jobs to fail. This is now fixed by pulling all dependencies from pip where possible (which is what is maintained by snakemake developers so should be correct compared to bioconda). Another issue was that the slurm log files reported by snakemake are not correct anymore. Where before all wildcards were included in the logging path, it is now only the rule name, due to limitations of the slurm array jobs, however, paths are still reported including the wildcard part, so these need to be manually edited when copying out of the snakemake log to investigate. |
There was a problem hiding this comment.
Pull request overview
This PR reorganizes Snakemake execution profiles by moving Rosa-specific Slurm settings into a dedicated profile/rosa profile, while adjusting the default profile to be less tied to a specific partition, and updates the conda environment setup for Snakemake installation.
Changes:
- Added a Rosa-specific Snakemake Slurm profile (
profile/rosa/config.yaml) and partition capability mapping (profile/rosa/partitions.yaml). - Updated the default profile to remove a hard-coded
slurm_partitionand add Slurm log/account-related flags. - Updated
environment.ymlto installsnakemake, the Slurm executor plugin, andmatplotlibvia pip instead of conda packages.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| profile/rosa/partitions.yaml | Adds a Rosa partition capability map intended for partition selection/validation. |
| profile/rosa/config.yaml | Introduces a Rosa-specific Slurm executor configuration pointing at the partition config. |
| profile/default/config.yaml | Makes the default profile less tied to a specific partition; adds Slurm flags. |
| environment.yml | Switches Snakemake + Slurm plugin (and matplotlib) installation from conda to pip. |
Comments suppressed due to low confidence (1)
environment.yml:19
- Switching
snakemake,snakemake-executor-plugin-slurm, andmatplotlibfrom conda to pip makes the environment less reproducible (unpinned versions) and increases the risk of dependency conflicts in a conda-managed environment. Consider pinning compatible versions (at least major/minor) and/or preferring the Bioconda/conda-forge packages for these dependencies to keep resolution consistent.
- pip:
- snakemake
- snakemake-executor-plugin-slurm
- matplotlib
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Closes #19