Skip to content

Update profile#20

Open
gmanthey wants to merge 9 commits intomainfrom
19-update-profile
Open

Update profile#20
gmanthey wants to merge 9 commits intomainfrom
19-update-profile

Conversation

@gmanthey
Copy link
Copy Markdown
Owner

Closes #19

Co-authored-by: Copilot <copilot@github.com>
@gmanthey gmanthey linked an issue Apr 29, 2026 that may be closed by this pull request
@gmanthey gmanthey requested a review from matwe340 April 29, 2026 13:01
Copy link
Copy Markdown
Collaborator

@matwe340 matwe340 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noice

@gmanthey
Copy link
Copy Markdown
Owner Author

Currently testing with a different pipeline, still probably needs some adjustments

@gmanthey
Copy link
Copy Markdown
Owner Author

gmanthey commented May 5, 2026

Seems to be working now in https://github.com/gmanthey/vcfstats-snakemake, though the array jobs seem to sometimes have issues determining if jobs are running or not, might be that including this setting requires more babysitting the pipeline. This will probably get clear with some more testing.

Also there was a version mismatch in bioconda where the slurm executor plugin for snakemake was calling for the wrong version in a dependency, causing the array jobs to fail. This is now fixed by pulling all dependencies from pip where possible (which is what is maintained by snakemake developers so should be correct compared to bioconda).

Another issue was that the slurm log files reported by snakemake are not correct anymore. Where before all wildcards were included in the logging path, it is now only the rule name, due to limitations of the slurm array jobs, however, paths are still reported including the wildcard part, so these need to be manually edited when copying out of the snakemake log to investigate.

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reorganizes Snakemake execution profiles by moving Rosa-specific Slurm settings into a dedicated profile/rosa profile, while adjusting the default profile to be less tied to a specific partition, and updates the conda environment setup for Snakemake installation.

Changes:

  • Added a Rosa-specific Snakemake Slurm profile (profile/rosa/config.yaml) and partition capability mapping (profile/rosa/partitions.yaml).
  • Updated the default profile to remove a hard-coded slurm_partition and add Slurm log/account-related flags.
  • Updated environment.yml to install snakemake, the Slurm executor plugin, and matplotlib via pip instead of conda packages.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
profile/rosa/partitions.yaml Adds a Rosa partition capability map intended for partition selection/validation.
profile/rosa/config.yaml Introduces a Rosa-specific Slurm executor configuration pointing at the partition config.
profile/default/config.yaml Makes the default profile less tied to a specific partition; adds Slurm flags.
environment.yml Switches Snakemake + Slurm plugin (and matplotlib) installation from conda to pip.
Comments suppressed due to low confidence (1)

environment.yml:19

  • Switching snakemake, snakemake-executor-plugin-slurm, and matplotlib from conda to pip makes the environment less reproducible (unpinned versions) and increases the risk of dependency conflicts in a conda-managed environment. Consider pinning compatible versions (at least major/minor) and/or preferring the Bioconda/conda-forge packages for these dependencies to keep resolution consistent.
  - pip:
    - snakemake
    - snakemake-executor-plugin-slurm
    - matplotlib

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread profile/default/config.yaml Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update profile

3 participants