exit 1: /bin/sh: pbs-submit.py: command not found

[jperales]

Hi,
I am trying to use this profile in our HPC facility (PBS-torque schedule system).
However it crashes when I use snakemake --config ~/.config/snakemake/pbs-torque/config.yaml, with the following error:

/bin/sh: pbs-submit.py: command not found
Error submitting jobscript (exit code 127):

I don't understand the problem. I have checked that shell and python are installed in our HPC. I have also tried to edit the first line of pbs-submit.py and pbs-jobscript.sh to point out to alternative different paths for the interpreters shell and python3; but it still does not work.

I have checked that the rule for the job run well in an interactive job session in a node of the cluster. So it is just the profile what it is not working.
Thanks in advanced for your help!
Javier

---

The log and jobscript for one of the jobs is the following:

rule seurat:
    input: data/00_scRNAseq/GSM4191942/matrix.mtx.gz, data/00_scRNAseq/GSM4191942/features.tsv.gz, data/00_scRNAseq/GSM4191942/barcodes.tsv.gz, index/samples/GSM4191942.yaml
    output: data/01_Seurat/GSM4191942_S.rds, data/01_Seurat/GSM4191942_nPCs.txt, data/01_Seurat/GSM4191942_nCells.txt
    jobid: 17
    wildcards: id=GSM4191942

Jobscript:
#!/bin/sh
# properties = {"type": "single", "rule": "seurat", "local": false, "input": ["data/00_scRNAseq/GSM4191942/matrix.mtx.gz", "data/00_scRNAseq/GSM4191942/features.tsv.gz", "data/00_scRNAseq/GSM4191942/barcodes.tsv.gz", "index/samples/GSM4191942.yaml"], "output": ["data/01_Seurat/GSM4191942_S.rds", "data/01_Seurat/GSM4191942_nPCs.txt", "data/01_Seurat/GSM4191942_nCells.txt"], "wildcards": {"id": "GSM4191942"}, "params": {}, "log": [], "threads": 1, "resources": {}, "jobid": 17, "cluster": {}}
 cd /beegfs/work/CKD && \
/beegfs/work/CKD/envs/01_Seurat/bin/python3.7 \
-m snakemake data/01_Seurat/GSM4191942_S.rds --snakefile /beegfs/work/CKD/Snakefile \
--force -j --keep-target-files --keep-remote \
--wait-for-files /beegfs/work/CKD/.snakemake/tmp.jzedn2wh data/00_scRNAseq/GSM4191942/matrix.mtx.gz data/00_scRNAseq/GSM4191942/features.tsv.gz data/00_scRNAseq/GSM4191942/barcodes.tsv.gz index/samples/GSM4191942.yaml --latency-wait 5 \
 --attempt 1 --force-use-threads \
--wrapper-prefix https://github.com/snakemake/snakemake-wrappers/raw/ \
   --allowed-rules seurat --nocolor --notemp --no-hooks --nolock \
--mode 2  && exit 0 || exit 1

[willprice]

This appears to be an issue with the way the config.yaml is written. The cluster specification is a command line invocation rather than the name of file. If you change cluster from

cluster: "pbs-submit.py --depend \"{dependencies}\""

to

cluster: "pbs-submit.py"

You should find that it now submits jobs (you might also have to make pbs-submit.py executable (chmod +x ~/.config/snakemake/myprofile/pbs-submit.py)

Once I've figured how to add dependencies back in to the script invocation I'll update with another comment

[zihhuafang]

Hi @willprice ,

I hope you could also help me to understand how to use this profile on cluster.

I am not sure where I should put the parameters for cluster config.
In my profile/config.yaml, I tried to add something like cluster: "pbs-submit.py -q tiny" and chmod +x pbs-submit.py

When I ran snakemake --profile fangz_proflie -rp
Then I got the error of /bin/sh: pbs-submit.py: command not found
Below is the job script printed by snakemake:

Jobscript:
#!/bin/sh
# properties = {"type": "single", "rule": "fastp_qc", "local": false, "input": ["/beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fastq/PPMI51971_R1.fastq.gz", "/beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fastq/PPMI51971_R2.fastq.gz"], "output": ["qc/PPMI51971_R1.fastq.gz", "qc/PPMI51971_R2.fastq.gz", "qc/PPMI51971_fastp.json"], "wildcards": {"sample": "PPMI51971"}, "params": {"qualified_quality_phred": 15, "unqualified_percent_limit": 40, "nova_seq": "-g"}, "log": [], "threads": 10, "resources": {"mem": "1gb", "walltime": "04:00:00", "node": 1}, "jobid": 10, "cluster": {}}
 cd /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call && \
PATH='/home/tu/tu_tu/tu_knkph01/miniconda3/envs/snakemake/bin':$PATH /home/tu/tu_tu/tu_knkph01/miniconda3/envs/snakemake/bin/python3.9 \
-m snakemake qc/PPMI51971_fastp.json --snakefile /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/Snakefile \
--force -j --keep-target-files --keep-remote --max-inventory-time 0 \
--wait-for-files /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/tmp.tybds2yj /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fastq/PPMI51971_R1.fastq.gz /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fastq/PPMI51971_R2.fastq.gz /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/conda/d71cd897 --latency-wait 60 \
 --attempt 1 --force-use-threads --scheduler ilp \
--wrapper-prefix https://github.com/snakemake/snakemake-wrappers/raw/ \
  -p --allowed-rules fastp_qc --nocolor --notemp --no-hooks --nolock \
--mode 2  --use-conda  --use-singularity  && touch /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/tmp.tybds2yj/10.jobfinished || (touch /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/tmp.tybds2yj/10.jobfailed; exit 1)

/bin/sh: pbs-submit.py: command not found
Error submitting jobscript (exit code 127):

I think it did not recognize the cluster configs in profile/config.yaml (ex: -q tiny).
Did I add it at the wrong place?

Thanks in advance.
Best,
Zih-Hua

[willprice]

Hi Zih-Hua,
I'm not the author, but have tried using these scripts.
Try changing your cluster line to cluster: "pbs-submit.py"

If that doesn't work tell me

• Where is your profile located
• Where pbs-submit.py is located
• Where your snakefile is located
  (relative locations are fine)

I

[zihhuafang]

Hi @willprice,

Thanks a lot for your reply. I understand you're not the author, but I would appreciate that you could share the experience with me.

So I changed the cluster line in profile/config.yaml to cluster: "pbs-submit.py".
Then I got the error qsub: submit error (No default queue specified MSG=requested queue not found).

Jobscript:
#!/bin/sh
# properties = {"type": "single", "rule": "fastp_qc", "local": false, "input": ["/beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fastq/PPMI51971_R1.fastq.gz", "/beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fastq/PPMI51971_R2.fastq.gz"], "output": ["qc/PPMI51971_R1.fastq.gz", "qc/PPMI51971_R2.fastq.gz", "qc/PPMI51971_fastp.json"], "wildcards": {"sample": "PPMI51971"}, "params": {"qualified_quality_phred": 15, "unqualified_percent_limit": 40, "nova_seq": "-g"}, "log": ["logs/fastp_PPMI51971.log"], "threads": 10, "resources": {"mem": "1gb", "walltime": "04:00:00", "node": 1}, "jobid": 10, "cluster": {}}
 cd /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call && \
PATH='/home/tu/tu_tu/tu_knkph01/miniconda3/envs/snakemake/bin':$PATH /home/tu/tu_tu/tu_knkph01/miniconda3/envs/snakemake/bin/python3.9 \
-m snakemake qc/PPMI51971_fastp.json --snakefile /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/Snakefile \
--force -j --keep-target-files --keep-remote --max-inventory-time 0 \
--wait-for-files /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/tmp.o3d0fpyf /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fastq/PPMI51971_R1.fastq.gz /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fastq/PPMI51971_R2.fastq.gz /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/conda/d71cd897 --latency-wait 60 \
 --attempt 1 --force-use-threads --scheduler ilp \
--wrapper-prefix https://github.com/snakemake/snakemake-wrappers/raw/ \
  -p --allowed-rules fastp_qc --nocolor --notemp --no-hooks --nolock \
--mode 2  --use-conda  --use-singularity  && touch /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/tmp.o3d0fpyf/10.jobfinished || (touch /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/tmp.o3d0fpyf/10.jobfailed; exit 1)

qsub: submit error (No default queue specified MSG=requested queue not found)
Traceback (most recent call last):
  File "/beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fangz_profile/pbs-submit.py", line 155, in <module>
    raise e
  File "/beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fangz_profile/pbs-submit.py", line 153, in <module>
    res = subprocess.run(cmd, check=True, shell=True, stdout=subprocess.PIPE)
  File "/home/tu/tu_tu/tu_knkph01/miniconda3/envs/snakemake/lib/python3.9/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'qsub  -l "nodes=1:ppn=10,mem=1gb,walltime=04:00:00" /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/tmp.o3d0fpyf/snakejob.fastp_qc.10.sh' returned non-zero exit status 191.
Error submitting jobscript (exit code 1):

I then created a cluster_config.yml in the profile directory.

__default__:
    -j: oe
    -q: short

I ran snakemake --profile /path/to/fangz_profile --cluster-config /path/to/fangz_profile/cluster_config.yml -rp.
Then the parameters were correctly parsed "cluster": {"-j": "oe", "-q": "short"}, but I still got the error of qsub: submit error (No default queue specified MSG=requested queue not found). cmd did not print my cluster config.

Jobscript:
#!/bin/sh
# properties = {"type": "single", "rule": "fastp_qc", "local": false, "input": ["/beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fastq/PPMI51971_R1.fastq.gz", "/beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fastq/PPMI51971_R2.fastq.gz"], "output": ["qc/PPMI51971_R1.fastq.gz", "qc/PPMI51971_R2.fastq.gz", "qc/PPMI51971_fastp.json"], "wildcards": {"sample": "PPMI51971"}, "params": {"qualified_quality_phred": 15, "unqualified_percent_limit": 40, "nova_seq": "-g"}, "log": ["logs/fastp_PPMI51971.log"], "threads": 10, "resources": {"mem": "1gb", "walltime": "04:00:00", "node": 1}, "jobid": 10, "cluster": {"-j": "oe", "-q": "short"}}
 cd /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call && \
PATH='/home/tu/tu_tu/tu_knkph01/miniconda3/envs/snakemake/bin':$PATH /home/tu/tu_tu/tu_knkph01/miniconda3/envs/snakemake/bin/python3.9 \
-m snakemake qc/PPMI51971_fastp.json --snakefile /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/Snakefile \
--force -j --keep-target-files --keep-remote --max-inventory-time 0 \
--wait-for-files /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/tmp.31x25uer /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fastq/PPMI51971_R1.fastq.gz /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fastq/PPMI51971_R2.fastq.gz /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/conda/d71cd897 --latency-wait 60 \
 --attempt 1 --force-use-threads --scheduler ilp \
--wrapper-prefix https://github.com/snakemake/snakemake-wrappers/raw/ \
  -p --allowed-rules fastp_qc --nocolor --notemp --no-hooks --nolock \
--mode 2  --use-conda  --use-singularity  && touch /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/tmp.31x25uer/10.jobfinished || (touch /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/tmp.31x25uer/10.jobfailed; exit 1)

qsub: submit error (No default queue specified MSG=requested queue not found)
Traceback (most recent call last):
  File "/beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fangz_profile/pbs-submit.py", line 155, in <module>
    raise e
  File "/beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fangz_profile/pbs-submit.py", line 153, in <module>
    res = subprocess.run(cmd, check=True, shell=True, stdout=subprocess.PIPE)
  File "/home/tu/tu_tu/tu_knkph01/miniconda3/envs/snakemake/lib/python3.9/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'qsub  -l "nodes=1:ppn=10,mem=1gb,walltime=04:00:00" /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/tmp.31x25uer/snakejob.fastp_qc.10.sh' returned non-zero exit status 191.
Error submitting jobscript (exit code 1):

Below is my snakemake directory structure where I put my profile folder (fangz_profile):

.
├── config.yaml
├── envs
│   └── short_read_mapping.yml
├── fangz_profile
│   ├── cluster_config.yml
│   ├── config.yaml
│   ├── pbs-jobscript.sh
│   ├── pbs-status.py
│   └── pbs-submit.py
├── fastq
│   ├── PPMI54144_R1.fastq.gz
│   ├── PPMI54144_R2.fastq.gz
│   ├── PPMI56954_R1.fastq.gz
│   └── PPMI56954_R2.fastq.gz
├── hg38
│   ├── 1000G_omni2.5.hg38.vcf.gz
│   └──1000G_omni2.5.hg38.vcf.gz.tbi
├── rules
│   ├── align.smk
│   ├── haplotype_caller.smk
│   └── qc_split.smk
└── Snakefile

Appreciate your insights!

[willprice]

The first error is a good one, it tells us that snakemake can invoke pbs-submit.py correctly.
I'm not familiar with profiles in snakemake, but looking at the slurm profile I see they set some options in
https://github.com/Snakemake-Profiles/slurm/blob/master/%7B%7Bcookiecutter.profile_name%7D%7D/config.yaml
but it's not immediately obvious how their job submit scripts use them. Perhaps you can delve into those to see how it works and apply your findings to your scenario.
Sorry not to be of more help.
I just was lazy and set my defaults within the pbs-submit.py script which got the job done!

[zihhuafang]

I understand. Maybe I should just type all my qsub parameters coupled with the --cluster-config without using profile like in the old version of snakemake.
Unfortunately, I have a lot of parameters that need to be decided on the per-rule basis.
Thanks anyway! Appreciate your time!

[willprice]

No worries, in that case I'd try and reverse engineer one of the better maintained configs in this organisation (e.g. the slurm one) to see how they get the parameters from the profile. For overrides from your rule I think you can just use snakemake.utils.read_job_properties

[gibsramen]

Popping in to say that the solution outlined by @willprice worked for me when I also made the pbs-status.py executable. I am also trying to figure out how to best provide rule-specific cluster parameters.

[gibsramen]

@zihhuafang I found this repository that I think has a much more intuitive profile implementation.

I basically copied it and modified the qsub command as necessary and added a cluster.yaml file for my pipeline.

#  ~/.snakemake/pbs-torque/config.yaml
cluster-config: "config/cluster.yaml"
cluster: "qsub -N {cluster.name} -l nodes={cluster.nodes} -l mem={cluster.mem} -l walltime={cluster.walltime} -j oe -o {cluster.out}"
jobs: 10
latency-wait: 60
verbose: true

# ~/projects/myproject/config/cluster.yaml
__default__:
    name: default
    walltime: 4:00:00
    mem: 8gb
    nodes: 1:ppn=8
    out: pbs_out/

Then I run my command as snakemake <target> --profile pbs-torque.

[zihhuafang]

@gibsramen Thanks for the link. I actually found this repository as well and am currently using it as the intermediate solution for me. This was how I used to run my snakemake (the old version). I will probably find time to make pbs profile works as the slurm one as I do prefer to specify resources in the rules.