Pulsar job not running - empty tool script?

[OliverWoolland]

Hello everyone!

I'm having trying to set up Pulsar (0.15.7) to serve a Galaxy server (24.2.2) but I've been running into a bit of trouble!

I think I've narrowed it down to empty tool_script.shs being created on the Pulsar end? In configs and configs.orig the tool_script looks ok (has size, correct contents), but in the root of the job dir the file exists but has no contents and is 0 bytes. tool_script.sh looks ok on the Galaxy side.

To investigate a bit I set up an inotify watch:

$ inotifywait -m -r /pulsar-store/files/staging/ -e create -e modify -e delete -e move
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
/pulsar-store/files/staging/ CREATE,ISDIR 5
/pulsar-store/files/staging/5/ CREATE tool\_id
/pulsar-store/files/staging/5/ MODIFY tool\_id
/pulsar-store/files/staging/5/ CREATE tool\_version
/pulsar-store/files/staging/5/ MODIFY tool\_version
/pulsar-store/files/staging/5/ CREATE launch\_config
/pulsar-store/files/staging/5/ MODIFY launch\_config
/pulsar-store/files/staging/5/ CREATE tool\_script.sh
/pulsar-store/files/staging/5/ CREATE submitted
/pulsar-store/files/staging/5/ MODIFY submitted
/pulsar-store/files/staging/5/ CREATE command.sh
/pulsar-store/files/staging/5/ MODIFY command.sh
/pulsar-store/files/staging/5/ CREATE command\_line
/pulsar-store/files/staging/5/ MODIFY command\_line
/pulsar-store/files/staging/5/ CREATE preprocessed
/pulsar-store/files/staging/5/ DELETE command\_line
/pulsar-store/files/staging/5/ MODIFY preprocessed
/pulsar-store/files/staging/5/ CREATE pid
/pulsar-store/files/staging/5/ MODIFY pid
/pulsar-store/files/staging/5/ CREATE,ISDIR tmp
/pulsar-store/files/staging/5/ CREATE,ISDIR home
/pulsar-store/files/staging/5/ CREATE,ISDIR working.orig
/pulsar-store/files/staging/5/ CREATE,ISDIR outputs.orig
/pulsar-store/files/staging/5/ CREATE,ISDIR configs.orig
/pulsar-store/files/staging/5/configs.orig/ CREATE tool\_script.sh
/pulsar-store/files/staging/5/configs.orig/ MODIFY tool\_script.sh
/pulsar-store/files/staging/5/ CREATE galaxy\_5.ec
/pulsar-store/files/staging/5/ MODIFY galaxy\_5.ec
/pulsar-store/files/staging/5/ CREATE return\_code
/pulsar-store/files/staging/5/ MODIFY return\_code
/pulsar-store/files/staging/5/ DELETE submitted
/pulsar-store/files/staging/5/ DELETE pid
/pulsar-store/files/staging/5/ CREATE final\_status
/pulsar-store/files/staging/5/ MODIFY final\_status
/pulsar-store/files/staging/5/ CREATE postprocessed

It seems to me that tool_script.sh is the only file which is created but not then modified? Stat seems to confirm this

File: pulsar-store/files/staging/5/tool\_script.sh
Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 253,1   Inode: 14599278    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2025-04-02 07:09:37.869721411 +0000
Modify: 2025-04-02 07:09:37.849281951 +0000
Change: 2025-04-02 07:09:37.849281951 +0000
Birth: 2025-04-02 07:09:37.849281951 +0000

Has anyone seen this before? Would anyone have any suggestions for more debugging or for fixes?

I tried modifying PREPARE_DIRS_TEMPLATE in directory.py to add a copy from configs/tool_script.sh to tool_script.sh - this allowed the job to run. However, then the outputs were not being picked up! I'm sure there must be some config or other I am missing.

My job conf contains:

[...]
    <plugin id="pulsar_mq" type="runner" load="galaxy.jobs.runners.pulsar:PulsarMQJobRunner">
      <param id="galaxy_url">http://{galaxy-url}</param>
      <param id="amqp_url">amqp://{user:pass@rabbitmq-url}:5672//pulsar/pulsar_au</param>
[...]
    <destination id="pulsar_mq_dest" runner="pulsar_mq" >
      <param id="jobs_directory">/files/staging</param>
      <param id="default_file_action">remote_transfer</param>
    </destination>
[...]

Many thanks! :)