"engines" error message

[jesus-mavarez]

Hello
I've been using ipyrad on a cluster for a while with great results, but a couple of days ago it just stopped working. I receive this message now:

micromamba activate ipyrad
ipyrad -p params-Chor_5_test.txt -s 1 -c 16 -f

ipyrad [v.0.9.105]
Interactive assembly and analysis of RAD-seq data

Parallel connection |

Step 1: Demultiplexing fastq data to Samples

Encountered an Error.
Message: This operation requires engines. Try client.wait_for_engines(n) to wait for engines to register.

I tried:
micromamba activate ipyrad
ipcluster start --n 16 --daemonize
sleep 60
ipyrad -p params-Chor_5.txt -s 1 -c 16 -f --ipcluster

which seems to work until step "sleep 60" but not afterwards.

Do you know how to solve this?

Thanks in advance
Jesús

[isaacovercast]

What you have tried should work. How do you know the sleep 60 works but then the ipyrad call doesn't? What do you see in the output log/error log?

[jesus-mavarez]

This is the script used:

---

#!/bin/bash

Launch a job (Chor_6_s1) under project (criquets) using 16 cores

OAR instructions

#OAR -n Chor_6_s1
#OAR --project criquets
#OAR -l nodes=1/core=16,walltime=48:00:00
#OAR -O log_MyJob.%jobid%.stdout
#OAR -E log_MyJob.%jobid%.stderr

define some bash options

set -e ## exit the script as soon as a function return an error

Launch ipyrad and ipcluster

micromamba activate ipyrad_mamba
ipcluster start --n 16 --daemonize
sleep 60

run ipyrad script

ipyrad -p params-Chor_6.txt -s 1 -c 16 -f --ipcluster

stop ipcluster

ipcluster stop

---

[Here is what I get]

2025-11-14 12:28:32.421 [IPClusterStart] Starting ipcluster with [daemonize=True]
2025-11-14 12:28:41.903 [IPController] Hub listening on tcp://127.0.0.1:36489 for registration.
2025-11-14 12:28:41.906 [IPController] Hub using DB backend: DictDB
2025-11-14 12:28:42.159 [IPController] hub::created hub
2025-11-14 12:28:42.159 [IPController] writing connection info to /home/mavarezj/.ipython/profile_default/security/ipcontroller-client.json
2025-11-14 12:28:42.162 [IPController] writing connection info to /home/mavarezj/.ipython/profile_default/security/ipcontroller-engine.json
2025-11-14 12:28:42.165 [IPController] task::using Python leastload Task scheduler
2025-11-14 12:28:42.256 [IPClusterStart] Starting 16 engines with <class 'ipyparallel.cluster.launcher.LocalEngineSetLauncher'>
2025-11-14 12:28:42.252 [IPController] Heartmonitor beating every 3000ms
2025-11-14 12:28:42.253 [task] Task scheduler started [leastload]
2025-11-14 12:28:42.255 [broadcast-0] BroadcastScheduler 0 started
2025-11-14 12:28:42.255 [IPController] client::client b'\x00k\x8bEg' requested 'connection_request'
2025-11-14 12:28:42.256 [IPController] client::client [b'\x00k\x8bEg'] connected
2025-11-14 12:28:42.257 [broadcast-00] BroadcastScheduler 00 started
2025-11-14 12:28:42.259 [broadcast-01] BroadcastScheduler 01 started
2025-11-14 12:28:42.264 [IPController] heartbeat::waiting for subscription
2025-11-14 12:28:42.265 [IPController] heartbeat::subscription started
Leaving cluster running: /home/mavarezj/.ipython/profile_default/security/cluster-.json

[ipyrad step 1 should show results here but nothing happens between time 12:28 and 12:43. An then the lines below are printed]

2025-11-14 12:43:16.418 [IPClusterStop] Stopping cluster
2025-11-14 12:43:16.434 [IPClusterStop] Stopping controller
2025-11-14 12:43:16.596 [IPClusterStop] Stopping engine(s): 1763119714

[isaacovercast]

Try removing the -c 16 from the call to ipyrad, it is redundant in the case of using the --ipcluster flag, which attaches to an already running cluster. I think that will fix it, but let me know if not

[jesus-mavarez]

That option (without the -c 16) didn't work on an "interactive" mode.

ipyrad -p params-Chor_6.txt -s 1 -f --ipcluster

/home/mavarezj/micromamba/envs/ipyrad_mamba2/lib/python3.12/site-packages/ipyrad/main.py:8: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
from pkg_resources import get_distribution

---

ipyrad [v.0.9.105]
Interactive assembly and analysis of RAD-seq data

Waiting for connection file: /.ipython/profile_default/security/ipcontroller-client.json
Traceback (most recent call last):
File "/home/mavarezj/micromamba/envs/ipyrad_mamba2/bin/ipyrad", line 10, in
sys.exit(main())
^^^^^^
File "/home/mavarezj/micromamba/envs/ipyrad_mamba2/lib/python3.12/site-packages/ipyrad/main.py", line 605, in main
CLI()
File "/home/mavarezj/micromamba/envs/ipyrad_mamba2/lib/python3.12/site-packages/ipyrad/main.py", line 75, in init
self.run()
File "/home/mavarezj/micromamba/envs/ipyrad_mamba2/lib/python3.12/site-packages/ipyrad/main.py", line 496, in run
ipyclient = ipp.Client(profile=self.args.ipcluster)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mavarezj/micromamba/envs/ipyrad_mamba2/lib/python3.12/site-packages/ipyparallel/client/client.py", line 456, in init
raise OSError(msg)
OSError: Connection file '/.ipython/profile_default/security/ipcontroller-client.json' not found.
You have attempted to connect to an IPython Cluster but no Controller could be found.
Please double-check your configuration and ensure that a cluster is running.

[isaacovercast]

If you are trying to run in interactive mode you still need to launch the ipcluster and 'sleep' to wait for it to start:

ipcluster start --n 16 --daemonize
sleep 60

Then you can run the ipyrad call ipyrad -p params-Chor_6.txt -s 1 -f --ipcluster

[jesus-mavarez]

Sorry. I forgot to copy that part on my message. Yes, I did so:

(ipyrad_mamba2) mavarezj@dahu171:/bettik/PROJECTS/pr-criquets/mavarezj/Chorthippus_2$ ipcluster start --n 16 --daemonize

2025-11-14 18:42:23.032 [IPClusterStart] Starting ipcluster with [daemonize=True]
2025-11-14 18:42:28.242 [IPController] Hub listening on tcp://127.0.0.1:48715 for registration.
2025-11-14 18:42:28.244 [IPController] Hub using DB backend: DictDB
2025-11-14 18:42:28.497 [IPController] hub::created hub
2025-11-14 18:42:28.497 [IPController] writing connection info to /home/mavarezj/.ipython/profile_default/security/ipcontroller-client.json
2025-11-14 18:42:28.531 [IPController] writing connection info to /home/mavarezj/.ipython/profile_default/security/ipcontroller-engine.json
2025-11-14 18:42:28.593 [IPClusterStart] Starting 16 engines with <class 'ipyparallel.cluster.launcher.LocalEngineSetLauncher'>
2025-11-14 18:42:28.555 [IPController] task::using Python leastload Task scheduler
2025-11-14 18:42:28.585 [IPController] Heartmonitor beating every 3000ms
2025-11-14 18:42:28.586 [task] Task scheduler started [leastload]
2025-11-14 18:42:28.587 [broadcast-0] BroadcastScheduler 0 started
2025-11-14 18:42:28.588 [IPController] client::client b'\x00k\x8bEg' requested 'connection_request'
2025-11-14 18:42:28.588 [IPController] client::client [b'\x00k\x8bEg'] connected
2025-11-14 18:42:28.588 [broadcast-00] BroadcastScheduler 00 started
2025-11-14 18:42:28.591 [broadcast-01] BroadcastScheduler 01 started
2025-11-14 18:42:28.596 [IPController] heartbeat::waiting for subscription
2025-11-14 18:42:28.596 [IPController] heartbeat::subscription started
Leaving cluster running: /home/mavarezj/.ipython/profile_default/security/cluster-.json

(ipyrad_mamba2) mavarezj@dahu171:/bettik/PROJECTS/pr-criquets/mavarezj/Chorthippus_2$ sleep 60

(ipyrad_mamba2) mavarezj@dahu171:/bettik/PROJECTS/pr-criquets/mavarezj/Chorthippus_2$ ipyrad -p params-Chor_6.txt -s 1 -f --ipcluster

[isaacovercast]

And then what does it do? It just sits there and does nothing?

[jesus-mavarez]

In the interactive job with the "-c 16 --ipcluster" option I receive this message:

Encountered an Error.
Message: This operation requires engines. Try client.wait_for_engines(n) to wait for engines to register.

Interactive job without the "-c 16" option I receive the error in the previous message.

In the scripted jobs it just sits there and does nothing

[isaacovercast]

It sounds like your HPC is killing the ipcluster instance for some reason. Can you try looking to see if there are lingering ipcluster lock files: ipcluster list

You can also do ipcluster clean to clean up any lingering config files.

Then try this:

ipcluster start --n 16 --daemonize
sleep 60
ipcluster list

Which should show you this:

PROFILE          CLUSTER ID                       RUNNING ENGINES LAUNCHER
default          ''                               True          4 Local

If running engines is FALSE that means the cluster is dead. If it is TRUE you can go ahead and try running ipyrad again.

[holtbh]

I was having a potentially similar issue. I was actually about to post it but then i figured it out:

I started my cluster and ran the sleep 60 per this thread

ipcluster start -n=8 --cluster-id="test" --daemonize

and then started Jupyter (python v 3.12.8, ipython v 9.9.0, ipyrad v 0.9.107) and connected to the cluster

ipyclient = ipp.Client(cluster_id="test")
ipyclient.ids

ipyclient[:].apply_sync(lambda : 'Hello, World')

['Hello, World',
'Hello, World',
'Hello, World',
'Hello, World',
'Hello, World',
'Hello, World',
'Hello, World',
'Hello, World']

When trying to complete steps 1 and 2, I get the following error:

Traceback (most recent call last):
File "/opt/anaconda3/envs/ipyrad/lib/python3.12/site-packages/ipyrad/core/Parallel.py", line 172, in wait_for_connection
ipyclient = ipp.Client(**args)
^^^^^^^^^^^^^^^^^^
File "/opt/anaconda3/envs/ipyrad/lib/python3.12/site-packages/ipyparallel/client/client.py", line 456, in init
raise OSError(msg)
OSError: Connection file '~/.ipython/profile_default/security/ipcontroller-client.json' not found.
You have attempted to connect to an IPython Cluster but no Controller could be found.
Please double-check your configuration and ensure that a cluster is running.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/anaconda3/envs/ipyrad/lib/python3.12/site-packages/ipyrad/core/Parallel.py", line 302, in wrap_run
self.ipyclient = self.wait_for_connection()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/anaconda3/envs/ipyrad/lib/python3.12/site-packages/ipyrad/core/Parallel.py", line 217, in wait_for_connection
raise IPyradError(
ipyrad.assemble.utils.IPyradError:
ipcluster not found, use 'auto=True' or see docs.

running ipcluster list indicates my cluster is running:
PROFILE CLUSTER ID RUNNING ENGINES LAUNCHER
default test True 8 Local

When I navigate to my .ipython/profile_default/security folder, I have an ipcontroller-client.json file but it is named after my cluster id ('ipcontroller-test-client.json'). When I renamed the file (cp ipcontroller-test-client.json ipcontroller-client.json), the issue was resolved.

Hopefully this fixes it for others!

[isaacovercast]

@holtbh Thanks for sharing your troubleshooting info and glad you got the problem solved!