omero.LockTimeout exception on "omero admin email"

[ehrenfeu]

Hi all,

while sending emails to users using the omero admin email command is working fine in general for us, we seem to be having just too many users on our main OMERO instance.

When running e.g.

omero admin email --everyone "OMERO maintenance completed" < mailbody.txt

the process is taking "a bit" longer and at some point (25s) crashes with an omero.LockTimeout exception like this:

Using session for ehrenfeu@localhost:4064. Idle timeout: 2000 min. Current group: IMCF
Traceback (most recent call last):
  File "/opt/omero/server/venv/bin/omero", line 8, in <module>
    sys.exit(main())
  File "/opt/omero/server/venv/lib/python3.10/site-packages/omero/main.py", line 125, in main
    rv = omero.cli.argv()
  File "/opt/omero/server/venv/lib/python3.10/site-packages/omero/cli.py", line 1771, in argv
    cli.invoke(args[1:])
  File "/opt/omero/server/venv/lib/python3.10/site-packages/omero/cli.py", line 1208, in invoke
    stop = self.onecmd(line, previous_args)
  File "/opt/omero/server/venv/lib/python3.10/site-packages/omero/cli.py", line 1285, in onecmd
    self.execute(line, previous_args)
  File "/opt/omero/server/venv/lib/python3.10/site-packages/omero/cli.py", line 1367, in execute
    args.func(args)
  File "<string>", line 1629, in email
  File "/opt/omero/server/venv/lib/python3.10/site-packages/omero/clients.py", line 966, in submit
    return self.waitOnCmd(
  File "/opt/omero/server/venv/lib/python3.10/site-packages/omero/clients.py", line 989, in waitOnCmd
    callback.loop(loops, ms)  # Throw LockTimeout
  File "/opt/omero/server/venv/lib/python3.10/site-packages/omero/callbacks.py", line 258, in loop
    raise omero.LockTimeout(
omero.LockTimeout: exception ::omero::LockTimeout
{
    serverStackTrace = None
    serverExceptionClass = None
    message = Command unfinished after 25.0 seconds
    backOff = 5000
    seconds = 25
}

Unfortunately I failed in finding where to configure this timeout.

➡️ Is there a log / database entries that would tell me who received the email?

All OMERO components are on their latest version, running on Ubuntu 22.04.

Cheers,
Niko

[ehrenfeu]

Forgot to mention,we're currently having about 800 active (=enabled) user accounts on this instance...

[will-moore]

@ehrenfeu I'm afraid I don't know the answer to your specific questions, but in general we find it preferable to use the webclient admin tool to send e-mails to users as this doesn't suffer these timeout issues.

Looking at the code at https://github.com/ome/omero-py/blob/8d8108de02104c654741d65f0ecbbf11c85e329d/src/omero/plugins/admin.py#L1599 it looks like you can use a wait argument which is explained better in the help, but I don't know if that is directly related to the LockTimeout you're seeing.

$ omero admin email -h
...
 --wait WAIT              Number of seconds to wait for the processing to complete (Indefinite < 0; No wait=0).

[will-moore]

Petr tells me he's used the webclient to e-mail 3000 users at a time and guesses that 800 users would take 40 minutes.

[ehrenfeu]

Thanks a lot @will-moore, that's already super helpful. 🚀

We prefer using the CLI for sending emails, as this allows us to store the emails in a local folder on the server and re-use them later on (e.g. to announce maintenance windows, storage downtimes, ...) without having to go for lengthy excursions into one or the other email client's search nightmare.

I'll have a look into the --wait parameter next time and update this issue then.

Unfortunately it's nothing I can easily test under real conditions, as it would mean bothering about 800 people each time 😦

Cheers,
Niko

[pwalczysko]

without having to go for lengthy excursions into one or the other email client's search nightmare.

We are having the email texts in g.docs. There must be anyway a fresh editing step prior to each sending (?). I do not suggest that our OMERO.web UI workflow is the best, but possibly the best under the circumstances. The UI workflow in our hands never crashed in the middle of sending emails. Afaik, there is no way to ascertain to which users the email was already sent to. The 40 minutes wait is a pretty safe estimate for 800 users.

[ehrenfeu]

By g.docs you're referring to Google docs? Or to some database table?

[pwalczysko]

By g.docs you're referring to Google docs? Or to some database table?

Google docs indeed, sorry for the shorthand.

[ehrenfeu]

Just to report back on this, using the --wait flag prevents the command from ⏳ timing out. ✔️