High system load with concurrent Imapsync connectors

[nrauso]

Scenario:

• Single NS8 node with 4 CPU cores and 8 GB of RAM
• 40 Imapsync connectors configured to fetch remote mailboxes every 5 minutes

When all connectors start simultaneously, the system load spikes significantly, making the node unresponsive for other services (Webtop, Nextcloud, and the file server).

Below is a top output screenshot taken during a fetch operation:

Further investigation is needed to implement an effective solution.

Components

• imapsync:1.0.4

[stephdl]

QA

version to test ghcr.io/nethserver/imapsync:1.0.5-dev.3

case 1

• do a first sync with sync only INBOX, check the command of imapsync the --maxage is not present : journalctl -f _UID=$(id -u imapsync1)
• let the first sync goes to the end, a timestamp file with the ID of of task is present
• sync again with sync only INBOX, this time the --maxage=1 is present, stop the task before it goes to the end, the timestamp file is removed (since we have interrupted a task we do not know if it went to the end, better to remove the timestamp and do a whole sync the next time)
  when you do the full sync you could need a lot of time before to get the timestamp
  you can simulate during the sync is going on by adding manually the timestamp if needed: 35wj75 is the task id

podman exec -ti imapsync ash
 touch /etc/imapsync/administrator_35wj75.timestamp

• do a full sync again with sync only INBOX, delete the task, the timestamp file is removed

case2

• do a full sync with sync only INBOX
• create a cron task
• each time the sync must be done by the cron with the --maxage=1

case3

• do a full sync with sync only INBOX
• inside the container the file syncctl find the unix timestamp of today, you can increase it and verify when you sync that the --maxage= is now superior than 1 (it must be an integer)

case 4
real case test, on a real server try to use many concurrent tasks by cron and try to demonstrate the load is lesser than with the stable module.