Single procrastinate task

[vadim-zabolotniy]

We have two setting for Procrastination tasks:
queueing_lock - this blocks spam of tasks in queue. It allows only one task with status todo. This means that in DB at any time can be one(doing) or two tasks(doing + todo).
lock - forbidden only one doing task in any time.

If we use:
Only queueing_lock will stop todo task spam. But we will have 2 executing same tasks.
Only lock will have only one executing of some kind task, but will have todo task spam.
queueing_lock + lock gives as optimal result we have no spam in task queue and always only one executing task.

Now about problem: Logic of working lock pretty simple if task with status doing exists is skips start of executing new one. And this leas to "dead" task after server restart. Worker killed, status doing and new execution will never start.

This is significant issue that need to be discus. For that reason I have created this MR as draft.

[vadim-zabolotniy]

Hi @rtpro!

I have pushed commit to remove stalled jobs before Procrastination worker start.
I want to warn you about PostgreSQL message in logs:

postgres_1       | 2023-02-09 14:36:00.528 UTC [35] ERROR:  duplicate key value violates unique constraint "procrastinate_jobs_queueing_lock_idx"
postgres_1       | 2023-02-09 14:36:00.528 UTC [35] DETAIL:  Key (queueing_lock)=(application__check_health) already exists.
postgres_1       | 2023-02-09 14:36:00.528 UTC [35] CONTEXT:  SQL statement "INSERT INTO procrastinate_jobs (queue_name, task_name, lock, queueing_lock, args, scheduled_at)
postgres_1       |          VALUES (queue_name, task_name, lock, queueing_lock, args, scheduled_at)
postgres_1       |          RETURNING id"
postgres_1       |      PL/pgSQL function procrastinate_defer_job(character varying,character varying,text,text,jsonb,timestamp with time zone) line 5 at SQL statement
postgres_1       |      SQL statement "UPDATE procrastinate_periodic_defers
postgres_1       |              SET job_id = procrastinate_defer_job(
postgres_1       |                      _queue_name,
postgres_1       |                      _task_name,
postgres_1       |                      _lock,
postgres_1       |                      _queueing_lock,
postgres_1       |                      _args,
postgres_1       |                      NULL
postgres_1       |                  )
postgres_1       |              WHERE id = _defer_id
postgres_1       |              RETURNING job_id"
postgres_1       |      PL/pgSQL function procrastinate_defer_periodic_job(character varying,character varying,character varying,character varying,character varying,bigint,jsonb) line 17 at SQL statement
postgres_1       | 2023-02-09 14:36:00.528 UTC [35] STATEMENT:  SELECT procrastinate_defer_periodic_job('default', 'application__check_health', 'application__check_health', 'application__check_health', '', 1675953360, '{"timestamp": 1675953360}') AS id;

This error raised by constraint that was created by Procrastination. Procrastination knows about it and correctly handling it in Python code. You can consider it as INFO level message.