Automatically retry failed tasks

[ayalash]

When A Reconfigure task failed for a deployment, the UI assumes that the deployment uses the new configuration and doesn't allow rerunning the task without changing some other configuration value.

Example for such failure:

[popen2]

After some discussions, the backend should retry tasks automatically.

To do this, dependent reconfigure tasks should be created only after a task succeeds, otherwise it's impossible to know the correct order for executing tasks.

Once the first change is done, each k8s-agent can retry a deployment task as long as it's the latest task for a deployment, excluding pending tasks.

Then, when k8s-agent starts it should mark all non-retryable tasks as failed, since those are leftovers from previous runs before fixing this issue (or if any bugs in the future cause tasks to be marked incorrectly). This solves tasks left in "Running" status since they would either be retried or marked as failed.