Resilient background job retry & monitoring

[rohitdash08]

Improve reliability of async job execution.

Acceptance Criteria

• Production ready implementation
• Includes tests
• Documentation updated

[a827681306]

/attempt #130

I'll implement this with full test coverage and documentation. PR incoming shortly.