backup: PerNodeProcessorProgress stalls backup completion

[jeffswenson]

I was inspecting a backup job that was stuck at 100% completion and found the following stacks in backup:

This suggests backup is waiting for PerNodeProcessorProgress to complete. The backup processor pull from a channel that is sized with the capacity of numTotalSpans.

On a large multi-region cluster, it was observed the channel can take up to five hours to completely drain (360k spans, 50ms per update).

Jira issue: CRDB-53550

[blathers-crl]

Hi @jeffswenson, please add branch-* labels to identify which branch(es) this C-bug affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[blathers-crl]

cc @cockroachdb/disaster-recovery

[msbutler]

keeping open until backports have landed

[dt]

they landed.