Connection reset bug in distributed TrainingStateController

[sdrobert]

A recurring and irritating bug in TrainingStateController. See 924ff25 for an example.

Unclear if it's due to a deadlock or a timeout in AppVeyor.

[sdrobert]

It's still occurring sporadically after the switch to GitHub actions. Others have faced it, e.g. here and here.

Because I can't reproduce it reliably, I doubt I'll be able to fix it.

[sdrobert]

With #74 and #75, I believe this to be fixed.

So far as I can tell, the issue occurred because worker 0 was writing history to disk before some other worker would read it in order to determine when the last epoch was. With the new row of history, that worker believed it was one epoch later in the run and would terminate preemptively, causing worker 0 to wait indefinitely for results of the next epoch.

The solution is to assume that the cache is strongly consistent across workers (which it should be, so long as weights and measures are synched up) and just not read the disk for history.

However, as I mentioned, the bug occurred sporadically. It's possible that there are still issues with distributed training, so I'll leave this issue open for a bit.