Actually look into corrupted file #27
Description
I looked at the code for a bit #26
It seems like the issue was caused by a corruption in a file.
I'm not sure in which state the buffer appears after we advance past the problematic field.
But I think it doesn't necessary happen at the end of the file, and can happen in the middle.
There are 2 potentially bad outcomes:
- we misinterpret the rest of the file, and then make wrong decisions when finding bottleneck (very bad)
- we ignore rest of the file (bad but alright I guess)
I'm not sure it's actually easy to recover after the problem. @i7i can you take a look at bytes?
If it's not possible to reliably recover to a good state after passing one bad record, I think we should skip an entire file so that (1) doesn't happen.