Make DeltaTableWriter more robust to failures in the write-commit process #384
Description
Currently DeltaTableWriter, which is used for writing data to disk, can fail in the write-commit process which means data is not saved. We currently delete the files that we tried (but failed) to commit which means this data is lost. This should be changed so DeltaTableWriter is more robust to transient errors.
A discussion on this was started here: #382 (comment)