Make deterministic an option.

[kevinzakka]

For debugging and overfitting.

[kpertsch]

Did you test whether this actually makes the data loader fully deterministic? For very large model training it can be good to have a deterministic data loader to avoid re-training on data upon resuming (eg if you do a total of <=1 epoch), so having an option for determinism could be great even beyond debugging use cases.

[kevinzakka]

@kpertsch I did some testing in a notebook and it seemed deterministic but have to write a full unit test.

[peter-mitrano-bg]

I would find this addition useful, although possible we should also expose the arg in the from_tfrecords function not just from_rlds?

[peter-mitrano-bg]

And of course I tested and can confirm this works. Having this hard-coded to false is extra scary because the TF docs say that if shuffle_files=False then it will automatically set deterministic to true, but that I guess doesn't happen in this case 💀

https://www.tensorflow.org/datasets/determinism#with_shuffle_filestrue