Batched and distributed eval (with full determinism feature)

[TonyChen06]

Adds batched eval and distributed eval options.

Both the process of batching and distributing across GPUs themselves are fully deterministic and produces identical results as single GPU non-batched inferencing. (This is tested for 1-2 gpus, 1-8 batch size, and for ptbqa and instuct45k).

The two sources of nondeterminism are confirmed to be 1. RNG conditions changing when batching which leads to different logit choices for do_sample, and 2. slight bfloat16 calculation imprecisions that become apparent when batching and distributing are done since the GPU conditions slightly change.

This version of the PR contains a flag called --full_determinism that allows for fully deterministic inference (as in the results across gpu count and batch size will be identical when the flag is enabled). What it does is simply cast bfloat16 into float64 and set do_sample=false. This is not really too efficient for actual use because of the memory costs for float64 but may be nice if you want to confirm the problem is really there yourself.

The other version of the same PR doesn't contain this flag and is much simpler. Choose whichever one is preferable.

[TonyChen06]

Casting into float32 instead of float64 is a much more efficient way to get deterministic results (it is exponentially faster), and it works 99.9% of the time but occasionally a sample or two will differ when batch sizes are vastly different.