Implement data generator into pretrain.py pipeline #22
Description
At the stage where I've got too much game data and am not able to train on all of it because my RAM is not big enough.
2018-06-04 06:42:29.484700: E tensorflow/core/common_runtime/direct_session.cc:170] Internal: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_OUT_OF_MEMORY; total memory reported: 2099904512
Should implement some data generator framework. At the very least, data should be read on the fly, and not inserted as a massive numpy array.