Shuffle data while cycling?

[SebastianRiedel]

Do you see a possibility to shuffle the data while reading/cycling through it? Either on tftables or multitables-level?

As I don't see an option related to random access, I assume, you store your training data already shuffled?

[ghcollin]

I store my data pre-shuffled.

In principe, I don't think anything would prevent such functionality in multitables. But it does require some modifications, at which point it might be better to give multitables full random access capability.

There is also a slight complication that HDF5 stores compressed datasets in chunks, and so a whole chunk has to be read at once. If each row of the dataset spans multiple chunks, then this is no issue. But if a chunk spans multiple rows, then all rows in the chunk have to be read. That said, order that the chunks are read could be random.