Data mapping cache blocks function updates

[antoniorv6]

HuggingFace datasets use aggressive caching for map() operations, which creates a significant development workflow issue: when you modify a function used in dataset mapping, the cached results from the old function version are still returned, making it appear as if your code changes have no effect.

We need some sort of cache cleaning or debug_mode for development reasons, as users may have issues when trying to customize the data loading and preprocessing functions of the repository.

[eric-ayllon]

HuggingFace datasets implement a clear_cache_files() method that erases the cache. Calling this method before the map allows forcing the execution of the map operation. Furthermore, the map(...) method allows deactivating the cache loading to force the execution of the given function: load_from_cache_file, which defaults to True.

However, another issue that could be raised from using the map method is that it can consume a lot of memory. A balance between time and memory usage can be found by adjusting the writer_batch_size parameter, which defaults to 1000.

For debugging purposes, the dataset map operation might not be the best option. I would suggest disabling the map operation and performing the preprocessing on the fly when debugging. Then re-enable it after debugging the data loading. This might be a bit more cumbersome, but I cannot think of another way to circumvent the potential memory and time issues during debug runs.

[antoniorv6]

Can we put a --debug parameter on execution that may trigger that behavor? If possible, I think we should go through that route and specify it in the README.md

[eric-ayllon]

It is possible, though I would first recommend migrating to another configuration manager to keep command line options more organized but still allow advanced customization from the command line without the need to create a json or yaml file for each new test or experiment. I will start working on it as soon as the full-page pipeline is integrated in this repository.

[antoniorv6]

In the last PR merge, it seems that the map function blocks execution, even if cache is completely clean. We need to fix these behavours asap.

[eric-ayllon]

Such behavior did not seem to show up after some testing. The map function initially runs relatively fast but stops after a certain amount of samples to write their updated info into the cache. Writing to this cache is a slow process as it involves writing to disk. While the execution appears to be stalled, it is not blocked. After some time, the execution resumes.