Error loading a cached database stored with newer pandas version

[audeerington]

If a database has been loaded to cache with pandas version 2.3.2 previously, e.g.

import audb

test_cache_root = "audb_cache"
audb.load("emodb", version="1.4.1", cache_root=test_cache_root, only_metadata=True)

and then one tries to load the same database with an older pandas version, e.g. 2.2.2, using the same script,
it fails with the error TypeError: StringDtype.__init__() takes from 1 to 2 positional arguments but 3 were given.

The error originates from loading a pickled dataframe, when the constructor of StringDtype is called. In version 2.3.x, the constructor has two arguments, whereas in versions 2.2.x there is only one argument .

This makes it impossible to use the same audb cache with different pandas versions - one can either manually delete the pickle files in the cache or update all environments to use the newer pandas version. This is tricky with a cache that is shared between multiple users.

[hagenw]

I guess the error is happening at https://github.com/audeering/audformat/blob/main/audformat/core/table.py#L501-L516 where we do not handle TypeError at the moment.

In general, the current handling of the cache with different libraries is not ideal. The solution at the moment does always load the original table and overwrites the cache then. Which means if you have users that work with different libraries all the time, loading from cache will always be slow as we can never directly load from there.

[audeerington]

Ah, I see. So to keep with the current approach, we would just add TypeError to the caught exceptions.

I'm not sure how often it actually happens in practice for us, that the pickle file has to be overwritten. So I don't know if it would make sense to try to change this to always load from csv/parquet.

[hagenw]

Loading from pickle was always the fastest option, that was the reason why we continued using it, even after we started introducing parquet files. But I have to admit, I did not focus on the shared cache. My assumption simply was that you will switch to a newer pandas version at some point and then your cache is updated for every dataset you reuse.

But maybe we should reconsider this and not store parquet tables as pickle files?

/cc @frankenjoe

[frankenjoe]

But maybe we should reconsider this and not store parquet tables as pickle files?

Ok, I was not aware you can store parquet as pickle. I thought it was already a different and more compatible format than pickle.

And yes, relying on pickle seems dangerous.

[hagenw]

I thought it was already a different and more compatible format than pickle.

Yes, but we also did some benchmarking when introducing it in audeering/audformat#419

There you can see that loading from pickle is still up to 10x faster for large datasets.

[frankenjoe]

Ok, I was just confused that you wrote "store parquet as pickle files". Or do you mean store as parquet instead of pickle?

[hagenw]

When loading a table with audb it is downloaded and stored in its original format in cache (csv or parquet), the table is then loaded and written as pickle file to the cache as well. This is what I meant with "store parquet as pickle files". When you then load a second time the database, the tables are read from the pickle files in cache.

[frankenjoe]

Got it! So in your scenario you mention that you update to a newer pandas version. But this should not cause issues because pickle should be backward compatible, but if someone tries to load with an old pandas version this could cause problems, right?

[hagenw]

It can also break loading from pickle with a newer pandas or pyarrow version, e.g. if pandas or pyarrow has breaking API changes, or sometimes even if they rename some private functions that are used when loading from pickled files. I have never exactly looked how it works, but we had those issues already with several pandas versions.

We have the same issue in audb with the dependency table, see https://github.com/audeering/audb/blob/main/audb/core/api.py#L302-L317.
There we catch every possible exception, so we will not see there error @audeerington described above.

[frankenjoe]

OK! But loading the parquet will always work? So maybe if loading pickle fails fallback to parquet? And maybe dump the pickle again?

[hagenw]

Yes, we fall back to loading again from csv or pickle already. The reported issue here is because we do not catch all possible occurring errors in audformat yet (https://github.com/audeering/audformat/blob/main/audformat/core/table.py#L501-L516)

[frankenjoe]

Ok, but coming back to the original question: a speed up of 10x still seems to justify storing a pickled version to me.