Ability to access cache from multiple python instances

[mbkupfer]

Whenever I import a module, with a cached function, in more than one python instance, I get a Resource temporarily unavailable error on the module.py.cache file.

This is a bit of a nuisance since I then have to kill the first process to let a script run. Wouldn't this be fixed by setting the optional writeback parameter for the shelve object?

[ubershmekel]

Please test and let me know if it works.

[mbkupfer]

Sure, I'll give it a try and see if anything breaks.

[mbkupfer]

Never mind, I should of carefully read the shelve documentation as it explicitly states:

The shelve module does not support concurrent read/write access to shelved objects. (Multiple simultaneous read accesses are safe.) When a program has a shelf open for writing, no other program should have it open for reading or writing. Unix file locking can be used to solve this, but this differs across Unix versions and requires knowledge about the database implementation used.

That said, it does allow multiple reads so maybe there is something that can be done there. I'll have to noodle it over some more.

[mbkupfer]

@ubershmekel it doesn't appear that concurrent read/write access will work and/or fit with the intention of your filecache library. Have you considered a more robust key-value pair storage other than the built-in shelve module?

[ubershmekel]

Not really. At the time it felt like I should use the included batteries. Nowadays it seems that's a bit of an unreasonable requirement. It might make sense to update the underlying storage mechanism.

[mjstahlberg]

I've also hit this issue and I expected the fail_silently argument to disable caching in such a scenario. However, this also does not let the second instance of the program run while the first is within the decorated method.