Efficiency on large datasets #13
Description
As far as cacheing data, a compromise between a huge inefficient DataFrame and SQL might be HDF5 (pytables). This can provide hierarchial data storage on the drive freeing up memory and allowing for querying through pandas. Please do some research.
Heres a few links to get you going:
Fast Data Mining
HDF5 Pandas cookbook
Pytables
Large Datasets