Support for multi-file ckpt #13
Description
Great work! Love the goals of the library and the blogpost was very insightful. Hoping to be using flashpack for the foreseeable future for all weight loading needs.
I was wondering if the lib currently supports or plans to support loading from multiple files similar to safetensors / dcp.
If its in the works, i was also wondering what the plan was to optimize for training and inference, particularly how it compares to the choices made by safetensors and dcp.
HF Safetensors (optimized for inference -- ~4GB file sizes, parallel lfs / xet io throughput)
DCP (optimized for training -- rank specific files, low global metadata and resharding comms overhead)
