Batched File Storage #36
Description
Currently, we store everything per PDF or per page. This results in a massive number of tiny S3 files (roughly 60 million each for embedding, image embedding, image, and text) which is both slow and expensive. If they're only being used for bulk processing, we should just store them as ~1GB segments. This should be doable for all of them. However, the image segments will need to be de-compressed and stored per-page on the server's local filesystem.