Storage engine improvements #44
Description
The current storage engine is a simple abstraction over cloud object storage services such as S3. A file is received by the server and stored as-is in the backing service. This can be inefficient in case of large files.
The following activities can be carried out to improve the storage engine:
- Build an abstraction around the
holdcrate that can implement custom logic around files and stores metadata in CouchDB - Split files into equally sized chunks and store those as blobs
- Compress chunks before writing them to reduce storage space