Huge file size from borzoi_sad.py

[ratheraarif]

Hi,

Thank you for the nice and truly open source work you are doing!

I tested the code for VEP using a single example from https://github.com/calico/borzoi/blob/main/tutorials/legacy/score_variants/snps_expr.vcf, with the script https://github.com/calico/borzoi/blob/main/tutorials/legacy/score_variants/score_expr_sad.sh. However, I noticed that the size of the sad.h5 file generated was extremely large, approximately 450MB for just a single example.

Additionally, when I ran the code on my own VCF file containing about 135 examples, the resulting file size ballooned to 192GB. Am I making an error in my workflow, or is this file size expected?

Actually, I want to retain only ref and alt predictions and then compute the sad score manually. can this file size be decreased?

[davek44]

Hi, the REF and ALT codes are meant for your application, but they are indeed very large tensors, so we mainly use this mode for visualizing single interesting variants rather than large-scale analysis. The predictions are L=6144 x T=7611, stored as 16 bit for each of 2 alleles, which comes to 187 MB per variant. I guess the other MB are other scores? You could check the array dtypes and shapes in the HDF5 to figure it out.

We could consider adding an option to average pool the 32 bp predictions to a coarser resolution if that would be helpful.

[ratheraarif]

An option to average the predictions be highly appreciated.