Running out of memory issue

[MirkoBr]

Hi,

Thanks for the great tool. I'm exploring meta transcriptome assemblies with Altesch and I try to run it on a TCGA cohort with about 500 samples.

I first generate the profiles, using:

aletsch --profile -i $bams -p $profile_folder > profiles_out.txt -t 16 --max_group_size 200 --boost_precision

Then I try to create the meta assembly using:

aletsch -i $bams -o $assembly_folder -p $profile_folder -d gtf_folder -t 16 --max_group_size 200 --boost_precision

Doing this works fine for a smaller test set of about 20 samples. The initial results seem to be very messy, compared to for instance the StringTie approach (individual assemblies followed by stringtie --merge). But after ranking them with score.py and -p 0.5 the results look promising. However for all 500 samples I get an out of memory error. I run it on a HPC with 250GB ram via slurm and I allocated all 250GB to the job.

From the supplementary data of the manuscript it seems that you could run the SC-H2 data set, which has about 1000 cells in it with 25GB RAM.

Could you help me figure out what is going on here? Is the --max_group_size 200 maybe wrong or is the --boost_precision not used correctly?
Also what does --boost_precision actually do?

Any help is much appreciated!

Best, Mirko

[shaomingfu]

Hi Mirko,

Thanks for using Aletsch! I apologize for the delayed response.

Aletsch is memory-intensive, as it loads reads from all samples within a genomic region into memory. Therefore, it is not surprising that assembling 500 samples simultaneously exceeds the memory limit. I recommend splitting your samples into batches of 20-50 samples and assembling each batch separately. When doing so, it is preferable to group similar samples (e.g., biological or technical replicates from the same condition) within the same batch, rather than partitioning the 500 samples randomly.

The parameter --max_group_size controls the number of splice graphs allowed in each cluster; larger values require more memory. A value of 20 is a reasonable choice, but you may also experiment with values up to the number of samples in your input (i.e., the batch size).

In our paper, SC-H2 is a single-cell dataset, where each cell contains far fewer reads than bulk Illumina RNA-seq data. For such single-cell datasets, Aletsch can assemble thousands of cells simultaneously (with reasonable memory).

Aletsch indeed produces a large number of transcripts initially, as you observed. Scoring followed by filtering is recommended.

Please ignore the --boost_precision option; this flag is currently inactive in the current release.

Thanks!
Mingfu

[MirkoBr]

Hi Mingfu,

Thanks for the detailed reply. Any feedback is much appreciated!

I also guessed that the memory issues are caused by the higher number of sequencing reads in bulk RNA-seq compared to single-cell data. Can you give a rough estimate of how the memory usage of Aletsch scales with the number and size of samples? For example, for each bulk RNA-seq sample with about 40 million reads, does Aletsch require roughly X GB of RAM?

The batching approach you recommended could be an elegant solution to circumvent the RAM issues. What is your recommendation for merging the assemblies from the individual batches? Should this be done before or after the scoring step?

Regarding the max_group_size parameter: when you say that 20 is a reasonable choice, is this influenced by the number of samples being used?

Also, could you elaborate a bit on the --boost_precision flag? What is the intention behind it, and what is the underlying idea?

Thank you so much for your time and help!

Best,
Mirko