Speeding up polishing

[githubgig]

Hello,

I'm trying to do a hybrid polish of a large genome with Illumina and PacBio. Looking at the current rate using 20 threads, it will take me 3 months to complete the polishing. Is there a different and better way to set up the polishing so that I can complete this task faster?

Thanks.

[canfirtina]

Hello,

We are currently working on implementing Apollo on several architectures to accelerate the parts that seem to be a bottleneck. Hopefully, we will release these implementations very soon.

At the moment: We can only suggest either 1) reducing the depth of coverage but still keeping the hybrid data set and/or 2) increasing the number of threads (if possible), as we believe Apollo scales well with the increased number of threads. The former may decrease the polishing accuracy slightly depending how reads are selected in the reduced read set while the latter may require using clusters available in AWS or other platforms to get access to machines with high number of threads, which would be more costly compared to using an in-house machine.

Thanks,

Can

[Neato-Nick]

Is there a logfile or verbose mode we can enable to watch progress?

[Neato-Nick]

Sorry, asking because I've had a process running on 10 threads for about a week. I didn't enable --no-verbose but I'm not seeing progress logs. Trying to figure out how far in the polishing is, or if it'll be so slow I might as well kill the process and restart with double the threads.
In the meantime before performance upgrades, more benchmarks would be helpful to determine how many threads would need to be used for a reasonable runtime. Also, does performance scale linearly with each additional cpu or does it drop off?

[canfirtina]

Hi @Neato-Nick,

You can check your output file to see how many contigs are produced so far. Apollo outputs the contigs in the same order that you have in your input assembly file. I would first count the number of contigs in your original assembly file (assuming that it's a FASTA file) using the following command in bash

grep ">" input.fasta | wc -l

I'd also do the same for the output file. Assume that you specified -o polished.fasta. Then:

grep ">" polished.fasta | wc -l

These two numbers may more or less give you an idea about the progress. However, contig size is also important as smaller contigs are finished much faster than the big ones.

For your second question regarding benchmarking: We have some numbers we did not report in the original paper. We observed that Apollo can potentially scale to as many thread as you may have in a single machine. Specifically, we've seen that Apollo continues to scale linearly up until 256 threads. We have not tried with more threads so far. You can still control the memory scaling using the chunking, and filtering parameters (i.e., F/B and Viterbi filters).

[Neato-Nick]

@canfirtina Thanks, this is very helpful info

Scaling linearly up till 256 threads is great. Understandable that you haven't tried more than that, I think that's more than most folks have access to anyway, myself included. I won't be bashful in how many CPUs I use next time, thank you!