Scaffolding steps and checkpoints

[andreaswallberg]

Hi @markhilt !

I am running ARBitR on a very large and quite fragmented genome assembly. I have gotten to the point where the program runs many processes in parallel:

[Wed Oct 14 18:45:43 2020] Trimming contig ends...
[Wed Oct 14 20:25:35 2020] [ TRIMMING ] Completed: 100.0% (24098 out of 24098)
[Wed Oct 14 20:29:38 2020] Creating scaffolds...
[Wed Oct 14 20:29:38 2020] Number of paths: 24098
[Thu Oct 15 22:52:48 2020] [ SCAFFOLDING ] Completed: 99.67% (24018 out of 24098)

All my 16 processes are running with 55-99% CPU, respectively. However it seems as if the analysis has been sitting at 99.67% completed for over 10 hours or so by now. Up until this point the scaffolding completion was going quite fast.

Is there a particularly time/CPU consuming step at the very end of the scaffolding process?

Also, seeing as the program has produced various, including the GFA and path text files, I wonder if it is possible to resume the analysis from these checkpoint datasets if I need to cancel the analysis (after it has now been running for one week)?

[markhilt]

Hi Andreas,

ARBitR was designed with small genomes in mind, and the overlapping step during scaffolding will be of more use the shorter the repeat clusters are. Any genome larger than the human genome hasn't been tested, and those genome sizes will introduce particular challenges.

Unfortunately, ARBitR may get stuck during the scaffolding step if there are many short contigs that it attempts to assemble together. If you see the memory consumption run away, it might be better to terminate the run. A short-term fix would be to try a higher stringency for anchoring short contigs by increasing -Q, -B and -F.

At present time there is no way to resume a previous run, though I feel like this feature would improve usability quite a bit and will look to implement it in the future.

Markus