Optimize rule combine_samples

[victorlin]

Description of proposed changes

This rule became noticeably slower with Augur 31.2.1. I tracked down a few reasons. This PR along with Augur #1834 speeds up this rule significantly, shaving off ~1hr of run time from ncov/gisaid (details in testing section).

Testing

Full ncov/gisaid trial run results:

• tanglegram for ncov/gisaid/global/6m, I think this shows that there are no functional changes?
• Snakemake run stats compared with old runs (added rule subsample because it has notable differences too):
  • stats-augur-31.0.0.json from May 25 run
    • average subsample run time: 27m
    • average combine_samples run time: 33m
    • workflow run time: 3h15m
  • stats-augur-31.1.0.json from May 28 run
    • average subsample run time: 17m (improved by date function caching)
    • average combine_samples run time: 37m
    • workflow run time: 2h53m
  • stats-augur-31.2.1.json from June 22 run
    • average subsample run time: 11m
    • average combine_samples run time: 2h18m (increased due to concurrency bug)
    • workflow run time: 4h41m
  • stats-ncov-pr-1178.json from trial run linked above
    • average subsample run time: 10m
    • average combine_samples run time: 15m (improved by fixes in this PR)
    • workflow run time: 2h50m

Checklist

• Merge+release of filter: Improve concurrent run times augur#1834

Release checklist

If this pull request introduces backward incompatible changes, complete the following steps for a new release of the workflow:

• Determine the version number for the new release by incrementing the most recent release (e.g., "v2" from "v1").
• Update docs/src/reference/change_log.md in this pull request to document these changes and the new version number.
• After merging, create a new GitHub release with the new version number as the tag and release title.
• Document new minimum Augur version requirement

[joverlee521]

Thanks for tracking down the slowness and documenting all of the test runs!

[victorlin]

I've started a new ncov/gisaid trial run. If all goes well, I'll plan to merge tomorrow.

[victorlin]

The trial run showed faster run times for combine_samples jobs but slightly slower overall. Re-running without b6c233c to see if that helps.

[victorlin]

Average combine_samples run time from the trial run above is still 15 minutes, meaning it's not necessary to use 2 threads instead of 1. I'm going to drop 94f1858 + b6c233c and merge.

Side note: run time statistics are no longer available directly through a stats.json file, so I used this command in the benchmarks directory to get the numbers:

awk 'FNR == 2 {
    # For each file, on its 2nd line, print the first field
    print $1
}' combine_samples_* |

awk '{
    # Accumulate sum and track min/max
    sum += $1
    if (NR == 1 || $1 < min) min = $1
    if (NR == 1 || $1 > max) max = $1
}
END {
    # After reading all values, print statistics
    print "Min: " min " seconds"
    print "Max: " max " seconds"
    print "Mean: " sum/NR " seconds"
}'