truvari bench feature request: pick mode for breakends

[helrick]

Hi there,

I'm using truvari v5.4.0 to compare SVs reported in BND and SVTYPE notation against a truthset in SVTYPE notation. In the wiki, you mention that to allow all BNDs to match to a decomposed SV, --pick multi must be enabled. Would it be feasible to add a BND-specific --pick option, which would permit 2 BNDs to match to an SV with SVTYPE DEL/DUP/INS and 4 to match to an SV with SVTYPE INV?

I may be missing some nuances which prevent the implementation of this feature or make it more complex to interpret than the different cases I've listed above. However, I believe that this feature would enable a better comparison, as it would prevent overly penalising (by using the single option) or favouring (by using the multi option) calls which are in BND notation.

Let me know if I can provide more information.

[ACEnglish]

Hello,

The short answer is I don't think this is something I'll be able to get to in a timely manner. So it could be months before its fully implemented. The longer answer is the --pick modes are fairly well abstracted such that adding a new method of logic isn't terribly difficult. If you look at the truvari.bench.pick_*_matches methods, you'll see they're all reasonably straight forward. Each is given a matrix of truvari.MatchResult objects which hold all the measures of similarity and are orderable/sortable. Each returns a list of MatchResults with updated annotations to be output.

As an example, the easiest code is --pick multi which simply searches across each Row (baseline SV) or Column (comparison SV), grabs the maximum MatchResult and adds that to the returned list. You'll notice that MatchResult.base or MatchResult.comp (a truvari.VariantRecord) are updated and set to None in order to make sure the writer doesn't output the same VariantRecord twice. However, other pickers, such as pick_single_matches, doesn't always set one of the base/comp attributes to None because the writer can output two VariantRecord from a single MatchResult.

With the infrastructure in mind, the pick_bnd_matches method would need to check if any given baseline/comparision SVs is BND and if its pair is BND/resolved (and vise versa) while keeping track of how many times the baseline/comparison call has been used. For example, in the pick_single_matches, this is tracked with a boolean matrix where True/False corresponds to if the baseline/comparision call has used its single available match or not.

The logic gets a little more complicated when we have to consider the order in which we set calls to having been used. Luckily, because the MatchResult objects are sortable, we simply have to sort them by best-to-worst match and then update the tracking of if the call has been used in that same order.

However, for the described BND case, we may not be able to always rely on simple sorting which just uses a MatchResult.score - a flat average of (self.seqsim + self.sizesim + self.ovlpct) / 3.0 * 100. This is because BNDs don't have any of those similarity metrics, only their position and distance, so their 'score' is always zero. And, it gets even tricker when we start to consider edge cases where e.g. the baseline SV is an <INV> while the comparison SV set has the 4 inversion BNDs as well as an <INV> call. This may sound wild/unlikely, but in my years of maintaining truvari I've learned that almost any edge case you can dream up will be hit by at least one of the many SV callers that exist. Therefore, the pick_bnd_matches method will need to enforce a rule that says e.g. "when picking between a single-variant or BND representation, the picker will prefer the single-variant representation" or maybe ".. the picker will prefer the variant representation that is identical to its counterpart". Another edge cases I can imagine is a situation where say we still have a baseline <INV> and a comparison BNDs of that inversion as well as some random FP BND that is describing a deletion but - for whatever reason - aligns better to the <INV> position . The picker would need to prevent the 'less complete' DEL BND from becoming the preferred match by somehow weighing in 'completeness' evidence to the sorting/ordering. And of course these edge cases would need to be designed around for every possible symbolic/resolved/bnd pair.

I'm writing all these notes for two reasons. 1) If/when I'm able to sit down and work in this in a few months, I'll have something to refer to. 2) Perhaps you're feeling adventurous and would like to try develop the feature. I'd be more than happy to review pull requests.

Have a great day,
~/Adam English

[helrick]

Cheers, that makes sense, thanks for the detailed response! I will let you know if I attempt a PR, but otherwise, I will stay tuned for whether you decide to implement this feature.

All the best,
Hillary