File format conversion - tidy up and consistify

[benjeffery]

@nikbaya is working on FAM file import to tskit in #1487 and we'd like to document write_ms in #1464
In tskit we also have ts.write_vcf and in https://github.com/tskit-dev/tsconvert we have the to/from_ms to/from_newick and from_oriented_forest.

This has grown organically, so needs some tidying up. It seems cleanest to move all input/output conversions to https://github.com/tskit-dev/tsconvert and add new ones there moving forward. This would mean bringing tsconvert up to standard, e.g. getting the tsconvert CLI implemeted, supporting pipe streaming to tskit CLI, improving docs and tutorials.

The exception here is ts.write_vcf as it is documented in a release, to deal with this I propose a two phase deprecation:

1. Mark ts.write_vcf as deprecated and implement tsconvert.to_vcf
2. .... wait a year or two ....
3. Make ts.write_vcf call the tsconvert method, throwing a "You need to pip install tsconvert" error if it is not available.

In this way we don't break anything for a while, and when we do it is a quick obvious fix, and we don't have two VCF export codes forever.

Thoughts?

[jeromekelleher]

This is tricky. Tsconvert started really as a way to get some useful-ish functions into a repo that didn't need to be brought up to library quality. Also, any from_x functions usually entail a dependency in order to read format x, and we don't want any more dependencies on tskit.

My thoughts were that where we had popular and well known formats that don't incur any extra dependencies or lots of code, we could include these in tskit. These aren't particularly well thought out thoughts, though, so I'd be happy to think again about it!

Mark ts.write_vcf as deprecated and implement tsconvert.to_vcf

One advantage of this I guess is that we could in principle do things more efficiently by pulling in some dependency to write BCF directly (not that I suggest actually doing this).

[benjeffery]

Yes, I see the the logic that lead to simple, non-dependant conversions being in tskit and in some ways that makes sense. My main concern with splitting things up is it isn't clear from a users point of view where to look, and if all converters were in one place it makes them more discoverable.

Another option is to go the zarr route and have optional dependencies (either on tsconvert or tskit) that error out if not installed, as we now do with h5py on tskit.

[jeromekelleher]

Makes sense. Maybe we should discuss it at one of the meetings so we can get a community decision on the right way forward?