Speed up filtering/subsampling without replacing Pandas

[victorlin]

Context

See parent issue for context on how Pandas is used in augur filter and why it is slow.

There are some potential optimizations to the current code without a full rewrite that's necessary with #1574.

Progress

• Use builtin pandas functions to determine subsample groups #794
• io: Parse metadata with C engine, restrict to either CSV or TSV #812
• Read a subset of metadata columns #1294
• filter: Improve speed of checking duplicates #1466
• Cache get_year_month_day #1792
• More filter speedups #1811
• filter: Vectorize string concatenation #1845

Not pursued

• filter: Rewrite priority queue logic with pandas functions #809

[victorlin]

Another potential speedup here is to leverage the pyarrow+pandas integration. This should be more mature with pandas v2. Pandas is pushing more in this direction as well, slated to make pyarrow a required dependency in v3.

Unfortunately, it's not as simple as setting engine='pyarrow'. I tried briefly with 11743ac. If we want to go down this route, it might be best to convert the metadata TSV to parquet upfront, which would require rewriting some logic (I'm not sure how much). Previous discussion on using parquet for metadata: (1, 2)

[victorlin]

As mentioned in #1811, I believe all the easy gains have been addressed in the scope of this issue. My sense is that the pandas+pyarrow integration approach requires more of a rewrite, so I've moved it to #1574.