Array-buffer wrapper class for all-tree data (epoc & branch length)

[lentinj]

Whilst we may want to manage multiple sparse data files, don't do this yet, but sculpt for the possibility in the future.

We need a class that can:

• Wrap an ArrayBuffer (think raw bytes in RAM), potentially multiple arrays in future if we want to partially fetch data.
• Directs requests for data to the appropriate offset, or null if we don't have this data yet
• Have an async method to fetch data if it's been requested at least once (eventually this would be a list of cache misses)

On top of this, specialisms that can convert metacode/OZids to byte offsets, and return typed data (e.g. bool offset is divided by 8, then picks the relevant bit out)

Ultimately this will be used for epoc data & branch length, but could be retrofitted to e.g. IUCN data, removing special-case code.

[lentinj]

@hyanwong we'll need a generic mechanism in server_urls to fetch a versioned data resource here, so I was seeing if I could combine this with the existing mechanism to fetch the FinalOutputs/data/completetree_{}.js files in dynamic_scripts.

Over there we try to fetch completetree directly via <script> (if try_local_treefiles_version has been filled in, i.e. minlife). If not we fetch version.json, then fetch treefiles ourselves. The point here is, as I understand it, the <script> tags will be doctored into local URLs by the minlife machinations, so despite initial appearances your minlife setup will attempt to use a local completetree first.

Is this complexity we need? As a result we're backing up the loading of the live site behind a request to version.json, and latency of the live site seems more important than minlife.

We could just have sever-rendered script tags instead, and:

• Remove the rewriting, fetch FinalOutputs/data/completetree_{}.js from the server in minlife, assume that the browser caches will manage any performance loss
• Keep rewriting, fetch completetree locally in minlife, and if it goes wrong tell the user to update their local files rather than trying to do it transparently

Any thoughts?

[lentinj]

I've pondered a bit and sketched out what I want to do here (I've not even tried compiling, so 'scuse syntax errors):

a4710a6

The DataStore represents any number of arrays, which it calls "slices". For instance in the base class we have a slice for node & a slice for leaf. The others we only have one slice. We're free to break data down into many more slices if we want to in the future, but aren't just yet.

The IUCNDataStore example reads values indexed by OTT, as a result it only has one slice. It also reads 16bit integers, so has to think about endianness---we could actually generate both le & be files if we really wanted, but there's little point (sorry Eric).

The CutPositionMapDataStore goes as far as wedging a JSON map into the same framework, which should still work.

I'm not proposing we make use of these subclasses, or even if they make sense, they were just real examples I was borrowing for the time being. But it is something we could look at later.

[hyanwong]

I think we hardly ever use minlife, so happy to deprioritise that, especially if it reduced the latency of the live site.

I suppose IUCN is a reasonable test case, but in reality we would just provide this via the API, as it doesn't need to be known in order to position the branches in space. The main reason that we need dates to be accessible on the client side is because it's required for laying out higher-level nodes. Stuff that can be laid on dynamically via an API call is probably not worth wrapping in a downloadable array.

[hyanwong]

I suspect that the main work here will be figuring out how to test it. Will we need to sort out a tiny test tree for this?

[lentinj]

Stuff that can be laid on dynamically via an API call is probably not worth wrapping in a downloadable array.

The use case here really is the epoc & branch length data. But I do have a bit of an ulterior motive. A pipe-dream of mine is to see more of what currently happens via. the API work from static files. The chattyness of node_details won't be great for performance, both the amount of load it'll generate on the server, the network latency of each request & the amount of housekeeping client-side to make sure everything is present.

If we could reduce the content of node_details down to a series of column-based binary files, sliced up to some rough measure of locality, then bar search you could host a onezoom with cacheable flat files. The devil is invariably in the details, mind.

I suspect that the main work here will be figuring out how to test it. Will we need to sort out a tiny test tree for this?

For this? No. Generating a random array & inventing some fake nodes will be pretty simple, and will make it easier to test edge-cases we can't otherwise scratch.

In general we should have something, both for new developers & for running Python CI. It might be easier & a better job to skip the tiny-tree approach, and publish the real thing (.js files & a SQL dump) as a GitHub release, then script the ingestion of a new dump. Again, I may be glossing over details that make this impractical.

[hyanwong]

I do have a bit of an ulterior motive. A pipe-dream of mine is to see more of what currently happens via. the API work from static files

It's true that we could move some of this from the API, but there's too much data, I think, for it to work well for all to be static?

It might be easier & a better job to skip the tiny-tree approach, and publish the real thing

We could use the extinct tree for this?

[lentinj]

It's true that we could move some of this from the API, but there's too much data, I think, for it to work well for all to be static?

It depends on if we can come up with a good-enough algorithm to chop the tree into chunks, with each chunk being a reasonably local set of nodes. It doesn't have to be perfect, given each chunk would be quite large. For epoc data here (and IUCN) it's not worth the pain, but latin names in such a dump would be a non-starter without some kind of sharding.

Even if you did, you're creating your own cilent-side database and maybe should be trying something like client-side-sqlite first.

[lentinj]

I've pushed again, and made everything a bit more concrete, with linkages to the outside world: https://github.com/OneZoom/OZtree/compare/wip-issue-978?expand=1

Nodes could do something like data_store_api.iucn.get(node), and everything else should just work(tm).

I don't like how it's a somewhat separate world to everything else:

• We have our own timer on top of the node_details_api & image_details_api, rather than combining with one of them---not that I think having these requests in the same queue as node_details is necessarily sensible, but it would be good to remove complexity rather than add it. It does at least only spin up when needed.
• We do our own thing in api_manager rather than using api_wrapper. This is mostly so we can use window.fetch, but could be tidied up.
• The data_store_api singleton should probably live within api_manager, rather than being imported and used directly. But doing so seems a bit odd.

Overall it does feel like it should work though, at least in it's abstract sense atm.

[lentinj]

I've added tests (and fixed some gaffes) to the wip-issue-978 branch, and will leave it as-is for the time being. It largely seems to do what we want I think, but will see once we have real data to integrate.

[lentinj]

I stumbled on roaring bitmaps last night, and may be worth thinking about. The salient quote seems to be:

Roaring solves this problem. It works in the following manner. It divides the data into chunks of 65536 integers (e.g., [0, 65536), [65536, 2 x 65536, …). Within a chunk, it can use an uncompressed bitmap, a simple list of integers, or a list of runs.

...which on a quick glance sounds like it'd work quite well for our use case, and could be worth investigating more deeply at some point. There's Javascript libraries to work with the format too.