Potential memory/slowdown issues

[lk11235]

We are seeing evidence of excessive memory usage when attempting to store and calculate new quantities for very large sets of events. This is related to the event loop approach which currently requires the creation and maintenance of several large numpy arrays before values can be stored to temporary subtree files.

Due to Python's notoriously bad management of memory, each job looks like it requires a lot of resources and particularly large jobs cannot be run interactively without overloading the LXPLUS node. Variable deletion and garbage collection cannot be used to effectively free resources during runtime.

One solution which has worked in the past is to submit each calculation to a worker process and take advantage of multiprocessing's ability to clear and free resources which each worker has used. But this is arguably messy in other ways, and I don't want to implement it here. Scaling is also a concern, as we will be adding numerous additional calculations and interfacing with MELA.

[lk11235]

A hacky method which works for intensive jobs that you wish to run interactively is to take advantage of the subtree features which I purposefully added. Process each subtree separately, then run the command to concatenate the subtrees at the end. Subtrees can be deleted manually or the entire thing (including cleanup) can be automated using the merge code at the end of the tagger. This trick also works if a batched job fails and only successfully processes one of your subtrees---just run over the remaining subtrees as needed and then merge.

[lk11235]

A better long-term solution is to perform all calculations using awkward arrays after reading the mass-dependent spline values into new columns. The best way to do this is to overhaul the tool and utilize more efficient tools to read the values from the root files directly into immutable arrays. This would also be much safer than using numpy, and make it easier for future users to modify the code to target subsets of events, etc.

[lk11235]

Ideally, I want to take advantage of JIT and XLA through JAX which should speed up these numerous chained operations by orders of magnitude. But let's not force this on anyone until the introduction of additional AC calculations. Modification to use JAX should be straightforward and can be implemented after the next update (which will improve read/write operations and array-like objects).

[lk11235]

One consideration here is that calculations with existing branches can be vectorized but interfacing with MELA to create new branches might have to be treated separately. This is really the primary bottleneck in performance. It would be nice if we could vectorize MELA operations over Heshy's objects in immutable arrays. This must still be tested. In principle this should not cause conflicts with JAX but we will have to be careful to register objects with JAX and be mindful of object dimension, etc. (for example, recall that JAX and Awkward scalars look completely different).