Notes from IceCube Software Conversations

[nsoblath]

This issue is for recording notes about possible changes we could make to Nymph (and Katydid) based on input from the IceCube analysis/simulations framework and conversations with Ben, Sebastian, Alex Olivas, and myself.

[nsoblath]

Conversation on 6/29/17

Discussion Notes

IceTray

1. Hierarchy of data/analysis function categories, with higher categories being slower-changing than lower categories, and higher categories remaining available to the analysis in the lower categories:
   1. TrayInfo
   2. Geometry
   3. Calibration
   4. DetectorStatus
   5. DAQ
   6. Physics
2. Shared services available to all modules from a context object
3. Modules can be defined at runtime from functions or lambdas
4. Alex said if he were to do it all over again, he'd consider basing the framework in python and having some modules written in C++ for efficiency in running.
5. Alex also said he's open to the idea of making the IceCube software package open source, or granting us access, but one of the main things holding him back from the former is that there are some things that need cleaning-up.

Serialization

1. They use intrusive serialization
2. If they wanted to do non-intrusive, would involve more changes to the data classes than intrusive
3. HDF5 and ROOT writers are separate from the boost::serialization-based writers
4. The former are based on "tableio," which converts the data to a table format
5. From the table format, the data can be written to HDF5 files or ROOT TNtuples
6. I mentioned that I thought it was possible to write a boost::serialization archive that would work with HDF5 or ROOT, but I hadn't tried it. It would simplify things for our writers, though.
7. Alex is going to try implementing a JSON archive.

Post-Discussion Notes

1. I like the idea of having geometry-, calibration-, or detector-status-like data available to the DAQ- or physics-like data. I'm not sure I would implement it in the same way, though. I might prefer to have a database service, or something like that, which can be accessed by any of the processors and has some sense of a validity range for information. Information is requested by a processor for a particular date and time, and the service returns the information that is valid at that date/time.
2. The idea of a service could be very useful for something like the database or RNG (the latter especially if the framework is used for simulations).
3. I really like the idea of being able to define a processor on-the-fly with a function or a lambda. I think this is going to be possible in Nymph2 with the improved separation of the data/analysis layer and the application layer. Clearly this is something that would work well in the Python interface. I also think we could do it from a config file using the C++ interface.
4. I'm open to the idea of developing a more natively-Pythonic side of Nymph, and even transitioning some of the main functionality to that. Certainly the functionality of the processor toolbox could be done in Python. I think it's important to keep the signal/slot mechanism in C++ and probably the data classes in C++ (for uniformity of how the serialization is done, for one thing), but we'll have to see how things evolve. We would need to figure out the best unified build system.
5. I'm still pretty confidant that we can base our main HDF5 and ROOT input and output on boost::serialization, but it'll require some work to get right, I suspect.
6. It'll be nice to have a native read/write ability to allow analyses to be done in stages.
7. I was very interested in the TableIO Converter, which had, for every field in a data object, a string name, a type, a string unit label, and a string description. I wonder if we might just go ahead an implement something like that in our data classes themselves, so they're available for use by any writers or anything else that needs that information. Something like this:

   DATA_FIELD( double, AcqRate, "acq_rate", KTTypes.Float64, "MHz", "Acquisition rate" );

   This would expand to something like this in the header:

   double fAcqRate;
   struct AcqRateInfo
   {
       std::string fLabel, fUnits, fDescription;
       typedef KTTypes.Float64 Type;
   }
   static const AcqRateInfo fAcqRateInfo;

   And somehow something like this in the source:

   AcqRateInfo DataClass::fAcqRateInfo { "acq_rate", "MHz", "Acquisition rate" };

[laroque]

It will probably be more productive if @nsoblath and I discuss and post some notes back here, but I want to go ahead and post some thoughts related to the post-discussion notes above.

\1) I think that there are really two different cases here and we want to support both. Case 1 is that there is information from outside the fast DAQ which we want to incorporate (trap configuration, sensor values, whatever). Hopefully this information is captured in the snapshot files and could be read from there rather than actually submitting jobs that all do database interactions, but either way this is similar to their "service" and something we've talked about wanting for some time. Case 2 is that there could be information from earlier parts of the analysis which we wish were still available in later processors. The most obvious example being access to the egg header information downstream when looking at track or event objects (including at a later time when I've perhaps written an event file and am trying to read it back).
\2) I'm pretty iffy on having a database available to analysis software. We can probably make something that does what we want, but building and deploying it in a reasonable way is a challenge at best, and we would need to have some reasonable way of faking it for offline testing/development etc.
\4) I think that this item is the reason why my progress has stagnated. I'm understanding the structure a bit better but am not really clear on which design choices are important features and which represent choices made mostly because the implementation is in C++ and don't need to be retained if we migrate to a pythonic version. I think we could take an adiabatic approach, but that it will be much less painful if we pick which one way we want the interface to work and commit to that. I also think that if we choose the latter we will need to keep both for now, but I don't know that we will want to try to maintain both.
For the build system, I'm not sure if there is a clean way to merge cmake and setuptools. Probably the easiest thing would be just to call it a dependency and have the python implementations in their own repo.
\6) I suspect this upgrade will be similar to irc->slack or svn->git.
\7) Is typedef of Type going to make heads explode? My C++ is still not fluent, shouldn't the last header line be static const AcqRateInfo fAcqRateInfo and doesn't the source side need to initialize fAcqRate also? Is DATA_FIELD a macro that would be used to add parameters to new data classes? The macro seems nice, but I'm not sure how to get this to play with wanting to be able to create new data objects in python-implemented processors.

[nsoblath]

Regarding point (7), yes, that's what the header line should be. I've fixed it in the original comment.

This is probably a slightly more realistic implementation:

.hh file:

DECLARE_DATA_FIELD( KTTypes.Float64, AcqRate  );

expands to

double fAcqRate;
struct AcqRateInfo
{
    std::string fLabel, fUnits, fDescription;
    typedef KTTypes.Float64 Type;
}
static const AcqRateInfo fAcqRateInfo;

.cc file:

INSTANTIATE_DATA_FIELD( AcqRate, "acq_rate", "MHz", "Acquisition rate" );

expands to

AcqRateInfo DataClass::fAcqRateInfo { "acq_rate", "MHz", "Acquisition rate" };