Switch driver metadata files to pickle

[gilrrei]

Description of the Issue:

When the metadata files were introduced, they were only thought of as documentation of the current job, I went for a human-readable file format, YAML. This means that only standard types, i.e., floats, list etc can be exported. Since then, the metadata files have been used for restarts by returning the "outputs" stored in them, if they exist. However, since they are converted to standard types, some information might be lost.

Proposed Solution:

Change this to pickle files, which allows to store the objects directly. Pickle should also be faster to write and read.

The main downside is that it is no longer human-readable, which makes it more difficult to grep for data.

Action Items:

No response

Related Issues:

No response

Interested Parties:

No response

[rjoussen]

I think this should be optional only. At least I personally look at metadata files a lot to see what is actually going on at the moment, e.g. what result the driver read for a certain job.

[gilrrei]

@rjoussen

I think this should be optional only. At least I personally look at metadata files a lot to see what is actually going on at the moment, e.g. what result the driver read for a certain job.

As mentioned, this makes it more difficult to grep. Nevertheless, this is a different problem; since we don't have an overview of the current jobs we (mis)use the metadata files for this.

[sbrandstaeter]

I agree on the discussion, I myself and afak many others (mis-)use the metadata files for progress tracking.
On the other side, I also struggled with the limitations of the supported data types.
All in all a very good point of discussion is raised here!

This might also be related to #170. Where logging could fix the progress tracking.

Another idea: is there other formats than pickle that we could use?
Yes, they are very flexible and can serialise almost anything; however, they are also not well suited for long-term storage. Would you consider them as part of the output that might be interesting to store for a longer?

[gilrrei]

@sbrandstaeter

Another idea: is there other formats than pickle that we could use?

If we want to store every type of result object, I don't know how to do this without introducing serialisers for every object.

Yes, they are very flexible and can serialise almost anything; however, they are also not well suited for long-term storage. Would you consider them as part of the output that might be interesting to store for a longer?

Personally, I don't think we should use the metadata files for long-term storage. I think this is the users responsability.

@sbrandstaeter @rjoussen thanks for your inputs :)

Maybe just as an overview, what the metadata currently are used for:

• Store job id
• Store inputs
• Track current job status (currently not working perfectly)
• Track time for each subjob
• Store outputs
• Store additional custom job data
• (Mis)use for custom restarts

A sensible split of information:

• Job tracking (job_id, status, simulation times), ideally:

  • Be human-readable
  • Single file for all jobs (this is not so easy to do, requires parallel writing/reading)
  • Data should be structured, each job has the same

• Job data storage (inputs, outputs, custom data, restarts)

  • Needs to work with large inputs and outputs
  • Easy to add additional data
  • Should be fast to write and read → binary
  • Each job gets its own file, needed for restarts. Gathered data is done as a post-processing step if needed, as for online iterators, the gathered data across iterations does not necessarily make sense.

I think @sbrandstaeter, you are on to something. Logging the start and end (successful or failure) in a single experiment log file should be feasible and is the information we are generally interested in for job tracking. The rest could then be done in different separate files.

@rjoussen, I get the luxury of looking into inputs and outputs in an ASCII file; the only problem is that if you have 10000 inputs and outputs, this becomes unreadable and slow. Also, this comes at the price of general-purpose restarts... My gut feeling tells me there is a solution, but currently, I don't now how 😅