ways to reduce perturb output directories?

[jessecusack]

When I run perturb the following folders are created:

├── CTD_0000
├── CTD_combo_0000
├── database
├── diss_0000
├── diss_binned_0000
├── diss_combo_0000
├── logs
├── Matfiles_0000
├── profiles_0000
├── profiles_binned_0000
├── profiles_combo_0000
└── trimed_p_files_0000

@mousebrains can you explain the logic for all of these. Is there a way this could be streamlined? For example, I am not sure why there are CTD_combo, diss_combo, and profiles_combo. Could these all be just one combo directory?

[mousebrains]

They are split out by parameter set uniqueness. The parameters are in the JSON file in each of the directories, with a hash of the parameters in the filename. The JSON file not only contains the parameters for this particular directory, but also all the parameters of the predecessor steps.

├── CTD_0000 -- This is the scalar sensors from the instrument binned in time.
├── CTD_combo_0000 -- this is a .mat and .nc single file of the time binned scalar sensors that pass the filename pattern.
├── database -- this contains associative arrays of all P files seen, independent of any parameters.
├── diss_0000 -- this is the non-depth binned dissipation processed values.
├── diss_binned_0000 -- this is the depth binned dissipation values.
├── diss_combo_0000 -- this is a .mat and .nc single file of the binned profiles that pass the filename pattern.
├── logs -- this contains a log file corresponding to each P file's processing along with a combined log.
├── Matfiles_0000 -- this contains the output of odas_p2mat using a particular set of parameters.
├── profiles_0000 -- this is the non-depth binned individual profiles from the Matfiles+trimmed_p_files using a particular set of parameters.
├── profiles_binned_0000 -- this is the depth binned individual profiles using a particular set of parameters.
├── profiles_combo_0000 -- this is a .mat and .nc single file of the binned profiles that pass the filename pattern.
└── trimed_p_files_0000 -- this is is an overlay of the Matfiles for P files that were created from hitting the configured size limit in the instrument are combined together. This fixes timestamp issues and truncated FFT issues.

Yes, the directory count could be reduced at the cost of reprocessing time when parameters are tweaked. For example, if you only want to change the depth binning of the dissipation parameters, then only a new diss_binned_xxxx and diss_combo_xxxx directory are created using information from the existing diss_yyyy directory. Or if you only want to include one of many instruments in the CTD information, then only a new CTD_combo_xxxx is created using the existing CTD_yyyy information.

The objective function in this design is to minimize reprocessing time as parameters are tweaked.

How parameters are grouping is based on my experience and the structure of the ODAS library.

[mousebrains]

One of the parameter classes I've never been overly satisfied with is filename paths and using shared filesystems, like an NAS. I mostly convert them to absolute paths, but if two users are accessing the same underlying tree, they may have different absolute paths, so they would have a different hash of the JSON file contents. I have not spent enough time on this aspect of perturb to find a robust solution. The only one I've come up with so far is to construct a hash of the source files contents, and use that as a signature for the file. But this will require more IO operations, and I already feel there is a lot of IO.

Given modern computers, it is probably faster and acceptable from a space standpoint to have the "Processed" directory tree on one's local computer. This is why I have pursued the filename path issue too much.

[jessecusack]

Thank you for the explanation. I can see why it is useful to have all of these outputs when still playing around with the processing parameters. I find that in a typical workflow I tweak a few parameters a handful of times until I am happy with the output, and then I don't change them again. So these additional outputs are useful for a short time.

[jessecusack]

One of our guiding principles (issue #1) was to follow a hierarchy of processing levels, L1, L2 etc. Can we identify how the processing steps that we perform produce output that fits into this hierarchy? One way to simplify our output structure might be to streamline it into folders called L1, L2, L3.

To me, it seems like
L0 = essentially the p file, trimmed_p_file?
L1 = the Matfiles, maybe the profiles_ and diss_ folder?
L2 = binned?
L3 = combo?

I can't load the atomix wiki right now so I am not sure how they define these levels. I think that we can be a little flexible and still within the spirit of atomix.

It might also help if we could separate our processing parameters into those that are responsible for moving us between these levels.

[mousebrains]

To me, the level correlation is:

• L0 is the P files, with some nuances:

  • L0.0 is the P-files as generated by the instrument
  • L0.1 is the manually patched P-files with updated config information
  • L0.2 is the trimmed P-files, where bad records are trimmed from the P-files, this is done as part of perturb
  • L0.3 is the joined P-files, where multiple P-files that were generated the instrument due to a maximum file size limit are merged together.
  • L0.4 is output of odas_p2mat, a Mat file with the raw P file data converted to scientific/engineering units, but no other filtering.

• L1 is the first level filtering and has several parts:

  • L1.0 a time lead/lag adjustment is applied to the various sensors so they align up in time
  • L1.1 apply smoothing filters to reduce the high frequency jitter to temperature, pressure, conductivity
  • L1.2 an in-situ temperature calibration of the FP07 thermistors against the reference temperature
  • L1.3 calculate physical water properties, salinity, density, ...
  • L1.4 finds the profiles that meet various criteria
  • L1.5 assigns ship/glider position information to the profiles and VMP out-of-profile data
  • L1.6 time bin the scalars to generate the CTD_binned and combo files

• L2 deals with scalar data for each profile

  • L2.0 apply smoothing filters to the scalar data within each profile
  • L2.1 the profiles are depth binned and written out individually as a combo file

• L3 deals with turbulence calculations using output of L2.0:

  • L3.0 calculate dissipation related values in uniform time steps
  • L3.1 bin the dissipation related values into depth bins
  • L3.2 write out individual dissipation profiles and combination.