Questions about TrIOS Data Processing in Hyper-CP

[felipefariadesousa]

Hello everyone,

I am new to processing TrIOS data in Hyper-CP, and I have some questions about the configuration and data input workflow. My goal is to process the data from raw to L2, but I am facing some difficulties in certain steps.

Configuration of the .cfg file: Is the default configuration file sufficient for processing, or are there specific parameters that I need to adjust?. Data input (.mlb): I understand that the data must be provided in .mlb format, but I am unsure about the correct export procedure in XSDA_ME. My questions are:

• Do I need to export the measurements of each radiance separately?

• Do I need to generate separate files for each point and its respective measurements?

• Complete processing workflow: I would like to understand the step-by-step process from exporting the data in .mlb format to processing it in Hyper-CP.

I would greatly appreciate any guidance or reference materials you can provide.

Thank you for your help!

[oceancolorcoder]

Hello @felipefariadesousa,

There is a step-by-step guide for setting up your processing configuration provided here: Configuration Window. As it states, while some parameters are loaded to default values in the configuration file, there really is no default instrument set-up or processing regime. For instance, your individual sensor calibrations will need to be loaded in and associated with their appropriate orientation (i.e., Li, Lt, Es) and you will need to adjust the processing for either Factory-based calibrations, Class-based characterization, or Sensor-specific characterization. Other parameterizations my differ depending on the water type you are sampling (e.g., SimSpec NIR residual correction for optically complex waters). The documentation at the above link provides the citations for "default" parameters for reference, as well as alternate values used in special situations.

That documentation also links you to the D-6 which describes the process of extracting your data from (manually acquired) TriOS data acquired with MSDA_XE. Most applications for manually acquired TriOS (i.e., with no robot to control sensor azimuth relative to the sun) to date have exported ~5 minute casts from each instrument to .mlb files. However, it is also possible to work with longer files, e.g., one hour. These will be binned to 5-minute ensembles in L2.

Hope this helps!

Dirk

[felipefariadesousa]

Hi Dirk Aurin, thanks for your feedback.

I had not configured TriOS for my calibration files. Also, I was exporting my .mlb files incorrectly. By adjusting the different configuration levels I was able to achieve the expected results. It is an amazing software.

Thank you again.

[oceancolorcoder]

@felipefariadesousa that's great to hear. Don't hesitate to reach out if you run have other discussion questions or software issues.

[helena31415]

Hello @oceancolorcoder !
I am currently processing TriOS data in Hyper-CP and encountered an issue during the "ProcessL1a" step. The error message is as follows:

---

Process Single Level
ProcessL1a
ERROR: no identifier recognized in TRIOS L0 file name
L0 filename should have a cast to identify triplet instrument
ending in 4 digits before S.mlb
L1a processing failed. Nothing to output.

---

I am not sure how to format the filenames correctly for the L0 files to avoid this error. Could you please clarify how the filenames should be structured, specifically regarding the instrument identifier and the 4-digit cast before the "S.mlb" suffix?

Any advice or resources on resolving this issue would be greatly appreciated!

Thank you for your help!

[oceancolorcoder]

Hi @helena31415,

Thanks for pointing this out. I see now that the file naming restrictions for TriOS "raw" .mlb files is overly strict, conforming to standards adopted by some teams who helped to develop this portion of the software. We will work to reform this section to make it more filename agnostic.

In the meantime, the current convention requires one of three formats:

dddddddd_dddddd.mlb
ddddS.mlb
dddd-dd-dd_dd-dd-dd.mlb

where d is a digit 0-9. I do not believe these digits are parsed for use currently (I have a colleague checking), but would speculate that the first convention was meant to indicate:
YYYYMMDD_HHmmSS; Y-year, M-month, D-day, H-hour, m-minute, S-second.
And the last is similarly:
YYYY-MM-DD_HH-mm-SS
Not sure about the middle format, but someone will get back to you.

Thanks again for bringing this up and you patience while we work on a fix for you.

Best,
Dirk

[helena31415]

We have encountered some challenges, particularly with configuring the .cfg file and correctly naming the files so that the software can recognize them. We managed to convert the raw data to level L1A, but we are unable to proceed to L1AQC because we do not have a SOLARTRACKER available. For the SAS sensor, does processing up to level L2 require a similar input to the TriOS? Or is there a way to skip steps like L1AQC and L1BQC?

[oceancolorcoder]

@helena31415,

Do you have TriOS instruments as described in your post from 4 days ago or SAS (HyperSAS) instruments from SeaBird as indicated in yesterday's post? Or both?

It is possible to process either in HyperCP without any azimuth robot (SOLARTRACKER, pySAS, SoRAD, DALEC, etc.) See the sample data and configuration provided with the repository called "sample_TRIOS_NOTRACKER". However, please note that, as described in the README section 3, data collected without an autonomous platform requires an ancillary file to provide GPS position and relative azimuth, at a minimum. Example ancillary files are provided with the repository.

For setting up your configuration properly for your instruments and setup, I encourage you to read through the Configuration README.

[helena31415]

Hi @oceancolorcoder! 👋

First of all, thank you so much for your time and patience—we really appreciate your help!

Context:
We’re working with TriOS instruments (as mentioned a few days ago) and have successfully processed our data up to L1AQC and L1BQC. However, we’re stuck at the L2A processing step and encountered an error.

---

"Process Single Level
ProcessL2: /home/de-sousa/Documents/Doutorado/00_database/in_situ_data/fieldwork_lasa_simcosta/field_20250227/BG_10/L1BQC/20250227_101100_L1BQC.hdf
stationsEnsemblesReflectance
Binning datasets to ensemble time interval.
Convolving MODIS Aqua (ir)radiances in the slice
11 spectra in slice (ensemble).
Calculating M99 glint correction with complete LUT
Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc
TriOS factory uncertainties not implemented
Traceback (most recent call last):
File "/home/de-sousa/Documents/hypercp/Main.py", line 518, in singleL2Clicked
self.processSingle("L2")
File "/home/de-sousa/Documents/hypercp/Main.py", line 487, in processSingle
Controller.processFilesSingleLevel(
File "/home/de-sousa/Documents/hypercp/Source/Controller.py", line 921, in processFilesSingleLevel
Controller.processSingleLevel(pathOut, fp, calibrationMap, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/de-sousa/Documents/hypercp/Source/Controller.py", line 778, in processSingleLevel
root = Controller.processL2(root,outFilePath)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/de-sousa/Documents/hypercp/Source/Controller.py", line 457, in processL2
node = ProcessL2.processL2(root,station)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/de-sousa/Documents/hypercp/Source/ProcessL2.py", line 2363, in processL2
if not ProcessL2.stationsEnsemblesReflectance(node, root,station):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/de-sousa/Documents/hypercp/Source/ProcessL2.py", line 2252, in stationsEnsemblesReflectance
if not ProcessL2.ensemblesReflectance(node, sasGroup, referenceGroup, ancGroup,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/de-sousa/Documents/hypercp/Source/ProcessL2.py", line 1796, in ensemblesReflectance
L1B_UNC = instrument.ClassBased(node, uncGroup, stats)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/de-sousa/Documents/hypercp/Source/ProcessInstrumentUncertainties.py", line 228, in ClassBased
ind_rad_wvl, nan_mask = self.read_uncertainties(
^^^^^^^^^^^^^^^^^^^^^
TypeError: cannot unpack non-iterable bool object"

---

1. The error mentions TriOS factory uncertainties not implemented—could this be the root cause?
2. The TypeError suggests an issue with unpacking a boolean—any advice on how to debug this?
3. Are there known workarounds or missing dependencies for TriOS L2A processing?

We’re still getting familiar with the pipeline, so any tips would mean a lot!

We’ve tried:

• Verifying input HDF5 files (L1BQC looks valid).
• Testing different ancillary settings (no change in error).
  Next steps?

[oceancolorcoder]

Hi @helena31415,

The TriOS factory uncertainties message is unrelated. Unless you have a fully characterized set of radiometers and use the FRM Full Characterization regime, or your have an absolute radiometric calibration (RADCAL) with uncertainty and use the FRM Class-based regime, we unfortunately have no good way to propagate uncertainties for TriOS instruments. (The standard error should be propagated to the L2 file, however.) You appear to have a simple factory calibration and have it set to that regime in L1B, so that's okay. That said, I need to remind myself why ProcessInstrumentUncertainties.py (where your crash happens) is called at all for Factory regime with TriOS...

Here are a couple of things I noticed in your configuration that may not fix this issue, but you should be aware of them.

First, you are using an older version of HyperCP. I recommend you "git pull" to get up to date with the latest version (v1.2.12). This may require you to rebuild your environment, depending on how old your current version is (v1.2.10 or older, I believe; check for j6s dependency in your environment.yml). I would try updating and test if you still have the error.

There are a few unconventional settings, such as not filtering on relative azimuth or using models to fill in ancillary data that may be justifiable in your case. Not filtering on SZA minimum may be okay if you don't encounter low SZA, but could otherwise introduce large errors. Filtering out spectral outliers in L1BQC is recommended, though you may need to adjust the filter for you data (the sigma/window defaults tends to be robust for most collections). The short ensemble time can be problematic for slow acquisition systems, yielding low numbers of spectra in the average, particularly if the glitter filter (Percent Lt) is used (recommended in protocols). In you case, with the glitter filter off, you are still getting 11 spectra in the ensemble, which is okay and unlikely the cause of your error.

That is not an error I have encountered before, but I will bring it up with the team tomorrow to see if we can't get you back in business.

[helena31415]

ok, my team and I are looking into it and waiting for your future advise. If we figure it out I will let you know. Thank you very much.