RIOT acoustics

[jessecusack]

@s-pearce I think we should add a new 'riot' subcommand to glide that parses the tbd.nc + sbd.nc + ini file and spits out an acoustic csv file.

[jessecusack]

Here is my current understanding of the format of the CSV file. I deduced this from the GitLab documentation and the Spray glider csv.

A line from the Spray glider CSV looks like this. It consists of 11 columns.

$riotData,1772132200020,0,0,0,89710,0,2,125,10,0

Jenn said that she uses the following python code to create each line:

riot_data_times.append(int(riot_data_parts[1]))
riot_data_traveltime.append(int(riot_data_parts[2]))
riot_data_freq.append(int(riot_data_parts[3]))
riot_data_detectionlevel.append(int(riot_data_parts[4]))
riot_data_sequencenumber.append(int(riot_data_parts[5]))
riot_data_platformid.append(riot_data_parts[6])
riot_data_slot.append(riot_data_parts[7])
riot_data_group.append(riot_data_parts[8])
riot_data_platformstate.append(riot_data_parts[9])
riot_data_recordnumber.append(riot_data_parts[10])

I deduce this must produce the 10 columns following $riotData. I believe the $riotData element is literally returned in the console output after sending the command pollData. I imagine it was left in to distinguish from $riotStatus which is returned by the command pollStatus.

The GitLab documentation indicates the following C struct containing 9 variables. It does not include a record number which is being added to the Spray CSV output.

typedef struct riot_data_format_t{ // TODO add group and slot
    int64_t    epochMsecs;    // s- of interrogation (Active)
                              //       interrogation Rx (Listening Active and Passive)
    uint32_t   rtMsecs;       // tenths of a mS
    uint32_t   freq;          // Hz
    uint16_t   detectionLevel; // rx'd ping, std. dev. above mean noise level
    uint32_t   sequenceNumber; // as tracked by RIOT
    uint8_t    platformId;     // can use modem ID
    uint8_t    slot;
    uint8_t    group;
    platform_states platformState; // as enumerated above, really message type.
    // Current data size: 29 bytes
} riot_data_format_t;

I deduce the following structure and units for the RIOT data CSV. If we were to give the csv file verbose column headings they would be:

Output type (-), Time (posix), Two-way travel time (0.1ms), Frequency (Hz), Detection level (-), Sequence number (-), Platform ID (-), Slot (-), Group (-), Platform state (-), Record number (-)
$riotData,1772147825000,126355,9750,106,89835,0,2,125,1,3

[jessecusack]

Jessica pointed to the following info from GitLab

    raw_columns = [
        "riotData", "epochMsecs", "rtMsecs", "freq", "detectionLevel",
        "sequenceNumber", "platformId", "slot", "group", "platformState",
        "recNumInFile", "depth", "lat", "lon"
    ]
    
    final_order = [
        "platformName", "platformId", "epochMsecs", "lat", "lon", "depth", 
        "rtMsecs", "freq", "detectionLevel", "platformState", "slot", 
        "group", "sequenceNumber", "recNumInFile"
    ]

[jessecusack]

Trevor and Ben R clarified the record number:

Trevor: The record number indicates how many records were available on the riot bottle when the platform computer queried the riot bottle. In the documentation, Justin has included an example of the pollData queries, a process which - if I understand correctly - is performed on some schedule by the host platform. Each time, the log number increases from 0 until the queue is empty, but then resets to 0 once emptied, resulting in the non-monotonically increasing value over the entire assembled log. Also, at the end of the parsing the pollData section, he clarifies that it is purely a "if you want it." However, the other gliders are including it, so to maintain the log size, I'd request that you do as well.

Ben: I'll add some not-too-valuable two cents in on record number. Trevor - you are correct. yes, it is on a schedule dictated by host (for spray2s so far, that has been every 5 minutes - shore-settable).I included recNum as a field in the struct I store on the glider (2-byte signed int), and likewise pass it to shore and right into the log data files. agreed that it probably won't be used for much ever, but it's an easy way to see how many elements the "poll data batches".

[jessecusack]

To summarize the conversation from yesterday @s-pearce, implementation plan is (roughly):

• New arguments to l2. These might need to be something like riot_out=None, riot_extmap=None, riot_add_position=True.
• Some logic which does nothing if the output file is None.
• If output file is specified and extmap is provided then extract sci generic parameters from raw science file and format for csv.
• If add position is True then interpolate the adjusted dead-reckoned lat, lon and depth to the times of the RIOT acoustics and add as columns.
• If the riot output file already exists, append the output to the bottom (do not overwrite)? (I think there was a request for an a single appended file rather than many individual files)

@s-pearce what do you think?

[s-pearce]

I think we should keep the extmap feature simple and that is controlled by a riot_config.yml file for riot deployments since that is how you built the feature to add variables and rename them. If we really believe that extmap might change during all of this and you want to submit the extmap.ini file to update, I would instead suggest a command similar to glider cfg -o cfgfilename.yml that would take extmap.ini and produce a riot_config.yml.

Otherwise, that is all doable and I have already been working on it.

I am almost of the opinion that the riot acoustic data csv file should be a separate script from glide anyway, to keep glide generic and controllable by configuration files. But I have been working on a riot_csv_writer branch anyway. We can see how we like it and could even keep it just as a branch. Then we can just install the particular branch on our riot dataserver that will push to the riot AWS storage.