In tbd and sbd files, the first record in the binary file is an initialization record that sets all of the variables to an intial value. In the tbd file, this initialization record has some valid data (e.g. sci_m_disk_usage), but most of the instrument variables initialize with zeros (e.g. sci_water_cond, sci_oxy4_oxygen, etc.), which should be removed. In addition to the initialization record, occasionally, the same science instrument variables may all have a value of zero and these should be removed. After the initialization record however, this must be done for each instrument, since each instrument might have a different sampling rate (e.g. 0.5 Hz for a CTD and 1 Hz for a Fluorometer) and may not all align.
I find the best method is to find where all the primary variables (but not necessarily all variables) from an instrument are all valued at zero at the same timestamp. By finding where they are all valued at zero, it eliminates the case where one value may be valid at zero by itself, such as temperature or pressure. For example using the CTD, you can use
"""
assuming you already have temperature, conductivity, and pressure from the CTD
as variables temp, cndc, and pres respectively, set these values at these timestamps
zero.
Note: .reduce allows more than 2 conditionals to pass into the AND statement at once
zeros_indices = np.logical_and.reduce((temp == 0.0, cndc == 0.0, pres == 0.0))
temp[zeros_indices] = np.nan
cndc[zeros_indices] = np.nan
pres[zeros_indices] = np.nan
"""
For other instrument examples the primary variables you might use could be:
sci_oxy4_oxygen and sci_oxy4_saturation for the Aanderaa 4831sci_flbbcd_chlor_units, sci_flbbcd_bb_units, and sci_flbbcd_cdom_units for the WETLabs/Sea Bird ECO Puck FLBBCD
This isn't being done at present in glide, but would be a good addition.
In tbd and sbd files, the first record in the binary file is an initialization record that sets all of the variables to an intial value. In the tbd file, this initialization record has some valid data (e.g.
sci_m_disk_usage), but most of the instrument variables initialize with zeros (e.g.sci_water_cond,sci_oxy4_oxygen, etc.), which should be removed. In addition to the initialization record, occasionally, the same science instrument variables may all have a value of zero and these should be removed. After the initialization record however, this must be done for each instrument, since each instrument might have a different sampling rate (e.g. 0.5 Hz for a CTD and 1 Hz for a Fluorometer) and may not all align.I find the best method is to find where all the primary variables (but not necessarily all variables) from an instrument are all valued at zero at the same timestamp. By finding where they are all valued at zero, it eliminates the case where one value may be valid at zero by itself, such as temperature or pressure. For example using the CTD, you can use
"""
assuming you already have temperature, conductivity, and pressure from the CTD
as variables temp, cndc, and pres respectively, set these values at these timestamps
zero.
Note: .reduce allows more than 2 conditionals to pass into the AND statement at once
zeros_indices = np.logical_and.reduce((temp == 0.0, cndc == 0.0, pres == 0.0))
temp[zeros_indices] = np.nan
cndc[zeros_indices] = np.nan
pres[zeros_indices] = np.nan
"""
For other instrument examples the primary variables you might use could be:
sci_oxy4_oxygenandsci_oxy4_saturationfor the Aanderaa 4831sci_flbbcd_chlor_units,sci_flbbcd_bb_units, andsci_flbbcd_cdom_unitsfor the WETLabs/Sea Bird ECO Puck FLBBCDThis isn't being done at present in
glide, but would be a good addition.