Flow calc and initial site parser code mods by sauuyer · Pull Request #24 · WHOIGit/ooicgsn-data-tools

sauuyer · 2026-01-27T17:54:41Z

Files to Review

IFCB_Log-Discrete_IFCB_Data_Processing.ipynb
IFCB_Log-Underway_IFCB_Data_Processing.ipynb

Underway and Discrete Log Notebook Updates Summaries

Reasoning: syringeSamplingSpeed (McLane sets this to 20 minutes, i.e., this is a fixed 20 in the header file, which is not always accurate); 20 minutes is accurate in the cases where the sampling volume is 5 mL, but becomes inaccurate when the sampling times are altered. The more operationally precise way to view this is to focus on the default flow rate being 0.25 mL/min (which is true, and is usually accurately represented by the defaults (5 mL SyringeSampleVolume / syringeSamplingSpeed). 0.25 mL/min is only NOT accurate in cases where runSampleFast_Int is changed from 1 (the default) AND runSampleFast == TRUE.

SyringeSampleVolume (this is usually 5 ml unless the sample time is cut short by the operator (usually for testing reasons)
base_flow_rate = 0.25

underway_IFCB_hdr_output['flowRate_mins'] = np.where(
underway_IFCB_hdr_output['runSampleFast'],
base_flow_rate * underway_IFCB_hdr_output["RunFastFactor"],
base_flow_rate
)

New volume analyzed calculation:
underway_IFCB_hdr_output['volumeAnalyzed'] = underway_IFCB_hdr_output['flowRate_mins'] * (underway_IFCB_hdr_output['lookTime']/60)

Underway Notebook Change Summary
";" sep added between sites in cases where there are two sites (e.g., CP10CNSM;CP12CNSW)

ingest_csvs v2, adds users ability

ingest: fixes <dataFrame>.append deprecation

b1de362 Merge pull request #3 from sauuyer/notebook-cleaning 5318d90 flow rate calc updated. site indicator updated for underway file. ee64317 Rename IFCB_Log-Moored_IFCB_Data_Processing.ipynb to IFCB_Log-Underway_IFCB_Data_Processing.ipynb 44c3b34 Rename IFCB_Log-Underway_IFCB_Data_Processing.ipynb to IFCB_Log-Moored_IFCB_Data_Processing.ipynb d90457b added notes about fast factorsettings in md a6aa809 Merge pull request #2 from sauuyer/pioneer-22 1680167 Delete IFCB_Log_Notebooks/IFCB_Dashboard_Metadata_CSV_shipboard_AR87.ipynb c3aa3be Delete IFCB_Log_Notebooks/IFCB_Dashboard_Metadata_CSV_shipboard_AR82.ipynb ab2f2fb updated log notebooks 8cdffda moored and shipboard notebooks updated 0f9ae6c cleaned up variable updates 14e107a Merge pull request #1 from sauuyer/notebook-cleaning 9a7faba Updated code in jupyter notebooks that create dashboard metadata git-subtree-dir: ifcb-parsing-processing git-subtree-split: b1de362d6e2ca08d69a040177d4eae78273d5133

…-parsers Flow rate code changes added through subtree of personal repo

cooleyky

Updates summarized in the PR look good. A future PR should include adding the .ipynb_checkpoints directory to .gitignore

cooleyky · 2026-02-03T15:00:26Z

ifcb-parsing-processing/IFCB_Log_Notebooks/IFCB_Log-Underway_IFCB_Data_Processing.ipynb

Updates to site tags and volume analyzed calculation worked as expected.

cooleyky · 2026-02-03T15:01:14Z

ifcb-parsing-processing/IFCB_Log_Notebooks/IFCB_Log-Discrete_IFCB_Data_Processing.ipynb

Updates to volume analyzed calculation worked as expected.

sbeaulieu · 2026-02-03T19:29:43Z

ifcb-parsing-processing/IFCB_Log_Notebooks/IFCB_Log-Discrete_IFCB_Data_Processing.ipynb

    "\n",
    "# Create a column for volumeAnalyzed\n",
-    "discrete_IFCB_hdr_output['volumeAnalyzed'] = (discrete_IFCB_hdr_output['RunFastFactor'] * discrete_IFCB_hdr_output['runSampleFast_Int']) * discrete_IFCB_hdr_output['flowRate_mins'] * (discrete_IFCB_hdr_output['lookTime']/60)/5\n",
+    "#discrete_IFCB_hdr_output['volumeAnalyzed'] = (discrete_IFCB_hdr_output['RunFastFactor'] * discrete_IFCB_hdr_output['runSampleFast_Int']) * discrete_IFCB_hdr_output['flowRate_mins'] * (discrete_IFCB_hdr_output['lookTime']/60)/5\n",


confirmed with Sawyer in Slack that we no longer need the runSampleFast_Int variable

sbeaulieu · 2026-02-03T19:30:18Z

ifcb-parsing-processing/IFCB_Log_Notebooks/IFCB_Log-Discrete_IFCB_Data_Processing.ipynb

+    "## SyringeSampleVolume (this is usually 5 ml unless the sample time is cut short by the operator (usually for testing reasons)\n",
+    "base_flow_rate = 0.25\n",
    "\n",
+    "discrete_IFCB_hdr_output['flowRate_mins'] = np.where(\n",


sbeaulieu · 2026-02-03T19:30:45Z

ifcb-parsing-processing/IFCB_Log_Notebooks/IFCB_Log-Underway_IFCB_Data_Processing.ipynb

+    "## SyringeSampleVolume (this is usually 5 ml unless the sample time is cut short by the operator (usually for testing reasons)\n",
+    "base_flow_rate = 0.25\n",
+    "\n",
+    "underway_IFCB_hdr_output['flowRate_mins'] = np.where(\n",


sbeaulieu · 2026-02-03T19:30:59Z

ifcb-parsing-processing/IFCB_Log_Notebooks/IFCB_Log-Underway_IFCB_Data_Processing.ipynb

-    "    'lat': [35.95, 36.175, 35.725, 35.95, 35.95, 35.95, 36.175, 35.725, 36.0536, 35.8514],\n",
-    "    'lon': [-75.125, -74.8267, -74.853, -75.125, -75.3333, -74.8457, -74.8267, -74.853, \n",
-    "            -74.7776, -74.8482]\n",
+    "    'site': ['CP10CNSM;CP12CNSW', 'CP11NOSM;CP13NOPM', 'CP11SOSM;CP13SOPM', 'CP12WESW', 'CP13EAPM', \n",


… from notebooks, updated LAT/LON file

cooleyky

Looks good

cooleyky · 2026-02-03T20:07:35Z

.gitignore

 *.py,cover
 .hypothesis/
 .pytest_cache/
+.ipynb_checkpoints


cooleyky · 2026-02-03T20:13:34Z

...rsing-processing/IFCB_Log_Notebooks/IFCB_Log-Underway_IFCB_Data_Processing_with_LATLON.ipynb

updates look good

cooleyky · 2026-02-03T20:16:12Z

ifcb-parsing-processing/IFCB_Log_Notebooks/IFCB_Log-Discrete_IFCB_Data_Processing.ipynb

updates to remove runSampleFast_Int look good.

cooleyky · 2026-02-03T20:16:47Z

ifcb-parsing-processing/IFCB_Log_Notebooks/IFCB_Log-Underway_IFCB_Data_Processing.ipynb

Looks good with runSampleFast_Int removed

sbeaulieu · 2026-02-04T14:14:31Z

...rsing-processing/IFCB_Log_Notebooks/IFCB_Log-Underway_IFCB_Data_Processing_with_LATLON.ipynb

+    "## syringeSamplingSpeed (usually 20 mins), SyringeSampleVolume (usually 5 ml)\n",
+    "underway_IFCB_output_with_calcs['flowRate_mins'] = underway_IFCB_output_with_calcs['SyringeSampleVolume'] / underway_IFCB_output_with_calcs['syringeSamplingSpeed']\n",
+    "\n",
+    "underway_IFCB_hdr_output_with_calcs['flowRate_mins'] = np.where(\n",


sbeaulieu · 2026-02-04T14:17:10Z

...rsing-processing/IFCB_Log_Notebooks/IFCB_Log-Underway_IFCB_Data_Processing_with_LATLON.ipynb

+{
+ "cells": [
+  {
+   "cell_type": "code",


for next version perhaps consider adding markdown cell at top to describe that this one notebook combines the two notebooks "IFCB_Log-Underway_IFCB_Data_Processing.ipynb" and "Ship_Underway_Lat_and_Lon_Processing.ipynb"

s-pearce and others added 11 commits March 14, 2022 10:09

ingest_csvs v2, adds users ability

d1ace49

corrects extra typo

894d1ad

rearranges text indentation

d6a3904

removes unecessary repititions of netrc usage note and version info

5975e43

Merge pull request #15 from s-pearce/ingest_script_update

b05164f

ingest_csvs v2, adds users ability

ingest: fixes <dataFrame>.append deprecation

0b52aad

Merge pull request #16 from s-pearce/ingest_pd_warning_fix

8d372c7

ingest: fixes <dataFrame>.append deprecation

Merge commit '21b7bc568d3f66e2b78dd226b0a4effbac643162' into add-ifcb…

6b8b16e

…-parsers Flow rate code changes added through subtree of personal repo

flow rate calc changes and site seperator added

17f1b7b

Merge branch 'oceanobservatories:master' into add-ifcb-parsers

2df0b2a

sauuyer requested review from cooleyky and sbeaulieu January 27, 2026 17:54

sauuyer added the enhancement New feature or request label Jan 27, 2026

cooleyky previously approved these changes Feb 3, 2026

View reviewed changes

sbeaulieu previously approved these changes Feb 3, 2026

View reviewed changes

sauuyer added 2 commits February 3, 2026 14:54

added pythoncheckpoints file to .gitignore, removed runSampleFast_Int…

ef61f81

… from notebooks, updated LAT/LON file

pulled ingest_csvs.py update from remote

3619aff

sauuyer dismissed stale reviews from sbeaulieu and cooleyky via 3619aff February 3, 2026 20:00

sauuyer requested review from cooleyky and sbeaulieu February 3, 2026 20:02

cooleyky approved these changes Feb 3, 2026

View reviewed changes

sbeaulieu approved these changes Feb 4, 2026

View reviewed changes

Conversation

sauuyer commented Jan 27, 2026

Uh oh!

cooleyky left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cooleyky left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants