Archive data files

[RachaelDMueller]

Data files to archive:

• aggregated results
• random oil spills file used to reproduce results in manuscript
• inputs to Part1_plot_statistical_spills.ipynb
• any others?

From Karyn's document

Research Data Repository:

SalishSeaCast model results and model forcing fields are available online at http://salishsea.eos.ubc.ca/erddap/griddap/index.html and this link can be provided in your data availability statement. However, if aggregate files were created (e.g., monthly averages) or model experiments were carried out, these data files need to be deposited in a research data repository.

UBC Library provides information on research data repositories:

https://researchdata.library.ubc.ca/deposit/

MOAD group members commonly store data in The Federated Research Data Respository.

[RachaelDMueller]

Other adds:

• Look in oil_attribution.ipynb and include this notebook/.yaml in repository
• Add files on Salish:/data/MIDOSS
• https://github.com/MIDOSS/MIDOSS-MOHID-config/blob/main/monte-carlo/random-oil-spills.yaml
• make note of ERDAAP resource (winds, currents, waves)
• Add (copy) Casey's vessel track join script (/data/MIDOSS/origin_destination_analysis)

[RachaelDMueller]

Currently, the graphic in Part1_plot_statistical_spills.ipynb need to be generated on UBC machines. Need to consolidate source files so this can be run elsewhere too.

[RachaelDMueller]

Part 1: Open Research Section

Current Version

I’m not even going to write it because it is all the generic stuff in Part 2 and is missing any mention of data files or location of data files. Problem: This is the paper in which most of the data files were used.

If possible, this manuscript needs to be updated to reference data files.

Alternatively, I can link the data repository on Git and mention their location in the Part 1 notebooks/scripts

[RachaelDMueller]

Part 2: Open Research Section

Current Version

SalishSeaCast, HRDPS, and WW3 model output used in MOHID model runs is accessible through ERDDAP at https://salishsea.eos.ubc.ca/erddap/index.html. Washington state oil transfer data is publicly accessible by request through https://ecology.wa.gov/Footer/Public1445-records-requests. Aggregated model output from MOHID is accessible through The Federated Research Data Respository at https://www.frdr-dfdr.ca/. All software used to develop this manuscript is available through GitHub [reference to GitHub repo].

[RachaelDMueller]

Part1 datafiles required

Summary: 15.5 G total space needed with ~1G as the largest file size.

oil_attribution.ipynb files:

root = '/data/MIDOSS/'

• data = root / ‘marine_transport_data/‘
• shapefiles = root / 'shapefiles/‘
• spreadsheets = root / ‘spreadsheets’

1. [1.5M] marine_transport_data

• 976K: Largest filesize (Origin_Destination_Analysis_updated.xlsx)
• 18: Total number of files
• Location: /data/MIDOSS/marine_transport_data

2. [15G = 19G - 4G]shapefiles (Can’t add per copyright)

• 1017M: Largest filesize (barge_2018_12.dbf)

• 868: Total number of files

• Location: /data/MIDOSS/shapefiles

  There are three .zip files that I’m guessing should NOT be included in the paper archive because they read more as source data than the derived products that we used in this manuscript. . They are:

• [1.5G] VesselTrackShapefiles-WithDates.zip

• [1.4G] VesselTrackShapefiles-BadGeometry.zip

• [1.2G] VesselTrackShapefiles.zip

Side note: Something strikes me as odd in the barge .dbf files when listing with "ls -lhS | more". The largest three files are oct-dec (840-1020M) and the smallest two files are jan & feb (350-500M). Same with the .shp files. ATBs follow a similar pattern. Perhaps a large increase in traffic over this year? Market change? Regulation change? Probably more a market change because the tanker and ATB files all show a similar size increase through 2018 (i.e. not a change in cargo ship type). It is probably too late to mention in this paper but if the barge tank traffic can vary so much in a year, and if these trends differ between years, then our barge spills may be biased to some months (and environmental conditions) in a way that would be different in a different year. Barge volume and spill count is only exceeded by tanker traffic...so...this potential bias is worth noting, albeit perhaps too late for this paper.

3. [183K] spreadsheets

• 183K: Largest filesize (ATB_MMSIs.xlsx)
• 1: Total number of files
• Location: /data/MIDOSS/spreadsheets

Side note: There are 5 spreadsheets here but only one that we will archive b/c the others are from ECY (accessible by public request)

random_oil_spills (GeoTIFFs):

4. [50M] /data/MIDOSS/geotiffs

• 16M: Largest filesize (VesselTimeExposureGeoTIFFs.zip) <-- ONLY COUNTING THIS ONE IN TOTAL
• 591: Total number of files (this is odd because there are 593 files in the .zip file)
• Location: /data/MIDOSS/shapefiles

WW3 Forcing

Not counting. Not including. LMK if we ought to.

Spills files

5. [13.5M] /Users/rmueller/Projects/MIDOSS/analysis_files/

-rw-rw-r--  1 rmueller  staff   1.5M Sep 18  2021 SalishSea_oil_spills_6.csv
-rw-rw-r--  1 rmueller  staff   1.5M Sep 18  2021 SalishSea_oil_spills_7.csv
-rw-rw-r--  1 rmueller  staff   1.5M Sep 18  2021 SalishSea_oil_spills_2.csv
-rw-rw-r--  1 rmueller  staff   1.5M Sep 18  2021 SalishSea_oil_spills_3.csv
-rw-rw-r--@ 1 rmueller  staff   1.5M Sep 17  2021 SalishSea_oil_spills_1.csv
-rw-rw-r--  1 rmueller  staff   1.5M Sep 18  2021 SalishSea_oil_spills_8.csv
-rw-rw-r--  1 rmueller  staff   1.5M Sep 18  2021 SalishSea_oil_spills_4.csv
-rw-rw-r--  1 rmueller  staff   1.5M Sep 18  2021 SalishSea_oil_spills_9.csv
-rw-rw-r--  1 rmueller  staff   1.5M Sep 18  2021 SalishSea_oil_spills_5.csv

[RachaelDMueller]

Part2: Aggregated results

Summary: 4.1G total space needed with ~1G as the largest file size.

On my MacBook: /Users/rmueller/Projects/MIDOSS/analysis_files/aggregated_spills_output/

-rw-r-----  1 rmueller  staff   822M Jan  1  2022 fiftieth_55_oils.nc
-rw-r-----  1 rmueller  staff   822M Jan  1  2022 fiftieth_55_diesel.nc
-rw-r-----  1 rmueller  staff   821M Jan  1  2022 fiftieth_55_bunker.nc
-rw-r-----  1 rmueller  staff   821M Jan  1  2022 fiftieth_55_akns.nc
-rw-r-----  1 rmueller  staff   821M Jan  1  2022 fiftieth_55_dilbit.nc

[RachaelDMueller]

Globus -> FRDR archiving progress

Part 1

• geotiffs
  • VesselTimeExposureGeoTIFFs.zip
• spreadsheets
  • spill_event_probability.xlsx
  • ATB_MMSIs.xlsx
  • ATB_fullness.xlsx
  • ATB_oil_capacity.xlsx
  • Oil_Transfer_Facilities.xlsx
  • Origin_Destination_Analysis_updated.xlsx
  • Past_Spill_Events_and_Vessel_Info.xlsx
  • oil_capacity.xlsx
  • spill_event_probability.xlsx
• oil_attribution_yaml_files
  • CAD_origin.yaml
  • Pacific_origin.yaml
  • US_general.yaml
  • US_origin.yaml
  • WA_destination.yaml
  • WA_origin.yaml
  • barge_capacity.yaml
  • barge_capacity_weights.yaml
  • fill_fraction.yaml
  • fuel_by_vessel.yaml
  • handymax_capacity.yaml
  • oil_attribution.yaml
• 9 iterations of random spills
  • SalishSea_oil_spills_[1-9].csv

Part 2

• aggregated_spills_output
  • fiftieth_55_oils.nc
  • fiftieth_55_diesel.nc
  • fiftieth_55_bunker.nc
  • fiftieth_55_akns.nc
  • fiftieth_55_dilbit.nc

[RachaelDMueller]

• Reviewed and tidied up Part2 code for Figs 1-5.
• Added massbalance_byoil_16062022_03/20/07.yaml to data repository and corrected header information.
• Updated paths to be relative and with data repository in same directory as the analysis repository such that the code is more transferrable.

Still need to:

• Update/review Part 2 Figs 6+ code
• Review Part 1 code
• Review method for Fig 5(c) and make sure it’s what we want (I am worried I may have messed something up here that needs correcting. Hoping not but want to double check.)
• Update FRDR files and make recommended changes