Added Landsat 5 & 7 to TMS-OS; Added NSSC calculations to RAT

docs/Configuration/basins_metadata.md

@@ -6,9 +6,9 @@ To implement this feature RAT uses `basins_metadata` which is a csv file that is
 
 ![Screenshot of Basins_Metadata.csv](../images/configure/basins_metadata_sample.jpg)
 
-It should definitely have the index `basin_name` for section `BASIN` as it needs to be different for all basins and same is true for `basin_id`. Rest all those parameters which do not vary between basins can be provided in the configuration file and those which will vary must be provided in the `basins_metadata` csv file. 
+It should definitely have the index `basin_name` for section `BASIN` as it needs to be different for all basins and same is true for `basin_id`. Rest all those parameters which do not vary between basins can be provided in the configuration file and those which will vary must be provided in the `basins_metadata` csv file. A special index `run`for section `BASIN` must be used in `basins_metadata` and it's value should be set to 1 to indicate which river basins should RAT run, otherwise 0.
 
 !!! tip_note "Tip"
     1. To make use of `basins_metadata`, `multiple_basin_run` in `GLOBAL` section should be `true`.
-    2. The values of `basin_name` from  `basins_metadata` should be provided in the list `basins_to_process` for all the river basins for which you want to run RAT for.
+    2. The values of `run` in `BASIN` must be provided as either 1 or 0 for each basin in the `basins_metadata` where 1 is for all the river basins for which you want to run RAT for.
     3. A sample copy of `basins_metadata` is provided in 'params' folder inside `project_dir` with the name of 'basins_metadata_sample.csv'.

docs/Configuration/rat_config.md

@@ -37,7 +37,22 @@ RAT config file has 12 major sections that defines several parameters which are
             -1
             -2
     ```
-
+
+* <h6 class="parameter_heading">*`cleaning`* :</h6> 
+    <span class="requirement">Optional parameter</span>
+
+    <span class="parameter_property">Description </span>: Boolean flag indicating whether to do cleaning during RAT run. If True, it cleans the data specified by the [`CLEAN UP`](#clean-up) section in the configuration file.
+
+    <span class="parameter_property">Default </span>: False 
+
+    <span class="parameter_property">Syntax </span>: If you want to delete data produced by RAT and want to use `CLEAN UP` section, then
+    ```
+    GLOBAL:
+        cleaning: True
+    ```
+    !!! note
+    If you want to use [`CLEAN UP` section](#clean-up), `cleaning` must be True.
+
 * <h6 class="parameter_heading">*`project_dir`* :</h6> 
     <span class="requirement">Required parameter</span>
 
@@ -143,19 +158,6 @@ RAT config file has 12 major sections that defines several parameters which are
         basins_metadata: /Cheetah/rat_project/params/basins_metadata_sample.csv
     ```
 
-* <h6 class="parameter_heading">*`basins_to_process`* :</h6> 
-    <span class="requirement">Required parameter</span>
-
-    <span class="parameter_property">Description </span>: List of basins to run RAT for within the `basins_metadata`. The list values must match with the values of `basin_name` in `BASIN` section in `basins_metadata`.  For more information, please look [multiple basin run](../basins_metadata).
-
-    <span class="parameter_property">Default </span>: It is blank by default and can be filled by the user.
-
-    <span class="parameter_property">Syntax </span>: If you want to run RAT for basins Sabine and Nueces, then
-    ```
-    GLOBAL:
-        basins_to_process: ['Sabine','Nueces']
-    ```
-
 ### Basin 
 
 * <h6 class="parameter_heading">*`region_name`* :</h6> 
@@ -712,38 +714,74 @@ This section of the configuration file describes the parameters defined by `rout
     2. If `station_global_data` is `False`, AEC file names should be <'dam_name_column' value where spaces are replaced by '_'>. For example, the file name for a reservoir with 'dam_name' as 'Tehri Dam' will be 'Tehri_Dam.csv'.
     3. Each AEC file should have two columns with headers as 'Elevation' and 'CumArea'. 'Elevation' should be in meters and 'CumArea' should be in square Kilometers.
 
+* <h6 class="parameter_heading">*`catchment_vector_file`* :</h6> 
+    <span class="requirement">Optional parameter</span>
+
+    <span class="parameter_property">Description </span>: Absolute path of the catchment vector file where the geometry is represented by reservoirs' catchment polygons. It can be "global" (relative to basin) and will be automatically filtered for the basin. It can have unique id column and dam name column.  
+
+    <span class="parameter_property">Default </span>: It is blank by default and can be filled by the user.
+
+    <span class="parameter_property">Syntax </span>: If `catchment_vector_file` has the path *'/Cheetah/rat_project/custom_files/catchments.geojson/'*, then
+    ```
+    POST_PROCESSING:
+        catchment_vector_file : /Cheetah/rat_project/custom_files/catchments.geojson
+    ```
+
+* <h6 class="parameter_heading">*`catchment_vector_file_columns_dict`* :</h6> 
+    <span class="requirement">Optional parameter</span>
+
+    <span class="parameter_property">Description </span>: Dictionary of column names for `catchment_vector_file`. The dictionary must have keys 'id_column' and 'dam_name_column' and their values should be the actual name of the corresponding columns respectively.   
+
+    <span class="parameter_property">Default </span>: `{id_column : 'GRAND_ID', dam_name_column : 'DAM_NAME'}` 
+
+    <span class="parameter_property">Syntax </span>: If `catchment_vector_file` has column names 'GRAND_ID' and 'DAM_NAME', then 
+    ```
+    POST_PROCESSIN:
+        catchment_vector_file_columns_dict: {id_column : 'GRAND_ID', dam_name_column : 'DAM_NAME'}
+    ```
+    or
+    ```
+    POST_PROCESSIN:
+        catchment_vector_file_columns_dict: 
+            id_column: GRAND_ID
+            dam_name_column: DAM_NAME
+    ```
+
 ### Clean Up
 
-* <h6 class="parameter_heading">*`clean_preprocessing`* :</h6> 
-    <span class="requirement">Required parameter</span>
+!!!note
+    To use this section, `cleaning` should be set to True in [Global Section](#global) of the configuration file. This section is Optional if `cleaning` is `False` or not provided in Global section.
+
+* <h6 class="parameter_heading">*`clean_processing`* :</h6> 
+    <span class="requirement">Optional parameter</span>
 
-    <span class="parameter_property">Description </span>: `True` if  you want to delete intermediate pre-processed data for a river basin except global raw data downloaded from servers after the RAT run. Otherwise, `False`.
+    <span class="parameter_property">Description </span>: `True` if  you want to delete intermediate pre-processed and post-processed data for a river basin except global raw data downloaded from servers after the RAT run. Otherwise, `False`.
 
     <span class="parameter_property">Default </span>: `False`
 
-    <span class="parameter_property">Syntax </span>: If you want to delete intermediate pre-processed data for a river basin, 
+    <span class="parameter_property">Syntax </span>: If you want to delete intermediate pre-processed and post-processed data for a river basin, 
     ```
     CLEAN_UP:
-        clean_preprocessing: True
+        clean_processing: True
     ```
 
 * <h6 class="parameter_heading">*`clean_metsim`* :</h6> 
-    <span class="requirement">Required parameter</span>
+    <span class="requirement">Optional parameter</span>
 
-    <span class="parameter_property">Description </span>: `True` if  you want to delete intermediate metsim outputs for a river basin after the RAT run. Otherwise, `False`.
+    <span class="parameter_property">Description </span>: `True` if  you want to delete intermediate metsim inputs and outputs for a river basin after the RAT run. Otherwise, `False`.
 
     <span class="parameter_property">Default </span>: `False`
 
-    <span class="parameter_property">Syntax </span>: If you want to delete intermediate metsim outputs for a river basin, 
+    <span class="parameter_property">Syntax </span>: If you want to delete intermediate metsim inputs and outputs for a river basin, 
     ```
     CLEAN_UP:
         clean_metsim: True
     ```
 
 * <h6 class="parameter_heading">*`clean_vic`* :</h6> 
-    <span class="requirement">Required parameter</span>
+    <span class="requirement">Optional parameter</span>
 
-    <span class="parameter_property">Description </span>: `True` if  you want to delete intermediate vic inputs and outputs, and any vic initial soil state file that is older than 15 days, for a river basin after the RAT run. Otherwise, `False`.
+    <span class="parameter_property">Description </span>: `True` if  you want to delete intermediate vic inputs and outputs, and any vic initial soil state file that is older than 20 days, for a river basin after the RAT run. Otherwise, `False`.
 
     <span class="parameter_property">Default </span>: `False`
 
@@ -754,9 +792,9 @@ This section of the configuration file describes the parameters defined by `rout
     ```
 
 * <h6 class="parameter_heading">*`clean_routing`* :</h6> 
-    <span class="requirement">Required parameter</span>
+    <span class="requirement">Optional parameter</span>
 
-    <span class="parameter_property">Description </span>: `True` if  you want to delete intermediate routing inputs and outputs, and any routing initial state file that is older than 15 days, for a river basin after the RAT run. Otherwise, `False`.
+    <span class="parameter_property">Description </span>: `True` if  you want to delete intermediate routing inputs and outputs, and any routing initial state file that is older than 20 days, for a river basin after the RAT run. Otherwise, `False`.
 
     <span class="parameter_property">Default </span>: `False`
 
@@ -767,7 +805,7 @@ This section of the configuration file describes the parameters defined by `rout
     ```
 
 * <h6 class="parameter_heading">*`clean_gee`* :</h6> 
-    <span class="requirement">Required parameter</span>
+    <span class="requirement">Optional parameter</span>
 
     <span class="parameter_property">Description </span>: `True` if  you want to delete gee produced small chunk files of surface area time series for a river basin after the RAT run. Otherwise, `False`.
 
@@ -782,7 +820,7 @@ This section of the configuration file describes the parameters defined by `rout
         If `clean_gee` is `True`, it will not delete the final gee outputs that will be appended with new data in next RAT run. To delete that, use `clean_previous_outputs`.
 
 * <h6 class="parameter_heading">*`clean_altimetry`* :</h6> 
-    <span class="requirement">Required parameter</span>
+    <span class="requirement">Optional parameter</span>
 
     <span class="parameter_property">Description </span>: `True` if  you want to delete raw altimetry data that takes a lot of time to download for a river basin after the RAT run. Otherwise, `False`.
 
@@ -796,8 +834,36 @@ This section of the configuration file describes the parameters defined by `rout
     !!!note
         If `clean_altimetry` is `True`, it will not delete the extracted altimetry data that will be appended with new data in next RAT run. To delete that, use `clean_previous_outputs`.
 
+* <h6 class="parameter_heading">*`clean_basin_parameter_files`* :</h6> 
+    <span class="requirement">Optional parameter</span>
+
+    <span class="parameter_property">Description </span>: `True` if  you want to delete parameter files related to the basin. Otherwise, `False`.
+
+    <span class="parameter_property">Default </span>: `False`
+
+    <span class="parameter_property">Syntax </span>: If you want to delete parameter files related to basin, 
+    ```
+    CLEAN_UP:
+        clean_basin_parameter_files: True
+    ```
+
+* <h6 class="parameter_heading">*`clean_basin_meteorological_data`* :</h6> 
+    <span class="requirement">Optional parameter</span>
+
+    <span class="parameter_property">Description </span>: `True` if  you want to delete combined meteorological data of the basin which is required to hot-start RAT for next RAT run. Otherwise, `False`.
+
+    <span class="parameter_property">Default </span>: `False`
+
+    <span class="parameter_property">Syntax </span>: If you want to delete combined meteorological data of the basin, 
+    ```
+    CLEAN_UP:
+        clean_basin_meteorological_data: True
+    ```
+    !!! note
+        It should be noted that combined meteorological data is the data that helps in running VIC (hydrological model) without any spin-up for next run. Without this data, RAT needs to run the hydrological model for extra 2-3 years. You should use `clean_basin_meteorological_data` if you want to recreate combined meteorological data file for a river basin in the next RAT run.
+
 * <h6 class="parameter_heading">*`clean_previous_outputs`* :</h6> 
-    <span class="requirement">Required parameter</span>
+    <span class="requirement">Optional parameter</span>
 
     <span class="parameter_property">Description </span>: `True` if  you want to delete previous outputs, gee extracted surface area time series and altimetry extracted height data produced by last RAT run. Otherwise, `False`.
 
@@ -828,4 +894,8 @@ This section of the configuration file describes the parameters defined by `rout
         secrets: /Cheetah/rat_project/secrets/secrets.ini
     ```
     !!! note 
-        It will be left blank if '-s' argument is not provided in `rat init` command.
+        It will be left blank if '-s' argument is not provided in `rat init` command.
+
+### Plugins
+
+This section of the configuration file is optional and can be used to run different Plugins as mentioned [here](../../Plugins/). It should be used to describe the parameters required by a plugin that you want to use.

environment.yml

@@ -16,8 +16,14 @@ dependencies:
   - ruamel_yaml
   - yaml
   - gdown
+  - plotly
   - cfgrib
   - conda-build
+  - gfortran
+  - make
+  - gdal
+  - libnetcdf
+  - libgdal-netcdf
   - pip 
   - pip:
     - geonetworkx

src/rat/cli/rat_init_config.py

@@ -1,13 +1,14 @@
 DOWNLOAD_LINKS_DROPBOX = {
     'route_model': "https://www.dropbox.com/scl/fi/1kjivr13kyf6gn7wlhbzt/routing.zip?rlkey=zq8j501amqyirfprcgb9dquec&dl=1",
-    'params': "https://www.dropbox.com/scl/fi/9qk9bwrbawryx79o7cclg/params.zip?rlkey=j5dqxipkvwuo3nso4dl1cxaux&dl=1",
+    'params': "https://www.dropbox.com/scl/fi/zvluibsnhz36k4smzcavv/params.zip?rlkey=uqbxdi4ulr9dv4u4imktf605m&st=r0qrt2gd&dl=1",
     'global_data': "https://www.dropbox.com/scl/fi/dhy3y3e9dw6tg89vt6x66/global_data.zip?rlkey=uvy6772cj5bpdfvtvqa4u8enj&st=4sti44e7&dl=1",
     'global_vic_params': "https://www.dropbox.com/s/jsg2wu62qi2ltwz/global_vic_params.zip?dl=1",
 }
 
+## For google drive link, https://drive.google.com/file/d/<ID>/view?usp=sharing, use https://drive.google.com/uc?id=<ID>
 DOWNLOAD_LINKS_GOOGLE = {
     'route_model': "https://drive.google.com/uc?id=1zr3VH0wy-XN-yF2_n0xic89_PiT_V_Mb",
-    'params': "https://drive.google.com/uc?id=1LAGivWvgBdtJvDWzkGKorzfjPEKvO69k",
+    'params': "https://drive.google.com/uc?id=1o5TH8GBmP4rjvFFROQmbid-ILMJw7xiS",
     'global_data': "https://drive.google.com/uc?id=1Obm_cKPFoumJKhcNTvFxIZNkSU3OmgT_",
     'global_vic_params': "https://drive.google.com/uc?id=16P95eu2yG0i77ac_NrmTSVutNtVXWNeA",
 }

src/rat/cli/rat_test_config.py

@@ -1,10 +1,12 @@
 from datetime import date
 
 DOWNLOAD_LINK_DROPBOX = {
-    'test_data': "https://www.dropbox.com/scl/fi/f1pnyz9mo178kweh3agtf/test_data.zip?dl=1&rlkey=qxvsfrhc2li55dh4yj4a03bq2"
+    'test_data': "https://www.dropbox.com/scl/fi/q88v9qpg20rhzhj8v4ktq/test_data.zip?rlkey=j6y1mlhc8vgzaicdofj4hn8g4&st=l7bcqmx1&dl=1"
 }
+
+## For google drive link, https://drive.google.com/file/d/<ID>/view?usp=sharing, use https://drive.google.com/uc?id=<ID>
 DOWNLOAD_LINK_GOOGLE = {
-    'test_data': "https://drive.google.com/uc?id=1F5Z6f4rX7nBSXXo7XYGK_XC0i5-G6zBF"
+    'test_data': "https://drive.google.com/uc?id=1yV2zfqovPjAyWI4yWL5lhN7_GWaXL2Va"
 }
 
 PATHS = {

src/rat/cli/rat_test_verify.py

@@ -27,6 +27,9 @@ def verify_test_results(self):
 
         print("############################## Test-6 (Surface Area) ##############################",end="\n")
         self._verify_test_results_for_var('sarea_tmsos','Surface Area')
+
+        print("############################## Test-7 (NSSC) ##############################",end="\n")
+        self._verify_test_results_for_var('nssc','Normalized Suspended Sediment Concentration')
 
     # Comparing true and estimated files for a variable
     def _verify_test_results_for_var(self, var_to_observe, var_to_display):