Add conventional 2D stats

[ADCollard]

This is a parallel utility to ush/SpatialTemporalStatsTool/SpatialTemporalStats.py which works for conventional data.

In addition I have added a CONUS projection and changes the polar ones to polar stereographic.

I am also working on producing difference plots between two experiments.

Eventually I hope this can be combined with ush/SpatialTemporalStatsTool/SpatialTemporalStats.py.

[ADCollard]

Usage is:

python SpatialTemporalStats_Conv.py -h
usage: SpatialTemporalStats_Conv.py [-h] -input DIR -output DIR -geovar string -var string -pmax integer -pmin integer [-grid float] [-no_qc_flag] [-region integer] [-sdate string] [-edate string]
                                    [-filter_by_vars FILTER_BY_VARS [FILTER_BY_VARS ...]] [-input_ctl DIR] [-obs_types OBS_TYPES [OBS_TYPES ...]]

Python Tool for Spatial and Temporal Analysis

options:
  -h, --help            show this help message and exit
  -input DIR            REQUIRED: path to input config nc files
  -output DIR           REQUIRED: path to output files
  -geovar string        REQUIRED: geovar name
  -var string           REQUIRED: variable name
  -pmax integer         REQUIRED maximum pressure (hPa)
  -pmin integer         REQUIRED minimum pressure (hPa)
  -grid float           optional: size of grid for plotting (choices: 0.5, 1, 2)
  -no_qc_flag           Optional: qc flag for filtering
  -region integer       REQUIRED: region for mapplot. 1: global, 2: polar region, 3: mid-latitudes region, 4: tropics region, 5:southern mid-latitudes region, 6: southern polar region, 7: CONUS
  -sdate string         REQUIRED: start date of evaluation
  -edate string         REQUIRED: end date of evaluation
  -filter_by_vars FILTER_BY_VARS [FILTER_BY_VARS ...]
                        Optional: Filtering criteria in format 'var_name,comparison,threshold'. Example: Land_Fraction,lt,0.9
  -input_ctl DIR        Optional: Input path of comparison expt
  -obs_types OBS_TYPES [OBS_TYPES ...]
                        Optional: List specific obs types

An example call (plotting aircraft temperature OmFs over CONUS below 700hPa) is:

python SpatialTemporalStats_Conv.py -input /scratch2/NCEPDEV/stmp3/Andrew.Collard/diag/data/prsonde -output /scratch2/NCEPDEV/stmp3/Andrew.Collard/diag -geovar "t" -var "Obs_Minus_Forecast_adjusted" -pmax 1050 -pmin 700 -region 7 -sdate "2025-03-08" -edate "2025-03-18" -obs_types 130 131 132 133 134

[ADCollard]

Converting to an actual pull request. It would be good to have this merged into develop before trying to merge with the radiance version. This version has a number of capabilities that are not in the radiance version:

• Difference plots
• Zoomed in regional plots
• Ability to choose a list of individual obs types

However, the Summary Plot capability is not yet functional for conventional.

[dmerkova]

Looks good and works well (tested on aircraft). Not sure how to interpret -input_ctrl option. IThis is my command line: SpatialTemporalStats_Conv.py -input /lfs/h2/emc/obsproc/noscrub/dagmar.merkova/diag -output /lfs/h2/emc/obsproc/noscrub/dagmar.merkova -geovar "t" -var "Obs_Minus_Forecast_adjusted" -pmax 1050 -pmin 700 -region 7 -sdate "2025-04-20" -edate "2025-04-20" -obs_types 131 133 -input_ctl /lfs/h2/emc/ptmp/dagmar.merkova/tmp766/diag

example of result is here: Control_t_700_to_1050_hPA_Obs_Minus_Forecast_adjusted_Count_region_7.png

'Experiment - Control_t_700_to_1050_hPA_Obs_Minus_Forecast_adjusted_Count_region_7.png'

first one is density for control run and second one is density of Experiment minus control, that means only grid where both experiment and control had some flights will be diplyed? is that correct assumption? I expected more squares since my experiment should only add more aircrafts.

I think quotation is an error , this is list of files created

[ADCollard]

@dmerkova Thanks for looking into this. The difference plot looks correct to me assuming you are only adding a small number of aircraft. Plotting the experiment version of the count plot, it looks very similar to the control.

Maybe if you plot flight level the differences would be more obvious?

You seem to be interpreting the -input_ctrl option correctly. It is the path to the control for difference plots. If the path is not given, no difference plots are produced.

I will fix the issues with some of the output file names.

[ADCollard]

This section of code makes sure we do not plot the Average and RMS where the counts differ wildly between the control and experiment so we do not have spurious differences dominating the plot. But it should not affect the counts themselves.

(I also realise this is wrong, so I need to fix it).

       # Set RMS amd Average fields to missing in difference field where
        # counts are significantly different
        if comparison_plots:
            bool_test = ((self.obs_gdf_ctl[var_name + "_Count"].values +
                         self.obs_gdf_exp[var_name + "_Count"].values) /
                         self.obs_gdf[var_name + "_Count"].values) < 100.0
            self.obs_gdf[var_name + "_RMS"] = \
                np.where(bool_test, np.nan,
                         self.obs_gdf[var_name + "_RMS"].values)
            self.obs_gdf[var_name + "_Average"] = \
                np.where(bool_test, np.nan,
                         self.obs_gdf[var_name + "_Average"].values)