New BITS, Bridge Inundation Threat Service, layer inside QC FIM / FIM Pact service layers

[RobHanna-NOAA]

This is a new set of layers to be added to all services that have a QC FIM / FIM pact layer. Generally, for all layers that have building data, this layer will belong in the same services.

The service will be point based with the colors of gray, orange or red and is dynamically calculated based on the current flow data supporting that QC FIM service layer.

The data supplied by the FIM dev teams for the bridge points have to key columns, threshold_discharge_cfs and , threshold_discharge_75_cfs. These represent a flow value which can help identify if the bridge is under no / low risk of flooding, moderate risk of flooding or likely is inundated. Note: is it easiest to think of the threshold_discharge_cfs as 100, max discharge calculated by FIM and threshold_discharge_75_cfs as 75% of the max discharge before being threatened.

The logic scheme for the system is based on using actual flow data provide dynamically at run time and it's relationship to the two threshold discharge values above.

• If the actual flow value is above threshold_discharge_cfs (100%), then the point color is red and the message says "High Risk of Flooding" in the metadata.
• If the actual flow value is below threshold_discharge_cfs (100%) and above threshold_discharge_75_cfs, then the point color is orange and the meta data says "Potentially Flooded"
• If the actual flow is below threshold_discharge_75_cfs, then the point color is gray and meta data says "Low Risk of Flooding".

Services it is plugged into are:

• ana_inundation_extent_noaa.mapx
• ana_inundation_extent_hi_noaa.mapx
• ana_inundation_extent_prvi_noaa.mapx
• ana_past_14day_max_inundation_extent_noaa.mapx (layers for both 7 day and 14 day of course)
• mrf_nbm_10day_max_inundation_extent_noaa.mapx (layers for 3 day, 5 day and 10 day)
• mrf_gfs_10day_max_inundation_extent_noaa.mapx (layers for 3 day, 5 day and 10 day)
• rfc_based_5day_max_inundation_extent_noaa.mapx
• srf_18hr_max_inundation_extent_noaa.mapx
• srf_48hr_max_inundation_extent_hi_noaa.mapx
• srf_48hr_max_inundation_extent_prvi_noaa.mapx

[shawncrawley]

Rob, looks great! Although, one minor nitpick/debate is if we should update the bridge_threat.sql files to use the WITH common table expression rather than CREATE TEMP TABLE. See https://www.postgresql.org/docs/current/queries-with.html.

The tweak would look like this:

-- Ultimately, we want a unique bridge point (osmid) and the feature id
-- tied to it that has the highest risk_status_code and highest threshold_discharge_cfs

DROP TABLE IF EXISTS publish.ana_inundation_bridge_threats;

-- At this point, we can have dup osmid, but each tied to a different feature
-- and each record will calc a risk code based on if the source feature discharge:
-- if the source feature discharge has a higher discharge then the bp.threshold_discharge_cfs, then code 2
-- if the source feature discharge has a lower than bp.threshold_discharge_cfs but higher than
-- the bp.threshold_discharge_75_cfs (75% of the max bp.threshold_discharge_cfs), then the code is 1
-- if it is lower than threshold_discharge_75_cfs, then the code is 0.
-- Note.. we do the IS NULL test against bp.threshold_discharge_cfs as not all
-- features will have matching bridges and visa-versa
WITH tt_w_risk_codes AS (
    SELECT
        bp.osmid,
        bp.name,
        bp.feature_id,
        bp.threshold_hand_ft,
        bp.threshold_hand_75_ft,
        bp.threshold_discharge_cfs,
        bp.threshold_discharge_75_cfs,
        flows.feature_id as src_feature_id,
        flows.discharge_cfs as src_flow_dis,
        CASE
            WHEN bp.threshold_discharge_cfs IS NULL THEN -1
            WHEN flows.discharge_cfs < bp.threshold_discharge_cfs AND
                flows.discharge_cfs >= bp.threshold_discharge_75_cfs THEN 1
            WHEN flows.discharge_cfs >= bp.threshold_discharge_cfs THEN 2
            ELSE 0
            END AS risk_status_code,
        bp.oid,	
        bp.is_backwater,
        bp.has_lidar_tif,
        bp.hydroid,
        bp.branch,
        bp.mainstem,
        bp.bridge_type,
        bp.huc8,
        bp.geom
    FROM cache.max_flows_ana AS flows
    JOIN derived.bridge_points AS bp
    ON flows.feature_id = bp.feature_id
),

tt_osm_max_code AS (

-- For any rec that has a dup osmid, we want the osmid with the highest code
-- This might also result in dup osmids.
    SELECT rc.*
    FROM (
        select osmid,
                max (risk_status_code) as max_code
            from tt_w_risk_codes
            group by osmid
        ) s JOIN tt_w_risk_codes rc
    ON s.osmid = rc.osmid AND s.max_code = rc.risk_status_code;
)
-- now we can filter down to one osmid (bridge point) based which bridge point
-- had the highest bp.threshold_discharge_cfs. In theory, should be no dups
-- but leave it if that super rare occurence happens
select mc.*
INTO publish.ana_inundation_bridge_threats
FROM (
	select osmid, 
	max ( threshold_discharge_cfs) as max_dis
	from tt_osm_max_code
	group by osmid
) s JOIN tt_osm_max_code mc
    ON s.osmid = mc.osmid
   AND s.max_dis = mc.threshold_discharge_cfs; 

-- Drop the oid column? Only when in a production model and what is checked into git
-- Why? the publish schema gets copied over to a service schema which add a new oid
-- back in.
-- When dev testing against a local arcpro, dont' drop the column
ALTER TABLE publish.ana_inundation_bridge_threats
DROP COLUMN oid;

It basically just removes the need for the INTO and DROP statements for the temp tables. Thoughts?

[RobHanna-NOAA]

oh.. cool trick. I broke it up into smaller chunks as I like to keep things in smaller discrete sets, which is easier to debug and validate. But.. this is pretty cool. I am tempted to just leave it, as changing it does not give us any advantages. I don't know if it even saves us a memory footprint. Love the suggestion though. Thanks Shawn

[shawncrawley]

Actually, I completely agree on the debugging thing! I'm glad you pointed that out. I think just that thought has me essentially convinced that your way is better. Curious, are the drop statements even required? I know it's good defensive programming either way, but don't temp tables auto delete after the session they are executed in closes?

[RobHanna-NOAA]

Actually, I completely agree on the debugging thing! I'm glad you pointed that out. I think just that thought has me essentially convinced that your way is better. Curious, are the drop statements even required? I know it's good defensive programming either way, but don't temp tables auto delete after the session they are executed in closes?

-- opps.. you are right. It does not need the drop tables. It was some debugging statements I had in there was I was doing experiments with data combinations in pg_admin. I forgot them in there and yes.. they would auto drop when it leaves the script. I will go and take them out. Thanks for the heads up.